Migrating File Shares to OneDrive for Business

* Update 22-6-2015: Added migration tools from Dell and included section on new migration APIs and PowerShell commandlets

* Update 4-6-2015: Added link to blog/tool from Bill Baer

* Update 15-4-2015: Added MigrationWiz tool and fixed max amount of docs in library from 5 to 30 million

* Update 9-12-2014: Added O365Uploader tool (thx Jos)

* Update 3-12-2014: Added LepideMigrator tool (thx Tom)

* Update: Changed naming from SkyDrive Pro to OneDrive for Business

* Update: Removed "Note that this functionality is only available in SharePoint on-prem", as this is in Online as well.

* Update: Changed 100 Gig to 1000 Gig (1 TerraByte)

* Update: Added The Fullsuite 3rd party migration tool (thx Menno)

* Update: Added additional 3rd party tools and made some small corrections

---

It has been a while since I created a blog post, but recently I received a lot of questions and requests for advice on how to migrate file shares to SharePoint and use OneDrive for Business (ODFB). So I figured to create a blog post with the things you need to consider as a Small and Medium Business (SMB) partner when you are planning to migrate file share content into SharePoint and want to make use of ODFB for synchronizing the SharePoint content offline.

Note: that these steps are both valid for SharePoint 2013 on-premises (on-prem) and SharePoint Online (SPO).

Step 1 – Analyze your File Shares

As a first step, try to understand the data that resides on the file shares. Ask yourself the following questions:

What is the total size of the file share data that the customer wants to migrate?

How many files are there in total?

What are the largest file sizes?

How deep are the folder structures nested?

Is there any content that is not being used anymore?

What file types are there?

Let me try to explain why you should ask yourself these questions.

Total Size

If the total size of the file shares are more that the storage capacity that you have on SharePoint, you need to buy additional storage (SPO) or increase your disk capacity (on-prem). To determine how much storage you will have in SPO, please check the Total available tenant storage in the tables in this article. Another issues that may arise is that in SharePoint is that you reach the capacity per site collection. For SPO that is 1000 Gigabyte (changed from 100 GB to 1 TB), for on-premises the recommended size per site collection is still around 200 Gigabyte.

So, what should I do when my customer has more than 1000 Gigabyte?

Try to divide the file share content over multiple site collections when it concerns content which needs to be shared with others.

If certain content is just for personal use, try to migrate that specific content into the personal site of the user.

How Many Files

The total amount of files on the file shares is important as there are some limits in both SharePoint as well as ODFB that can result in an unusable state of the library or list within SharePoint but you also might end up with missing files when using the ODFB client.

First, in SPO we have a fixed limit of 5000 items per view, folder or query. Reasoning behind this 5000 limit boils all the way down to how SQL works under the hood. If you would like to know more about it, please read this article. In on-prem there is a way to boost this up, but it is not something we recommend as the performance can significantly decrease when you increase this limit.

There is also a limit of 30 million items within a document library, but I guess that most customer in SMB won’t reach that limit very easily.

So, what should I do if my data that I want to migrate to a document library contains more than 5000 items in one folder?

Try to divide that amount over multiple subfolders or create additional views that will limit the amount of documents displayed.

But wait! If I already have 5000 items in one folder, doesn’t that mean that the rest of the other document won’t get synchronized when I use ODFB?

Yes, that is correct. So if you would like to use ODFB to synchronize document offline, make sure that the total amount of documents per library in a team site, does not exceed 5000 documents in total.

So, how do I fix that?

Look at the folder structure of the file share content and see if you can divide that data across multiple sites and/or libraries. So if there is a folder marketing for example, it might make more sense to migrate that data into a separate site anyway, as this department probably wants to store additional information besides just documents (e.g. calendar, general info about the marketing team, site mailbox etc). An additional benefit of spreading the data over multiple sites/libraries is that it will give the ODFB users more granularity about what data they can take offline using ODFB. If you would migrate everything into one big document library (not recommended), it would mean that all users will need to synchronize everything which can have a severe impact on your network bandwidth.

Largest File Sizes

Another limit that exists in SPO and on-prem is the maximum file size. For both the maximum size per file is 2 Gigabyte. In on-prem the default is 250 MB, but can be increased to a maximum of 2 Gigabyte.

So, what if I have files that exceed this size?

Well, it won’t fit in SharePoint, so you can’t migrate these. So, see what type of files they are and determine what they are used for in the organization. Examples could be software distribution images, large media files, training courses or other materials. If these are still being used and not highly confidential, it is not a bad thing to keep these on alternative storage like a SAN, NAS or DVDs. If it concerns data that just needs to be kept for legal reasons and don’t require to be retrieved instantly, you might just put these on DVD or an external hard drive and store them in a safe for example.

Folder Structures

Another important aspect to look at on your file shares is the depth of nested folders and file length. The recommended total length of a URL in SharePoint is around 260 characters. You would think that 260 characters is pretty lengthy, but remember that URLs in SharePoint often has encoding applied to it, which takes up additional space. E.g. a space is one character but in Unicode this a %20, which takes up three characters. The problem is that you can run into issues when the URL becomes to large. More details about the exact limits can be found here, but as a best practice try to keep the URL length of a document under 260 characters.

So, what if I have files that will have more than 260 characters in total URL length?

Make sure you keep your site URLs short (the site title name can be long though). E.g. don’t call the URL Human Resources, but call it HR. If you land on the site, you would still see the full name Human Resources as Site Title and URL are separate things in SharePoint.

Shorten the document name (e.g. strip of …v.1.2, or …modified by Andre), as SharePoint has versioning build in. More information about versioning can be found here.

Idle Content

When migrating file shares into SharePoint is often also a good momentum to clean up some of the information that the organization has been collecting over the years. If you find there is a lot of content which is not been accessed for a couple of years, what would be the point of migrating that data it to SharePoint?

So, what should I do when I come across such content?

Discuss this with the customer and determine if it is really necessary to keep this data.

If the data cannot be purged, you might consider storing it on a DVD or external hard drive and keep it in a safe.

If the content has multiple versions, such as proposal 1.0.docx, proposal 1.1.docx, proposal final.docx, proposal modified by Andre.docx, you might consider just moving the latest version instead of migrating them all. This manual process might be time consuming, but can safe you lots of storage space in SharePoint. Versioning is also something that is build into the SharePoint system and is optimized to store multiple versions of the same document. For example, SharePoint only stores the delta of the next version, saving more storage space that way. This functionality is called Shredded Storage.

Types of Files

Determine what kind of files the customer is having. Are they mainly Office documents? If so, then SharePoint is the best place to store such content. However, if you come across developers code for example, it is not a good idea to move that into SharePoint. There are also other file extensions that are not allowed in SPO and/or on-prem. A complete list of blocked file types for both SPO and on-prem can be found here.

So, what if I come across such file extensions?

Well, you can’t move them into SharePoint, so you should either ask yourself, do I still need these files? And if so, is there an alternative storage facility such as a NAS, I can store these files on? If it concerns developer code, you might want to store such code on a Team Foundation Service Server instead.

Tools for analyzing and fixing file share data

In order to determine if you have large files or exceed the 5000 limit for example, you need to have some kind of tooling. There are a couple of approaches here.

First off, there is a PowerShell script that has been pimped up by a German colleague Hans Brender, which checks for blocked file types, bad characters in files and folders and finally for the maximum URL length. The script will even allow you to fix invalid characters and file extensions for you. It is a great script, but requires you to have some knowledge about PowerShell. Another alternative I was pointed at is a tool called SharePrep. This tool does a scan for URL length and invalid characters. There is also another great tool from colleage Bill Baer which is called filechecker.exe and can be found on his blog.

Secondly there are other 3rd party tools that can do a scan of your file share content such as Treesize. However such tools do not necessarily check for the SharePoint limitations we talked about in the earlier paragraphs, but at least they will give you a lot more insight about the size of the file share content.

Finally there are actual 3rd party migration tools that will move the file share content into SharePoint, but will check for invalid characters, extensions and URL length upfront. We will dig into these tools in Step 2 – Migrating your data.

Step 2 – Migrating your data

So, now that we have analyzed our file share content, it is time to move them into SharePoint. There are a couple of approaches here.

Open with Explorer

If you are in a document library you can open up the library in the Windows Explorer. That way you can just do a copy and paste from the files into SharePoint.

But, there are some drawbacks using this scenario. First of all, I’ve seen lots of issues trying to open up the library in the Windows Explorer. Secondly, the technology that is used for copying the data into SharePoint is not very reliable, so keep that in mind when copying larger chunks of data. Finally there is also drag & drop you can use, but this is only limited to files (no folders) and only does a maximum of 100 files per drag. So this would mean if you have 1000 files, you need to drag them 10 times in 10 chunks. More information can be found in this article. Checking for invalid characters, extensions and URL length upfront are also not addressed when using the Open with Explorer method.

OneDrive for Business

You could also use ODFB to upload the data into a library. This is fine as long as you don’t sync more than 5000 items per library. Remember though that ODFB is not a migration tool, but a sync tool, so it is not optimized for large chunks of data to be copied into SharePoint. Things like character and file type restrictions, path length etc. is on the list of the ODFB team to address, but they are currently not there.

The main drawbacks of using either the Open in Explorer option or using ODFB is that when you use these tools, they don’t preserve the metadata of the files and folder that are on the file shares. By this I mean, things like the modified date or owner field are not migrated into SharePoint. The owner will become the user that is copying the data and the modified date will be the timestamp of the when the copy operation was executed. So if this metadata on the files shares is important, don’t use any of the methods mentioned earlier, but use one of the third party tools below.

Pros: Free, easy to use, works fine for smaller amounts of data (max 5000 per team site library or 20000 per personal site)

PowerShell

A new migration API has been released back in May. The new API allows you to migrate content much quicker as it first uploads the content to Azure storage and from there imports it into SharePoint Online without being subject to CSOM throttling. With the new API we also released a preview of some PowerShell commandlets that will allow you to migrate content from fileshares into SharePoint Online using the new API. You can sign up for the preview via https://aka.ms/spomigrationpreview.

The list above is in random order, where some have a focus on SMB, while other more focused on the enterprise segment. We can’t speak out any preference for one tool or the other, but most of the tools will have a free trial version available, so you can try them out yourself.

Summary

So, when should I use what approach?

Here is a short summary of capabilities:

Open in Explorer

OneDrive for Business sync client

3rd party

Amount of data

Relatively small

No more than 5000 items per library

Larger data sets

Invalid character detection

No

No

Mostly yes1

URL length detection

No

No

Mostly yes1

Metadata preservation

No

No

Mostly yes1

Blocked file types detection

No

No

Mostly yes1

1This depends on the capabilities of the 3rd party tool.Note that in the table above, I have not included the new PowerShell commandlets yet as they are still in preview and am not sure about it's final capabilities.

Troubleshooting

ODFB gives me issues when synchronizing data Please check if you have the latest version of ODFB installed. There have been stability issues in earlier released builds of the tool, but most of the issues should be fixed by now. You can check if you are running the latest version, by opening up Word-> File-> Account and click on Update Options-> View Updates. If your current version number is lower than the one you have, click on the Disable Updates button (click yes if prompted), then click Enable updates (click yes if prompted). This will force downloading the latest version of Office and thus the latest version of the ODFB tool.

If you are running the stand-alone version of ODFB, make sure you have downloaded the latest version from here.

Why is the upload taking so long? This really depends on a lot of things. It can depend on:

The method or tool that is used to upload the data

The available bandwidth for uploading the data.

The amount of throttling that is applied on our end.

Tips:

Check your upload speed at http://www.speedtest.net and do a test for your nearest Office 365 data center. This will give you an indication of the maximum upload speed.

Often companies have less available upload bandwidth then people at home. If you have the chance, uploading from a home location might be faster.

Schedule the upload at times when there is much more bandwidth for uploading the data (usually at night)

Test your upload speed upfront by uploading maybe 1% of the data. Multiply it by 100 and you have a rough estimate of the total upload time.

The computers used for uploading the data. A slow laptop can become a bottle neck while uploading the data.

Make sure you are using a tool that is utilizing the new migration API (either PowerShell or 3rd party)

If you feel that there are things missing here, please let me know and I’ll try to add them to this blog post.

@Justin, there is no automated provisioning mechnism right now. I know that the product teams are working on a solution for the future.

@Radar, when you sync your first library, you are asked for a location to store you synchronized files. This can also be an external hard disk, as long as it is seen as a local hard disk. You can’t store on a network drive for example. If you currently have sync-ed libraries, the only way to move those is to disconnect them all from the sync process. When you then start the sync your first library again, it will ask for the location.

Awesome, thanks for sharing good information related to migrate file server to SharePoint Online. I tested an automated tool named LepideMigrator for Documents (http://www.lepide.com/lepidemigratordocuments/ ) that helps to migrate content from file server to SharePoint Online with the attributes of every file, folder, tasks, email, contact and other related
items.

wow, the takeaway from this is to not use skydrive pro for anything except small well-behaved document sets. When migrating one user, we ran into the URL character length, non-standard characters, file count and default size limitations.

Anyone have an approach to automate the provisioning of the SkyDrive MySite to allow the automated migration of files? Or does all of the above require that a user has logged into SkyDrive already to create the target and then these approaches can be used?

ok first time using this and transferred my files to skydrive from my external hard drives…it took up all my space on my internal hard drive so my pc is slow..cant skydrive be used only from external hard drive without taking up space on my internal hard drive?

It’s so funny that SkyDrive Pro (Paid version) is not good as SkyDrive (Free version). It’s big guy (25GB) without clever:
1. Almost of file extensions will be blocked
2. If you have folder and sub folder with long path, you can not upload to SkyDrive Pro.
….

Hi Andre, if I pretend to use OneDrive for Business Client in my computer, does my computer’s internal hard drive has to have 1TB or more of free space? This is because ODFBClient, only syncs to my computer’s internal HD. So, ODFB gives you 1TB of online
storage, however you can only sync up to the free space you have in your HD. Am I correct?

Does this also apply to OneDrive Professional? The statement below was found on a Microsoft web page, for troubleshooting uploading files to OneDrive. I need to know if the file size upload is dependent on the browser version.

The ODFB sync client is a truly poor piece of software. The reason it is so slow is in its code. For some reason my client was using 30% of a quad core cpu to do an rsync operation. The 1TB limit is irrelevant is you will reach the 20k file limit after
about 135GB. Until MS go try using something like dropbox or sugarsync and see how it is done properly, they wil fail completely with this product.

I agree with Mark. A 5,000 file limit for a Sharepoint share, and 20,000 limit for ODFB is too low for most users. It’s ridiculous to then require users to split this up into separate shares. Why should we go to so much work to make it fit under ODFB/SharePoint
constraints, when no other similar tool out there has such constraints? That tells me our needs are not abnormal, but that MS has not produced a product that meets customer’s needs well.

After messing around with different options for a week (and my ODFB client is still showing "Syncing 4 files" as it has for the past 2 days), I just decided to pay for Dropbox. No abnormally low limit on number of files, file length, etc. Why can’t MS offer
a product at least as good as the competition?

FYI: I’ve built a small tool in Dutch that does the uploading for you (including extensive logging) to any sharepoint (201x and OneDrive) Document Library. Based on Powershell. I’ll release it for free to anyone that wants to use it 🙂

How can you promote the script from Hans Brender when it’s full of errors and really bad rookie mistakes, for instance the progressbar is based on a $count variable which will go way above 100, it also reports that every single file has illegal extension…

Watch this video to see how to set up Onedrive and how to upload your files from your Drive into your OneDrive account. I will show you how to use the storage technology of microsoft and how to use and download the OneDrive hope you enjoy.

This is a véry nice blogpost. If everyone, before he starts migrating to O365 & ODFB, reads this, a lot, I mean a LOT of problems won’t even become reality.

One thing: Testing your upload speed through http://www.speedtest.net is’nt that reliable… A much better way is to create, let’s say, a 10 gig dummy file and upload it from the machine you’ll use for migrating to ODFB. The average speed is a way better indication
then speedtest.net of any other webservice.

Another thing that passed my mind: Don’t upload your data in very large batches. It could prevent a lot of problems when using 20 small batches, rather than 5 big ones.

has Microsoft looked at what their customers have typically setup regarding network shares? Most of my customers cannot migrate "as is" and this throws a huge monkey wrench into people’s immediate retreival of what they need. The users don’t care about
260 character url limits, directory nesting depth or file counts. Why would’nt microsoft allow their customers to migrate with a click of a mouse and go cloud and not look back?