Robocopy is slooow. Tips or alternative software?

We've set up a new SAN device and are moving our user data to this device, generally one user at a time. We always have used Robocopy to preserve all the ACLs of the files and folders being moved, but we're finding it to be very slow for large amounts of data - Moving a few hundred gigs takes an excruciatingly long time and never comes close to saturating the network card.

I was wondering if anybody had some advice on alternative pieces of software that will retain ACLs over SMB, or perhaps some tips on how we could increase the speed of robocopy.

We use Backup Exec to restore file shares/permissions across servers. Works like a charm. You could get the 60 day trial, backup, restore. However remote agent/open file option might require reboots on your various servers...

Originally posted by JerkyChew:but we're finding it to be very slow for large amounts of data - Moving a few hundred gigs takes an excruciatingly long time and never comes close to saturating the network card.

Are you using Windows' QoS on the adapter? Also, what happens if you copy from two sources - same throughput?

Accessing and copy ACLs is going to take a long time for a lot of individual files, there's no way around the cost. If you're moving large files, then you should get much better throuput.

Anyways, do some testing with some smaller sets for your own sanity first. Do some calculations of how many seconds you expect to copy ACLs on each file, then multiply that by several hundred thousand and that will easily equal hours for even a tenth of a second of overhead per file.

This won't help you yet but in Windows 7/2008 R2 robocopy now supports multithreaded copying instead of synchronous. Not that anything is stopping you from running the Betas to do this right now for your project except the risk of an unsupported OS. I use this all day though to sync folders from our Redmond campus and it works a charm.

/MT[:n] :: Do multi-threaded copies with n threads (default 8). n must be at least 1 and not greater than 128. This option is incompatible with the /IPG and /EFSRAW options Redirect output using /LOG option for better performance.

One thing you could do in the meantime is roll your own multi-threading. Run multiple copies of robocopy via a batch file that starts at (perhaps) the subfolder level so that you get more bang than a single synchronous process. If you are blocking on very large files, this could really add up in savings.

Originally posted by MonsieurEvil:One thing you could do in the meantime is roll your own multi-threading. Run multiple copies of robocopy via a batch file that starts at (perhaps) the subfolder level so that you get more bang than a single synchronous process. If you are blocking on very large files, this could really add up in savings.

I used to do this when I was doing 500-1000 user homeshare moves a night. I sliced them up into chunks and had several CMD windows open, each with a batchfile running robocopy (with logfiles) for hours. Which was way better than hours AND hours, if I had done it in one job.

Originally posted by MonsieurEvil:This won't help you yet but in Windows 7/2008 R2 robocopy now supports multithreaded copying instead of synchronous. Not that anything is stopping you from running the Betas to do this right now for your project except the risk of an unsupported OS. I use this all day though to sync folders from our Redmond campus and it works a charm.

/MT[:n] :: Do multi-threaded copies with n threads (default 8). n must be at least 1 and not greater than 128. This option is incompatible with the /IPG and /EFSRAW options Redirect output using /LOG option for better performance.

One thing you could do in the meantime is roll your own multi-threading. Run multiple copies of robocopy via a batch file that starts at (perhaps) the subfolder level so that you get more bang than a single synchronous process. If you are blocking on very large files, this could really add up in savings.

Is there a way to get that version of robocopy without pulling and installing Windows 7? I have access to Server 2008 - does that have the updated version that supports multi-threading?

If the file-ACL's are the issue, an effective workaround would be to pre-cfg the target with dirs that have the necessary inheritance perms set. (Even if it's a ton of users dirs, it shouldn't be too hard to script the ACL'ing.) Then run the robocopy job without security. This is much less overhead.

This won't help you yet but in Windows 7/2008 R2 robocopy now supports multithreaded copying instead of synchronous. Not that anything is stopping you from running the Betas to do this right now for your project except the risk of an unsupported OS. I use this all day though to sync folders from our Redmond campus and it works a charm.

/MT[:n] :: Do multi-threaded copies with n threads (default 8).n must be at least 1 and not greater than 128.This option is incompatible with the /IPG and /EFSRAW optionsRedirect output using /LOG option for better performance.

Anyway to move the robocopy from Windows 7 to a Vista/2008 box? Copying the executable Vista says it's not a valid Win32 app. Copying the mui files and such doesn't help either.

Most (all?) enterprise backup apps use agents on the protected machines. Unlike native SMB which is really inefficient, the backup agents are bottlenecked only by the disk and network performance. So the performance is very good. The backup app will let you preserve ACLs, copy empty folders, etc etc...

Instead of running it multiple times at the same time, you could run it multiple times sequentially. First pass gets the bulk of data, and is time consuming, but isn't slow to your users since they're still working while it runs. Multiple passes allow robocopy to just get the files that were changed, so they'll be much faster and eventually, when you move a user, it should only take a short amount of time.

Last february I had to copy about 7tb of data from one machine to another across the network. Doing a full restore from our aging TSM setup would have taken too long, so I ended up making a script to spawn a bunch of different robocopies. My need was to copy four root file shars, but on different weekends, so I just ran this one time for each.

We ran robocopy several times during a recent file server migration. Shares and home drives about 500-600GBs I think, took about 4 or so hours I think. The nice thing being robocopy on sobsequent runs would just copy changes, with the bulk of data moved it only had to move a few files that were open at the time of the initial copy runs.

That was nice, we didnt have to copy it on the weekend, just reset the shares.

These where pretty new servers on GB net., still looked like the disk (scsi raids) throughput was the bottle neck though as the NICs didnt ever seem to max out.

EDIT: I think my point was if you have time, run robocopy pre-emptively, then run it again a final time if not to many file changes have occured, it may only need to copy a fraction of the files during the final run.