We have a few thousand small files (2-20KB) to copy between servers in one folder for backup purposes. Currently the process is automated as a cron job and does of SCP copy of each file to the remote server.

These files are are mostly text based so I was thinking SCP -C ... may work however I'm not sure if there is a way to batch up all these small files (obviously trying to compress each 5KB file is pointless).

I could zip these files up however I would need to unzip them on the other side (need some process running to monitor when the SCP copy had completed and trigger and unzip).

The other option would be to do a ssh and pipe the zip and unzip

So the question is what is the most efficient way to copy many many many small files to a remote server over a WAN while minimizing the number of changes which need to be made on the remote host?

1 Answer
1

The most efficient way is to stream them through the connection. On Unix, the easiest path is:

tar vc source-dir/ | ssh -C target.example.com 'tar xC /target-dir'

That will use tar to bundle and unbundle the files into a single stream, allowing the most efficient compression and TCP bandwidth use over the link that you can get out of SSH - with zero space overhead for archiving the files.