The scenario is this: Machine A has files I want to copy to Machine C. Machine A can't access C directly, but can access Machine B that can access Machine C. I am using scp to copy from Machine A to B, and then from B to C.

Machine B has limited storage space, so as files come in, I need to copy them to C and delete them from B. The second copy is much faster, so this is no problem with bandwidth.

I could do this by hand, but I am lazy. What I would like is to run a script on B or C that will copy each file to C as each one finishes. The scp job is running from A.

So what I need is a way to ask (preferably from a bash script) if file X.avi is "done" copying. Each of these files is a different size, and I can't really predict size or time of completion.

Edit: by the way, the file transfer times are something about 1 hour from A to B and about 10 minutes from B to C, if time scale matters at all.

The problem with this is that on machine A I want to do scp * user@host:~/ and the files being copied would more than fill machine B, so I can't move/rename files after they are copied from A.
–
Mike CooperSep 22 '09 at 18:25

Ah, so if I understand correctly, you're coping more than one file at once? That's where the problem comes from?
–
JoshSep 22 '09 at 19:01

Yes, that is the problem. I can't store all the files on B, but this copy will take long enough that I don't want to sit and babysit it by watching for each file to finish and then copying it. Maybe I will go with your expanded version.
–
Mike CooperSep 22 '09 at 19:08

Does lsof on machine B show that scp has the file open? if so, you could watch lsof and see when scp closes the file. If not, you could watch the size of the file and after it hasn't changed for a given period of time (5 minutes, for example), copy it from B to C.

A third option would be to copy the files from A to to an "in_progress" directory on C. After the copy finishes on A, execute a mv command to move out of the "in_progress" directory.

Sadly lsof doesn't seem to exist on machine B. About your third option, that won't work because if I let the copy finish before I do anything, I will vastly overwhelm my allowed space on B (something like 10:1). So whatever solution I have has to work during the copy.
–
Mike CooperSep 22 '09 at 18:24

The wait 5 minutes idea is a good one (I thought of something similar) but I'm not quite sure how it would be done. Any ideas?
–
Mike CooperSep 22 '09 at 18:27

Yeah, I could write a quick and dirty ruby script for you -- does machine B have ruby?
–
JoshSep 22 '09 at 19:00

Nope. It isn't under my control and doesn't seem to have any of the things I try and use. I was hoping bash might have something built in to do this. Though I might be able to run the script from my local machine... I will experiment with that.
–
Mike CooperSep 22 '09 at 19:07

I'll see if I can code something like that in bash alone. My ruby prowess exceeds my bash prowess however. stackoverflow.com might be able to assist with such a script.
–
JoshSep 22 '09 at 19:13

The way you explain this it would seem to work... but it seems a bit "dark magic" for my liking. I think found another solution that works the original way, but I will definitely keep this snippet for later use.
–
Mike CooperSep 22 '09 at 19:32

This way isn't really "dark magic", it's just using pipes which is a standard OS facility. But I do like your other solution better, it's less code and probably easier to maintain.
–
JoshSep 22 '09 at 19:55

Oh, not I just mean to me it is magic-y because I have never played with fifo pipes and it is something that isn't used quite as often as other things. Though something to learn now. Also, maintenance isn't much of an issue. This will be set up for about a day is all.
–
Mike CooperSep 23 '09 at 7:18

After thinking about the answers posted (in particular @Josh's idea of watching modified times) I was trying to run manipulate B's files on C. See, B is anaemic as far as available tools, so nothing that seemed to be able to do the job was there. I came upon this solution. This idea is not mine, I found it in google searches before this question. I discarded it earlier, since machine B did not have the find utility.

First, mount the appropriate directory on B onto C, so it appears as a local file system. I used sshfs for this (awesome tool, by the way). This will let me use C's utilities instead of B's.

Secondly, the command find /the/folder/* -mmin +5 will match all files modified over 5 minutes ago. So the command find /the/folder/* -mmin +5 -exec {} /the/other/folder \; will move all files that have been modified over 5 minutes ago to the other folder (which is actually on C, instead of sshfs mounted from B.

Finally, I set up a cron script to run the above script every 10 minutes today and tomorrow. The line in my crontab looks like this.

Hopefully this will work. The next file has yet to complete, so I can't comment on if it really works when combined with the cron script, but I made some files by hand and seeded them and they moved fine. cross my fingers

Edit: This is working, though how it was originally had some errors, those are corrected now.

The copy will either run as a another process, or you could force it to, using a subshell. Then, you could use ps to "watch" the process and see when it disappears.

Also, I believe that in *nix, you can delete the file while it's being copied. The system won't delete it until the copy program closes it. Of course, if the copy doesn't succeed, you lose the file, so, not the best idea.