Move file to another directory once it is done transferring

Question

I have a video encoding script that I would like to run as soon as a file is moved into a specific directory.

If I use something like inotify, how do I ensure that the file isn't encoded until it is done moving?

I've considered doing something like:

• Copy (rsync) file into a temporary directory.
• Once finished, move (simple 'mv') into the encode directory.
• Have my script monitor the encode directory.

However, how do I get step #2 to work properly and only run once #1 is complete?

I am using Ubuntu Server 11.10 and I'd like to use bash, but I could be persuaded to use Python if that'd simplify issues.

I am not "downloading" files into this directory, per se; rather I will be using rsync the vast majority of the time.

Additionally, this Ubuntu Server is running on a VM.

I have my main file storage mounted via NFS from a FreeBSD server.

Answer 1

The easiest way would be to download using a program like wget or curl that doesn't exit until the file is done downloading. Failing that, I'm not sure if it's the absolute best solution, but you could use a script that loops, checking whether the file is open using lsof and grep:

while lsof | grep /path/to/download >/dev/null; do sleep 5; done
mv /path/to/download /path/to/encode/dir

Answer 2

One technique I use works with FTP. You issue a command to the FTP server to transfer the file to an auxiliary directory. Once the command completes, you send a second command to the server, this time telling it to rename the file from from the aux directory to the final destination directory.If you're using inotify or polling the directory, the filename won't appear until the rename has completed, thus, you're guaranteed that the file is complete.

I'm not familar with rsync so I don't know if it has a similar rename capability.

Answer 3

inotifywait looks promising. You have to be somewhat careful because if the download finishes before the notification is set up, it'll wait forever.

inotifywait -e close_write "$download_file"
mv "$download_file" "$new_location"