Optimize time when creating archive

Question

Currently I am using the following command to create an archive with files older than 7 days:

find /var/tunningLog/ -type f -mtime +7 -print0 | tar -czf "/var/tunningLog/$(date '+%Y-%m-%d').tar.gz" --null -T - && echo "OK" || echo "NOK"

But it is taking to long (currently /var/tunningLog/ has 49G). Is there any way to speed up the process or to improve the command? Thx

Answer 1

Since you're creating your archive of /var/TuningLog in /var/TuningLog, the archive you create contains the archive you created a week ago, or earlier. You have an ever growing archive of archives.

Either modify your find to exclude your archives (\! -name '*.tar.gz') if that doesn't exclude wanted files.

OR

Save your archives elsewhere.

Answer 2

Using a find | tar pipe is something from the stoneage of UNIX when the only way to reuse code was to combine commands via pipelines.

This is slow because it requires to run stat(2) twice for every file.

• Once from within the find(1) command to identify the files of interest

• Another time from within tar(1) because tar needs the meta data for archiving

Since 33 years, there is another way to combine reusable code by using shared libraries and since 16 years, there is libfind that includes all features of the find command in a way that allows to reuse the functionality.

Since you are using non-standard tar options like -T, it seems that you are using gtar instead of tar and gtar is known for it's ineffective code for creating the tar headers.

I recommend you to have a look at star, the oldest free tar implementation that includes libfind support since 2005 and includes support for a -find option that permits to use find(1) syntax to the right side of -find on the command line.

Check e.g. the man page from the schilytools web page at: http://schilytools.sourceforge.net/man/man1/star.1.html

This leads to a command line similar to:

star -c -f /tmp/xxx.tar -find . -type f -mtime +7