I have to pack circa 7,000,000 small files. I'm looking for an efficient way to do this.
Is tar the best way? What about benchmark tests or any other suggestion?
I have to pack circa 7,000,000 small files. I'm looking for an efficient way to do this.
Is tar the best way? What about benchmark tests or any other suggestion?
TAR is actually doesn't have it's own compression. TAR is just merging directories/files into one big file. The speed of creation of tar archive depends more on filesystem and I/O subsystem than on tar itself. The best way to get the highest compression ratio is to use 7zip (7zr command on *nix/Linux). The bigger dictionary size you choose - the smaller archive you receive.
For Linux you can try:
7zr a -t7z -m0=lzma -mx=9 -mfb=64 -md=128m -ms=on /path/to/put/archive.7z folder_to_compress
jar is also very nice, as it's a zip (compression) of the files. If not for the manifest inclusion, it would be close to perfect (as jar seems to exist just about everywhere now)