I'm trying to use Zlib::Deflate.deflate on a massive file (4 gigs). There are obvious problems with doing that, the first of which being that I can't load the entire file into memory all at once. Zlib::GzipWriter would work, since it works with streams, but it's not zlib compression. Any ideas?
1 Answer
You could try instantiating a Zlib::Deflate stream and feeding it data from your big file piecemeal. Zlib::Deflate::deflate purports to do that sort of thing behind the scenes.
It would look something like this:
z = Zlib::Deflate.new
File.open "big_uncompressed_file" do |f|
File.open "big_compressed_file", "w" do |w|
f.each do |str|
w << z.deflate str, Zlib::SYNC_FLUSH
end
end
end
z.finish
z.close
-
Zlib by itself does work incrementally in this way - you give it buffers of data incrementally and it spits out compressed data. Commented Apr 12, 2010 at 16:41
-
You'd want the output of finish at the end of your big_compressed_file ("w").– toothrotCommented Oct 21, 2013 at 15:39
-
@ConcernedOfTunbridgeWells i.e.
w << z.deflate(str, Zlib::NO_FLUSH)
? @toothrot i.e.f.write(z.finish)
? Commented Feb 12, 2016 at 23:51