S3 fails to unzip uploaded file

Question

I'm following this example

// Load the stream
var fs = require('fs'), zlib = require('zlib');
var body = fs.createReadStream('bigfile').pipe(zlib.createGzip());

// Upload the stream
var s3obj = new AWS.S3({params: {Bucket: 'myBucket', Key: 'myKey'}});
s3obj.upload({Body: body}, function(err, data) {
  if (err) console.log("An error occurred", err);
  console.log("Uploaded the file at", data.Location);
})

And it "works" in that it does everything exactly as expected, EXCEPT that the file arrives on S3 compressed and stays that way.

As far as I can tell there's no auto facility for it to unzip it on S3, so, if your intention is to upload a publicly available image or video (or anything else that the end user is meant to simply consume) the solution appears to leave the uploaded file unzipped like so...

// Load the stream
var fs = require('fs'), zlib = require('zlib');
var body = fs.createReadStream('bigfile');//.pipe(zlib.createGzip()); <-- removing the zipping part

// Upload the stream
var s3obj = new AWS.S3({params: {Bucket: 'myBucket', Key: 'myKey'}});
s3obj.upload({Body: body}, function(err, data) {
  if (err) console.log("An error occurred", err);
  console.log("Uploaded the file at", data.Location);
})

I'm curious if I'm doing something wrong and if there IS an automatic way to have S3 recognize that the file is arriving zipped and unzip it?

Answer 1

The way this works is that s3 has now way of knowing that the file is gziped without a bit of help. You need to set the metadata on the file when uploading telling S3 that it's gzipped. It will do the right thing is this is set.

you need to set Content-Encoding: gzip and Content-Type: <<your file type>> in the object metadata when uploading.

Later edit: Found these which explains how to do it for Cloudfront, but basically the same: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html#CompressedS3
http://www.cameronstokes.com/2011/07/20/compressed-content-amazon-s3-and-cloudfront/

However note that as per this blogpost S3 will serve the file gzipped and rely on the browser to unzip it. This works fine in many cases but as the blogger notes will fail in curl (since curl will have no idea what to do with the gzipped file). So if your intention is to simply upload a file for raw consumption by the user your best bet is to skip the gzipping and upload the file in its uncompressed state.