5

I have web storage set up for some files in Amazon S3. I'm using its PHP SDK to manage these files. I have an automation script running to delete files from a certain bucket after a certain period of time. I've just realized that I don't believe it's actually deleting the files, but replacing them with a delete marker. When I use a program like Transmission (Mac) to view the bucket, it lists all files since 2013 when I set up the script. But if I go to the management interface in a browser, it only lists files as far back as the cutoff I have set in the script.

My question is: am I paying much more money for storage of these markers when I have no need for actually keeping any of these files? And how do I permanently delete these files? Everything I've found is that this is only an issue with versioned buckets, but this particular bucket is not versioned. According to the documentation, I need to include the version ID of the object in the delete call to delete the marker, but at this point, how do I retrieve that information since 'listObjects' does not return these.

If I use Transmission to pull down a 'deleted' file, the file still opens and functions as if it were never deleted.

Improve this question
1
  • You say the bucket is not versioned, but this sounds like the behavior of a versioned bucket. Check the console? It could be that the bucket has versioning enabled but suspended. I'm unfamiliar with the php sdk but you might look for something along the lines of ListObjectVersions which will show you the objects and version ids.
    – Michael - sqlbot
    Commented Nov 19, 2014 at 3:07

1 Answer 1

Reset to default
9

If Versioning is enabled on an Amazon S3 bucket, then all uploads of an object to an existing key name will result in the creation of an additional version of the object. The prior versions will be kept on Amazon S3. Deleting an object will add a Delete Marker, so that the object appears to have been deleted, but prior versions are still available for download.

If a Bucket has Versioning enabled, you can view object versions and delete markers in the console:

  • Access your Bucket in the Amazon S3 console
  • Look for the Versions buttons at the top of the screen. It should have two buttons: Show, Hide
  • Selecting Show will display delete markers and all prior versions of objects

A simpler way to delete objects from Amazon S3 after a certain period of time is:

  • Select your Bucket
  • Open Properties
  • Click Lifecycle
  • Create a rule set to Permanently Delete n days after the object's creation date
Improve this answer
3
  • As far as I learned, the Lifecycle rule will only permanently delete objects if they are not the "current latest" version, i.e. they'll only delete versions of the key which were overridden. To delete the "current latest deleted version" you'd still need to run a script.
    – Capt. Crunch
    Commented Nov 24, 2016 at 0:03
  • @AmosShapira: With the current lifecycle rule version you can delete old versions and/or current versions by age
    – Jonathan
    Commented Oct 15, 2017 at 16:20
  • @Jonathan it's been a while since I had to deal with this but I wouldn't be surprised if S3 and CloudFormation has added the option since I wrote that answer. Back when I wrote this I used a Lambda function and a custom resource to implement this.
    – Capt. Crunch
    Commented Oct 15, 2017 at 19:17

Not the answer you're looking for? Browse other questions tagged or ask your own question.