Can i trigger an AWS lambda function when a sagemaker labeling job is completed using CloudWatch events?

Question

I'm new using CloudWatch events and I'm trying to create an event rule that triggers a lambda function when a Sagemaker Ground Truth Labeling Job is completed. It seems to me that this is possible, from reading this articles from aws:

React to Amazon SageMaker Job Status Changes with CloudWatch Events

Creating a CloudWatch Events Rule That Triggers on an Event

I followed the tutorial to create a cloudwatch event rule and set the event pattern to Sagemaker:Sagemaker Training Job State Change and created a Lambda Function and added it in Targets. But when i create a labeling job i don't see any output coming from the lambda function in the logs, neither when the job status changes to Completed, Failed or Stopped.

I've also tried setting the event pattern to Sagemaker:All Events, but this didn't work either.

I'm checking the metrics for the rule and it seems like it hasn't recorded any event.

If someone has done something similar to this in the past, any suggestion or recommendation will be very appreciated. Thanks

Answer 1

Yes, this should be possible. You can do it like this:

1. Hook up a lambda trigger, triggered by S3. Use "All Object Create Events".
2. As Trigger Suffix choose: /manifests/output/output.manifest (which is the generic suffix for all final groundtruth output manifests).
3. Now the problem with this is, it will create two events when a job is finally finished / stopped (Two events of eventName "ObjectCreated:CompleteMultipartUpload")
4. Therefore I would do the following: In the lambda script make sure it is polling the labeljob status, which you can get from here: Boto3 Documentation, 'LabelingJobStatus'. I would use both "Completed" and "Failed". Waittime until one of those two states is reached (should be 1-2 minutes, 0.01 cents)
5. Once the job is done, operate on the output manifest file, e.g. by getting it from s3 bucket should be simple.

Not the simplest approach, and it is not using CloudWatch. But it is using lambda on finished groundtruth job, so serverless, only with minimal waittime.