Devops Zero to Hero #6— A complete guide to serverless in AWS with Lambda
Learn AWS Lambda by doing a real-world example
What to expect in this blog post?
This is the 6th blog post of the series Devops Zero to Hero, and it is about the serverless AWS Lambda function. I will go through fundamental concepts around Lambda and use a real-world example to deploy it.
What is Lambda Function?
It is a compute service provided by AWS that runs your code/application without you having to jump through hoops to manage the Infra where your code will be running.
You must be thinking that people always talk about Lambda in the sense of not having to manage the infrastructure but never talk about the problems in managing the Infra — It shouldn’t be that hard, right?
Managing an EC2 instance( infrastructure, i.e.server) has its challenges. You need to ensure multiple things like
- You are using the latest OS image, and all the installed software is up to date, not to forget every new vulnerability that pops up now and then. You need to take care of routine patching and all the mess that comes with it.
- You have to tighten the security around the EC2 VM and not leave any security holes for intruders. Things like not opening unnecessary ports, managing SSH keys, or having all access m, management stuff bugging you.
- You have configured high-availability, autoscaling, SSL, and the cost of everything you need to pay to run your application. ALB, Nat, Public IP, etc.
- And not to forget the cost of human resources you will hire to do all this. Don’t expect AI to do that for you anytime soon.
It makes sense to use managed services from a cloud provider(AWS lambda) instead of provisioning servers on EC2.
Features of AWS Lambda
- Write your application code in any language you like.
- Follows event-driven architecture, meaning it runs your code when some events take place
- Seamlessly integrate with multiple AWS services and 3rd party SDK.
- Lambda scales in response to events. It is fault-tolerant and highly available by default.
- It provides fine-grain access control through AWS IAM. You can assign specific permissions to the Lambda function.
- Only pay for every millisecond you run your code.
When should you use AWS Lambda and when you shouldn’t?
AWS Lambda is suitable for —
- Event-driven and asynchronous tasks: Ideal for executing code as a response to events such as file upload in an Amazon S3 bucket, updates to a DynamoDB table, HTTP requests via API Gateway, or messages from an Amazon SQS queue.
- Short-lived and stateless functions: Lambda functions are stateless and short-lived (with a maximum execution time of 15 minutes), making them well-suited for processing individual tasks or events.
- Microservices architecture: Suitable to implement individual microservices within a distributed, loosely coupled application.
- Integration with other AWS services: Lambda integrates seamlessly with various AWS services, enabling you to build serverless applications that use other AWS services such as S3, DynamoDB, SNS, etc.
And Not suitable for —
- Long-running tasks: Lambda functions are limited to a maximum execution time of 15 minutes, making them unsuitable for long-running or continuously executing tasks.
- Tasks requiring high CPU/memory requirements: If your application requires consistently high CPU or memory resources, you are better off using EC2 instances or containers managed by Amazon ECS or EKS.
- Applications with complex dependencies: If your application has complex dependencies or requires custom runtimes, Lambda is not for you.
- Workloads with predictable and consistent traffic: It would be more cost-effective to use provisioned EC2 instances for this type of workload
- Legacy applications: Migrating legacy applications to a serverless architecture may require significant refactoring and may not be practical in some cases.
Events to trigger Lambda
Three types of events can invoke a Lambda function.
Synchronous Invocation
Lambda runs the function and waits for a response. There are no built-in retries in this type of Lambda trigger.
Amazon API Gateway, Amazon Cognito, AWS CloudFormation, Amazon Alexa, etc, services invoke Lambda synchronously.
Asynchronous invocation
When you call a function asynchronously, events get lined up, and the requester doesn’t wait for the function to finish. This method works well when the client doesn’t need an instant reply.
Instead of an instant reply to the client, it uses destinations to send responses to other services.
Amazon SNS, Amazon S3, and Amazon EventBridge can invoke Lambda asynchronously
Polling invocation
Lambda will poll (or watch) AWS streaming and queuing-based services, retrieve matching events, and invoke your functions.
This invocation model supports Amazon Kinesis, Amazon SQS, and Amazon DynamoDB Streams.
Now that we have covered the basics, let’s do some hands-on with Lambda — That is the best way to learn.
AWS Lambda Example using Event Bridge
Scenario: Suppose you are an admin for an AWS account used by many people, who can create and use EC2 machines. You want a way to identify the EC2 images with the person who created them.
We want to create a Lambda function that tags the EC2 instances when they are created with the name of the person who created the EC2 VM.
- We will use AWS EventBridge to invoke the Lambda whenever any EC2 instance is created.
- We will use the AWS cloud trail to log the API calls in AWS accounts, filter the API logs related to the EC2 instance creation, and pass them to EventBridge. This will ensure that the Lambda function is triggered whenever anyone creates an EC2 instance.
- The lambda function will run some Python code to fetch the instance ID and owner information and tag the AWS instance with the owner’s name.
Enough with the theory, let’s get to it.
1. The first thing we will do is create a cloud trail. Go to the AWS console look for cloud trail and create a new trail.
- CloudTrail -> Create trail
- Create a new bucket to store cloud trail logs.
- Enable the cloud watch log, and create a new log group for logs specific to this cloud trail.
- Go to the next
- Select management events as event type
- Select read and write as API activity.
- Go to next, review, and create a cloud trail.
2. Now we will create a Lambda function
We will use Python runtime for this Lambda function
3. Amazon EventBridge to trigger the Lambda
- Event source as
AWS events or EventBridge partner events
Now select the event pattern
- Event source as
AWS services - AWS services as
EC2 - Event types as
AWS API Call via CloudTrail - Choose events as
specific operationsand writeRunInstancesas operation
The last thing is to choose the target for this Event rule, and for this example, it is the Lambda function ec2-tag-lambda function we created earlier.
You can go to the Lambda and it will look like this.
We need to provide Lambda access to create ec2 tags.
We will attach a policy to the Lambda execution role, this role will give Lambda to perform the desired operations. Click on the Role link and it will redirect you to the role
Edit the policy, and add the below statement to the existing role policy
{
"Sid": "ec2tag",
"Effect": "Allow",
"Action": "ec2:CreateTags",
"Resource": "*"
}If you are new to AWS and not comfortable editing policies, you can attach an existing policy for this demo.
The last thing is to write the Python code to tag the new EC2 instance with the username of the user who created the EC2 instance.
- We will use boto3 for this use case. Boto3 is the Python library for AWS.
- We will fetch the user name and Instance ID from the event that will trigger the Lambda
- Whenever anyone creates an EC2 instance, the AWS cloud trail will log that event. Amazon Eventbridge will look for
RunInstancesthe type of event associated with the EC2 launch. - It will send the event to Lambda, and invoke the Lambda function.
- The lambda function will run the Python code that in turn will tag the EC2 instance.
Lambda python code
# Import the Boto3 library, which is the Amazon Web Services (AWS) SDK for Python.
import boto3
# Define the Lambda handler function that will be executed when the Lambda function is invoked.
def lambda_handler(event, context):
# Extract the owner's username from the event data.
try:
# If logged in as AWS IAM user
owner = event["detail"]["userIdentity"]["userName"]
except Exception as e:
# if logged in as AWS root user
owner = event["detail"]["userIdentity"]["arn"].split(":")[-1]
# Extract the ID of the EC2 instance from the event data.
InstanceId = event["detail"]["responseElements"]["instancesSet"]["items"][0]["instanceId"]
# Create an EC2 client using the boto3 library.
ec2 = boto3.client("ec2")
# Create tags for the EC2 instance. Tags provide metadata for the instance.
ec2.create_tags(
# Specify the resources (in this case, the EC2 instance) to which the tags will be applied.
Resources=[
InstanceId, # ID of the EC2 instance
],
# Specify the tags to be applied to the resources.
Tags=[
{
'Key': 'OWNER', # Tag key
'Value': owner # Tag value (the owner's username)
},
]
)Once you paste the code, deploy the Lambda. And we are all set.
Let’s test this by launching an EC2 instance and see if Lambda does the job.
I created EC2 as a root user and Lambda successfully tagged the EC2 instance with the owner name.
Here is an AWS Lambda implementation using Terraform — Streamlining Email Notifications with AWS Lambda, Python, and SES
That is all for this blog post. See you on the next one
AWS Lambda has way more stuff into its armour and I will cover those areas in the upcoming blog post.
Don’t forget to follow, and subscribe so you won’t miss any of that content.
About me
I am Akhilesh Mishra, a self-taught Devops engineer with 11+ years working on private and public cloud (GCP & AWS)technologies.
Writing is my way of learning new tools and technologies around Devops. It helps me explore various tools, and libraries and try new ways of doing things.
I also mentor DevOps aspirants in their journey to devops by providing guided learning and mentorship on Topmate.
Connect with me on Linkedin: https://www.linkedin.com/in/akhilesh-mishra-0ab886124/
