AWS Lambda, Step Functions & MongoDB Atlas Tutorial

Editor's Notes

1. Hello my name is Raphael Londner with MongoDB and I am here with Paul Sears from Amazon Web Services. Today we are going to discuss building serverless applications with MongoDB Atlas, AWS Lambda and AWS Step Functions.
2. During this talk we will be covering the following topics An overview of AWS Lambda A deep dive into the code A demo of Lambda and MongoDB Atlas Some best practices for using Lambda and MongoDB Atlas And finally an introduction to AWS Step Functions
3. Thank you Raphael So what is Serverless computing? Serverless computing allows you to build and run applications and services without thinking of servers. With serverless computing, your application still runs on servers, but all the server management is done by a service provider. You no longer have to provision, scale, and maintain servers; install and operate databases and storage systems; or manage connections and messages from mobile and IoT devices with serverless architectures. You also no longer need to worry about application fault tolerance and availability. Instead, a service provider does all of this for you. And AWS Lambda lets you use serverless computing.
4. Some benefits of AWS Lambda are no servers to manage -- no worrying about provisioning, availability or fault-tolerance of the infrastructure. AWS Lamba provides continuous scaling as well as subsecond metering, so you only pay for the resources you actually use.
5. So do you use AWS Lambda? Some common patterns are web applications, data processing, scalable back ends, Amazon Alexa integration, and chatbots.
6. Let's talk about Data Processing. For example, a common use case is to automatically generate various image sizes from an image uploaded into S3. When the upload is completed, you can have S3 trigger AWS Lambda to process the image and generate the desired formats
7. Another example of processing data is to use AWS Lambda to act upon data from a stream, such as from Kinesis. Kinesis can trigger a Lambda function and the Lambda function can operate on the data, such as creating metadata, and then store it somewhere, such as MongoDB Atlas.
8. Another popular use case for AWS Lambda is with Alexa skills. Alexa calls a Lambda function to do something, in this example, Alexa triggers Lambda to post a message in a Slack channel, and the the lambda function can poll for a response that Alexa can say to back to the user.
9. AWS Lambda can also be used for Autonomous IT such as policy engines, or managing your infrastructure. So now I will hand the discussion over to Raphael to explain how he is using AWS Lambda and MongoDB Atlas.
10. Sign up for Atlas at www.mongodb.com/atlas Create an M0 free cluster Grab your Atlas connection string Initialize a Node.js project Import MongoDB Node.js driver (or Mongoose) AWS SDK for JavaScript if necessary Write code! (using your Atlas connection string)
11. Use the lambda-local NPM package to test locally Zip your Node.js code with ALL modules Except aws-sdk if used Create your Lambda function in the AWS Console Use the AWS CLI, or packaging tools such as ClaudiaJS or Serverless.com Upload your package to AWS Lambda Create your environment variables (if necessary) Test your function in the AWS Console
12. Now we will share some best practices for AWS Lambda
13. Some things to remember when using AWS Lamba. First - Memory is your performance dial. When running within Lambda, you control how much CPU and memory is available to your function by configuring its memory. If your function is CPU bound, higher settings equals faster runtimes! Second – functions don’t have a built in notion of state. If you need configuration persisted across invocations, or need to persist a session across multiple visits to your web app, you should store it in an external store such as MongoDB Atlas or S3. Another important aspect to remember is that you have granular controls to secure the interactions between each of these components independently. You control which functions can be invoked by which event source, using resource policies on the Lambda function, and you control exactly what AWS services your function has access to using an IAM, or appropriate VPC configuration, and leverage KMS to store secrets for downstream endpoints.
14. One of the key benefits of the serverless approach to application development is that scalability and reliability are built in. Lambda will automatically spin up enough copies of your function to handle the incoming event rate; however, this can cause an impedance mismatch with downstream services that don’t have this kind of scaling behavior. You can either ensure sufficient write capacity for downstream databases, or appropriately configured API call limits for any services being invoked; or, introduce a buffer between the Lambda function and the downstream service using a Kinesis Stream. Scaling also becomes an important consideration when leveraging the recently announced Lambda feature that allows functions to access resources within a VPC. To use this feature, you specify the VPC subnets that a function can access, ideally one within each availability zone. When the function executes, it leverages an Elastic Network Interface or ENI to connect to resources within the specified subnets. Each concurrent request could require its own ENI, which in turn could require an IP, so ensure you size your subnet to match your expected concurrency. This makes scaling a shared responsibility for VPC based functions. And again, a reminder on Lambda’s retry behavior – Asynchronous invokes and invokes from Stream event sources have retried built in. For synchronous invocations such as those originating from API Gateway, you control the retries in the client as required.
15. There are some patterns where Lambda isn’t the optimal choice. For example, enterprise applications that require licensing agreements typically don’t work well in Lambda functions. Also, if your application stack relies on Windows RPCs calls, these won’t work well with Lambda. And if your application relies on specialized or custom hardware, such as GPU acceleration, then it isn’t a good candidate as a lambda function.
16. Now we will share some best practices for AWS Lambda
17. As discussed, Lambdas are stateless, event driven functions that exist in the cloud. But there aren’t that many apps with only one function, one entry point, one module, one component. So there’ll be more than one function. NEXT In fact, it’ll be common to have LOTS of functions, lots of them talking to each other. NEXT And in fact applications, serverless or not, tend to have databases. NEXT And in the cloud, I notice that a lot of them have queues of one kind or another. NEXT Some of the Lambda functions, connect to servers. NEXT This is more like what an actual modern serverless app might really look like.
18. You’ll notice that those arrows are in different colors –because there are lots of ways for functions and other serverless components to talk with each other and control each other and orchestrate each other. Before we dive into the “how”, let’s talk about WHAT we want to do with functions when we have more than one.
19. These are the things that people are trying to accomplish with those different-colored arrows, when coordinating a set of functions into an application.
20. ProductivitySpend more time thinking about innovating the business logic that makes your application unique, and your applications are easier to operate and maintain. AgilityAWS Step Functions records a history of each execution, so you can review in one place, all the events in sequence, in one location. Scale instantly from one execution to hundreds of thousands of concurrent executions, especially when used with other serverless AWS resources such as AWS Lambda, Amazon S3, and MongoDB Atlas. With AWS Step Functions, you pay only for what you use, when you use it. ResilienceAWS Step Functions supports automatic error handling for graceful exits, and operates at scale without you needing to configure or manage its underlying resources.
21. So one example is you are planning to travel to the Grand Canyon. You can break this down into a series of activities, such as booking a flight, and booking a hotel, and booking a rental car. You can run each step in sequence.
22. But if your plans fail, for example you cannot book a car, you want to go back and cancel your hotel and your flight. You will need some kind of decision or coordination to complete all the steps. You don't want waste money on your flight if you don't have any way to drive to the Grand Canyon. In other words, you can use a state machine, such as AWS Step Functions
23. A coordination solution must have several characteristics It needs to scale out as demand grows. You should be able to run one execution, or run thousands. You can never lose state. It deals with errors and time out, and implements things like try/catch/finally It is easy to use and easy to manage It keeps a record of its operations and is completely auditable
24. With AWS Step Functions, you define your workflow as a series of steps and transitions between each step, also known as a state machine. In Step Functions, we call these “states” and “state transitions”. A simple state machine looks like this, and you define it using simple commands in JSON. When you start a state machine, you pass it input in the form of JSON, and each state changes or adds to this JSON blob as output, which becomes input to the next state. The console provides this visualization and uses it to provide near-real-time information on your state machine execution.
25. This is a classic Hello World example. This JSON code on the left, generates this graph on the right. We specify where we start, we define each state, and we define each state transition. And then we get back to working on the code that makes our apps unique.
26. The power of Step Functions is that you can define a state machine once, and then run one or thousands and thousand of concurrent executions. This allows you to break down big tasks into a set of smaller, simpler tasks.
27. The console provides all kinds of information. Anything you see in the console, with the exception of the graph, is also accessible through the API. Let’s step through the elements. In the upper left, I can tab between my graph and my JSON code. This graph is color coded to show me successfully executed states, my current state in progress, failed states, or states not passed during execution. In the upper right panel, I get details about my execution. I can see general info, the input I gave to the execution, in the form of JSON key/value pairs, and the final output of the execution. The Step Details toggle at the bottom, allows me to inspect individual states. I can see the input and the output state by state. This is really useful in debugging when something unexpected occurs. Finally, at the bottom is the complete history of my execution, step by step with timestamps. Again, this is accessible from the API as well as the console.
28. AWS Step Functions supports seven kinds of states today: Task, Choice, Parallel, Wait, Pass, Fail, and Success, and we will have more states in the future. Let’s look a few. Task states do your work. These call on your application components and microservices. There are two kinds of Task states today: one pushes a call to AWS Lambda functions, and the other dispatches tasks to applications, which we call “activity workers”, when these activity workers long-poll for work. Choice states allow you to introduce branching logic to your state machines. Parallel states allow you fork the same input across multiple states, and then join the results into a combined output. This is really useful when you want to apply several independent manipulations to your data, such as image processing or data reduction. When you combine these states, you can build really interesting state machines. Let’s look at a few.
29. For example, with Step Functions, you can build visual work flows. This workflow, which is available as a reference architecture, take a photo and processes using Rekognition. First it extracts the image metadata, then it checks the file type. It exits with a fail if its not a supported file type. It if it is supported, it stores the metadata, and then in parallel, creates a thumbnail and also used recognition to tag objects in the picture.
30. Here are some common patterns for AWS Step Functions Order management Batch processing Replacing shell scripts Enterprise application workflows Data collection and processing
31. With AWS Step Functions you can easily integrate with other AWS Services such as creating your state machine and activities with AWS CloudFormation. You can use Amazon API Gateway to call Step Functions or you can start state machines from a schedule or a response to events with CloudWatch Events. And of course you can monitor the execution of your state machine with Cloud Watch. And for logging, all API calls are captured in CloudTrail
32. Now for some best practices. For lambda the best overall choice is Python due to having the lowest startup latency. However, for warmed functions, Java can sometimes be faster. However, there are some potential downsides such as a high cold start latency. If you are using Java, only include classes that are absolutely necessary.
33. This is another example that is available in a blog on the link on the slide. This is simple employee promotion process which involves a single task: getting a manager’s approval through email. When an employee is nominated for promotion, a new execution starts. The name of the employee and the email address of the employee’s manager are provided to the execution. We use the design pattern shown to implement the manual approval step, and SES to send the email to the manager. After acquiring the task token, the Lambda function generates and sends an email to the manager with embedded hyperlinks to URIs hosted by API Gateway. When the manage select the appropriate URL through API GW, the token is passed to Step Functions and the execution continues.
34. Keep in mind that all Lambda functions run in a VPC, all the time You never need to “turn on” security – it’s always on You can also grant Lambda functions access to resources in your own VPC Functions configured for VPC access lose internet access… unless you have managed NAT or a NAT instance in the VPC
35. FoodPanda is a take-out food is a food delivery service that is making excellent use of Cloud Infrastructure. They have been in the Step Functions beta. They get food orders and have delivery people that take the food to hungry people. The solve this delivery problem with Lambda functions.