AWS IoT: From Testing to Scaling
•Download as PPTX, PDF•
0 likes•186 views
Companies create IoT proof of concepts (PoCs) or small tests to fine-tune IoT designs before deploying new technology across a plant or plants. Large-scale deployments present challenges that might not be uncovered during the PoC stage. In this session, we cover the most common challenges companies fall victim to when they move from testing to deployment and how AWS IoT services give customers flexible and scalable solutions that help them scale to meet their IoT needs regardless of the number of devices connected.
Report
Share
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
More Related Content
What's hot
What's hot (9)
Similar to AWS IoT: From Testing to Scaling
Similar to AWS IoT: From Testing to Scaling (20)
Recently uploaded
Recently uploaded (20)
AWS IoT: From Testing to Scaling
- 2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. From testing to scaling I O T 3 0 7 - R Catalin Vieru Sr. Specialized Architect - IoT AWS Neel Sendas Sr. Technical Account Manager AWS Russell Potapinski Head of Intelligent and Autonomous Systems Woodside Energy
- 3. Related breakouts IOT207-R - [REPEAT] Digital transformation and IoT monetization (ft. AB InBev) IOT209-R - [REPEAT] Building smarter devices for a better life (ft. Belkin) IOT210-R - [REPEAT] Post-launch planning for IoT deployments (ft. iRobot) IOT312-R - [REPEAT] Bringing AWS IoT and robotics together (ft. Amazon Robotics)
- 4. Agenda • What makes industrial IoT challenging • Lessons learned: Woodside Energy • Industrial AWS IoT architecture • From 0 to 1: prototyping • From 1 to N: scaling
- 5. What makes industrial IoT challenging Insular • Same machine, different places, different performance. Why? • Same facility: How to stitch data together? • Difficult to optimize operations Legacy • Heterogenous solutions • Proprietary integrations • Share little data • Specialized know-how Culture • Focuses on individual pieces of machinery • Promotes machine- specific knowledge • Difficult to embrace new paradigms
- 6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 9. Woodside IoT devices Photo: Courtesy of Woodside Energy
- 10. Data ingestion AWS Cloud AWS IoT AnalyticsAWS IoT Core AWS IoT Greengrass LNG Plant Device Gateway Sensors Edge Processing Amazon S3 AWS IoT Events
- 11. Scaling from 1… to 100s… to 1,000s
- 12. Monitoring architecture Enriching Monitoring / Alerting Amazon DynamoDB AWS LambdaAmazon Kinesis Data Analytics Amazon Elasticsearch Service AWS IoT Core IoT rule Amazon CloudWatch AWS IoT Events AWS Lambda WebEx Teams Ingestion Grafana
- 13. AWS IoT Events – state machines made easy Quickly build state machines Notifications on state changes Less error-prone than coding by hand
- 14. ChatOps - making the data accessible
- 15. Dashboards
- 16. Takeaways to your scaling success - Real life is different from the lab - Good monitoring is paramount to your success - Native IoT services help you scale - AWS IoT Events; state machines made easy - Use an event-based architecture - allows monitoring to evolve separately from development
- 17. Conveyor belt condition monitoring Factory Conveyor Belt Conveyor Belt NVIDIA Jetson TX2 Gateway IoT MQTT topic Over-the- air update Lambda function Inference Model AWS AWS IoT Core AWS IoT Greengrass AWS Lambda IoT rule IoT action AWS IoT Greengrass Resources Amazon Sagemaker NotebookModel Train Amazon Simple Storage Service Amazon DynamoDB AWS Elastic Beanstalk – web app Users IoT topic Camera Camera OPC-UA Adaptor
- 18. Path to scalability starts with 1 device
- 19. Path to scalability: automation with AWS CDK AWS Cloud Development Kit (CDK): a software development framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation • Author AWS CDK projects which are executed to generate CloudFormation templates. AWS CDK projects can be executed using the AWS CDK command line or in a continuous delivery system AWS CDK Stack Lambda function Construct Lambda Construct Greengrass AWS IoT Greengrass AWS CloudFormation Template AWS Resources AWS AWS CDK CLI
- 20. Using CDK to create an AWS IoT Greengrass core npm install -g aws-cdk mkdir aws-cdk-greengrass-sample cd aws-cdk-greengrass-sample cdk init --language typescript npm install --save @aws-cdk/aws-lambda npm install --save @aws-cdk/aws-greengrass npm install --save @aws-cdk/aws-iot
- 21. Using CDK to create an AWS IoT Greengrass core import cdk = require('@aws-cdk/core'); import lambda = require('@aws-cdk/aws-lambda'); export class GreengrassLambdaStack extends cdk.Stack { public readonly greengrassLambdaAlias: lambda.Alias; constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) { super(scope, id, props);
- 22. Using CDK to create an AWS IoT Greengrass core // Create and Deploy Lambda to Greengrass const greengrassLambda = new lambda.Function(this, 'GreengrassSampleHandler', { runtime: lambda.Runtime.PYTHON_3_7, code: lambda.Code.asset('handlers'), handler: 'handler.handler', }); const version = greengrassLambda.addVersion('GreengrassSampleVersion'); // Greengrass Lambda specify the alias this.greengrassLambdaAlias = new lambda.Alias(this, 'GreengrassSampleAlias', { aliasName: 'nvidiaLambda', version: version }) } }
- 23. Using CDK to develop an app #!/usr/bin/env node import 'source-map-support/register'; import cdk = require('@aws-cdk/core'); import { GreengrassNvidiaStack } from '../lib/greengrass-nvidia-stack'; import { GreengrassLambdaStack } from '../lib/greengrass-lambda-stack'; const app = new cdk.App(); const lambdaStack = new GreengrassLambdaStack(app, 'GreengrassLambdaStack'); new GreengrassNvidiaStack(app, 'GreengrassNvidiaStack', { greengrassLambdaAlias: lambdaStack.greengrassLambdaAlias });
- 24. Deploy AWS IoT Greengrass app cdk bootstrap npm run build cdk GreengrassNvidiaStack deploy
- 25. Deploy AWS IoT Greengrass app Including dependency stacks: GreengrassLambdaStack GreengrassLambdaStack This deployment will make potentially sensitive changes according to your current security approval level (--require-approval broadening). Please confirm you intend to make the following modifications: IAM Statement Changes ┌───┬────────────────────────────────────────────┬────────┬─────────┬────────────────────── ────────┬───────────┐ │ │ Resource │ Effect │ Action │ Principal │ Condition │ creating CloudFormation changeset...
- 26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 27. Path to scalability: best practices—data processing Know the difference between data buffering and queuing • Kinesis can buffer data up to 7 days • Real-time processing of data • Multiple applications can read from the same stream: fanning out helps prevent a fragile downstream architecture • Enables stream-driven architecture AWS IoT Core Kinesis Data source Lambda Data source IoT Data Buffering
- 28. Path to scalability: best practices—data processing Stream processing at scale • Have a safe way to persist your data until you can process it, up to 15 days • Be able to handle data spikes seamlessly • Allow downstream services to scale • When throttled, hold on to data • Consider batch processing and cost optimization • Enables event-driven architecture AWS IoT Core Data Source Lambda Data Source Amazon SNS Amazon SQS Integration to event streams IoT Data Queuing
- 29. Path to scalability: summary Automate resource creation • AWS CDK + CI/CD + Amazon CloudFormation Architect for data ingestion at scale • Build a common framework with interoperability in mind • Introduce monitoring early on Rely on native platform services and features • AWS IoT Events • AWS IoT SiteWise • AWS IoT Analytics
- 30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. 25+ additional free digital courses cover topics related to IoT, including: Take the free digital curriculum, Internet of Things (IoT) Foundation Series, to build IoT skills and work through common scenarios Learn IoT with AWS Training and Certification • AWS IoT Core • AWS IoT Greengrass • AWS IoT Analytics • AWS IoT Device Management • AWS IoT Events Visit the Learning Library at https://aws.training Resources created by the experts at AWS to help you build IoT skills
- 31. Thank you! © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Editor's Notes
- Who is Woodside Energy? Woodside is the pioneer of the LNG industry in Australia and the largest Australian natural gas producer. We have a global portfolio and are recognised for our world-class capabilities as an integrated upstream supplier of energy. Our operated assets are renowned for their safety, reliability and efficiency and we have a strong track record in project development. As Australia's premier LNG operator, we produce 6% of global LNG supply. We operate two floating production storage and offloading (FPSO) facilities. Technology, combined with our pioneering, innovative spirit, is a key supporter of our corporate strategy. From the first LNG plant in the southern hemisphere, to commissioning the world’s largest not-normally crewed offshore platform, innovation is in our DNA. We’ve long been a leader in applying conventional oil and gas technologies, supported by our growing capabilities in data science, analytics and machine learning. Now, we are taking the industry to new frontiers by adopting and developing innovations that build on our strengths, yet challenge our thinking. Our technology strategy increases focus on carbon management and developing new energy markets and sources, ensuring the resilience of our business for decades to come.
- We are installing a data-driven digital nerve system at our operating facilities that will provide real-time insights, enabling better decision making, cost reductions and higher reliability. To achieve this, we are combining the use of IoT, Robotics and Data Science (AI?) to build an “Intelligent Asset” with a goal of achieving “Better than human awareness”, allowing our staff to make better decisions, faster from where every they work. Being aware that providing more data by its self does not add value, our approach for the intelligent asset follows our model of : Sense – Insight – Action. For sensing we are utilising low cost wireless sensors and robotics for mobile awareness, aiming to trivialise the cost of acquiring data. For Insight we work with our data science team to apply advanced analytics from recommendations and predictions. We also utilize computer vision and machine learning to automate visual inspections. We also know people are still our key decision makers and leverage advanced visualisation like AR and VR to providing engaging interfaces. In action we are working how to present this information to field workers to help them in their tasks and reduce the amount of manual data entry. We are also working on robotic manipulation with task autonomy and how this all comes together to provide an integrated control environment.
- In house built devices: - ETPCam (audio snip and still image – wifi) - wake up every 6 hours, take image, transmit, sleep. battery lasts indefinitely with solar energy harvesting - Vibtemp - Ex zone 1 rated (LoRaWAN) – tx every 15 mins, battery lasts for 2-5 years All connecting back to AWS via IoT Core. Paramount importance to us is this data: firmware version power cycles battery level connectivity info
- We need to get the images, temp, vibration data into a place where we can easily retrieve and process it. Starting on prem at the LNG plant, the sensors securely send the data to a local device gateway, which then send into AWS Greengrass. Greengrass extends AWS to the edge, where we can perform compute and ML inferences, such as object detection, locally, which allows for a faster time to act. Also, Greengrass offers data caching, so in the event of network loss, no data is lost.. And is sent on reconnection. VERY useful for these remote sites When connection is available, data is sent from Greengrass into IoT Core – here you can specify actions depending on the data received. We always store the raw data in S3, enrich using Lambda’s with data such as device location, type etc We currently use Elasticsearch to query the data, but are looking forward to replacing this with IoT Analytics – a one stop shop where we can enrich, store and analyse the data. On top of all this and not depicted here are API’s and other processing which allows the data to be consumed by upstream applications. The front end queries these API’s and displays to the user. Woodside thought big here.. They did not settle that the only way to sense was to buy expensive off the shelf equipment. Building your own, and using the AWS IoT Device SDKs, they were able to create an Ex rated sensor that is in the order of a hundred bucks as opposed to many thousands.
- The real world is different! The wifi on our assets was not the same as the lab more powercycles as it switched between different access points monitoring gave us the data to retune and adapt (screenshots/quotes from the AWS Well Architected Framework ) Monitoring is key.. I’ll now talk about how we do this at Woodside
- Starting from the left.. Our data lands in the cloud in IoT Core. From here, we use Kinesis Data Analytics to batch up messages and send to a Lambda In the lambda we enrich the data eg. We get ”rssi” wifi strength. That figure on it’s own is not too useful. we want to know if it differs and is trending We look this up via Elasticsearch and dynamoDB. We can then push an enriched input into IoT Events I’ll speak more about IoT Events later, but this is where we determine the state of our devices. As we hit each state, we send out a message via lambda, hooked into Webex teams This means we get proactive alerting to our developers and admins. Additionally, we have dashboards, and we use Grafana for that – (next slide.. examples of dashboards)
- IoT Events, if you have not used it, was released at last reinvent. There are many IoT services now, and it’s hard to keep up! But this one we have found particularly useful for managing state IOT Events has a bunch of triggers, from SNS, SQS and also Lambda. We use it to create CloudWatch metrics which power our dashboards, and also to trigger the webhooks. Using IOTE over coding this up, we make it easy for our devs to update the machine.. it’s very visual, and takes away much of the room for error as opposed to coding it oh btw.. you can use the visual editor to create your state machine, but once you scale you want to export it and then convert to CloudFormaton (bottom right) I’ll give you a quick walkthrough of one of our state machines, and you can see how easy it is when using IOTE to create a useful state machine.
- (click to start Video... talk while it is in background) So the dashboards are good, but alerting is better Webex Teams is company chat tool, and we have a webhook and a bot to help bring the data closer Our bot allows admins and developers to query device data easily, especially useful for quick investigations. Making the data accessible via the bot has much improved investigation times, and we’d recommend implementing something similar from the start as you scale out your deployments. (transition) Now, the states you see being reported are being generated using IoT Events... (next slide)
- As well as dashboards, we have our alerting. Example on left : - Power cycles are really important - These devices wake up every 6 hours or so, so 4 cycles per day - normal operation, a linear graph. If we detect a spike, there is an issue more power cycles per day == less battery life. Something is wrong. Example on right: a simple graph of the status of our cameras. This is our staging environment, and you can see we have a few offline. but again, alerting is proactive, and we alert via webex teams (examples on next slide)
- Speaker Notes: You came to re:Invent to learn. There’s no need to stop when you go home. Keep re:Inventing with resources from AWS Training and Certification for IoT - for you and your teams. Using videos and hands-on exercises, you’ll explore a variety of AWS IoT service. On-demand and digital, IoT courses are convenient, available for free, and developed by the experts at AWS. The popular Internet of Things Foundation Series is a good place to start with 7 hours of on-demand instruction for everyone interested in the topic. For more information, visit https://aws.training and see the IoT courses in the Learning Library.