BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
•
1 like•1,455 views
Everything generates logs. Applications, infrastructure, security ... everything. Keeping track of the flood of log data is a big challenge, yet critical to your ability to understand your systems and troubleshoot (or prevent) issues. In this session, we will use both Amazon CloudWatch and application logs to show you how to build an end-to-end log analytics solution. First, we cover how to configure an Amazon Elaticsearch Service domain and ingest data into it using Amazon Kinesis Firehose, demonstrating how easy it is to transform data with Firehose. We look at best practices for choosing instance types, storage options, shard counts, and index rotations based on the throughput of incoming data and configure a secure analytics environment. We demonstrate how to set up a Kibana dashboard and build custom dashboard widgets. Finally, we dive deep into the Elasticsearch query DSL and review approaches for generating custom, ad-hoc reports.
Report
Share
![What produces data?
• Metering
Records
• Mobile
Apps
• IoT
Sensors
Web
Clickstream
• Enterprise
Documents
• Application
Logs
[Wed Oct 11 14:32:52
2000] [error] [client
127.0.0.1] client
denied by server
configuration:
/export/home/live/ap/ht
docs/test](https://cdn.statically.io/img/www.slideshare.net/data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
- 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Jon Handler, Principal Solutions Architect, AWS August 14, 2017 Deep Dive: Log Analytics with Amazon Elasticsearch Service
- 2. What produces data? • Metering Records • Mobile Apps • IoT Sensors Web Clickstream • Enterprise Documents • Application Logs [Wed Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server configuration: /export/home/live/ap/ht docs/test
- 3. The problem with all of that data
- 4. The solution
- 5. Amazon Elasticsearch Service is a cost-effective managed service that makes it easy to deploy, manage, and scale open-source Elasticsearch for log analytics, full-text search, and more. Amazon Elasticsearch Service
- 6. Data source / Kinesis Firehose Agent Amazon Kinesis Firehose Amazon Elasticsearch Service Kibana Log analytics architecture
- 7. Easy to Use Deploy a production-ready Elasticsearch cluster in minutes Simplifies time-consuming management tasks such as software patching, failure recovery, backups, and monitoring Open Get direct access to the Elasticsearch open-source API Fully compatible with the open-source Elasticsearch API, for all code and applications Secure Secure Elasticsearch clusters with AWS Identity and Access Management (IAM) policies with fine-grained access control access for users and endpoints Automatically applies security patches without disruption, keeping Elasticsearch environments secure Available Provides high availability using Zone Awareness, which replicates data between two Availability Zones Monitors the health of clusters and automatically replaces failed nodes, without service disruption AWS Integrated Integrates with Amazon Kinesis Firehose, AWS IoT, and Amazon CloudWatch Logs for seamless data ingestion AWS CloudTrail for auditing, AWS Identity and Access Management (IAM) for security, and AWS CloudFormation for cloud orchestration Scalable Scale clusters from a single node up to 20 nodes Configure clusters to meet performance requirements by selecting from a range of instance types and storage options, including SSD-powered EBS volumes Amazon Elasticsearch Service Benefits
- 8. Amazon Elasticsearch Service leading use cases Log Analytics & Operational Monitoring • Monitor the performance of applications, web servers, and hardware • Easy to use, powerful data visualization tools to detect issues quickly • Dig into logs in an intuitive, fine-grained way • Kibana provides fast, easy visualization Search • Application or website provides search capabilities over diverse documents • Tasked with making this knowledge base searchable and accessible • Text matching, faceting, filtering, fuzzy search, autocomplete, highlighting, and other search features • Query API to support application search
- 9. Leading enterprises trust Amazon Elasticsearch Service for their search and analytics applications Media & Entertainment Online Services Technology Other
- 10. Adobe Developer Platform (Adobe I/O) P R O B L E M • Cost-effective monitor for XL amount of log data • Over 200,000 API calls per second at peak - destinations, response times, bandwidth • Integrate seamlessly with other components of AWS ecosystem S O L U T I O N • Log data is routed with Amazon Kinesis to Amazon Elasticsearch Service, then displayed using AES Kibana • Adobe team can easily see traffic patterns and error rates, quickly identifying anomalies and potential challenges B E N E F I T S • Management and operational simplicity • Flexibility to try out different cluster config during dev and test Amazon Kinesis Streams Spark Streaming Amazon Elasticsearch Service Data Sources 1
- 11. McGraw-Hill Education P R O B L E M • Supporting a wide catalog across multiple services in multiple jurisdictions • Over 100 million learning events each month • Tests, quizzes, learning modules begun / completed / abandoned S O L U T I O N • Search and analyze test results, student/teacher interaction, teacher effectiveness, student progress • Analytics of applications and infrastructure are now integrated to understand operations in real time B E N E F I T S • Confidence to scale throughout the school year. From 0 to 32 TB in 9 months • Focus on their business, not their infrastructure
- 12. Get set up right
- 13. Easy to use and scalable AWS SDK AWS CLI AWS CloudFormation Elasticsearch data nodes Elasticsearch master nodes Elastic Load Balancing AWS IAM CloudWatchCloudTrail Amazon Elasticsearch Service domain
- 16. Data pattern Amazon ES cluster logs_01.21.2017 logs_01.22.2017 logs_01.23.2017 logs_01.24.2017 logs_01.25.2017 logs_01.26.2017 logs_01.27.2017 Shard 1 Shard 2 Shard 3 host ident auth timestamp etc. Each index has multiple shards Each shard contains a set of documents Each document contains a set of fields and values One index per day
- 17. Deployment of indices to a cluster • Index 1 – Shard 1 – Shard 2 – Shard 3 • Index 2 – Shard 1 – Shard 2 – Shard 3 Amazon ES cluster 1 2 3 1 2 3 1 2 3 1 2 3 Primary Replica 1 3 3 1 Instance 1, Master 2 1 1 2 Instance 2 3 2 2 3 Instance 3
- 18. How many instances? The index size will be about the same as the corpus of source documents • Double this if you are deploying an index replica Size based on storage requirements • Either local storage or up to 1.5 TB of Amazon Elastic Block Store (EBS) per instance • Example: 2 TB corpus will need 4 instances – Assuming a replica and using EBS – Given 1.5 TB of storage per instance, this gives 6TB of storage
- 19. How many shards? • Best practice: shards should be < 50GB • Divide index size by ~40GB to get initial shard count • Adjust for workload: more shards for more write capacity, fewer shards for better read latency • Target ~3 active shards per instance • Always use at least 1 replica for production! • Example: 2 TB corpus will need 50 shards – 2,000 GB / 40GB per shard = 50 shards
- 21. Instance type recommendations Instance Workload T2 Entry point. Dev and test. OK for dedicated masters. M3, M4 Equal read and write volumes. R3, R4 Read-heavy or workloads with high memory demands (e.g., aggregations). C4 High concurrency/indexing workloads I2 Up to 1.6 TB of SSD instance storage.
- 23. Cluster with no dedicated masters Amazon ES cluster 1 3 3 1 Instance 1, Master 2 1 1 2 Instance 2 3 2 2 3 Instance 3
- 24. Cluster with dedicated masters Amazon ES cluster 1 3 3 1 Instance 1 2 1 1 2 Instance 2 3 2 2 3 Instance 3Dedicated master nodes Data nodes: queries and updates
- 25. Master node recommendations Number of data nodes Master node instance type < 10 m3.medium+ < 20 m4.large+ <= 50 c4.xlarge+ 50-100 c4.2xlarge+ Always use an odd number of masters, >= 3
- 27. Cluster with zone awareness Amazon ES cluster 1 3 Instance 1 2 1 2 Instance 2 3 2 1 Instance 3 Availability Zone 1 Availability Zone 2 2 1 Instance 4 3 3
- 28. Small use cases • Logstash co-located on the Application instance • SigV4 signing via provided output plugin • Up to 200 GB of data • m3.medium + 100G EBS data nodes • 3x m3.medium master nodes Application Instance
- 29. Large use cases Amazon DynamoDB AWS Lambda Amazon S3 bucket Amazon CloudWatch • Data flows from instances and applications via Lambda; CWL is implicit • SigV4 signing via Lambda/roles • Up to 5 TB of data • r3.2xlarge + 512 GB EBS data nodes • 3x m3.medium master nodes
- 30. XL use cases Amazon Kinesis • Ingest supported through high-volume technologies like Spark or Kinesis • Up to 60 TB of data today • R3.8xlarge + 640GB data nodes • 3x m3.xlarge master nodes Amazon EMR
- 31. Best practices Data nodes = Storage needed/Storage per node Use GP2 EBS volumes Use 3 dedicated master nodes for production deployments Enable Zone Awareness Set indices.fielddata.cache.size = 40
- 32. Kinesis Firehose
- 33. Kinesis Firehose overview Delivery Stream: Underlying AWS resource Destination: Amazon ES, Amazon Redshift, or Amazon S3 Record: Put records in streams to deliver to destinations
- 34. Kinesis Firehose delivery architecture with transformations S3 bucket source records data source source records Amazon Elasticsearch Service Firehose delivery stream transformed records delivery failure Data transformation function transformation failure
- 35. Lambda blueprints for common use cases
- 36. Transform this 199.72.81.55 - - [01/Jul/1995:00:00:01 -0400] "GET /history/apollo/ HTTP/1.0" 200 6245 To this { "verb": "GET", "ident": "-", "bytes": 6245, "@timestamp": "1995-07-01T00:00:01", "request": "GET /history/apollo/ HTTP/1.0", "host": "199.72.81.55", "authuser": "-", "@timestamp_utc": "1995-07-01T04:00:01+00:00", "timezone": "-0400", "response": 200 }
- 37. Kinesis Firehose features for ingest Serverless scale Error handling S3 Backup
- 38. Best practices Use smaller buffer sizes to increase throughput, but be careful of concurrency Use index rotation based on sizing Default: stream limits: 2,000 transactions/second, 5,000 records/second, and 5 MB/second
- 39. Log analysis with aggregations
- 40. host:199.72.81.55 with <histogram of verb> 1, 4, 8, 12, 30, 42, 58, 100 ... Look up 199.72.81.55 Field data GET GET POST GET PUT GET GET POST Buckets GET POST PUT 5 2 1 Counts
- 41. Amazon ES aggregations Buckets – a collection of documents meeting some criterion Metrics – calculations on the content of buckets Bucket: time Metric:count
- 42. A more complicated aggregation Bucket: ARN Bucket: Region Bucket: eventName Metric: Count
- 43. Demo
- 44. Best practices Elasticsearch provides statistical evaluations based on field data gathered from matching documents Visualizations are based on buckets/metrics Use a histogram on the x-axis first, then sub-aggregate
- 45. Run Elasticsearch in the AWS cloud with Amazon Elasticsearch Service Use Kinesis Firehose to ingest data simply Kibana for monitoring, Elasticsearch queries for deeper analysisAmazon Elasticsearch Service
- 46. What to do next Centralized logging solution: https://aws.amazon.com/answers/logging/centralized-logging/ Our overview page on AWS: https://aws.amazon.com/elasticsearch-service/ Questions? Contact me at handler@amazon.com
- 47. Thank you!