Monitor All Your Things: Amazon CloudWatch in Action with BBC (DEV302) - AWS re:Invent 2018

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Monitor All Your Things: Amazon
CloudWatch in Action with BBC
Brian Dennehy
Director of Engineering
AWS
D E V 3 0 2
Christopher Darlaston
Development Lead
BBC

Monitoring matters because …
Visibility Real-time
troubleshooting
#Customer
experience
Applications
= $$
BusinessOperational

Full stack
visibility
Short-lived
resources
^Devices
^Data
Monolithic to
microservice
Faster release
velocity

Cloud native
defaults
Single solution for
metrics and logs
Highly scalable
Monitor with
automation
Logs
Metrics
Alarms
Events
Dash-
boards
Agent
& APIs

and Log analytics
Collect Monitor Act Analyze

Christopher Darlaston—BBC
• Development lead in interactive TV
• Seven years in interactive TV on BBC
iPlayer, Sport, News and Frameworks
• Previous 13 years working at Sun
Microsystems in their web teams

BBC Interactive TV
overview
Giving users access to additional TV
programming.
Press the red button on your TV
remote control to enjoy additional
coverage from the big events:
• Glastonbury Festival (Music)
• Wimbledon (Tennis, Grand Slam)
• Olympic Games

Simplified architecture—Unconnected Red Button
AWS Direct
Connect
Main
Data Playout
Carousel Injection
Amazon EC2
Amazon EC2
Private
Public
Amazon EFS
Carousel Storage
Carousel Creation
Amazon EC2
Amazon
DynamoDB
Amazon
Kinesis
AWS
Lambda
Amazon
S3
Amazon
CloudWatch

Collecting metrics and logs via CloudWatch agent
{
"metrics": {
"aggregation_dimensions": [ ["AutoScalingGroupName", "InstanceId"], ["AutoScalingGroupName”] ],
"append_dimensions": { "InstanceId": "${aws:InstanceId}”, "AutoScalingGroupName": "${aws:AutoScalingGroupName}” },
"metrics_collected": {
"mem": { "measurement": ["mem_used", "mem_cached", "mem_used_percent", "mem_available_percent”] },
"processes": { "measurement": ["running", "sleeping", "dead”] },
"disk": {"resources": ["/"], "measurement": ["free", "used_percent”] },
"netstat": {"measurement": ["tcp_established”] },
"cpu": { "totalcpu": false, "resources": ["*"], "measurement": ["cpu_usage_iowait", "cpu_usage_idle", "cpu_usage_nice”] }
},
"namespace": "live-broadcast-red-button-linkmanager-api"
},
"logs": {
"logs_collected": {
"files": {
"collect_list": [{
“file_path": "/var/log/broadcast-red-button-linkmanager-api/output.log",
"log_group_name": "live-broadcast-red-button-linkmanager-api-infrastructure-ApplicationLog-J8FGOWKDFOE8",
"log_stream_name": "{instance_id}-{ip_address}-output.log"
}]
}
},
"log_stream_name": "{instance_id}-{hostname}"
},
"agent": { "logfile": "/var/log/amazon-cloudwatch-agent/amazon-cloudwatch-agent.log”, "metrics_collection_interval": 60 }
}

Collecting metrics from log extraction

Monitoring view—Typical day

Alerting on issues using CloudWatch alarms

Monitoring view—Day of trouble

Diagnosing—Is it downstream or on premise ?

Diagnosing—Is it upstream of us?

Flexibility—Dashboard created during incident

Monitoring view—Full day

Split
the problem space
Log
everything
Do you have the right
dashboards?
What did we learn?

Why do we use CloudWatch?
“Our interactive services, just like
picking up your phone and making a
call, needs to just work at all times.
We deliver journalistic content and
news, which are fundamental services
that our users expect in real-time and
on-demand without failure.”
1. End-to-end visibility for on-premise
and cloud
Log analytics for both on-premise & Amazon Web
Services (AWS)
2. Monitoring with automation
Resource optimization, snapshot graphs
3. Correlate & investigate issues in real
time
CloudWatch agent & dashboards
4. More time back to focus on BBC
innovation

Reinvent & simplify: Lessons learned
inform our future
What’s new

NEW: CloudWatch Automatic Dashboards
CloudWatch simplifies infrastructure monitoring with a default, getting started
experience
Dynamic, self-
updating AWS
infrastructure
dashboards

Building operational dashboards takes time &
experience
“I just want a quick, summary view …”
“I just want some default recommendations …”
“Oh, not all statistics and visualizations are created equal …”
“I create dashboards one by one and someone always forgets …”

Automatic
Explore account &
resource-based views of
health and performance
metrics
Smart
Browse defaults with
built-in AWS best
practices, including
metrics, statistics, and
visualizations
Dynamic
Auto-scrub metrics of
resources that no longer
exist to reduce stale
views via resource-aware
updates
Granular
Easily drill down for
troubleshooting with
AWS or resource group
filtering

Collect
everything with ease
using defaults for building
operational visibility
Automate
monitoring
with new CloudWatch
automated operational
dashboards
Session key takeaways
Correlate
metrics and logs
for faster
troubleshooting and
understanding root
cause

More sessions:
AWS booth for demos
DEV375 “Amazon CloudWatch Logs Is Making an Exciting Announcement!”
DEV311 “Breaking Observability Chaos: Best Practices to Monitor AWS Cloud Native Apps”
DEV301R “AIOPs: Find Your Needle in the Haystack”
DEV306R1 “Monitoring for Operational Outcomes and Application Insights: Best Practices Workshop”
DEV303R “Instrumenting Kubernetes for Observability Using AWS X-Ray and Amazon CloudWatch”
WIN202L “Leadership Session: Learn about 10 Years’ of Windows and .NET Innovation on AWS with
10 New Launches”
What else is new:
Metric Math alarms
Log insights
CloudWatch agent with collectd and StatsD
integration
Snapshot graphs
Events support for AWS organizations

Thank you!
Brian Dennehy
Christopher Darlaston

Monitor All Your Things: Amazon CloudWatch in Action with BBC (DEV302) - AWS re:Invent 2018

Related slideshows

More Related Content

Monitor All Your Things: Amazon CloudWatch in Action with BBC (DEV302) - AWS re:Invent 2018