Cassandra Day Atlanta 2016 - Monitoring Cassandra
•
5 likes•1,467 views
This document discusses monitoring Apache Cassandra using metrics. It describes various metric types like gauges, histograms, meters, and timers that can be used to monitor Cassandra. It provides examples of specific metrics that can be monitored for requests, latencies, memory usage, clients, errors, inconsistencies, compactions, thread pools, commit logs, and more. Reporting mechanisms like Graphite and JMX are also covered. The presentation aims to help understand how to gain insights into a Cassandra cluster through effective metric collection and monitoring.
Report
Share
Cassandra Day Atlanta 2016 - Monitoring Cassandra
- 1. CASSANDRA DAY ATLANTA 2016 MONITORING CASSANDRA Aaron Morton @aaronmorton CEO Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License
- 2. AboutThe Last Pickle. Work with clients to deliver and improve Apache Cassandra based solutions. Apache Cassandra Committer and DataStax MVPs. Based in New Zealand,Australia, France & USA.
- 4. codehale / yammer / drop wizard
- 7. Metrics Collection Metrics are always collected.
- 8. Metrics Metrics have a dotted notation name, timestamp, and value e.g. com.thelastpickle.presenters.count=2
- 10. MetricTypes Ratio Gauge. A ratio between two values.
- 11. MetricTypes Histograms. The distribution of values in a stream of data.
- 12. Histograms Quantiles (e.g. 75th, 95th) calculated using reservoir sampling. (Check docs.)
- 13. Histograms Default Exponentially Decaying Reservoirs, (roughly) the last five minutes of data, exponential weighting towards newer data. (Check docs.)
- 14. MetricTypes Meter Measures the per second rate at which a set of events occur.
- 15. Meter Three different exponentially- weighted moving average rates: 1, 5, and 15 minutes
- 16. MetricTypes Timer. Histogram of duration and rate of events .
- 17. Reporting Reporters run in the Cassandra process, pushing metrics to external services.
- 19. Reporters In Cassandra Configuration file: metrics-reporter-config- sample.yaml
- 20. Reporters In Cassandra graphite: - period: 10 timeunit: 'SECONDS' prefix: 'cassandra.prod.ip_1_2_3_4.' hosts: - host: '1.2.3.4' port: 2003 predicate: color: "white" useQualifiedName: true patterns: - "^org.apache.cassandra.metrics.+"
- 23. JMX Cassandra creates JMX MBeans for each Metric.
- 24. JMX
- 25. Reporters Reporters may change the name of measures, e.g. 95thPercentile == p95
- 27. Monitoring and Alerting Use what you like and what works for you.
- 28. Monitoring Platforms OpsCentre, Grafana & Graphite, DataDog, Riemann
- 31. Scale ? Latency? microseconds Rates? per second Data? bytes
- 34. RequestThroughput - All Requests ClientRequest. $REQUEST.Latency.1MinuteRate CASRead, CASWrite, RangeSlice, Read, ViewWrite, Write
- 35. A Note On Requests We will focus on Read, Write But there are others CAS*, RangeSlice, ViewWrite
- 37. Request Latency - All Requests ClientRequest. Write.Latency.95percentile Read.Latency.95percentile
- 38. Request Latency - PerTable Table.$KEYSPACE.$TABLE. CoordinatorReadLatency.95percentile
- 39. Local Latency - PerTable Table.$KEYSPACE.$TABLE. WriteLatency.95percentile ReadLatency.95percentile
- 45. Inconsistency Will also want to monitor dropped messages, later…
- 54. Thanks.
- 55. Aaron Morton @aaronmorton Co-Founder & Principal Consultant www.thelastpickle.com