Мониторинг. Опять, rootconf 2016
•
2 likes•590 views
This document discusses metrics monitoring systems. It provides an overview of the Graphite monitoring system architecture, including the carbon-cache and carbon-relay components. It then evaluates various open source alternatives for handling high volumes of metrics data, finding that a combination of go-carbon and carbon-c-relay can process over 1.4 million requests per second. The document also discusses tagging metrics with metadata and time series databases that support tags.
Report
Share
Мониторинг. Опять, rootconf 2016
- 2. Platform Engineer . Grammarly ctrlok.com
- 6. Успешность
- 7. Количество
- 8. Время
- 10. Взаимодействие
- 14. Алерты
- 15. Аналитика
- 17. Graphite
- 19. what? • RRD-like (gram.ly/gfsx) • so.it.is.my.metric → /so/it/is/my/metric.wsp • Fixed retention (by namepattern) • Fixed size (actually no)
- 20. Retention and size • 1s:1d → 1 036 828 bytes • 10s:10d → 1 036 828 bytes • 1s:365d → 378 432 028 bytes (1 TB ~ 3 000) • 10s:365d → 37 843 228 bytes (1 TB ~ 30 000) whisper calc
- 21. Retention and size • 10s:30d,1m:120d,10m:365d → 4 564 864 bytes • 240 864 metrics in 1 TB • aggregation: average, sum, min, max, and last. • can be assign per metric
- 22. How • terraform (https://www.terraform.io/) • docker (https://www.docker.com/) • ansible (https://www.ansible.com/) • rocker (https://github.com/grammarly/rocker) • rocker-compose (https://github.com/grammarly/rocker-compose)
- 24. carbon-cache.py • single-core • many options in config file • default link
- 26. Start load testing • m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2) • retentions = 1s:1d • MAX_CACHE_SIZE, MAX_UPDATES_PER_SECOND, MAX_CREATES_PER_MINUTE = inf • defaults • almost 1.5h to get limit :(
- 27. carbon-cache.py cache size → 75k ms
- 30. results • 75 000 ms max • 60 000 ms flagman speed • IO :(
- 31. Try to tune! • WHISPER_SPARSE_CREATE = true (don’t allocate space on creation) non-linear IO load. • CACHE_WRITE_STRATEGY = sorted (default)
- 32. cache size 1k → 195k ms
- 33. results • 120 000 ms flagman speed • cache flush problem :(
- 34. Try to tune! • CACHE_WRITE_STRATEGY = max will give a strong flush preference to frequently updated metrics and will also reduce random file-io.
- 35. from 1k to 150k
- 36. results • 90 000 ms flagman speed • cache flush problem :(
- 37. Try to tune! • CACHE_WRITE_STRATEGY = naive just flush. Better with random IO.
- 38. from 45k to 135k
- 39. results • 120 000 ms flagman speed • still CPU
- 40. sorted max naive
- 41. • Maybe it’s IO EBS limitation? → 512 GB disk. • No.
- 42. go-carbon • multi-core single daemon • written in golang • not many options to tune :( link
- 43. Start load testing • m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2) • retentions = 1s:1d • max-size = 0 • max-updates-per-second = 0 • almost 1h to get limit :(
- 44. 1k → 130k ms ~3k/min
- 46. results • 120 000 ms flagman speed • but it’s without sparse. • try to implement
- 47. try to tune! remaining := whisper.Size() - whisper.MetadataSize() whisper.file.Seek(int64(remaining-1), 0) whisper.file.Write([]byte{0}) chunkSize := 16384 zeros := make([]byte, chunkSize) for remaining > chunkSize { // if _, err = whisper.file.Write(zeros); err != nil { // return nil, err // } remaining -= chunkSize } if _, err = whisper.file.Write(zeros[:remaining]); err != nil { return nil, err }
- 48. Уже есть в go-carbon
- 49. 180 000 ms !
- 51. try to tune! • max update operation = 1500
- 52. results • TLDR 210 000 - 240 000 ms flagman speed • 31 000 000 cache size!
- 54. try to tune! • max update operation = 0 • input-buffer = 400 000
- 55. results • 270 000 ms flagman speed • 10-20kk cache size!
- 57. try to tune! • vm.dirty_background_ratio=40 • vm.dirty_ratio=60
- 58. 300 000 reqs
- 59. results • 300 000 ms flagman speed • 180k+ ms ±without cache
- 60. Re:Lays
- 62. arch forward
- 63. arch namedregexp
- 64. arch hash
- 65. arch hash replicafactor: 2
- 66. carbon-relay.py • twisted based • native
- 67. Start load testing • c4.xlarge instance (4 CPU, 7.5 GB ram) • ~1 Gb lan • default parameters • hashing • 10 connections
- 68. WTF!
- 69. carbon-relay-ng • golang-based • web-panel • live-updates • aggregators • spooling link
- 70. <150 000 reqs
- 71. carbon-c-relay • написан на C • advanced cluster management
- 72. from 100 000 to 1 600 000 reqs
- 73. 1 400 000 flagman speed. Or not?
- 74. Итак… go-carbon + carbon-c-relay = ♡
- 80. Контейнеры
- 81. Всё перепутано
- 82. Различия • Окружение • Роль • Трек (Модификатор) • IP • Датацентр • Что-угодно
- 83. Теги
- 84. TSDB с тегами • influxDB • openTSDB (hbase) • cyanite (cassandra) • newTS (cassandra) • Prometheus
- 85. (cluster) influx, 130k metrics увеличить график
- 87. openTSDB single instance + hbase cluster = upto 150k metrics
- 92. Compaction
- 94. Graphite
- 99. Найти уникальное
- 107. Работает с Grafana
- 108. Zipper • https://github.com/grobian/carbonserver • https://github.com/dgryski/carbonzipper • https://github.com/dgryski/carbonapi
- 111. ALSO • https://github.com/jssjr/carbonate • https://github.com/jjneely/buckytools • https://github.com/dgryski/carbonmem • https://github.com/grobian/carbonwriter
- 112. Планы • Патч statsd → ES • Патч carbonserver → carbonlink
- 113. feel free to ask • Vsevolod Polyakov • ctrlok@gmail.com • skype: ctrlok1987 • github.com/ctrlok • twitter.com/ctrlok • slack: HangOps • Gitter: dev_ua/devops • skype: DevOps from Ukraine • slack.ukrops.club Мы хайрим!