Scylla Summit 2017: A Deep Dive on Heat Weighted Load Balancing
•
1 like•1,160 views
This presentation discusses the "cold node problem" that occurs when a node restarts in a Cassandra cluster. When a node restarts, it loses its cached data and becomes a bottleneck. The presentation proposes a "heat weighted load balancing" solution where the cluster tracks each node's cache hit ratio and redistributes requests based on this ratio after a restart. Testing shows this solution significantly improves throughput after a node restart by distributing requests more evenly across nodes based on their "heat" or cache contents.
Report
Share
More Related Content
What's hot
What's hot (16)
Viewers also liked
Viewers also liked (9)
Similar to Scylla Summit 2017: A Deep Dive on Heat Weighted Load Balancing
Similar to Scylla Summit 2017: A Deep Dive on Heat Weighted Load Balancing (7)
More from ScyllaDB
More from ScyllaDB (20)
Recently uploaded
Recently uploaded (20)
Scylla Summit 2017: A Deep Dive on Heat Weighted Load Balancing
- 1. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Heat Weighted Load Balancing Software Engineer, ScyllaDB Gleb Natapov
- 2. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Gleb Natapov 2 I am a long-time open source developer who previously worked on the open source routing suite “Zebra”, the OpenMPI HPC library, KVM hypervisor for Linux, the OSv unikernel, and now Scylla.
- 3. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Cold Node Problem
- 4. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Read Query Flow 4 Client Request Coordinator cql Replica 1 Replica 3 Replica 2 rpc rpc RF=3, CL=QUORUM
- 5. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Replica Selection 5 If a coordinator is also a replica - select it. The rest are selected randomly. Replicas are doing equal amount of work.
- 6. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Steady State 6 Over time all replicas will have hot data in cache. Given constant workload cache miss ratio will stabilize
- 7. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company The Problem 7 What Happens After One Node Restarts?
- 8. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Read Query Flow 8 Client Request Coordinator cql Replica 1 Replica 3 Replica 2 rpc rpc RF=3, CL=QUORUM restart
- 9. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Test Setup 9 ■ Three node cluster ■ Slow storage (to make cache miss effect more pronounced) ■ Warm-Up runs READ with CL=ALL to fully populate row cache on all nodes ■ Test runs is a sequential READ with CL=QUORUM rate limited to 1000 op/s ■ During the test run, one node is restarted
- 10. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Total Requests Served After Node Restart 10 Restart
- 11. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Reads Distribution After Node Restart 11
- 12. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Solution
- 13. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Heat Weighted Load Balancing 13 Smarter request redistribution based on cache hit ratio
- 14. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Cache hit ration info redistribution 14 Node’s cache hit ratio is calculated per column family. It is propagated over the cluster by gossiper and piggybacked to RPC READ request replies.
- 15. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company The Rest Is Easy, No? 15 In general, if we have N (the replication factor, a.k.a RF) different replicas with hit rates (hits/requests) Hi , with a little algebra it can be shown that the share of requests that we need to send to node i is:
- 16. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company But There Are Complications... 16 ■ If a coordinator is also a replica, local communication is prefered and we want to keep it that way ■ CL > 1 makes it harder to redistribute according to probabilities since each replica can be contacted only once
- 17. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Good News 17 We Have A Solution* * Heat Weighted Load Balancing Blog
- 18. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Reads Distribution After Node Restart 18
- 19. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Total Requests Served After Node Restart 19 Before: After:
- 20. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company 99 percentile comparison 20
- 21. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company THANK YOU gleb@scylladb.com Please stay in touch Any questions?