SlideShare a Scribd company logo

Cache Rules Everything Around Me - DevIntersection - December 2022

•Download as PPTX, PDF•
•
M
Matthew Groves

Caching can bring speed to the slow systems in your enterprise. In this session, we'll explore how to avoid expensive computations and reduce database (or other systems) load with caching. Considerations include when to cache (cache policy), how to cache, and when/how to evict. This session will also explore and discuss problems and gotchas like cache sizing, eviction mistakes, and where your cache should live. Finally, we'll look at how caching is implemented in the "memory-first" Couchbase architecture. Presented at DevIntersection 2022 in Las Vegas

Related slideshows

Developing High Performance and Scalable ColdFusion Application Using Terraco...
Developing High Performance and Scalable ColdFusion Application Using Terraco...Developing High Performance and Scalable ColdFusion Application Using Terraco...
Developing High Performance and Scalable ColdFusion Application Using Terraco...
 
Developing High Performance and Scalable ColdFusion Applications Using Terrac...
Developing High Performance and Scalable ColdFusion Applications Using Terrac...Developing High Performance and Scalable ColdFusion Applications Using Terrac...
Developing High Performance and Scalable ColdFusion Applications Using Terrac...
 
Northeast PHP - High Performance PHP
Northeast PHP - High Performance PHPNortheast PHP - High Performance PHP
Northeast PHP - High Performance PHP
 
1 of 48
C.R.E.A.M. get the memory (dollar dollar bill ya'll) August 2022 Matthew D. Groves | Product Marketing Manager aka “Passionate Bravo” Cache Rules Everything Around Me
2 Latency Numbers Every Programmer Should Know https://stackoverflow.com/questions/4087280/approximate-cost-to-access-various-caches-and-main-memory/4087315#4087315
3 The Need for Speed https://www.slideshare.net/devonauerswald/walmart-pagespeedslide
4 Cache Rules Everything Around Me Dollar dollar bill ya'll Android apps iPhone apps CPU cache CDNs Web browser DNS
01/ 02/ 03/ 04/ 05/ 06/ What is caching? Use Cases Reading from Cache Writing into Cache Cache Invalidation Gotchas Agenda
6 ho am I? • Matthew D. Groves • Microsoft MVP, author at Pluralsight, Manning, Apress • Product Marketing Manager at Couchbase • https://twitter.com/mgroves • https://www.linkedin.com/in/mgroves/ • C# Advent: https://csadvent.christmas/
7 1 hat is Caching?
8 Caching Collection of duplicated data to serve future requests faster. • First formally described by Maurice Wilkes in 1965 • "Slave Memories and Dynamic Storage Allocation" https://bit.ly/WilkesPaper What is Caching? Key Data 1 Proteck Ya Neck 2 C.R.E.A.M. 3 Brooklyn Zoo 4 Criminology … … Key Data 2 C.R.E.A.M. 4 Criminology Records Cache
9 Powerpoint as a Cache What is Caching? Key Value What year was Wu-Tang Clan Formed? 1992
10 2 Caching Use Cases
11 Caching Use Cases What is Caching used for?
12 Caching Use Case 1: Faster Database Access • Databases can rely heavily on disk • Reading the same data over again could mean accessing the disk over again • Store frequently accessed data in a cache, to reduce pressure on database/disk
14 Caching Use Case 1: Faster Database Access DATABASE APP LAN CACHE CACHE CACHE
15 Caching Use Case 2: Faster Web Browsing • Any given URL could result in hundreds of files that are loaded over HTTP • A single JS file may be loaded by every page on a site. • Cache static assets (JS, images, sound, icons, etc) on a user's laptop to avoid loading them over the network every time
16 Caching Use Case 2: Faster Web Browsing CACHE
17 Caching Use Case 3: Efficient API Use • API usage may involve an internet call (with latency) • API calls may cost per use, or may be limited • Storing results in cache may help reduce latency AND reduce costs
19 Caching Use Case 3: Efficient API Use API APP LAN/WAN CACHE CACHE CACHE
20 Caching Use Case 4: Maintaining Availability • Retrieving data from other processes • Microservices • Latency is secondary concern
21 Caching Use Case 4: Maintaining Availability CACHE
22 Cache Rules Everything Around Me Dollar dollar bill ya'll Disconnected Availability Slow Repetition Constrained Costs
23 3 Reading From Cache
24 Using a Cache 101 Your Code Key Value A Protect Ya Neck B C.R.E.A.M. C Brooklyn Zoo Cache Key Value A Protect Ya Neck B C.R.E.A.M. C Brooklyn Zoo D Criminology E Triumph F Uzi G Method Man H Wu-Tang Clan Ain't . . . . . . . . . Database 1. Lookup "A" 2. Lookup "B" 3. Lookup "C" HIT! HIT! MISS! 4. Lookup "C" again? WALK OF SHAME
25 4 Writing Into Cache
26 What about writes? • Data is duplicated, stored in cache and record • The system is responsible for keeping the cache updated • Commons methods: • write-through • write-around • write-back Stuff needs to get in the cache somehow
27 write-through 1. System gets a new/updated piece of data 2. Write to cache and record "at the same time" 1. Write to record 2. If that didn't succeed, the write failed. 3. Write to cache 4. If that didn't succeed, the write failed. 5. If the system reaches this point, the write succeeded! Why or why not use this?
28 write-around 1. System gets a new/updated piece of data 2. Write the data to the record ONLY 3. Invalidate the record in the cache Why or why not use this?
29 write-back 1. System gets a new/updated piece of data 2. Write the data to the cache. 3. Update the record asynchronously (don't wait on it to finish) 4. If the cache write succeeded, then consider the write succeeded. Why or why not use this?
30 Cache Writing Methods Method Write performance Write integrity Best for…? write-through ➖ ➕ High % reads write-around ➖ ➕ High % read later write-back ➕ ❔ Mixed reads / writes
31 Couchbase A database with a built-in managed cache https://docs.couchbase.com/server/current/learn/buckets-memory-and-storage/memory-and-storage.html
32 Couchbase and Writing to the Cache await collection.InsertAsync("A", new { title = "Protect Ya Neck" }); await collection.InsertAsync("A", new { title = "Protect Ya Neck" }, options => { options.Durability(DurabilityLevel.None); // OR: options.Durability(DurabilityLevel.Majority); // OR: options.Durability(DurabilityLevel.MajorityAndPersistToActive); // OR: options.Durability(DurabilityLevel.PersistToMajority); });
33 Durability (with 3 replicas) None Memory Disk Node 1 Node 2 Node 3 ✔ ⚠ ⚠ ⚠ ⚠ ⚠ Node 4 ⚠ ⚠
34 Durability (with 3 replicas) Majority Memory Disk Node 1 Node 2 Node 3 ✔ ⚠ ⚠ ⚠ ✔ ⚠ Node 4 ✔ ⚠
35 Durability (with 3 replicas) MajorityAndPersistToActive Memory Disk Node 1 Node 2 Node 3 ✔ ✔ ⚠ ⚠ ✔ ⚠ Node 4 ✔ ⚠
36 Durability (with 3 replicas) PersistToMajority Memory Disk Node 1 Node 2 Node 3 ✔ ✔ ⚠ ⚠ ✔ ✔ Node 4 ✔ ✔
37 Latency Numbers Every Programmer Should Know https://stackoverflow.com/questions/4087280/approximate-cost-to-access-various-caches-and-main-memory/4087315#4087315
38 5 Cache Invalidation
39 Powerpoint as a Cache Key Value What year was Wu-Tang Clan Formed? 1992 What was Wu-Tang's first album called? Enter the Wu-Tang What year was Maurice Wilkes born? 1913 What year did Maurice Wilkes die? 2010 What is the most recent Wu- Tang studio album? Once Upon a Time in Shaolin
40 Cache Invalidation is Hard There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton How to decide what to invalidate (what data to "evict"): cache policy https://www.nndb.com/people/400/000031307/
41 Cache Policy 1: LRU (Least Recently Used) Key How new? A 0 B 1 3 C 2 D 4 Cache max size: 3 Order of reads: A, B, C, B, D X ---------------------------------------------------------------------------------
42 Cache Policy 2: NRU (Not Recently Used) Lower scores evicted (at random) Key Referenced? Modified? Score A 0 0 0 B 0 0 0 C 0 1 1 D 1 0 2 E 1 1 3
43 Cache Policy 3: None Don't evict anything
44 More Cache Policies • RR – random replacement • FIFO and LIFO – treat a cache like a queue • MRU – MOST recently used • LFU – least frequently used • Belady's algorithm – not possible to implement • Machine learning • https://en.wikipedia.org/wiki/Cache_replacement_policies
45 6 Gotchas
46 Gotcha 1: Cache Size • Problem: too small or too big cache • Symptom: High turnover and churn (many evictions) or long lifetimes (few evictions) • Solutions: • Monitoring • Size the cache • "Value" vs "Full" eviction
47 Gotcha 2: "Close" Caching • Problem: running the cache on the same machine as the application • Symptom: redundancy, hot spots, inconsistency, availability • Solutions: • Use a distributed cache • Running on its own machine(s) / cluster • Couchbase, of course!
48 Gotcha 3: Bad Eviction Policy • Problem: using the wrong eviction policy • Symptom: lots of overhead, cache checking • Solutions: • Don't Write an Eviction Policy (or better yet don't write a Cache) • Benchmarking: "hit ratio" and "latency" • "Hit ratio" – how often a hit vs miss (e.g. "45% hit ratio") • "Latency" – how long it takes from request to response (e.g. 10μs latency)
THANK YOU
50 References • https://www.geeksforgeeks.org/not-recently-used-nru-page-replacement-algorithm/ • https://shahriar.svbtle.com/Understanding-writethrough-writearound-and-writeback- caching-with-python • https://en.wikipedia.org/wiki/Maurice_Wilkes • https://safari.ethz.ch/digitaltechnik/spring2021/lib/exe/fetch.php?media=wilkes.pdf • https://stackoverflow.com/a/4087315/40015 • https://www.techtarget.com/searchstorage/definition/cache • https://en.wikipedia.org/wiki/Cache_replacement_policies • https://docs.couchbase.com/dotnet-sdk/current/howtos/kv-operations.html#durability • https://docs.couchbase.com/dotnet-sdk/current/concept-docs/durability-replication-failure- considerations.html#durable-writes

More Related Content

Cache Rules Everything Around Me - DevIntersection - December 2022

  • 1. C.R.E.A.M. get the memory (dollar dollar bill ya'll) August 2022 Matthew D. Groves | Product Marketing Manager aka “Passionate Bravo” Cache Rules Everything Around Me
  • 2. 2 Latency Numbers Every Programmer Should Know https://stackoverflow.com/questions/4087280/approximate-cost-to-access-various-caches-and-main-memory/4087315#4087315
  • 3. 3 The Need for Speed https://www.slideshare.net/devonauerswald/walmart-pagespeedslide
  • 4. 4 Cache Rules Everything Around Me Dollar dollar bill ya'll Android apps iPhone apps CPU cache CDNs Web browser DNS
  • 5. 01/ 02/ 03/ 04/ 05/ 06/ What is caching? Use Cases Reading from Cache Writing into Cache Cache Invalidation Gotchas Agenda
  • 6. 6 ho am I? • Matthew D. Groves • Microsoft MVP, author at Pluralsight, Manning, Apress • Product Marketing Manager at Couchbase • https://twitter.com/mgroves • https://www.linkedin.com/in/mgroves/ • C# Advent: https://csadvent.christmas/
  • 7. 7 1 hat is Caching?
  • 8. 8 Caching Collection of duplicated data to serve future requests faster. • First formally described by Maurice Wilkes in 1965 • "Slave Memories and Dynamic Storage Allocation" https://bit.ly/WilkesPaper What is Caching? Key Data 1 Proteck Ya Neck 2 C.R.E.A.M. 3 Brooklyn Zoo 4 Criminology … … Key Data 2 C.R.E.A.M. 4 Criminology Records Cache
  • 9. 9 Powerpoint as a Cache What is Caching? Key Value What year was Wu-Tang Clan Formed? 1992
  • 11. 11 Caching Use Cases What is Caching used for?
  • 12. 12 Caching Use Case 1: Faster Database Access • Databases can rely heavily on disk • Reading the same data over again could mean accessing the disk over again • Store frequently accessed data in a cache, to reduce pressure on database/disk
  • 13. 14 Caching Use Case 1: Faster Database Access DATABASE APP LAN CACHE CACHE CACHE
  • 14. 15 Caching Use Case 2: Faster Web Browsing • Any given URL could result in hundreds of files that are loaded over HTTP • A single JS file may be loaded by every page on a site. • Cache static assets (JS, images, sound, icons, etc) on a user's laptop to avoid loading them over the network every time
  • 15. 16 Caching Use Case 2: Faster Web Browsing CACHE
  • 16. 17 Caching Use Case 3: Efficient API Use • API usage may involve an internet call (with latency) • API calls may cost per use, or may be limited • Storing results in cache may help reduce latency AND reduce costs
  • 17. 19 Caching Use Case 3: Efficient API Use API APP LAN/WAN CACHE CACHE CACHE
  • 18. 20 Caching Use Case 4: Maintaining Availability • Retrieving data from other processes • Microservices • Latency is secondary concern
  • 19. 21 Caching Use Case 4: Maintaining Availability CACHE
  • 20. 22 Cache Rules Everything Around Me Dollar dollar bill ya'll Disconnected Availability Slow Repetition Constrained Costs
  • 22. 24 Using a Cache 101 Your Code Key Value A Protect Ya Neck B C.R.E.A.M. C Brooklyn Zoo Cache Key Value A Protect Ya Neck B C.R.E.A.M. C Brooklyn Zoo D Criminology E Triumph F Uzi G Method Man H Wu-Tang Clan Ain't . . . . . . . . . Database 1. Lookup "A" 2. Lookup "B" 3. Lookup "C" HIT! HIT! MISS! 4. Lookup "C" again? WALK OF SHAME
  • 24. 26 What about writes? • Data is duplicated, stored in cache and record • The system is responsible for keeping the cache updated • Commons methods: • write-through • write-around • write-back Stuff needs to get in the cache somehow
  • 25. 27 write-through 1. System gets a new/updated piece of data 2. Write to cache and record "at the same time" 1. Write to record 2. If that didn't succeed, the write failed. 3. Write to cache 4. If that didn't succeed, the write failed. 5. If the system reaches this point, the write succeeded! Why or why not use this?
  • 26. 28 write-around 1. System gets a new/updated piece of data 2. Write the data to the record ONLY 3. Invalidate the record in the cache Why or why not use this?
  • 27. 29 write-back 1. System gets a new/updated piece of data 2. Write the data to the cache. 3. Update the record asynchronously (don't wait on it to finish) 4. If the cache write succeeded, then consider the write succeeded. Why or why not use this?
  • 28. 30 Cache Writing Methods Method Write performance Write integrity Best for…? write-through ➖ ➕ High % reads write-around ➖ ➕ High % read later write-back ➕ ❔ Mixed reads / writes
  • 29. 31 Couchbase A database with a built-in managed cache https://docs.couchbase.com/server/current/learn/buckets-memory-and-storage/memory-and-storage.html
  • 30. 32 Couchbase and Writing to the Cache await collection.InsertAsync("A", new { title = "Protect Ya Neck" }); await collection.InsertAsync("A", new { title = "Protect Ya Neck" }, options => { options.Durability(DurabilityLevel.None); // OR: options.Durability(DurabilityLevel.Majority); // OR: options.Durability(DurabilityLevel.MajorityAndPersistToActive); // OR: options.Durability(DurabilityLevel.PersistToMajority); });
  • 31. 33 Durability (with 3 replicas) None Memory Disk Node 1 Node 2 Node 3 ✔ ⚠ ⚠ ⚠ ⚠ ⚠ Node 4 ⚠ ⚠
  • 32. 34 Durability (with 3 replicas) Majority Memory Disk Node 1 Node 2 Node 3 ✔ ⚠ ⚠ ⚠ ✔ ⚠ Node 4 ✔ ⚠
  • 33. 35 Durability (with 3 replicas) MajorityAndPersistToActive Memory Disk Node 1 Node 2 Node 3 ✔ ✔ ⚠ ⚠ ✔ ⚠ Node 4 ✔ ⚠
  • 34. 36 Durability (with 3 replicas) PersistToMajority Memory Disk Node 1 Node 2 Node 3 ✔ ✔ ⚠ ⚠ ✔ ✔ Node 4 ✔ ✔
  • 35. 37 Latency Numbers Every Programmer Should Know https://stackoverflow.com/questions/4087280/approximate-cost-to-access-various-caches-and-main-memory/4087315#4087315
  • 37. 39 Powerpoint as a Cache Key Value What year was Wu-Tang Clan Formed? 1992 What was Wu-Tang's first album called? Enter the Wu-Tang What year was Maurice Wilkes born? 1913 What year did Maurice Wilkes die? 2010 What is the most recent Wu- Tang studio album? Once Upon a Time in Shaolin
  • 38. 40 Cache Invalidation is Hard There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton How to decide what to invalidate (what data to "evict"): cache policy https://www.nndb.com/people/400/000031307/
  • 39. 41 Cache Policy 1: LRU (Least Recently Used) Key How new? A 0 B 1 3 C 2 D 4 Cache max size: 3 Order of reads: A, B, C, B, D X ---------------------------------------------------------------------------------
  • 40. 42 Cache Policy 2: NRU (Not Recently Used) Lower scores evicted (at random) Key Referenced? Modified? Score A 0 0 0 B 0 0 0 C 0 1 1 D 1 0 2 E 1 1 3
  • 41. 43 Cache Policy 3: None Don't evict anything
  • 42. 44 More Cache Policies • RR – random replacement • FIFO and LIFO – treat a cache like a queue • MRU – MOST recently used • LFU – least frequently used • Belady's algorithm – not possible to implement • Machine learning • https://en.wikipedia.org/wiki/Cache_replacement_policies
  • 44. 46 Gotcha 1: Cache Size • Problem: too small or too big cache • Symptom: High turnover and churn (many evictions) or long lifetimes (few evictions) • Solutions: • Monitoring • Size the cache • "Value" vs "Full" eviction
  • 45. 47 Gotcha 2: "Close" Caching • Problem: running the cache on the same machine as the application • Symptom: redundancy, hot spots, inconsistency, availability • Solutions: • Use a distributed cache • Running on its own machine(s) / cluster • Couchbase, of course!
  • 46. 48 Gotcha 3: Bad Eviction Policy • Problem: using the wrong eviction policy • Symptom: lots of overhead, cache checking • Solutions: • Don't Write an Eviction Policy (or better yet don't write a Cache) • Benchmarking: "hit ratio" and "latency" • "Hit ratio" – how often a hit vs miss (e.g. "45% hit ratio") • "Latency" – how long it takes from request to response (e.g. 10μs latency)
  • 48. 50 References • https://www.geeksforgeeks.org/not-recently-used-nru-page-replacement-algorithm/ • https://shahriar.svbtle.com/Understanding-writethrough-writearound-and-writeback- caching-with-python • https://en.wikipedia.org/wiki/Maurice_Wilkes • https://safari.ethz.ch/digitaltechnik/spring2021/lib/exe/fetch.php?media=wilkes.pdf • https://stackoverflow.com/a/4087315/40015 • https://www.techtarget.com/searchstorage/definition/cache • https://en.wikipedia.org/wiki/Cache_replacement_policies • https://docs.couchbase.com/dotnet-sdk/current/howtos/kv-operations.html#durability • https://docs.couchbase.com/dotnet-sdk/current/concept-docs/durability-replication-failure- considerations.html#durable-writes