Wikimedia Content API: A Cassandra Use-case
•
0 likes•598 views
Among the resources offered by Wikimedia is an API providing low-latency access to full-history content, in many formats. Its results are often the product of computationally intensive transforms, and must be pre-generated and stored to meet latency expectations. Unsurprisingly, there are many challenges to providing low-latency access to such a large data-set, in a demanding, globally distributed environment. This presentation covers the Wikimedia content API and its use of Apache Cassandra as storage for a diverse and growing set of use-cases. Trials, tribulations, and triumphs, of both a development and operational nature will be discussed.
Report
Share
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Wikimedia Content API: A Cassandra Use-case
Similar to Wikimedia Content API: A Cassandra Use-case (20)
More from Eric Evans
Recently uploaded
Recently uploaded (20)
Wikimedia Content API: A Cassandra Use-case
- 1. Wikimedia Content API: A Cassandra Use-case Eric Evans <eevans@wikimedia.org> @jericevans Apache Bigdata | May 10, 2016
- 3. Our Vision: A world in which every single human can freely share in the sum of all knowledge.
- 4. About: ● Global movement ● Largest collection of free, collaborative knowledge in human history ● 16 projects ● 16.5 billion total page views per month ● 58.9 million unique devices per day ● More than 13k new editors each month ● More than 75k active editors month-to-month
- 5. About: Wikipedia ● More than 38 million articles in 290 languages ● Over 10k new articles added per day ● 13 million edits per month ● Ranked #6 globally in web traffic
- 7. LAMP
- 8. THE ARCHITECTURE ALL OF IT
- 11. Wikitext = Star Wars: The Force Awakens = Star Wars: The Force Awakens is a 2015 American epic space opera film directed, co-produced, and co-written by [[J. J. Abrams]].
- 12. HTML <h1> Star Wars: The Force Awakens </h1> <p> Star Wars: The Force Awakens is a 2015 American epic space opera film directed, co-produced, and co-written by <a href="/wiki/J._J._Abrams" title="J. J. Abrams"> J. J. Abrams </a> </p>
- 13. Wikitext
- 14. HTML
- 16. WYSIWYG
- 20. Metadata [[Foo|{{echo|bar}}]] <a rel="mw:WikiLink" href="./Foo"> <span about="#mwt1" typeof="mw:Object/Template" data-parsoid="{...}" >bar</span> </a>
- 21. Parsoid ● Node.js service ● Converts wikitext to HTML/RDFa ● Converts HTML/RDFa to wikitext ● Semantics, and syntax (avoid dirty diffs)! ● Expensive (slow) ● Resulting output is large
- 22. RESTBase ● Services aggregator / proxy (REST) ● Durable cache (Cassandra) ● Wikimedia’s content API (e.g. https://en.wikipedia.org/api/rest_v1?doc)
- 24. Other use-cases ● Mobile content service ● Math formula rendering service ● Dumps ● ...
- 25. Cassandra
- 26. Environment ● 2 datacenters ● 3 racks per datacenter ● 18 hosts (16 core, 128G, SSDs) ● 36 nodes ● Deflate compression (~14-18%) ● 31T storage (~206T uncompressed) ● Cassandra 2.1.13 (moving to 2.2.6) ● Read-heavy workload (5:1)
- 27. Data model
- 28. Data model CREATE TABLE data ( domain text, title text, rev int, tid timeuuid, value blob, PRIMARY KEY ((domain, title), rev, tid) ) WITH CLUSTERING ORDER BY (rev DESC, tid DESC)
- 29. Data model en.wikipedia.org + Star_Wars:_The_Force_Awakens 717862573 717873822 ...97466b12...7c7a913d3d8a1f2dd66c...7c7a913d3d8a ... 09877568...7c7a913d3d8a bdebc9a6...7c7a913d3d8a827e2ec2...7c7a913d3d8a
- 30. Compression
- 33. Brotli compression ● Brought to you by the folks at Google; Successor to deflate ● Cassandra implementation (https://github.com/eevans/cassandra-brotli) ● Initial results very promising ● Better compression, lower cost (apples-apples) ● And, wider windows are possible (apples-oranges) ○ GC/memory permitting ○ Example: level=1, lgblock=4096, chunk_length_kb=4096, yields 1.73% compressed size! ○ https://phabricator.wikimedia.org/T122028 ● Stay tuned!
- 34. Compaction
- 35. Compaction ● The cost of having log-structured storage ● Asynchronously (post-write) optimize data on disk for reads ● At a minimum, reorganize into fewer files ○ Dropping what is obsolete ○ Expiring TTLs ○ Removing deleted (aka tombstoned) data (after a fashion) ● Reorganize data so results are nearer each other
- 36. Compaction strategies ● Size-tiered ○ Combines tables of similar size ○ Oblivious to column distribution; Works best for workloads with no overwrites/deletes ○ Needs working space for largest possible compaction (100%) ○ Minimal IO ● Leveled ○ Small, fixed size files in levels of exponentially increasing size ○ Files have non-overlapping ranges within a level ○ Amortized; Lots of continuous compaction ○ Very efficient reads, but also quite IO intensive ● Date-tiered
- 37. Date-tiered compaction ● Newest of the bunch; Introduced in 2.1 ● For append-only workloads (no overwrites, no deletes) ● Where data is ordered sort-ascending, (and not written out-of-order) ● Windows of time, arranged in tiers ● Avoids mixing “old” data with “new” ● Cold data eventually ceases to be compacted
- 38. 1 1 2 2 31 2-51 2 3 41 1 2 min_threshold (4) base_time_seconds 3-6
- 39. Date-tiered compaction ● Newest of the bunch; Introduced in 2.1 ● For append-only workloads (no overwrites, no deletes) ● Where data is ordered sort-ascending, (and not written out-of-order) ● Windows of time, arranged in tiers ● Avoids mixing “old” data with “new” ● Cold data eventually ceases to be compacted ● Hard to reason about ● Optimizations are easily defeated ● https://phabricator.wikimedia.org/T126221
- 41. DTCS: So now what? ● Size-tiered compaction? Might as well. ● TimeWindowCompactionStrategy (https://github.com/jeffjirsa/twcs)? Maybe... ● Reduce node density?
- 43. GC ● Early adopters of G1 (Garbage 1st) ● Successor to Concurrent Mark-sweep (CMS) ● Incremental parallel compacting collector ● More predictable than CMS ● Configurable pause-time target
- 44. Concurrent Mark-sweep Heap Old generation Young generation Eden Survivor
- 45. G1 Survivor Old-Gen Eden Eden Survivor Old-Gen Old-Gen Eden Survivor Old-Gen
- 46. Humongous objects ● Anything >= ½ region size is classified as Humongous ● Humongous objects are allocated into Humongous Regions ● Only one object for a region (wastes space, creates fragmentation) ● Until 1.8u40, humongous regions collected only during full collections (Bad) ● Since 1.8u40, end of the marking cycle, during the cleanup phase (Better) ● Treated as exceptions, so should be exceptional ○ For us, that means 8MB regions ● Enable GC logging and have a look!
- 47. Multi-instance
- 48. “Many smaller-sized Cassandra nodes is always better than fewer, dense ones.” — Everyone
- 50. What we do ● Processes (yup) ● Puppetized configuration ○ /etc/cassandra-a/ ○ /etc/cassandra-b/ ○ systemd units ○ Etc ● Shared RAID-0
- 51. What we should have done ● Virtualization ● Containers ● Blades
- 52. The Good ● Fault-tolerance ● Availability ● Datacenter / rack awareness ● Nice, helpful people (tickets, IRC, etc)
- 53. The Bad ● Usability ○ Compaction ○ Streaming ○ JMX ● Vertical scaling ● JVM
- 54. The Ugly ● Release process ● Upgrades