Where to put_my_data
•
4 likes•835 views
This document discusses data storage and management. It covers various data storage models like relational, document, key-value, and graph databases. It emphasizes that the most important factors are providing fast response times under all loads, and selecting a consistency model appropriate for the application's needs. New data storage technologies have made consistency more complex, so applications need to make their consistency rules explicit rather than relying on the storage engine. Data should be engineered to have good response time distributions and be agile, adaptable, and sustainable over the long term as applications and technologies change.
Report
Share
Where to put_my_data
- 1. WHERETO PUT DATA – or – What are we going to do with all this stuff?
- 2. About The Speaker Application Developer/Architect – 21 years Web Developer – 15 years Web Operations – 7 years
- 4. Scalability Consistency Relational Not Only SQL Flexible Rigid
- 5. App Web Media Web API Gateway CMSApp layer Cache farm Search farm Indexer Search Admin Jobbers Caching Proxy/LB Caching Proxy/LB CDN Egress ISP
- 10. OS 2200 Hierarchical ("Network") Database
- 11. OS 2200 Hierarchical ("Network") Database Relational Mapper
- 12. OS 2200 Hierarchical ("Network") Database Relational Mapper POSIX.1 Virtual Machine
- 13. OS 2200 Hierarchical ("Network") Database Relational Mapper POSIX.1 Virtual Machine COBOL Compiler ANSI SQL Library
- 14. Given enough time, and perversity, you can create any query model on top of any storage model.
- 15. SAY WHEN The Importance of ResponseTime Distribution
- 17. THERE ARETHINGSYOU CANNOT KNOW Will a response arrive? When? Was it stored or computed? Is it still true?
- 19. Send Request 100 ms 200 ms ASYMMETRY OFTIME
- 20. Send Request 100 ms 200 ms 300 ms 400 ms 500 ms 600 ms ASYMMETRY OFTIME
- 21. Send Request 100 ms 200 ms 300 ms 400 ms 500 ms 600 ms Time Out ASYMMETRY OFTIME
- 22. 9000 100 200 300 400 500 600 700 800 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Time Elapsed (ms) ConfidenceofResponsebeforeTimeout
- 23. Send Request 100 ms 200 ms 300 ms 400 ms 500 ms 600 ms Time Out Response Arrives ASYMMETRY OFTIME
- 24. To the observer, there is no difference between “too slow” and “not there”.
- 25. 0% 6.25% 12.50% 18.75% 25.00% 0 100 200 300 400 500 600 700 800 900 1000 Response Time Histogram%ofResponses Response Time (ms)
- 26. 0% 6.25% 12.50% 18.75% 25.00% 0 100 200 300 400 500 600 700 800 900 1000 Response Time Histogram%ofResponses Response Time (ms)
- 27. 0% 6.25% 12.50% 18.75% 25.00% 0 100 200 300 400 500 600 700 800 900 1000 Response Time Histogram%ofResponses Response Time (ms)
- 28. 0% 25.00% 50.00% 75.00% 100.00% 0 100 200 300 400 500 600 700 800 900 ∞ Response Time Histogram%ofResponses Response Time (ms)
- 29. This is a talk about data and data storage. scalability? What about Why am I talking so much about observers and response time?
- 30. Why do we worry about scalability?
- 32. 0% 25.00% 50.00% 75.00% 100.00% 0 100 200 300 400 500 600 700 800 900 1000 Response Time Histogram%ofResponses Response Time (ms) Empty Box Time Response time of a single request on an unloaded system.
- 33. Scalability is a means, not an end. What we need is fast response time, under all loads.
- 34. App Web Media Web API Gateway CMSApp layer Cache farm Search farm Indexer Search Admin Jobbers Caching Proxy/LB Caching Proxy/LB CDN Egress
- 35. HOWTRUE?
- 36. App Web Media Web API Gateway CMSApp layer Cache farm Search farm Indexer Search Admin Jobbers Caching Proxy/LB Caching Proxy/LB CDN Egress
- 37. App Web Media Web API Gateway CMS App layer Cache farm Search farm Indexer Search Admin Jobbers Caching Proxy/LB Caching Proxy/LB CDN Egress ISP
- 38. You cannot be sure the data is unchanged since your observation, except by making another observation. P(unch) = F(dC/dt, dt) UNCERTAINTY PRINCIPLE
- 39. THE ROLE OF SURPRISE Unlikely answers are often more interesting.
- 40. SAYS WHO? Thoughts on Consistency
- 41. OBSERVABILITY
- 44. OBSERVABILITY Steve Brian # Steve Brian 1 R R 2 L R 3 R L
- 45. STATE SPACE X1 = {L, R} X2 = {L, R}
- 46. SUPER-OBSERVER Has a view which dominates the views of all other observers.
- 47. SUPER-OBSERVER There are no one-to-many mappings from the super- observer’s states to any other observer’s states.
- 48. SUPER-OBSERVER A super-observer is maximally present if it can discriminate among the Cartesian product of all other observations. Observer Set of States Steve {L, R} Brian {L, R} Super-Observer {L → B, R → F} × {L, R}
- 49. OBSERVABILITY # Steve Brian Super-Observer 1 R R {F, R} 2 L R {B, R} 3 R L {F, L}
- 50. PORKY PIG’S WINDOW SHADE If Porky Pig is looking at the window shade, he always observes it to be down. If he is looking away from the window shade, it rolls up.
- 51. FIRST DIMENSION X1 = {looking, not looking}
- 52. SECOND DIMENSION X1 = {looking, not looking} X2 = {shade open, shade closed}
- 53. FORBIDDEN STATES X1 = {looking, not looking} X2 = {shade open, shade closed}
- 54. STATE SPACE Cartesian product of all possible sets of states. Example 1,000,000 bytes of RAM 8 bits per byte 2 states per bit 8,000,000 dimensions with 2 values each or 1,000,000 dimensions with 256 values each
- 55. STATE SPACE 10,000,000 rows in a table 20 columns Whole database is a single point in a 200,000,000 dimensional space. Changes to data are transforms of that point. State over time is the trajectory of that point.
- 56. CONSISTENCY Not every point in state space is allowed.
- 58. BLACK BOX HYPOTHESIS External observers can only ever ask for projections of the state space, at defined points in time.
- 59. BLACK BOX HYPOTHESIS State space trajectories may cross into forbidden states, as long as those are not revealed to observers.
- 60. PROJECTION X1 = {looking, not looking} Is Porky looking at the window shade? P1 P2
- 61. BLACK BOX HYPOTHESIS Even two clustered machines have their own state spaces. It’s impossible for either to be a superobserver.
- 62. OBSERVED CONSISTENCY Sufficient to ensure that forbidden states cannot be observed.
- 63. DOES A SUPEROBSERVER EXIST? Only if there is exactly one single-threaded CPU, in exactly one computer.
- 64. CONSEQUENCES Consistency doesn’t exist in most systems today. Sometimes we can fake it. Many times, it doesn’t really matter.
- 65. WHAT ABOUT CAP? Consistency: “...there must exist a total order on all operations such that each operation looks as if it were completed at a single instant.” Seth Gilbert and Nancy Lynch. 2002. Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services. SIGACT News 33, 2 (June 2002), 51-59. DOI=10.1145/564585.564601 http://doi.acm.org/10.1145/564585.564601
- 66. WHAT ABOUT CAP? Linearizability Seth Gilbert and Nancy Lynch. 2002. Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services. SIGACT News 33, 2 (June 2002), 51-59. DOI=10.1145/564585.564601 http://doi.acm.org/10.1145/564585.564601
- 67. The data base consists of entities which are related in certain ways. These relationships are best thought of as assertions about the data.
- 68. Examples of such assertions are: “Names is an index forTelephone_numbers.” “The value of Count_of_X gives the number of employees in department X.”
- 69. The data base is said to be consistent if it satisfies all its assertions. In some cases, the data base must become temporarily inconsistent in order to transform it to a new consistent state. From "Granularity of Locks and Degrees of Consistency in a Shared Data Base", J.N. Gray, R.A. Lorie, G.R. Putzolu, I.L.Traiger, 1976 From "Granularity of Locks and Degrees of Consistency in a Shared Data Base", J.N. Gray, R.A. Lorie, G.R. Putzolu, I.L.Traiger, 1976
- 70. Consistency is a predicate C on entities and their values.The predicate is generally not known to the system but is embodied in the structure of the transactions. From "Transactions and Consistency in Distributed Database Systems", I.L.Traiger, J.N. Gray, C.A. Galtieri, and B.G. Lindsay, 1982
- 71. “C” VERSUS “A”? Response Time Consistency Resilience See also: http://goo.gl/1Yv3 → http://dbmsmusings.blogspot.com/2010/04/problems-with-cap-and-yahoos-little.html
- 72. WHAT’STHAT? Data Models and Composability
- 73. DATA MODEL DEFINED The system’s representation of the consistency predicate C.
- 74. LATENT MODEL Implicit in the structure of application code.
- 75. EXPLICIT Visible to storage engine or applications, expressed in machine-readable form.
- 76. HOMOICONIC Explicit, and available for expressions, computations, and validation together with statements about the data itself.
- 81. ENFORCING C Must account for overlapping wavefronts of information. There is no master clock. Simultaneity is positional. Make C explicit in the application, don’t rely on storage engine.
- 82. HOW LONG? On Lifecycles and Lifespans
- 92. DOWN WITHTHE IRON FIST Throw out the DBAs Throw out the schemas Unstructured Semi-structured
- 93. DOWN WITHTHE IRON FIST Put the application in charge.
- 94. DOWN WITHTHE IRON FIST but...
- 95. DIFFICULTIES Application versions Validating correct behavior Capturing knowledge about that behavior
- 96. UGLYTRUTH Data routinely outlives applications.
- 98. WHERE NOW?
- 99. WHERETO PUTYOUR DATA? Data exists everywhere.
- 100. WHERETO PUTYOUR DATA? Nothing lasts forever.
- 101. WHERETO PUTYOUR DATA? Understand freshness.
- 102. WHERETO PUTYOUR DATA? Engineer a good response time distribution.
- 103. WHERETO PUTYOUR DATA? Select the consistency model you need.
- 104. WHERETO PUTYOUR DATA? Be agile and adaptable.
- 105. WHERETO PUTYOUR DATA? Make it sustainable.
- 106. Michael T. Nygard michael.nygard@n6consulting.com @mtnygard © 2010 N6 Consulting, LLC.All Rights Reserved.