Json data modeling june 2017 - pittsburgh tech fest

Editor's Notes

1. Spend just a little time on why people are using NoSQL Talk about how data is modeled differently in JSON Let’s talk about why SQL is good and why SQL for JSON is needed Let’s talk about the exciting stuff happening in the database ecosystem Including but not limited to the stuff Couchbase is doing If we have time, we’ll look at how a .NET developer (or Java developer, etc) would interact with SQL for JSON
2. This session is a WIP. It’s based on my knowledge of Couchbase, SQL server experience, and David Segleau’s engagement and lessons learned with customers, all combined into an hour presentation. David likes bullet points, I like to break up bullet points and use lots of pictures. David works with customers, I work with dev community. So you’re going to see a meshing of that, hopefully it works.
3. What’s also interesting is that we’re seeing the use of NoSQL expand inside many of these companies. Orbitz, the online travel company, is a great example – they started using Couchbase to store their hotel rate data, and now they use Couchbase in many other ways. Same with ebay, they recently presented at the Couchbase conference with a chart tracking how many instances of various nosql databases are in use, and we see growth in Cassandra, mongo, and couchbase has actually surpassed them within ebay
4. SQL (relational) databases are great. They give you LOT OF functionality. Great set of abstractions (tables, columns, data types, constraints, triggers, SQL, ACID TRANSACTIONS, stored procedures and more) at a highly reasonable cost. Change is inevitable One thing RDBMS does not handle well is CHANGE. Change of schema (both logical and physical), change of hardware, change of capacity. NoSQL databases ESPECIALLY ONES DESIGNED TO BE DISTRIBUTED tend to help solve problems with: agility, scalability, performance, and availability
5. Let’s talk about what NoSQL is, first. NoSQL generally refers to databases which lack SQL or don’t use a relational model Once the SQL language, transaction became optional, flurry of databases were created using distinct approaches for common use-cases. KEY-Value simply provided quick access to data for a given KEY. Wide Column databases can store large number of arbitrary columns in each row Graph databases store data and relationships as first class concepts Document databases aggregate data into a hierarchical structure. With JSON is a means to the end. Document databases provide flexible schema,built-in data types, rich structure, implicit relationships using JSON.
6. When we look at document databases, they originally came with a Minimal set of APIs and features But as they continue to mature, we’re seeing more features being added And generally I’m seeing a convergent trend between SQL and NoSQL But anyway, this set of minimal features, lacking a SQL language and tables gives us the buzzword “nosql”
7. Elastic scaling Size your cluster for today Scale out on demand Cost effective scaling Commodity hardware On premise or on cloud Scale OUT instead of Scale UP [example: changing the channel to a soccer game or Game of Thrones, everyone makes the same API request in the same 5 minutes] [example: TV show lets watchers vote during some period of the week, so you can scale up during that period of time] [example: black Friday]
8. Schema flexibility Easier management of change in the business requirements Easier management of change in the structure of the data Sometimes you're pulling together data, integrating from different sources (e.g. ELT) and that flexibility helps Document database means that you have no rigid schema. You can do whatever the heck you want. That being said, you SHOULDN’T. You should still have discipline about your data.
9. NoSQL systems are optimized for specific access patterns Low response time for web & mobile user experience Millisecond latency Consistently high throughput to handle growth [perf measures can be subjective – talk about architecture, integrated cache, maybe mention MDS too]
10. If one machine goes down, customers can still use the other. Or if you need to perform maintenance, upgrade, etc, you don't have to take the whole system down This is related to scaling Built-in replication and fail-over No application downtime when hardware fails Online maintenance & upgrade No application downtime
11. Let’s talk about data modeling a bit, because storing data in JSON Is different that storing in tables.
12. So I want to compare the approaches over 4 key areas. I’m going to fill in this table, traditional SQL on the left and JSON on the right
13. Let’s look at modeling Customer data. This is an example of what a customer might look like There is a rich structure: attributes, potentially sub-attributes (first name and last name) Relationships: to other data (other customers, to products perhaps) Value evolution: Maybe we’d start with one connection, change to multiple (data is updated) Structure evolution: Maybe we start without connections and add those later, or we evolve name field to be more than first and last name (data is reshaped)
14. Rich Structure In relational database, this customers data would be stored in five normalized tables. Each time you want to construct a customer object, you JOIN the data in these tables; Each time you persist, you find the appropriate rows in relevant tables and insert/update. Relationship Enforcement is via referential constraints. Objects are constructed by JOINS, EACH time. Value Evolution Additional values of the SAME TYPE (e.g. additional phone, additional address) is managed by additional ROWS in one of the tables. Customer:contacts will have 1:n relationship. Structure Evolution: Imagine we didn't start with a billing table. This is the most difficult part. Changing the structure is difficult, within a table, across tables. While you can do these via ALTER TABLE, requires downtime, migration and application versioning. This is one of the problem document databases try to handle by representing data in JSON.
15. Let’s see how to represent customer data in JSON. The primary (CustomerID) becomes the DocumentKey Column name-Column value becomes KEY-VALUE pair.
16. We aren’t normal form anymore Rich Structure & Relationships Billing information is stored as a sub-document There could be more than a single credit card. So, use an array.
17. Value evolution Simply add additional array element or update a value.
18. Structure evolution Simply add new key-value pairs No downtime to add new KV pairs Applications can validate data Structure evolution over time. Relations via Reference
19. So, finally, you have a JSON document that represents a CUSTOMER. In a single JSON document, relationship between the data is implicit by use of sub-structures and arrays and arrays of sub-structures.
20. Reference slide
21. What types of relationships are being modeled? How are the relationships accessed?
22. Let’s talk about data modeling a bit, because storing data in JSON Is different that storing in tables.
23. We’ll focus on N1QL for now.
24. Notice I’m using Guid That may not be a good idea
25. N1QL is powerful in it's flexibility, declarative nature, familiar to developers, JOINs, etc. Indexing is very important, as it's not as performant as key/value or map/reduce (Maybe talk about indexing on a SQL table vs indexing on a whole bucket)
26. Couchbase 5.0 has introduced some tools for analyzing query performance So you can see what indexes are being used, where the biggest costs are in the query And so on. There are a lot of different types of indexes for N1QL
27. This is kinda like a materialized view It's powerful in that it can be run in parallel, can use JavaScript to do filtering/mapping, great for aggregation. It's limited in that it can't do anything like a JOIN, can't get input from other views, and more
28. Let’s talk about data modeling a bit, because storing data in JSON Is different that storing in tables.
29. Are you going to take the time to clean up the data? Do you need to? Do you need to enrich or restructure the data to take advantage of Json? Duration v resources: how long is it going to take? What tools and resources are available to you? Data governance: what are the rules for moving data, auditing, etc?
30. Duration v resources: how long is it going to take? What tools and resources are available to you? What’s your biggest constraint – time or resources? Do you need to get the migration done in 1 hr (and have it use as many parallel resources as needed) or do you need to minimize/manage the resource impact on the existing system and it doesn’t matter how long it takes?
31. Data governance: what are the rules for moving data, auditing, etc? Do you need to keep track of where the data came from and who is allowed to access it? Many newer systems need to track where sensitive data originated. 
32. A whole bunch at a time, or one at a time Single threaded – easier Multi-threaded – faster, complicated is the migration a one-time event or does it need to happen incrementally (every day or over a 2-3 month period where both the old system and new system are both operating in parallel)? Do you plan to do the data migration as a single thread (read all the data, write all of the data) or using a multi-threaded or multi-process approach where each thread or process reads some percentage of the data.
33. If you're writing your own, Entity Framework can be helpful, because it can do the mapping of aggregate root C# objects for you, which you can then write to a document database So if you already have EF mappings created, you're part way there.
34. KISS: Either export to CVS and use N1QL to do any ETL that’s required (assuming that it’s Simple) or use SQL to do simple ETL on export and then just import into CB. Basically keep it as simple as you can and plan for failure. Developers often think of the migration process as “One and Done”, but the reality is that data migration is often an ongoing headache that DevOps needs to monitor and manage in a production environment. Make everyone’s life easier by thinking about the long game as much as possible.
35. From NoSQL to relational
36. From relational to NoSQL: Goldendate is from oracle Cdata for SSIS and Couchbase https://github.com/mahurtado/CouchbaseGoldenGateAdapter https://www.cdata.com/drivers/couchbase
37. Make it part of your application directly May or may not be reusable This is a lot of work, so make sure you have a good reason
38. Let’s talk about data modeling a bit, because storing data in JSON Is different that storing in tables.
39. Focus on SOA, application/use case specific
40. Use Document type, Versionid Create optimized, understandable keys Weigh nested, referenced or mixed designs Add indexes: Simple, Compound, Functional, Partial, Array, Covering, Memory Optimized
41. N1QL, Key-value, Views,
42. Focus, Success Criteria, Review Architecture consider using a tool like Hackolade to define models rigorously and collaboratively
43. Start the animation
44. Mongo: Features N1QL, XDCR, Full Text Search, Mobile & Sync. Memory-first architecture and proven, easy scaling. CouchDb: Couchbase started as a whole new piece of software that was basically a combination of memcache and CouchDb a long time ago, but has grown far beyond that. Couchbase isn’t a fork or vice versa. They share an acronym and they are both NoSQL. Like MySQL and SQL Server, for instance. Open source apache license for community edition, enterprise edition on a faster release schedule, some advanced features, and support license. Couchbase is software you can run in the cloud on a VM or on your own data center. CosmosDb is a manage cloud service, but there is a emulator you can run locally. Transactions: if you can use nested modeling, you don't need multi-document transactions.