SlideShare a Scribd company logo

Mongo db eveningschemadesign

•
•
MongoDB APAC
MongoDB APAC

MongoDB is a document-oriented, open source database that is high performing, horizontally scalable, and full featured. It uses a flexible schema and stores data in flexible JSON-like documents which allows for an evolving schema. MongoDB can be easily scaled out across commodity servers and provides high availability with automatic replication and recovery. It supports dynamic queries and indexing and has drivers for many languages.

Related slideshows

How Thermo Fisher is Reducing Data Analysis Times from Days to Minutes with M...
How Thermo Fisher is Reducing Data Analysis Times from Days to Minutes with M...How Thermo Fisher is Reducing Data Analysis Times from Days to Minutes with M...
How Thermo Fisher is Reducing Data Analysis Times from Days to Minutes with M...
 
MongoDB on Azure
MongoDB on AzureMongoDB on Azure
MongoDB on Azure
 
Webinar: What's New in MongoDB 3.2
Webinar: What's New in MongoDB 3.2Webinar: What's New in MongoDB 3.2
Webinar: What's New in MongoDB 3.2
 
1 of 70
Download to read offline
Consulting Engineer, MongoDB Prasoon Kumar #MongoDBDays #MongoDBIndia MongoDB Introduction; Schema Design
What is MongoDB?
MongoDB is a ___________ database 1.  Document 2.  Open source 3.  High performance 4.  Horizontally scalable 5.  Full featured
1. Document Database •  Not for .PDF & .DOC files •  A document is essentially an associative array •  Document = JSON object •  Document = PHP Array •  Document = Python Dict •  Document = Ruby Hash •  etc
1. NoSQL Data Model Key-Value Store Riak Memcache Project Voldemort Redis BerkeleyDB Document Database MongoDB CouchDB OrientDB Column-Family Stores Amazon SimpleDB Cassandra Hbase Hypertable Graph Databases Neo4J FlockDB OrientDB
2. Open Source •  MongoDB is an open source project •  On GitHub •  Licensed under the AGPL •  Started & sponsored by MongoDB Inc (formerly known as 10gen) •  Commercial licenses available •  Contributions welcome
3. High Performance •  Written in C++ •  Extensive use of memory-mapped files i.e.read-through write-through memory caching. •  Runs nearly everywhere •  Data serialized as BSON (fast parsing) •  Full support for primary & secondary indexes •  Document model = less work
Better Data Locality 3. Performance In-Memory Caching In-Place Updates
4. Scalability Auto-Sharding •  Increase capacity as you go •  Commodity and cloud architectures •  Improved operational simplicity and cost visibility
4. High Availability •  Automated replication and failover •  Multi-data center support •  Improved operational simplicity (e.g., HW swaps) •  Data durability and consistency
4. Scalability: MongoDB Architecture
5. Full Featured •  Ad Hoc queries •  Real time aggregation •  Rich query capabilities •  Strongly consistent •  Geospatial features •  Support for most programming languages •  Flexible schema
MongoDB is Fully Featured
mongodb.org/downloads
$ tar –zxvf mongodb-osx-x86_64-2.6.0.tgz $ cd mongodb-osx-i386-2.6.0/bin $ mkdir –p /data/db $ ./mongod Running MongoDB
MacBook-Pro-:~ $ mongo MongoDB shell version: 2.6.0 connecting to: test > db.cms.insert({text: 'Welcome to MongoDB'}) > db.cms.find().pretty() { "_id" : ObjectId("51c34130fbd5d7261b4cdb55"), "text" : "Welcome to MongoDB" } Mongo Shell
_id •  _id is the primary key in MongoDB •  Automatically indexed •  Automatically created as an ObjectId if not provided •  Any unique immutable value could be used
ObjectId •  ObjectId is a special 12 byte value •  Guaranteed to be unique across your cluster •  ObjectId("50804d0bd94ccab2da652599") |----ts-----||---mac---||-pid-||----inc-----| 4 3 2 3
Document Database
Terminology RDBMS MongoDB Table, View ➜ Collection Row ➜ Document Index ➜ Index Join ➜ Embedded Document Foreign Key ➜ Reference Partition ➜ Shard
Let’s Build a Blog
First step in any application is Determine your entities
Entities in our Blogging System •  Users (post authors) •  Article •  Comments •  Tags,Category •  Interactions (views,clicks)
In a relational base app We would start by doing schema design
Typical (relational) ERD User ¡Name ¡Email address Category ¡Name ¡URL Comment ¡Comment ¡Date ¡Author Article ¡Name ¡Slug ¡Publish date ¡Text Tag ¡Name ¡URL
In a MongoDB based app We start building our app and let the schema evolve
MongoDB ERD User ¡Name ¡Email address Article ¡Name ¡Slug ¡Publish date ¡Text ¡Author Comment[] ¡Comment ¡Date ¡Author Tag[] ¡Value Category[] ¡Value
Seek = 5+ ms Read = really really fast Post Author Comment Disk seeks and data locality
Post Author Comment Comment Comment Comment Comment Disk seeks and data locality
MongoDB Language Driver
Real applications are not built in the shell
MongoDB has native bindings for over 12 languages
Drivers & Ecosystem Drivers Support for the most popular languages and frameworks Frameworks Morphia MEAN Stack Java Python Perl Ruby
MongoDB Drivers •  Official Support for 12 languages •  Community drivers for tons more •  Drivers connect to mongo servers •  Drivers translate BSON into native types •  mongo shell is not a driver,but works like one in some ways •  Installed using typical means (maven,npm,pecl,gem, pip)
Working With MongoDB
# Python dictionary (or object) >>> article = { ‘title’ : ‘Schema design in MongoDB’, ‘author’ : ‘prasoonk’, ‘section’ : ‘schema’, ‘slug’ : ‘schema-design-in-mongodb’, ‘text’ : ‘Data in MongoDB has a flexible schema. So, 2 documents needn’t have same structure. It allows implicit schema to evolve.’, ‘date’ : datetime.utcnow(), ‘tags’ : [‘MongoDB’, ‘schema’] } >>> db[‘articles’].insert(article) Design schema.. In application code
>>> img_data = Binary(open(‘article_img.jpg’).read()) >>> article = { ‘title’ : ‘Schema evolutionin MongoDB’, ‘author’ : ‘mattbates’, ‘section’ : ‘schema’, ‘slug’ : ‘schema-evolution-in-mongodb’, ‘text’ : ‘MongoDb has dynamic schema. For good performance, you would need an implicit structure and indexes’, ‘date’ : datetime.utcnow(), ‘tags’ : [‘MongoDB’, ‘schema’, ‘migration’], ‘headline_img’ : { ‘img’ : img_data, ‘caption’ : ‘A sample document at the shell’ }} Let’s add a headline image
>>> article = { ‘title’ : ‘Favourite web application framework’, ‘author’ : ‘prasoonk’, ‘section’ : ‘web-dev’, ‘slug’ : ‘web-app-frameworks’, ‘gallery’ : [ { ‘img_url’ : ‘http://x.com/45rty’, ‘caption’ : ‘Flask’, ..}, .. ] ‘date’ : datetime.utcnow(), ‘tags’ : [‘Python’, ‘web’], } >>> db[‘articles’].insert(article) And different types of article
>>> user = { 'user' : 'prasoonk', 'email' : 'prasoon.kumar@mongodb.com', 'password' : 'prasoon101', 'joined' : datetime.utcnow(), 'location' : { 'city' : 'Mumbai' }, } } >>> db[‘users’].insert(user) Users and profiles
Retrive using Comparison Operators $gt,$gte,$in,$lt,$lte,$ne,$nin •  Use to query documents •  Logical:$or,$and,$not,$nor Element:$exists,$type •  Logical:$or,$and,$not,$nor Element:$exists,$type •  Evaluation:$mod,$regex,$where Geospatial:$geoWithin,$geoIntersects,$near,$nearSphere
Modelling comments (1) •  Two collections–articles and comments •  Use a reference (i.e. foreign key) to link together •  But.. N+1 queries to retrieve article and comments { ‘_id’: ObjectId(..), ‘title’:‘Schema design in MongoDB’, ‘author’:‘mattbates’, ‘date’: ISODate(..), ‘tags’: [‘MongoDB’, ‘schema’], ‘section’:‘schema’, ‘slug’:‘schema-design-in-mongodb’, ‘comments’:[ObjectId(..),…] } { ‘_id’: ObjectId(..), ‘article_id’: 1, ‘text’: ‘A great article,helped me understand schema design’, ‘date’: ISODate(..),, ‘author’:‘johnsmith’ }
$gt,$gte,$in,$lt,$lte,$ne,$nin •  Use to query documents •  Logical:$or,$and,$not,$nor Element:$exists,$type •  Evaluation:$mod,$regex,$where Geospatial:$geoWithin,$geoIntersects,$near,$nearSphere Comparison Operators db.articles.find( { 'title' : ’Intro to MongoDB’ } ) db.articles.find( { ’date' : { ‘$lt’ : {ISODate("2014-02-19T00:00:00.000Z") }} ) db.articles.find( { ‘tags’ : { ‘$in’ : [‘nosql’,‘database’] } } );
Modelling comments (2) •  Single articles collection– embed comments in article documents •  Pros •  Single query, document designed for the access pattern •  Locality (disk, shard) •  Cons •  Comments array is unbounded; documents will grow in size (remember 16MB document limit) { ‘_id’: ObjectId(..), ‘title’:‘Schema design in MongoDB’, ‘author’:‘mattbates’, ‘date’: ISODate(..), ‘tags’: [‘MongoDB’,‘schema’], … ‘comments’:[ { ‘text’: ‘Agreatarticle,helpedme understandschemadesign’, ‘date’:ISODate(..), ‘author’:‘johnsmith’ }, … ] }
Modelling comments (3) •  Another option: hybrid of (2) and (3),embed top x comments (e.g.by date,popularity) into the article document •  Fixed-size (2.4 feature) comments array •  All other comments ‘overflow’ into a comments collection (double write) in buckets •  Pros –  Document size is more fixed – fewer moves –  Single query built –  Full comment history with rich query/aggregation
Modelling comments (3) { ‘_id’:ObjectId(..), ‘title’:‘SchemadesigninMongoDB’, ‘author’:‘mattbates’, ‘date’:ISODate(..), ‘tags’:[‘MongoDB’, ‘schema’], … ‘comments_count’:45, ‘comments_pages’:1 ‘comments’:[ { ‘text’: ‘Agreatarticle,helpedme understandschemadesign’, ‘date’:ISODate(..), ‘author’:‘johnsmith’ }, … ] } Total number of comments •  Integer counter updated by update operation as comments added/removed Number of pages •  Page is a bucket of 100 comments (see next slide..) Fixed-size comments array •  10 most recent •  Sorted by date on insertion
Modelling comments (3) { ‘_id’: ObjectId(..), ‘article_id’: ObjectId(..), ‘page’: 1, ‘count’: 42 ‘comments’: [ { ‘text’: ‘A great article,helped me understand schema design’, ‘date’: ISODate(..), ‘author’:‘johnsmith’ }, … } One comment bucket (page) document containing up to about 100 comments Array of 100 comment sub- documents
Modelling interactions •  Interactions –  Article views –  Comments –  (Social media sharing) •  Requirements –  Time series –  Pre-aggregated in preparation for analytics
Modelling interactions •  Document per article per day– ‘bucketing’ •  Daily counter and hourly sub- document counters for interactions •  Bounded array (24 hours) •  Single query to retrieve daily article interactions; ready-made for graphing and further aggregation { ‘_id’: ObjectId(..), ‘article_id’: ObjectId(..), ‘section’:‘schema’, ‘date’: ISODate(..), ‘daily’: {‘views’: 45,‘comments’: 150 } ‘hours’: { 0 : {‘views’: 10 }, 1 : {‘views’: 2 }, … 23 : {‘comments’: 14,‘views’: 10 } } }
JSON and RESTful API Client-side JSON (eg AngularJS, (BSON) Real applications are not built at a shell–let’s build a RESTful API. Pymongo driver Python web app HTTP(S) REST Examples to follow: Python RESTful API using Flask microframework
myCMS REST endpoints Method URI Action GET /articles Retrieve all articles GET /articles-by-tag/[tag] Retrieve all articles by tag GET /articles/[article_id] Retrieve a specific article by article_id POST /articles Add a new article GET /articles/[article_id]/comments Retrieve all article comments by article_id POST /articles/[article_id]/comments Add a new comment to an article. POST /users Register a user user GET /users/[username] Retrieve user’s profile PUT /users/[username] Update a user’s profile
$ git clone http://www.github.com/prasoonk/mycms_mongodb $ cd mycms-mongodb $ virtualenv venv $ source venv/bin/activate $ pip install –r requirements.txt $ mkdir –p data/db $ mongod --dbpath=data/db --fork --logpath=mongod.log $ python web.py [$ deactivate] Getting started with the skeleton code
@app.route('/cms/api/v1.0/articles', methods=['GET']) def get_articles(): """Retrieves all articles in the collection sorted by date """ # query all articles and return a cursor sorted by date cur = db['articles'].find().sort('date’) if not cur: abort(400) # iterate the cursor and add docs to a dict articles = [article for article in cur] return jsonify({'articles' : json.dumps(articles, default=json_util.default)}) RESTful API methods in Python + Flask
@app.route('/cms/api/v1.0/articles/<string:article_id>/comments', methods = ['POST']) def add_comment(article_id): """Adds a comment to the specified article and a bucket, as well as updating a view counter "”” … page_id = article['last_comment_id'] // 100 … # push the comment to the latest bucket and $inc the count page = db['comments'].find_and_modify( { 'article_id' : ObjectId(article_id), 'page' : page_id}, { '$inc' : { 'count' : 1 }, '$push' : { 'comments' : comment } }, fields= {'count' : 1}, upsert=True, new=True) RESTful API methods in Python + Flask
# $inc the page count if bucket size (100) is exceeded if page['count'] > 100: db.articles.update( { '_id' : article_id, 'comments_pages': article['comments_pages'] }, { '$inc': { 'comments_pages': 1 } } ) # let's also add to the article itself # most recent 10 comments only res = db['articles'].update( {'_id' : ObjectId(article_id)}, {'$push' : {'comments' : { '$each' : [comment], '$sort' : {’date' : 1 }, '$slice' : -10}}, '$inc' : {'comment_count' : 1}}) … RESTful API methods in Python + Flask
def add_interaction(article_id, type): """Record the interaction (view/comment) for the specified article into the daily bucket and update an hourly counter """ ts = datetime.datetime.utcnow() # $inc daily and hourly view counters in day/article stats bucket # note the unacknowledged w=0 write concern for performance db['interactions'].update( { 'article_id' : ObjectId(article_id), 'date' : datetime.datetime(ts.year, ts.month, ts.day)}, { '$inc' : { 'daily.{}’.format(type) : 1, 'hourly.{}.{}'.format(ts.hour, type) : 1 }}, upsert=True, w=0) RESTful API methods in Python + Flask
$ curl -i http://localhost:5000/cms/api/v1.0/articles HTTP/1.0 200 OK Content-Type: application/json Content-Length: 335 Server: Werkzeug/0.9.4 Python/2.7.5 Date: Thu, 10 Apr 2014 16:00:51 GMT { "articles": "[{"title": "Schema design in MongoDB", "text": "Data in MongoDB has a flexible schema..", "section": "schema", "author": "prasoonk", "date": {"$date": 1397145312505}, "_id": {"$oid": "5346bef5f2610c064a36a793"}, "slug": "schema-design-in-mongodb", "tags": ["MongoDB", "schema"]}]"} Testing the API – retrieve articles
$ curl -H "Content-Type: application/json" -X POST -d '{"text":"An interesting article and a great read."}' http://localhost:5000/cms/api/v1.0/articles/52ed73a30bd031362b3c6bb3/ comments { "comment": "{"date": {"$date": 1391639269724}, "text": "An interesting article and a great read."}” }     Testing the API – comment on an article
Schema iteration New feature in the backlog? Documents have dynamic schema so we just iterate the object schema. >>> user = {‘username’:‘matt’, ‘first’:‘Matt’, ‘last’:‘Bates’, ‘preferences’: {‘opt_out’: True } } >>> user.save(user)
Next Steps
Further reading •  ‘myCMS’ skeleton source code: http://www.github.com/prasoonk/mycms_mongodb •  Data Models http://docs.mongodb.org/manual/data-modeling/ •  Use case-metadata and asset management: http://docs.mongodb.org/ecosystem/use-cases/metadata-and- asset-management/ •  Use case-storing comments: http://docs.mongodb.org/ecosystem/use-cases/storing- comments/
docs.mongodb.org
Online Training at MongoDB University
For More Information Resource Location MongoDB Downloads mongodb.com/download Free Online Training education.mongodb.com Webinars and Events mongodb.com/events White Papers mongodb.com/white-papers Case Studies mongodb.com/customers Presentations mongodb.com/presentations Documentation docs.mongodb.org Additional Info info@mongodb.com Resource Location
We've introduced a lot of concepts here
Schema Design @ User ¡Name ¡Email address Article ¡Name ¡Slug ¡Publish date ¡Text ¡Author Comment[] ¡Comment ¡Date ¡Author Tag[] ¡Value Category[] ¡Value
Replication @ Secondary Secondary Primary Client Application Driver Write Read Read
Indexing @ 7 16 1 2 5 6 9 12 18 21
Sharding @ www.etiennemansard.com
Questions?
Consulting Engineer, MongoDB Prasoon Kumar #MongoDBDays #MongoDBIndia @prasoonk Thank You

More Related Content

Mongo db eveningschemadesign

  • 1. Consulting Engineer, MongoDB Prasoon Kumar #MongoDBDays #MongoDBIndia MongoDB Introduction; Schema Design
  • 3. MongoDB is a ___________ database 1.  Document 2.  Open source 3.  High performance 4.  Horizontally scalable 5.  Full featured
  • 4. 1. Document Database •  Not for .PDF & .DOC les •  A document is essentially an associative array •  Document = JSON object •  Document = PHP Array •  Document = Python Dict •  Document = Ruby Hash •  etc
  • 5. 1. NoSQL Data Model Key-Value Store Riak Memcache Project Voldemort Redis BerkeleyDB Document Database MongoDB CouchDB OrientDB Column-Family Stores Amazon SimpleDB Cassandra Hbase Hypertable Graph Databases Neo4J FlockDB OrientDB
  • 6. 2. Open Source •  MongoDB is an open source project •  On GitHub •  Licensed under the AGPL •  Started & sponsored by MongoDB Inc (formerly known as 10gen) •  Commercial licenses available •  Contributions welcome
  • 7. 3. High Performance •  Written in C++ •  Extensive use of memory-mapped les i.e.read-through write-through memory caching. •  Runs nearly everywhere •  Data serialized as BSON (fast parsing) •  Full support for primary & secondary indexes •  Document model = less work
  • 9. 4. Scalability Auto-Sharding •  Increase capacity as you go •  Commodity and cloud architectures •  Improved operational simplicity and cost visibility
  • 10. 4. High Availability •  Automated replication and failover •  Multi-data center support •  Improved operational simplicity (e.g., HW swaps) •  Data durability and consistency
  • 11. 4. Scalability: MongoDB Architecture
  • 12. 5. Full Featured •  Ad Hoc queries •  Real time aggregation •  Rich query capabilities •  Strongly consistent •  Geospatial features •  Support for most programming languages •  Flexible schema
  • 13. MongoDB is Fully Featured
  • 15. $ tar –zxvf mongodb-osx-x86_64-2.6.0.tgz $ cd mongodb-osx-i386-2.6.0/bin $ mkdir –p /data/db $ ./mongod Running MongoDB
  • 16. MacBook-Pro-:~ $ mongo MongoDB shell version: 2.6.0 connecting to: test > db.cms.insert({text: 'Welcome to MongoDB'}) > db.cms.find().pretty() { "_id" : ObjectId("51c34130fbd5d7261b4cdb55"), "text" : "Welcome to MongoDB" } Mongo Shell
  • 17. _id •  _id is the primary key in MongoDB •  Automatically indexed •  Automatically created as an ObjectId if not provided •  Any unique immutable value could be used
  • 18. ObjectId •  ObjectId is a special 12 byte value •  Guaranteed to be unique across your cluster •  ObjectId("50804d0bd94ccab2da652599") |----ts-----||---mac---||-pid-||----inc-----| 4 3 2 3
  • 20. Terminology RDBMS MongoDB Table, View ➜ Collection Row ➜ Document Index ➜ Index Join ➜ Embedded Document Foreign Key ➜ Reference Partition ➜ Shard
  • 22. First step in any application is Determine your entities
  • 23. Entities in our Blogging System •  Users (post authors) •  Article •  Comments •  Tags,Category •  Interactions (views,clicks)
  • 24. In a relational base app We would start by doing schema design
  • 25. Typical (relational) ERD User ¡Name ¡Email address Category ¡Name ¡URL Comment ¡Comment ¡Date ¡Author Article ¡Name ¡Slug ¡Publish date ¡Text Tag ¡Name ¡URL
  • 26. In a MongoDB based app We start building our app and let the schema evolve
  • 27. MongoDB ERD User ¡Name ¡Email address Article ¡Name ¡Slug ¡Publish date ¡Text ¡Author Comment[] ¡Comment ¡Date ¡Author Tag[] ¡Value Category[] ¡Value
  • 28. Seek = 5+ ms Read = really really fast Post Author Comment Disk seeks and data locality
  • 31. Real applications are not built in the shell
  • 32. MongoDB has native bindings for over 12 languages
  • 33. Drivers & Ecosystem Drivers Support for the most popular languages and frameworks Frameworks Morphia MEAN Stack Java Python Perl Ruby
  • 34. MongoDB Drivers •  Ofcial Support for 12 languages •  Community drivers for tons more •  Drivers connect to mongo servers •  Drivers translate BSON into native types •  mongo shell is not a driver,but works like one in some ways •  Installed using typical means (maven,npm,pecl,gem, pip)
  • 36. # Python dictionary (or object) >>> article = { ‘title’ : ‘Schema design in MongoDB’, ‘author’ : ‘prasoonk’, ‘section’ : ‘schema’, ‘slug’ : ‘schema-design-in-mongodb’, ‘text’ : ‘Data in MongoDB has a flexible schema. So, 2 documents needn’t have same structure. It allows implicit schema to evolve.’, ‘date’ : datetime.utcnow(), ‘tags’ : [‘MongoDB’, ‘schema’] } >>> db[‘articles’].insert(article) Design schema.. In application code
  • 37. >>> img_data = Binary(open(‘article_img.jpg’).read()) >>> article = { ‘title’ : ‘Schema evolutionin MongoDB’, ‘author’ : ‘mattbates’, ‘section’ : ‘schema’, ‘slug’ : ‘schema-evolution-in-mongodb’, ‘text’ : ‘MongoDb has dynamic schema. For good performance, you would need an implicit structure and indexes’, ‘date’ : datetime.utcnow(), ‘tags’ : [‘MongoDB’, ‘schema’, ‘migration’], ‘headline_img’ : { ‘img’ : img_data, ‘caption’ : ‘A sample document at the shell’ }} Let’s add a headline image
  • 38. >>> article = { ‘title’ : ‘Favourite web application framework’, ‘author’ : ‘prasoonk’, ‘section’ : ‘web-dev’, ‘slug’ : ‘web-app-frameworks’, ‘gallery’ : [ { ‘img_url’ : ‘http://x.com/45rty’, ‘caption’ : ‘Flask’, ..}, .. ] ‘date’ : datetime.utcnow(), ‘tags’ : [‘Python’, ‘web’], } >>> db[‘articles’].insert(article) And different types of article
  • 39. >>> user = { 'user' : 'prasoonk', 'email' : 'prasoon.kumar@mongodb.com', 'password' : 'prasoon101', 'joined' : datetime.utcnow(), 'location' : { 'city' : 'Mumbai' }, } } >>> db[‘users’].insert(user) Users and proles
  • 40. Retrive using Comparison Operators $gt,$gte,$in,$lt,$lte,$ne,$nin •  Use to query documents •  Logical:$or,$and,$not,$nor Element:$exists,$type •  Logical:$or,$and,$not,$nor Element:$exists,$type •  Evaluation:$mod,$regex,$where Geospatial:$geoWithin,$geoIntersects,$near,$nearSphere
  • 41. Modelling comments (1) •  Two collections–articles and comments •  Use a reference (i.e. foreign key) to link together •  But.. N+1 queries to retrieve article and comments { ‘_id’: ObjectId(..), ‘title’:‘Schema design in MongoDB’, ‘author’:‘mattbates’, ‘date’: ISODate(..), ‘tags’: [‘MongoDB’, ‘schema’], ‘section’:‘schema’, ‘slug’:‘schema-design-in-mongodb’, ‘comments’:[ObjectId(..),…] } { ‘_id’: ObjectId(..), ‘article_id’: 1, ‘text’: ‘A great article,helped me understand schema design’, ‘date’: ISODate(..),, ‘author’:‘johnsmith’ }
  • 42. $gt,$gte,$in,$lt,$lte,$ne,$nin •  Use to query documents •  Logical:$or,$and,$not,$nor Element:$exists,$type •  Evaluation:$mod,$regex,$where Geospatial:$geoWithin,$geoIntersects,$near,$nearSphere Comparison Operators db.articles.nd( { 'title' : ’Intro to MongoDB’ } ) db.articles.nd( { ’date' : { ‘$lt’ : {ISODate("2014-02-19T00:00:00.000Z") }} ) db.articles.nd( { ‘tags’ : { ‘$in’ : [‘nosql’,‘database’] } } );
  • 43. Modelling comments (2) •  Single articles collection– embed comments in article documents •  Pros •  Single query, document designed for the access pattern •  Locality (disk, shard) •  Cons •  Comments array is unbounded; documents will grow in size (remember 16MB document limit) { ‘_id’: ObjectId(..), ‘title’:‘Schema design in MongoDB’, ‘author’:‘mattbates’, ‘date’: ISODate(..), ‘tags’: [‘MongoDB’,‘schema’], … ‘comments’:[ { ‘text’: ‘Agreatarticle,helpedme understandschemadesign’, ‘date’:ISODate(..), ‘author’:‘johnsmith’ }, … ] }
  • 44. Modelling comments (3) •  Another option: hybrid of (2) and (3),embed top x comments (e.g.by date,popularity) into the article document •  Fixed-size (2.4 feature) comments array •  All other comments ‘overflow’ into a comments collection (double write) in buckets •  Pros –  Document size is more xed – fewer moves –  Single query built –  Full comment history with rich query/aggregation
  • 45. Modelling comments (3) { ‘_id’:ObjectId(..), ‘title’:‘SchemadesigninMongoDB’, ‘author’:‘mattbates’, ‘date’:ISODate(..), ‘tags’:[‘MongoDB’, ‘schema’], … ‘comments_count’:45, ‘comments_pages’:1 ‘comments’:[ { ‘text’: ‘Agreatarticle,helpedme understandschemadesign’, ‘date’:ISODate(..), ‘author’:‘johnsmith’ }, … ] } Total number of comments •  Integer counter updated by update operation as comments added/removed Number of pages •  Page is a bucket of 100 comments (see next slide..) Fixed-size comments array •  10 most recent •  Sorted by date on insertion
  • 46. Modelling comments (3) { ‘_id’: ObjectId(..), ‘article_id’: ObjectId(..), ‘page’: 1, ‘count’: 42 ‘comments’: [ { ‘text’: ‘A great article,helped me understand schema design’, ‘date’: ISODate(..), ‘author’:‘johnsmith’ }, … } One comment bucket (page) document containing up to about 100 comments Array of 100 comment sub- documents
  • 47. Modelling interactions •  Interactions –  Article views –  Comments –  (Social media sharing) •  Requirements –  Time series –  Pre-aggregated in preparation for analytics
  • 48. Modelling interactions •  Document per article per day– ‘bucketing’ •  Daily counter and hourly sub- document counters for interactions •  Bounded array (24 hours) •  Single query to retrieve daily article interactions; ready-made for graphing and further aggregation { ‘_id’: ObjectId(..), ‘article_id’: ObjectId(..), ‘section’:‘schema’, ‘date’: ISODate(..), ‘daily’: {‘views’: 45,‘comments’: 150 } ‘hours’: { 0 : {‘views’: 10 }, 1 : {‘views’: 2 }, … 23 : {‘comments’: 14,‘views’: 10 } } }
  • 49. JSON and RESTful API Client-side JSON (eg AngularJS, (BSON) Real applications are not built at a shell–let’s build a RESTful API. Pymongo driver Python web app HTTP(S) REST Examples to follow: Python RESTful API using Flask microframework
  • 50. myCMS REST endpoints Method URI Action GET /articles Retrieve all articles GET /articles-by-tag/[tag] Retrieve all articles by tag GET /articles/[article_id] Retrieve a specic article by article_id POST /articles Add a new article GET /articles/[article_id]/comments Retrieve all article comments by article_id POST /articles/[article_id]/comments Add a new comment to an article. POST /users Register a user user GET /users/[username] Retrieve user’s prole PUT /users/[username] Update a user’s prole
  • 51. $ git clone http://www.github.com/prasoonk/mycms_mongodb $ cd mycms-mongodb $ virtualenv venv $ source venv/bin/activate $ pip install –r requirements.txt $ mkdir –p data/db $ mongod --dbpath=data/db --fork --logpath=mongod.log $ python web.py [$ deactivate] Getting started with the skeleton code
  • 52. @app.route('/cms/api/v1.0/articles', methods=['GET']) def get_articles(): """Retrieves all articles in the collection sorted by date """ # query all articles and return a cursor sorted by date cur = db['articles'].find().sort('date’) if not cur: abort(400) # iterate the cursor and add docs to a dict articles = [article for article in cur] return jsonify({'articles' : json.dumps(articles, default=json_util.default)}) RESTful API methods in Python + Flask
  • 53. @app.route('/cms/api/v1.0/articles/<string:article_id>/comments', methods = ['POST']) def add_comment(article_id): """Adds a comment to the specified article and a bucket, as well as updating a view counter "”” … page_id = article['last_comment_id'] // 100 … # push the comment to the latest bucket and $inc the count page = db['comments'].find_and_modify( { 'article_id' : ObjectId(article_id), 'page' : page_id}, { '$inc' : { 'count' : 1 }, '$push' : { 'comments' : comment } }, fields= {'count' : 1}, upsert=True, new=True) RESTful API methods in Python + Flask
  • 54. # $inc the page count if bucket size (100) is exceeded if page['count'] > 100: db.articles.update( { '_id' : article_id, 'comments_pages': article['comments_pages'] }, { '$inc': { 'comments_pages': 1 } } ) # let's also add to the article itself # most recent 10 comments only res = db['articles'].update( {'_id' : ObjectId(article_id)}, {'$push' : {'comments' : { '$each' : [comment], '$sort' : {’date' : 1 }, '$slice' : -10}}, '$inc' : {'comment_count' : 1}}) … RESTful API methods in Python + Flask
  • 55. def add_interaction(article_id, type): """Record the interaction (view/comment) for the specified article into the daily bucket and update an hourly counter """ ts = datetime.datetime.utcnow() # $inc daily and hourly view counters in day/article stats bucket # note the unacknowledged w=0 write concern for performance db['interactions'].update( { 'article_id' : ObjectId(article_id), 'date' : datetime.datetime(ts.year, ts.month, ts.day)}, { '$inc' : { 'daily.{}’.format(type) : 1, 'hourly.{}.{}'.format(ts.hour, type) : 1 }}, upsert=True, w=0) RESTful API methods in Python + Flask
  • 56. $ curl -i http://localhost:5000/cms/api/v1.0/articles HTTP/1.0 200 OK Content-Type: application/json Content-Length: 335 Server: Werkzeug/0.9.4 Python/2.7.5 Date: Thu, 10 Apr 2014 16:00:51 GMT { "articles": "[{"title": "Schema design in MongoDB", "text": "Data in MongoDB has a flexible schema..", "section": "schema", "author": "prasoonk", "date": {"$date": 1397145312505}, "_id": {"$oid": "5346bef5f2610c064a36a793"}, "slug": "schema-design-in-mongodb", "tags": ["MongoDB", "schema"]}]"} Testing the API – retrieve articles
  • 57. $ curl -H "Content-Type: application/json" -X POST -d '{"text":"An interesting article and a great read."}' http://localhost:5000/cms/api/v1.0/articles/52ed73a30bd031362b3c6bb3/ comments { "comment": "{"date": {"$date": 1391639269724}, "text": "An interesting article and a great read."}” }     Testing the API – comment on an article
  • 58. Schema iteration New feature in the backlog? Documents have dynamic schema so we just iterate the object schema. >>> user = {‘username’:‘matt’, ‘first’:‘Matt’, ‘last’:‘Bates’, ‘preferences’: {‘opt_out’: True } } >>> user.save(user)
  • 60. Further reading •  ‘myCMS’ skeleton source code: http://www.github.com/prasoonk/mycms_mongodb •  Data Models http://docs.mongodb.org/manual/data-modeling/ •  Use case-metadata and asset management: http://docs.mongodb.org/ecosystem/use-cases/metadata-and- asset-management/ •  Use case-storing comments: http://docs.mongodb.org/ecosystem/use-cases/storing- comments/
  • 62. Online Training at MongoDB University
  • 63. For More Information Resource Location MongoDB Downloads mongodb.com/download Free Online Training education.mongodb.com Webinars and Events mongodb.com/events White Papers mongodb.com/white-papers Case Studies mongodb.com/customers Presentations mongodb.com/presentations Documentation docs.mongodb.org Additional Info info@mongodb.com Resource Location
  • 64. We've introduced a lot of concepts here
  • 65. Schema Design @ User ¡Name ¡Email address Article ¡Name ¡Slug ¡Publish date ¡Text ¡Author Comment[] ¡Comment ¡Date ¡Author Tag[] ¡Value Category[] ¡Value
  • 66. Replication @ Secondary Secondary Primary Client Application Driver Write Read Read
  • 67. Indexing @ 7 16 1 2 5 6 9 12 18 21
  • 70. Consulting Engineer, MongoDB Prasoon Kumar #MongoDBDays #MongoDBIndia @prasoonk Thank You