Development to Production with Sharded MongoDB Clusters
•
12 likes•7,960 views
Severalnines presentation at MongoDB Stockholm Conference. Presentation covers: - mongoDB sharding/clustering concepts - recommended dev/test/prod setups - how to verify your deployment - how to avoid downtime - what MongoDB metrics to watch - when to scale
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to Development to Production with Sharded MongoDB Clusters
Similar to Development to Production with Sharded MongoDB Clusters (20)
More from Severalnines
More from Severalnines (20)
Recently uploaded
Recently uploaded (20)
Development to Production with Sharded MongoDB Clusters
- 1. From Dev to Prod with Sharded Mongo DB Clusters May 2013 Johan Andersson Severalnines AB johan@severalnines.com
- 2. Topics ¤ Best Practice Deployment ¤ Keeping it running ¤ Scaling with Shards Note: Most features mentioned in this presentation are applicable only on MongoDB 2.2 or later. 2 Copyright 2011 Severalnines AB
- 3. 3 Copyright 2011 Severalnines AB “An OS check-up a day keeps downtime away” Ancient Chinese wisdom
- 4. Distributed Computing 101 ¤ Network latency is not 0 ¤ Network is unreliable ¤ Network bandwidth is not infinite ¤ CPUs are not infinitely fast ¤ Disks are not infinitely fast ¤ Free RAM decreases fast ¤ Virtualization does not make the underlying hardware more powerful 4 Copyright 2011 Severalnines AB
- 5. MongoDB Concepts ¤ Mongos ¤ Query routers, routes queries to shards ¤ Caches data from Config Servers ¤ Config Servers ¤ Records which shard holds what chunks ¤ Shard Servers ¤ Stores the actual data 5 Copyright 2011 Severalnines AB
- 6. Developer Setup ¤ A good test setup for a developer: ¤ One or three config Servers ¤ Thee shard servers ¤ One or more Mongos ¤ But do set it up as a shard server so you can validate queries (explain) to make sure they behave as intended. ¤ Using correct indexes ¤ Vagrant image - http://alexyu.se/content/2013/03/vagrant-box-severalnines- database-cluster-deployment-scripts 6 Copyright 2011 Severalnines AB
- 7. Test Setup ¤ Recommendations ¤ One or three config Servers ¤ >1 shard - thee shard servers ¤ One or more Mongos ¤ Stay close to your production system. ¤ Two shards (even if prod use one only) allows you to: ¤ Verify queries use the correct indexes ¤ Check if one or all shards are hit ¤ Extremely useful when adding shards to Prod to avoid performance regression. 7 Copyright 2011 Severalnines AB
- 8. Production Setup ¤ Start with a good and sound architecture from day one! ¤ A sharded architecture will allow you to scale easily ¤ Avoid un-necessary craziness and risk to switch a non- sharded setup to a sharded ¤ You do want three copies of your data 8 Copyright 2011 Severalnines AB
- 9. Production Setup 9 Copyright 2011 Severalnines AB PRIMARY SECONDARY repl
- 10. Production Setup 10 Copyright 2011 Severalnines AB Hard to scale, too crowded Go for Best Practice instead PRIMARY SECONDARY repl
- 11. Production Setup 11 Copyright 2011 Severalnines AB … Shard0 Shard1 ShardN …
- 12. Production Setup ¤ Allows you to scale - Add more shards and mongos easily ¤ It is proven – used by many large organizations ¤ Deploy… ¤ Mongos ¤ Config servers ¤ Shard servers (mongod --shardsvr) ¤ … on individual servers 12 Copyright 2011 Severalnines AB
- 13. When processes fails ¤ Config Server process fails ¤ If one config server fails, no chunk migration between shards ¤ No chunk splitting ¤ Shards are still read/writeable ¤ Mongos fails ¤ You will have many ¤ Shard server fails ¤ If PRIMARY fails à One SECONDARY will become PRIMARY ¤ If SECONDADY fails à Restart and it will resync with PRIMARY ¤ Restart the failed process! ¤ Troll the logs to see it starts without errors 13 Copyright 2011 Severalnines AB
- 14. Keeping it running ¤ RAM is a scarce resource – keep as much as possible in it! ¤ Use short field names in the Documents to reduce size: ¤ { first_name:’not good’} ¤ {fn:’better’} ¤ Important when you have many Documents ¤ Compact collections: ¤ Run on SECONDARY ¤ db.mycollection.runCommand(“compact”) ¤ Stop PRIMARY, SECONDARY becomes new PRIMARY. 14 Copyright 2011 Severalnines AB
- 15. Keeping it running ¤ Replication Lag ¤ Producer / Consumer problem ¤ Temporary or Permanent Problem? ¤ Permanent problem ¤ Expunge the oplog? ¤ Magic wand? ¤ Disk, network, or load-dependent problem? ¤ Remedy: ¤ Faster disks: Improve disk subsystem -> IOPS or SSD, RAID configuration ¤ Faster CPUs ¤ Can you reduce writes to the shard(s)? ¤ Scale with shards 15 Copyright 2011 Severalnines AB
- 16. Keeping it running ¤ Page Faults ¤ MongoDB uses mmap:ed files, allocated from the VMM address space. Mmapped files does not need to fit in RAM ¤ OS manages what pages are mapped in RAM and not ¤ Accessing data on a page not in RAM cause a page fault ¤ Data will be loaded from DISK ¤ DISK is slow ¤ Performance will degrade ¤ If the number of PFs is constant, no worry, else if increasing, problem in the horizon. ¤ Remedy: ¤ Increase RAM in server (Active working set should fit in RAM) ¤ Scale with shards ¤ Add SSD 16 Copyright 2011 Severalnines AB
- 17. Not cool 17 Copyright 2011 Severalnines AB
- 18. Not cool 18 Copyright 2011 Severalnines AB
- 19. You are done for! 19 Copyright 2011 Severalnines AB
- 20. Avoid the situation ¤ Keep data set in RAM ¤ Use fast io-subsystem since OS is managing the paging ¤ SSD ¤ IOPS on EC2 ¤ Make sure the Host OS (Virtualized env) does not swap. ¤ Lock mongod’s to CPUs, check /etc/interrupts, bind to CPUs not handling ETH interrupts, avoids ctx switching. ¤ Add shards before its too late ¤ More disks/CPUs,RAM to handle the load Copyright 2011 Severalnines AB 20
- 21. Scaling with Shards ¤ Shard Keys ¤ When to scale? ¤ Add a shard 21 Copyright 2011 Severalnines AB
- 22. Database Size ¤ db.stats() ¤ Size of one db = dbStats.dataSize + dbStats.indexSize ¤ Does it fit in RAM? ¤ Plan ahead: ¤ avgObjSize x expectedNoOfDocuments ¤ Does it fit in RAM? ¤ How many documents can you store? 22 Copyright 2011 Severalnines AB
- 23. Working set ¤ Working Set Analyzer in 2.4 helps: !"workingSet" : {! ! !"note" : "thisIsAnEstimate",! ! !"pagesInMemory" : 33688, //4KB pg_size! ! !"computationTimeMicros" : 15283,! ! !"overSeconds" : 1923 //dist pg_new to pg_oldest },! ¤ Trend overSeconds and pagesInMemory 23 Copyright 2011 Severalnines AB
- 24. Shard Key ¤ Collections must have a Shard Key ¤ The shard key must be indexed ¤ Hash-based or Range based Sharding ¤ Hash-bashed great for exact match ¤ Subscriber/user databases ¤ Queries must use the shard key ¤ If not, all shards will be queried à slow ¤ Check and verify with .explain() 24 Copyright 2011 Severalnines AB
- 25. Adding Shards ¤ Adding a new Shard is easy: ¤ Deploy a new Replica-set: ¤ PRIMARY - 192.168.0.151 SECONDARY - 192.168.0.152 SECONDARY - 192.168.0.153 ¤ Register the Replica Set: ¤ sh.addShard(“myshard2/192.168.0.151:27017”); ¤ Finding the time to add shards is harder ¤ Rebalancing costs and query performance will suffer due to chunk migration. ¤ Do it off-peak if possible ¤ MongoDB will automatically rebalance the shards. 25 Copyright 2011 Severalnines AB
- 27. Resources 27 Copyright 2011 Severalnines AB ¤ MongoDB Configurator http://www.severalnines.com/mongodb-configurator/ ¤ -ClusterControl for MongoDB http://www.severalnines.com/resources/cmon-mongodb- cluster-management-monitoring-tool ¤ Installing ClusterControl on top of existing MongoDB Cluster http://www.severalnines.com/blog/install-clustercontrol-top- existing-mongodb-sharded-cluster ¤ Vagrant image - http://alexyu.se/content/2013/03/vagrant-box-severalnines- database-cluster-deployment-scripts
- 28. Follow us on 28 Copyright 2011 Severalnines AB ¤ Twitter : @severalnines ¤ Facebook: www.facebook.com/severalnines ¤ Slideshare : www.slideshare.net/severalnines ¤ Linked-in: www.linkedin.com/company/severalnines
- 29. Thank you! 29 Copyright 2011 Severalnines AB