CartoDB Inside Out
•
13 likes•2,643 views
Technical overview of the CartoDB platform, describing the most important pieces of an architecture that serves millions of maps every day.
Report
Share
CartoDB Inside Out
- 1. CartoDB Inside Out Jorge Sanz · jorge@cartodb.com · @xurxosanz SASIG · OSGeo.pt · Lisbon · 2015 A technical overview of a web mapping platform
- 2. Agenda ● Why ● CartoDB nowadays ● Architecture of the platform ○ Components ○ Setup ● How to serve dynamic data at scale No maps in this talk sorry
- 5. Why ● @jatorre and @saleiva start Vizzuality in April 2008 ● Company focused on visualization projects ● Every geospatial project had the same stack: ○ Database + Map Server + APIs + client code ● CartoDB: internal product to deploy faster this geo-stack
- 6. Mission With CartoDB everybody can analyse, visualise and share insights on location data. We are democratising location intelligence
- 7. CartoDB nowadays ● A VC founded company of ~70 people (NY + Madrid) ● A leading mapping service with +130.000 users ● An Open Source product that can be deployed on the cloud or on just one server (on-premises) ● Full stack solution from storage to client side libraries ● It’s a mapping editor supported by a platform (APIs)
- 11. The CartoDB platform ● Spatial database for users ● NoSQL database for configurations and cache ● Rendering, imports, database and other services ● Cache and Content Delivery Network ● JavaScript Library ● Map Editor Platform On-premises or in the Cloud Data services Your data Editor
- 12. The CartoDB platform CartoDB.js Odyssey.jsTorque.js Leaflet /Gmaps PostgreSQL PostGIS SQL API Redis Maps API TorqueMapnik Varnish Caching Fastly CDN CartoDBonAmazon elasticRecipes CartoDBon Amazonelastic Recipes PostgreSQL PostGIS CartoDB Server API Rest API Imports API CartoDB Editor Browser
- 13. Cloud setup ● 1 - load balancer (nginx) ● 1 - caching (varnish) ● N - UI servers (CartoDB UI) ● N - database servers (user data) ● 1 - Metadata server (CartoDB metadata) ● 1 - redis server (+slave) ● N - Maps API + API SQL servers
- 15. Postgres + Postgis ● Geodata, users and editor metadata storage ● Tiny Well Known Binary ● CartoDB - Postgres extension Redis ● key-value highly performant storage ● configuration for APIs ● cache
- 16. Backend services ● Ruby on Rails ● Import API (ogr2ogr) ● Background management tasks Invalidation service ● Node.js + HA Proxy + Consul ● Invalidates caches by request ● Multi-agent, replicated
- 17. SQL API ● node.js ● Exposes an entry point for running queries ● Acts as an export API Maps API · Windshaft ● node.js + mapnik ● renders tiles from CartoCSS + SQL ● SQL API client
- 18. nginx ● web server ● load balancer Varnish + Fastly/GCP ● web cache ● Content Delivery Network
- 19. CartoDB Map Editor the easiest web mapping tool ever
- 20. Map Editor ● Ruby on Rails + JavaScript ● Consumes the Imports, Maps, SQL API and its own API ● Highly focused on UX-UI ● Manage datasets (schema, privacy, etc) ● Manage maps (basemap, infowindows, other visual elements, privacy, etc) ● Define layer data source with filters and SQL editor ● Define symbology through wizards and CartoCSS editor ● Publish maps as embeds, links or with CartoDB.js
- 21. CartoDB.js ● Based on jQuery + Backbone + Underscore ● Leverages Leaflet or Google Maps API ● Easy interaction with the SQL and Maps API ● Manages transparently utf-grids for interactivity ● Allows easy customizations on infowindows, layer SQL and CartoCSS definitions, etc
- 22. Serving data at scale
- 25. Story of a tile: first request
- 26. Story of a tile: next request
- 27. Story of a tile: insert & next request
- 28. Cache invalidation Write queries fire a trigger that updates cdb_tablemetadata and starts the invalidation process for the tiles that use that table
- 29. Summary ● CartoDB is an Open Source, full stack mapping product ● Runs on a single server or distributed ● Delivers to thousands of users the superpowers of Postgis and Mapnik, not just pretty maps but also new insights ● Provides tools for users and developers ● It’s designed to scale, but we are always improving it
- 31. CartoDB Inside Out Jorge Sanz · jorge@cartodb.com · @xurxosanz SASIG · OSGeo.pt · Lisbon · 2015 http://bit.ly/cartodb-sasig2015 Muito obrigado! · Questions?
- 32. Links ● https://cartodb.com ● https://github.com/CartoDB/cartodb ● https://github.com/CartoDB/cartodb.js ● https://github.com/CartoDB/cartodb-postgresql ● https://github.com/CartoDB/Windshaft-cartodb ● https://github.com/CartoDB/CartoDB-SQL-API ● http://docs.cartodb.com ● http://cartodb.readthedocs.org