SlideShare a Scribd company logo

Os Gottfrid

•
•
oscon2007
oscon2007

DBSlayer is a middleware that allows web applications to access databases through HTTP and JSON. It provides a simple way to query databases by sending SQL statements in a JSON-formatted HTTP request and receiving the results of the query in the HTTP response also formatted in JSON. It aims to be independent, scalable, flexible and simple. Some features include connection pooling, load balancing queries across multiple databases, and automatic failover to alternate databases. It works by acting as an intermediary between web applications and databases, handling the network communication and database access transparently through its HTTP/JSON interface.

Related slideshows

Complex queries in a distributed multi-model database
Complex queries in a distributed multi-model databaseComplex queries in a distributed multi-model database
Complex queries in a distributed multi-model database
 
MongoDB Europe 2016 - Using MongoDB to Build a Fast and Scalable Content Repo...
MongoDB Europe 2016 - Using MongoDB to Build a Fast and Scalable Content Repo...MongoDB Europe 2016 - Using MongoDB to Build a Fast and Scalable Content Repo...
MongoDB Europe 2016 - Using MongoDB to Build a Fast and Scalable Content Repo...
 
Introduction to Foxx by our community member Iskandar Soesman @ikandars
Introduction to Foxx by our community member Iskandar Soesman @ikandarsIntroduction to Foxx by our community member Iskandar Soesman @ikandars
Introduction to Foxx by our community member Iskandar Soesman @ikandars
 
1 of 32
DBSlayer Introducing DBSlayer http://code.nytimes.com/projects/dbslayer Derek Gottfrid The New York Times
Problem: Provide Web Apps Access to the Database
Alternate Problem: Avoid evil horrible doctor show.
Solution JSON + HTTP + MYSQL
Simple Example select ci.* from City ci, Country c where c.name = 'United States' and c.code = ci.CountryCode order by Population desc
HTTP Request http://localhost:9090/db?%7B%22SQL%22: %22select%20ci.*%20from%20City%20ci,% 20Country%20c%20where%20c.name%20 =%20'United%20States'%20and%20c.code %20=%20ci.CountryCode%20order%20by% 20Population%20desc%20limit%2010;%22 %7D
HTTP Response HTTP/1.0 200 OK Date: Wed, 27 Jun 2007 20:55:59 GMT Server: dbslayer/server beta-9 Connection: Close Content-type: text/plain; charset=utf-8 Content-Length: 746
JSON Payload {quot;RESULTquot; : {quot;TYPESquot; : [quot;MYSQL_TYPE_LONGquot; , quot;MYSQL_TYPE_STRINGquot; , quot;MYSQL_TYPE_STRINGquot; , quot;MYSQL_TYPE_STRINGquot; , quot;MYSQL_TYPE_LONGquot;] , quot;HEADERquot; : [quot;IDquot; , quot;Namequot; , quot;CountryCodequot; , quot;Districtquot; , quot;Populationquot;] , quot;ROWSquot; : [[3793 , quot;New Yorkquot; , quot;USAquot; , quot;New Yorkquot; , 8008278] , [3794 , quot;Los Angelesquot; , quot;USAquot; , quot;Californiaquot; , 3694820] , [3795 , quot;Chicagoquot; , quot;USAquot; , quot;Illinoisquot; , 2896016] , [3796 , quot;Houstonquot; , quot;USAquot; , quot;Texasquot; , 1953631] , [3797 , quot;Philadelphiaquot; , quot;USAquot; , quot;Pennsylvaniaquot; , 1517550] , [3798 , quot;Phoenixquot; , quot;USAquot; , quot;Arizonaquot; , 1321045] , [3799 , quot;San Diegoquot; , quot;USAquot; , quot;Californiaquot; , 1223400] , [3800 , quot;Dallasquot; , quot;USAquot; , quot;Texasquot; , 1188580] , [3801 , quot;San Antonioquot; , quot;USAquot; , quot;Texasquot; , 1144646] , [3802 , quot;Detroitquot; , quot;USAquot; , quot;Michiganquot; , 951270]]}}
Python Sample Client import urllib,urllib2,cjson db = lambda q: cjson.decode( urllib2.urlopen( “http://localhost:9090/db? ”+urllib.quote('{“SQL ”:”%s”}' % q)).read())
Other Solutions MySQLProxy ● SQLRelay ● mysqli ●
Requirements Independent ● Scalable ● Flexible ● Simple ●
Disclaimer Works for us.
Problems Not Solved Back end scripts – ● bulk import/export Large Data Retrievals ● – cursors Native Binds / ● Prepared Statements Enterprise Buzzword ● Compliance A Sober Lindsay ●
DBSlayer is the middle-man Connection Pooling ● Multi-threaded ● Round robin distribution of queries ● Automatic fail over to alternate database ● Easy configuration ● Statistics / Logging ● Tiny code base ●
Design Considerations
Flexiblity Loose coupling of front and middle ● easy upgrade – abstract location of db – makes re-config a – snap Language independent ● JSON easily shared across languages – Common place to add additional features ●
Scalability Round robin across slaves ● Automatic fail over for slaves ● Easily add more slaves ● HTTP plays nice with load balancers ● JSON is very cachable ● Stateless is required for effective scaling ●
Simplicity HTTP and JSON - everyone knows them ● Transparent protocol – capture, interpret ● Lots of tools to work with HTTP ● Works with your browser ● HTTP server base – easy to deploy new ● endpoints
Browser Testing
Stats
Pretty Pictures
Os Gottfrid
Os Gottfrid
View Source Built on top of MySQL C API and Apache ● Runtime Libraries (APR) Why APR? ● Platform agnostic – Memory Pools – Networking / Threading Abstractions – Proven –
3 Parts server – handles network, HTTP parsing, ● threading, logging db – handle mysql access, type mapping, ● connection failover util – json de/serializer ●
Future / Your Chance improve HTTP caching layer ● ● complaince partition / shard ● more database support ● backends (APR new language ● dbd) bindings alternate serializer ● more stats / ● XML logging security ● scriptable filters ●
Other (tighten up them ....)
Another amazing talk... Rails Under The Knife Jacob Harris July 26th – 2:35pm Portland 255
BoF Amazon S3/EC2 computing against large ● datasets – 7:30 F150 Future Newspapers – 8:30 E143-144 ●
NYTIMES.COM open.nytimes.com – developers blog ● code.nytimes.com – code repository ●
Free Stuff Visit the NYTimes Booth
The End Questions: derek@nytimes.com

More Related Content

Os Gottfrid

  • 1. DBSlayer Introducing DBSlayer http://code.nytimes.com/projects/dbslayer Derek Gottfrid The New York Times
  • 2. Problem: Provide Web Apps Access to the Database
  • 3. Alternate Problem: Avoid evil horrible doctor show.
  • 5. Simple Example select ci.* from City ci, Country c where c.name = 'United States' and c.code = ci.CountryCode order by Population desc
  • 7. HTTP Response HTTP/1.0 200 OK Date: Wed, 27 Jun 2007 20:55:59 GMT Server: dbslayer/server beta-9 Connection: Close Content-type: text/plain; charset=utf-8 Content-Length: 746
  • 8. JSON Payload {quot;RESULTquot; : {quot;TYPESquot; : [quot;MYSQL_TYPE_LONGquot; , quot;MYSQL_TYPE_STRINGquot; , quot;MYSQL_TYPE_STRINGquot; , quot;MYSQL_TYPE_STRINGquot; , quot;MYSQL_TYPE_LONGquot;] , quot;HEADERquot; : [quot;IDquot; , quot;Namequot; , quot;CountryCodequot; , quot;Districtquot; , quot;Populationquot;] , quot;ROWSquot; : [[3793 , quot;New Yorkquot; , quot;USAquot; , quot;New Yorkquot; , 8008278] , [3794 , quot;Los Angelesquot; , quot;USAquot; , quot;Californiaquot; , 3694820] , [3795 , quot;Chicagoquot; , quot;USAquot; , quot;Illinoisquot; , 2896016] , [3796 , quot;Houstonquot; , quot;USAquot; , quot;Texasquot; , 1953631] , [3797 , quot;Philadelphiaquot; , quot;USAquot; , quot;Pennsylvaniaquot; , 1517550] , [3798 , quot;Phoenixquot; , quot;USAquot; , quot;Arizonaquot; , 1321045] , [3799 , quot;San Diegoquot; , quot;USAquot; , quot;Californiaquot; , 1223400] , [3800 , quot;Dallasquot; , quot;USAquot; , quot;Texasquot; , 1188580] , [3801 , quot;San Antonioquot; , quot;USAquot; , quot;Texasquot; , 1144646] , [3802 , quot;Detroitquot; , quot;USAquot; , quot;Michiganquot; , 951270]]}}
  • 9. Python Sample Client import urllib,urllib2,cjson db = lambda q: cjson.decode( urllib2.urlopen( “http://localhost:9090/db? ”+urllib.quote('{“SQL ”:”%s”}' % q)).read())
  • 10. Other Solutions MySQLProxy ● SQLRelay ● mysqli ●
  • 11. Requirements Independent ● Scalable ● Flexible ● Simple ●
  • 13. Problems Not Solved Back end scripts – ● bulk import/export Large Data Retrievals ● – cursors Native Binds / ● Prepared Statements Enterprise Buzzword ● Compliance A Sober Lindsay ●
  • 14. DBSlayer is the middle-man Connection Pooling ● Multi-threaded ● Round robin distribution of queries ● Automatic fail over to alternate database ● Easy configuration ● Statistics / Logging ● Tiny code base ●
  • 16. Flexiblity Loose coupling of front and middle ● easy upgrade – abstract location of db – makes re-config a – snap Language independent ● JSON easily shared across languages – Common place to add additional features ●
  • 17. Scalability Round robin across slaves ● Automatic fail over for slaves ● Easily add more slaves ● HTTP plays nice with load balancers ● JSON is very cachable ● Stateless is required for effective scaling ●
  • 18. Simplicity HTTP and JSON - everyone knows them ● Transparent protocol – capture, interpret ● Lots of tools to work with HTTP ● Works with your browser ● HTTP server base – easy to deploy new ● endpoints
  • 20. Stats
  • 24. View Source Built on top of MySQL C API and Apache ● Runtime Libraries (APR) Why APR? ● Platform agnostic – Memory Pools – Networking / Threading Abstractions – Proven –
  • 25. 3 Parts server – handles network, HTTP parsing, ● threading, logging db – handle mysql access, type mapping, ● connection failover util – json de/serializer ●
  • 26. Future / Your Chance improve HTTP caching layer ● ● complaince partition / shard ● more database support ● backends (APR new language ● dbd) bindings alternate serializer ● more stats / ● XML logging security ● scriptable filters ●
  • 28. Another amazing talk... Rails Under The Knife Jacob Harris July 26th – 2:35pm Portland 255
  • 29. BoF Amazon S3/EC2 computing against large ● datasets – 7:30 F150 Future Newspapers – 8:30 E143-144 ●
  • 30. NYTIMES.COM open.nytimes.com – developers blog ● code.nytimes.com – code repository ●
  • 31. Free Stuff Visit the NYTimes Booth