High concurrency, Low latency analytics using Spark/Kudu
- 1. High concurrency, Low latency analytics using Spark/Kudu Chris George
- 2. Who is this guy?
- 3. Tech we will talk about: Kudu Spark Spark Job Server Spark Thrift Server
- 5. Apache Kudu
- 7. Columnar vs other types of storage
- 8. What if you could update parquet/ORC easily?
- 9. HDFS vs Kudu vs HBase/Cassandra/xyz
- 10. Kudu is purely a storage engine accessible through api
- 11. To add sql queries/more advanced sql like operations
- 12. Impala vs Spark
- 14. Master and Tablets in Kudu
- 16. Number of cores = number of partitions
- 17. Partitioning can be on 1+ columns
- 18. Composite primary keys important to filter on the key in order A, B, C i.e. don’t scan for just B if possible it will be expensive
- 19. Scans on a tablet is single threaded but you can do 200+ scans on a tablet concurrently
- 20. To find your scale... load up a single tablet Insertion, Update, Deletes.. concurrently Until it doesn't meet your performance
- 22. Kudu client is java Python connectors coming C++ client
- 23. Java Client loops through tablets, but not concurrently
- 24. But you can code the multithread or contribute
- 27. Predicates/Projections on any column very quickly at scale
- 28. Spark
- 30. Reads CSV val df = sqlContext.read .format("com.databricks.spark.csv") .option("header", "true") // Use first line of all files as header .option("inferSchema", "true") // Automatically infer data types .load("cars.csv") Writes CSV val selectedData = df.select("year", "model") selectedData.write .format("com.databricks.spark.csv") .option("header", "true") .save("newcars.csv")
- 31. But these are often simlified as: val parquetDataframe = sqlContext.read.parquet(“people.parquet") parquetDataframe.write.parquet("people.parquet")
- 32. I wrote the current version of the Kudu Datasource/Spark Integration
- 33. There are limitations with the datasource api
- 34. Save Modes for datasource api: append, overwrite, ignore, error
- 35. append = insert
- 36. overwrite = truncate + insert
- 37. ignore = create if not exists
- 38. error = throw exception if the data exists
- 39. What if I want to update? Nope
- 40. What about deletes? Not individually
- 41. So how do you support updates/deletes?
- 42. By not using the datasource api.. but I'll talk more about that in a minute
- 44. So why use datasource api?
- 45. Because it's smarter than it appears for reads
- 47. Pushdown predicates: val df = sqlContext.read.options(Map("kudu.master" -> "kudu.master:7051","kudu.table" -> “kudu_table")).kudu df.filter("id" >= 5).show()
- 48. Datasource has knowledge of what can be pushed down to underlying store and what can not.
- 49. Why am I telling you this?
- 50. Cause if you want things to be fast you need to know what is not pushed down!
- 52. Did you notice whats missing? EqualTo GreaterThan GreaterThanOrEqual LessThan LessThanOrEqual And
- 53. "OR" So spark will use it's optimizer to run two separate kudu scans for the OR "IN" is coming very soon, nuanced performance details
- 54. btw if you register the dataframe as a temp table in spark "select * from someDF where id>=5" will also do pushdowns
- 55. "select * from someDF where id>=5" will also do pushdowns
- 56. things like select * from someDF where lower(name)="joe" will pull the entire table into memory probably a bad thing
- 57. Projections will also be pushed down to kudu so your not retrieving the entire row df.select("id", "name") select id, name from someDf
- 58. Looked at lots of existing datasource’s to design kudu’s
- 59. How does kudu do updates/deletes in spark?
- 60. // Use KuduContext to create, delete, or write to Kudu tables val kuduContext = new KuduContext("kudu.master:7051")
- 61. // Insert data kuduContext.insertRows(df, "test_table") // Delete data kuduContext.deleteRows(filteredDF, "test_table") // Upsert data kuduContext.upsertRows(df, "test_table") // Update data val alteredDF = df.select("id", $"count" + 1) kuduContext.updateRows(alteredDF, “test_table") http://kudu.apache.org/docs/developing.html
- 62. Upserts are handled server side for performance Upserts can also be handled through datasource api: df.write.options(Map("kudu.master"-> "kudu.master:7051", "kudu.table"- >"test_table")).mode("append").kudu
- 63. You can also create, check existence and delete tables through api
- 64. Additional notes: Kudu datasource currently works with spark 1.x Next release it will support both 1.x and 2.x It's being improved on regular basis
- 65. Number of partitions on the dataframe is related to how many tablets/partitions are related to the filter. Partition scans are parallel and have locality awareness in spark
- 66. Be sure to set spark locality wait to something other small for low latency (3 seconds is the spark default)
- 68. Created for low latency jobs on spark
- 69. Persistent contexts Reduces runtime of hello world type of job from 1 second to 10 ms
- 70. Rest based api to: Run Jobs Create contexts Check status of job both async/sync
- 71. Creating a context calls spark submit (in separate jvm mode) Uses akka to communicate between rest and spark driver
- 72. To create a persistent context you need: cpu cores + memory footprint name to reference it by factory to use for the context. ie HiveContextFactory vs SqlContextFactory
- 73. Our average job time is 30ms when coming through api for simpler retrievals
- 74. Jobs need to implement an interface context will be passed in DON’T CREATE YOUR OWN SQLCONTEXT!!
- 75. Currently only supports spark 1.x 2.x is coming soonish
- 76. Keeps track of job runtimes in nice ui along with additional metrics
- 77. You can cache data and it will be available to later jobs
- 78. You can also load objects and they are available to later jobs via NamedObject interface
- 79. Persistent context can be run in seperate JVM or within SJS
- 80. It does have some sharp edges though...
- 81. Due to jvm classloader contexts need to be restarted on deploy to pick up new code
- 82. Some settings: spark.files.overwrite = true context-per-jvm = true spray-can: parsing.max-content-length = 256m spray-can: idle-timeout = 600 s spray-can: request-timeout = 540 s spark.serializer = "org.apache.spark.serializer.KryoSerializer" filedao vs sqldao backend have to build from source/no binary for SJS
- 83. hive-site.xml <configuration> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:derby:memory:myDB;create=true</value> <description>JDBC connect string for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>org.apache.derby.jdbc.EmbeddedDriver</value> <description>Driver class name for a JDBC metastore</description> </property>
- 85. I run the following on a persistent context: sc.getConf.set("spark.sql.hive.thriftServer.singleSes sion", "true") sqlContext.setConf("hive.server2.thrift.port", port) // port to run thrift server on HiveThriftServer2.startWithContext(sqlContext)
- 86. Now I can connect using hive-jdbc odbc (microsoft or simba)
- 87. Run a job with joins/ or even just a basic dataframe through datasource api and registerTempTable
- 88. val df = sqlContext.read.options(Map("kudu.master" -> "kudu.master:7051","kudu.table" -> “kudu_table")).kudu df.registerTempTable
- 89. You could also potentially cache/persist via spark and register that way assuming joins are expensive
- 90. Now you can run queries as if it was a traditional database
- 91. Hey thats great, but how fast? 500 ms average response time 200 concurrent complex queries 1+ Billion rows with 200+ columns sql queries with 5 predicates, min,max,count some values and group by on 5 columns No spark caching
- 92. We take this a step farther and do complex dataframes and it is made available as a registered temp table
- 93. Questions… If we run out of time send me questions on slack