Queryable State for Kafka Streamsを使ってみた
•
2 likes•1,675 views
The document discusses queryable state for Apache Kafka Streams. It introduces Kafka Streams and stateful transformations. It then describes state for Kafka Streams, including how state is stored in RocksDB and tracked with a changelog in Kafka. Finally, it covers the new queryable state feature in Kafka Streams 0.10.1, which provides APIs to access state stores and retrieve values by key for windowed state.
Report
Share
Queryable State for Kafka Streamsを使ってみた
- 1. Queryable State for Kafka Streams / @laclefyoshi / ysaeki@r.recruit.co.jp
- 2. • • Kafka Streams • State for Kafka Streams • Queryable State for Kafka Streams • • 2
- 3. • 2011/04 • 2015/09 • • Druid (KDP, 2015) • RDB NoSQL ( , 2016; : HBase ) • ESP8266 Wi-Fi IoT (KDP, 2016) • • (DEIM 2014) • (WebDB Forum 2014) • Spark Streaming (Spark Meetup December 2015) • Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016) • (FutureOfData; 2016) 3
- 7. Kafka 7
- 8. Apache Kafka Version 0.10.1.0 • New Features • [KAFKA-1464] - Add a throttling option to the Kafka replication tool • [KAFKA-3176] - Allow console consumer to consume from particular partitions when new consumer is used. • [KAFKA-3492] - support quota based on authenticated user name • [KAFKA-3776] - Unify store and downstream caching in streams • [KAFKA-3858] - Add functions to print stream topologies �� [KAFKA-3909] - Queryable state for Kafka Streams • [KAFKA-4015] - Change cleanup.policy config to accept a list of valid policies • [KAFKA-4093] - Cluster id 8
- 9. Kafka Streams • Version 0.10.0.0 • Kafka API 9
- 10. Kafka Streams Stateless transformations Stateful transformations 10
- 11. State for Kafka Streams • Stateful transformations • 11
- 12. State • Kafka Streams • 2 • State Store RocksDB • Kafka StateChangelog • Kafka Streams • Key Value • Window 12
- 13. • Record Stream • Changelog Stream • State (Table) Record Changelog State 13
- 14. Queryable State for Kafka Streams • Version 0.10.1.0 • State Store API 14
- 15. Queryable State • ReadOnlyKeyValueStore • Window Streams State • Key • Value • ReadOnlyWindowStore • Window Streams State • Key, From (Long), To (Long) • Iterator<KeyValue<Long, Value>> 15
- 17. Kafka Streams • Stateful transformations count KafkaStreams streams = null; KStreamBuilder builder = new KStreamBuilder(); KStream<String, String> textLines = builder.stream(stringSerde, stringSerde, inputStream); KStream<String, Long> wordCounts = textLines .flatMapValues(value -> ...) .groupBy((key, word) -> word) .count(storeName) .toStream(); wordCounts.to(stringSerde, longSerde, outputStream); streams = new KafkaStreams(builder, conf); 17
- 18. Queryable State Streams • Streams o.a.k.streams.KafkaStreams KafkaStreams streams = ...; String key = "kafka"; ReadOnlyKeyValueStore<String, Long> store = streams.store( storeName, QueryableStoreTypes.<String, Long>keyValueStore()); Long value = store.get(key); return value; 18
- 19. Queryable State Streams • API KafkaStreams streams = ...; get("/results/:key", (req, res) -> { String key = req.params(":key"); ReadOnlyKeyValueStore<String, Long> store = streams.store( storeName, QueryableStoreTypes.<String, Long>keyValueStore()); Long value = store.get(key); return value; }); 19
- 22. $ java -cp target/kafka_streams_example-1.0-SNAPSHOT.jar org.saekiyoshiyasu.App kafka -> null ... kafka -> 5 $ ./bin/kafka-console-producer.sh --topic input-stream --broker-list 127.0.0.1:9092 What is Kafka good for? ... 22
- 23. Queryable State for Kafka Streams • Kafka Streams ? 23
- 24. Queryable State for Kafka Streams • Kafka Streams ? State 24
- 25. • Queryable State • Kafka Streams 25