Streaming with Spring Cloud Stream and Apache Kafka - Soby Chacko
- 3. Spring Cloud Stream
Overview
• General purpose framework for writing event
driven/stream processing micro services
• Destination based bindings on your choice of
middleware
- 7. Apache Kafka
• Single Source of Truth for data
• Fault Tolerant
• Based on the ubiquitous log
data structure
• Producer/Consumer/Streams
API
• Transactions
• Exactly once semantics
processing guarantees
- 10. Spring Cloud Stream
Event-Driven microservices framework
Pluggable Binder
Implementations
Opportunities: Same code; Same tests; Drop-in replacement for a variety of Message Brokers
Rabbit MQ Apache Kafka Google PubSub
Amazon Kinesis Azure Event Hubs Solace
- 11. Spring Cloud Stream
Application level features
• Message conversions are handled by the framework
• Support for error handling - DLQ
• Auto provisioning of topics (destinations)
• Schema evolution
• Health indicators
• And a whole lot of other convenient features….
- 12. Spring Cloud Stream
Programming model
• Java 8 based functional programming model
• Spring Cloud Function support as the foundation
• Reactive programming support using Project
Reactor
• Framework provided annotations based
programming (EnableBinding, StreamListener)
- 13. Spring Cloud Stream
application types
• Sources - java.util.function.Supplier
• Sinks - java.util.function.Consumer
• Processors - java.util.function.Function
- 18. “We call an application data-
intensive if data is its primary
challenge—the quantity of
data, the complexity of data,
or the speed at which it is
changing.”
- 19. Kafka Streams
• Client library
• No need for dedicated processing cluster
• All guarantees of Kafka are applicable
• Per record processing
• Stateful stream processing
• And many more features…
- 20. Spring Cloud Stream
with Kafka Streams
Spring Boot Application
Kafka Streams binder from Spring
Cloud Stream
Kafka Streams library
Kafka Broker
Spring Kafka
- 21. Major types available for
stream processing
• KStream
• KTable
• GlobalKTable
Spring Cloud Stream
provides binding capabilities
for all these three types.
- 22. Stream processing concepts
• Stream-Table duality
• Time based windowing
• Joins and Aggregation
• Stateless map and filter like operations on data
- 23. Stateful Stream Processing
• Kafka Streams provide built in capabilities for
stateful stream processing
• Default state store - RocksDB
• Spring Cloud Stream touch points for interactive
queries in Kafka Streams
- 24. Spring Cloud Stream programming
model for Kafka Streams apps
• Java 8 based functional programming model
• Processors can be written as
java.util.function.Function or
java.util.function.Consumer.
• Multiple input/output binding capabilities
- 25. Kafka Streams app as a
function…
@Bean
public Function<KStream<Object, String>, KStream<?, WordCount>> process()
{
return input -> input
.flatMapValues(value ->
Arrays.asList(value.toLowerCase().split("W+")))
.map((key, value) -> new KeyValue<>(value, value))
.groupByKey(Serialized.with(Serdes.String(), Serdes.String()))
.windowedBy(TimeWindows.of(5000))
.count(Materialized.as("foo-WordCounts"))
.toStream()
.map((key, value) -> new KeyValue<>(null, new
WordCount(key.key(), value,
new Date(key.window().start()), new
Date(key.window().end()))));
}
- 27. Resources
• Code used for the demo
• Samples Repository
• Project page
• Dataflow microsite
• Confluent Blogs on Spring with Kafka:
Part 1 | Part 2 | Part 3 | Part 4