SlideShare a Scribd company logo

Introduction to Kafka with Spring Integration

•Download as PPTX, PDF•
•
B
Borislav Markov

A brief example that demonstrates asic usage of Kafka and Spring Integration when you build a microservice.

Related slideshows

Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
 
Building Microservices with Apache Kafka
Building Microservices with Apache KafkaBuilding Microservices with Apache Kafka
Building Microservices with Apache Kafka
 
Stream processing using Kafka
Stream processing using KafkaStream processing using Kafka
Stream processing using Kafka
 
1 of 35
Introduction to Kafka with Spring Integration
Introduction to Kafka with Spring Integration • Kafka (Mihail Yordanov) • Spring integration (Borislav Markov) • Students Example (Mihail & Borislav) • Conclusion
Motivation • Real time data being continuously generated • Producers and consumers relationships • Streaming systems • Messaging systems
Apache Kafka • Developed by LinkedIn • “We built Apache Kafka at LinkedIn with a specific purpose in mind: to serve as a central repository of data streams.” - Jay Kreps • Kafka improvements – Transporting data between systems – Richer analytical processing • “A publish-subscribe messaging system rethought as a distributed commit log”
Kafka Vocabulary • Brokers • Producers • Consumers • “A publish-subscribe messaging system rethought as a distributed commit log”
Kafka Vocabulary • Topics • Partitions • “A publish-subscribe messaging system rethought as a distributed commit log”
Kafka Speed
Kafka Speed
Kafka Speed
Consumer groups • Definition - “A publish-subscribe messaging system rethought as a distributed commit log” • Publish - Subscribe • Queueing
Spring Integration • Spring Integration enables lightweight messaging • Supports integration with external systems via declarative adapters • Primary goal is to provide a simple model for building enterprise integration solutions • Separation of concerns • Maintainable, testable code
Features • Implementation of most of the Enterprise Integration Patterns • Endpoint • Channel (Point-to-point and Publish/Subscribe) • Aggregator • Filter • Transformer • Control Bus • … • Integration with External Systems • ReST/HTTP • ...
The theory... The patterns can be found at http://www.enterpriseintegrationpatterns.com/patterns/messag ing/index.html
Message • Messages are objects that carry information between two applications • They are constructed by the producer and they are consumed/deconstructed at the consumer/subscriber • Message consists of: – payload – header public interface Message<T> { T getPayload(); MessageHeaders getHeaders(); }
Message endpoints Common endpoints: • Service Activator - invokes a method on a bean • Message Bridge - couples different messaging modes/adapters. Example: (P2P with Publish/Subscribe) • Message Enricher - enrich the incoming message with additional information. – Header Enricher - add header attributes – Payload Enricher - enrich payload
Message endpoints (continued) • Gateway - Used when we don’t have knowledge for the messaging system. – Synchronous Gateway - void or returns T – Asynchronous Gateway - returns Future<T> • Delayer - introduce delay between sender and receiver • Consumers – Polling Consumers - poll messages in a timely fashion – Polling Using Triggers - poll messages with PeriodicTrigger and with CronTrigger – Event-Driven Consumers - they wait for someone to deliver the message (framework’s responsibility)
Message endpoints (continued) • Channel Adapter - endpoint that connects a Message Channel to some other system or transport. –inbound –outbound
Message Channels • interface MessageChannel – boolean send(Message<?> message); – boolean send(Message<?> message, long timeout); • Point-to-Point Mode -> PollableChannel – Message<?> receive(); – Message<?> receive(long timeout); • Publish/Subscribe Mode -> SubscribableChannel – boolean subscribe(MessageHandler handler); – boolean unsubscribe(MessageHandler handler);
Message Channels(continued) •Queue Channel - has capacity •Priority Channel - queue channel with prioritization •PublishSubscribe Channel •DirectChannel - mixed type P2P and Pub/Sub
Message Channels(continued)
Flow components • Filters – Custom filters are tied to the framework, can be bean method returning boolean – Framework MessageSelector - use of method boolean accept(Message m) – Using annotation - @Filter • Routers – PayloadTypeRouter – HeaderValueRouter – Custom Routers - any bean method can be router, the result will be the channel name. – Recipient List Router - the message goes to all statically defined channels
Flow components (continued...) • Splitters - splits a message into pieces – Custom splitters - any bean method; – Framework AbstractMessageSplitter - use of method Object splitMessage(Message m); – Using annotation - @Splitter ; • Aggregator – assemble multiple messages to create a single parent message – Complex task - all messages of a set have to arrive before aggregators can start work – CorrelationStrategy - defines the key for grouping of the messages, the default grouping is based on CORRELATION_ID – ReleaseStrategy - dictates at which point the group of messages should be sent or released for aggregation. Default is SequenceSizeReleazeStrategy – Message Store - aggregators hold messages until all of them arrived. Options are: • in-memory • external database • Resequencer - fix order of the messages(work on SEQUENCE_NUMBER)
Transformers • Built-in transformers – String Transformers - invokes toString() method of the payload – Map Transformers - transformes POJO to a name-value pair of Map – Serializing and Deserializing Transformers - converts payload to byte array – JSON Transformers - object-to-json converts POJO to readable JSON format; json-to-object transformer needs additional property “type” – XML Transformers - requires Spring OXM (Object-to-XML); org.springframework.oxm.Marshaller/Unmarshaller
Transformers(continued...) • Built-in transformers – XPath Transformers - decodes XML using XPath expressions, ex.: xpath- expression=”/mytag/@prop” • Custom Transformers - can be any spring bean method • Using @Transformer
Students Example • Example application will demo usage of Kafka and Spring Integration • App is built with maven • Ideal candidate for Microservice • Idea: takes students from outside and calculates their average score
Maven dependencies
Project Structure • Package “api” – POJO models • Package “config” – beans with factory methods • Package “service” – the business logic • Folder “resources” – application.properties
Spring Integration DSL
Students Example
Service activator
Introduction to Kafka with Spring Integration
Running the app and giving some inputs
Check the logs
Conclusion Why Kafka and Spring ? • Easy for microservices • Clean and testable code • Widely adopted technologies • Easy to be dockerized
Introduction to Kafka with Spring Integration

More Related Content

Introduction to Kafka with Spring Integration

  • 1. Introduction to Kafka with Spring Integration
  • 2. Introduction to Kafka with Spring Integration • Kafka (Mihail Yordanov) • Spring integration (Borislav Markov) • Students Example (Mihail & Borislav) • Conclusion
  • 3. Motivation • Real time data being continuously generated • Producers and consumers relationships • Streaming systems • Messaging systems
  • 4. Apache Kafka • Developed by LinkedIn • “We built Apache Kafka at LinkedIn with a specific purpose in mind: to serve as a central repository of data streams.” - Jay Kreps • Kafka improvements – Transporting data between systems – Richer analytical processing • “A publish-subscribe messaging system rethought as a distributed commit log”
  • 5. Kafka Vocabulary • Brokers • Producers • Consumers • “A publish-subscribe messaging system rethought as a distributed commit log”
  • 6. Kafka Vocabulary • Topics • Partitions • “A publish-subscribe messaging system rethought as a distributed commit log”
  • 10. Consumer groups • Definition - “A publish-subscribe messaging system rethought as a distributed commit log” • Publish - Subscribe • Queueing
  • 11. Spring Integration • Spring Integration enables lightweight messaging • Supports integration with external systems via declarative adapters • Primary goal is to provide a simple model for building enterprise integration solutions • Separation of concerns • Maintainable, testable code
  • 12. Features • Implementation of most of the Enterprise Integration Patterns • Endpoint • Channel (Point-to-point and Publish/Subscribe) • Aggregator • Filter • Transformer • Control Bus • … • Integration with External Systems • ReST/HTTP • ...
  • 13. The theory... The patterns can be found at http://www.enterpriseintegrationpatterns.com/patterns/messag ing/index.html
  • 14. Message • Messages are objects that carry information between two applications • They are constructed by the producer and they are consumed/deconstructed at the consumer/subscriber • Message consists of: – payload – header public interface Message<T> { T getPayload(); MessageHeaders getHeaders(); }
  • 15. Message endpoints Common endpoints: • Service Activator - invokes a method on a bean • Message Bridge - couples different messaging modes/adapters. Example: (P2P with Publish/Subscribe) • Message Enricher - enrich the incoming message with additional information. – Header Enricher - add header attributes – Payload Enricher - enrich payload
  • 16. Message endpoints (continued) • Gateway - Used when we don’t have knowledge for the messaging system. – Synchronous Gateway - void or returns T – Asynchronous Gateway - returns Future<T> • Delayer - introduce delay between sender and receiver • Consumers – Polling Consumers - poll messages in a timely fashion – Polling Using Triggers - poll messages with PeriodicTrigger and with CronTrigger – Event-Driven Consumers - they wait for someone to deliver the message (framework’s responsibility)
  • 17. Message endpoints (continued) • Channel Adapter - endpoint that connects a Message Channel to some other system or transport. –inbound –outbound
  • 18. Message Channels • interface MessageChannel – boolean send(Message<?> message); – boolean send(Message<?> message, long timeout); • Point-to-Point Mode -> PollableChannel – Message<?> receive(); – Message<?> receive(long timeout); • Publish/Subscribe Mode -> SubscribableChannel – boolean subscribe(MessageHandler handler); – boolean unsubscribe(MessageHandler handler);
  • 19. Message Channels(continued) •Queue Channel - has capacity •Priority Channel - queue channel with prioritization •PublishSubscribe Channel •DirectChannel - mixed type P2P and Pub/Sub
  • 21. Flow components • Filters – Custom filters are tied to the framework, can be bean method returning boolean – Framework MessageSelector - use of method boolean accept(Message m) – Using annotation - @Filter • Routers – PayloadTypeRouter – HeaderValueRouter – Custom Routers - any bean method can be router, the result will be the channel name. – Recipient List Router - the message goes to all statically defined channels
  • 22. Flow components (continued...) • Splitters - splits a message into pieces – Custom splitters - any bean method; – Framework AbstractMessageSplitter - use of method Object splitMessage(Message m); – Using annotation - @Splitter ; • Aggregator – assemble multiple messages to create a single parent message – Complex task - all messages of a set have to arrive before aggregators can start work – CorrelationStrategy - defines the key for grouping of the messages, the default grouping is based on CORRELATION_ID – ReleaseStrategy - dictates at which point the group of messages should be sent or released for aggregation. Default is SequenceSizeReleazeStrategy – Message Store - aggregators hold messages until all of them arrived. Options are: • in-memory • external database • Resequencer - fix order of the messages(work on SEQUENCE_NUMBER)
  • 23. Transformers • Built-in transformers – String Transformers - invokes toString() method of the payload – Map Transformers - transformes POJO to a name-value pair of Map – Serializing and Deserializing Transformers - converts payload to byte array – JSON Transformers - object-to-json converts POJO to readable JSON format; json-to-object transformer needs additional property “type” – XML Transformers - requires Spring OXM (Object-to-XML); org.springframework.oxm.Marshaller/Unmarshaller
  • 24. Transformers(continued...) • Built-in transformers – XPath Transformers - decodes XML using XPath expressions, ex.: xpath- expression=”/mytag/@prop” • Custom Transformers - can be any spring bean method • Using @Transformer
  • 25. Students Example • Example application will demo usage of Kafka and Spring Integration • App is built with maven • Ideal candidate for Microservice • Idea: takes students from outside and calculates their average score
  • 27. Project Structure • Package “api” – POJO models • Package “config” – beans with factory methods • Package “service” – the business logic • Folder “resources” – application.properties
  • 32. Running the app and giving some inputs
  • 34. Conclusion Why Kafka and Spring ? • Easy for microservices • Clean and testable code • Widely adopted technologies • Easy to be dockerized