Apache kafka
- 1. Apache Kafka The Big Data Messaging Tool
- 2. Index ● Need of Apache Kafka ● Kafka At LinkedIn ● What is Apache Kafka and its features ● Components of Apache Kafka ● Architecture and Flow in Apache Kafka ● Uses of Apache Kafka ● Kafka in Real World ● Comparison with other messaging system ● Demo
- 3. Need Of Apache Kafka In Big Data, an enormous volume of data is used. Regarding data, we have two main challenges.The first challenge is how to collect large volume of data and the second challenge is to analyze the collected data. To overcome those challenges, you must need a messaging system.
- 4. Kafka at Linkedin If data is the lifeblood of high technology, Apache Kafka is the circulatory system in use at LinkedIn -Todd Palin The Kafka ecosystem at LinkedIn is sent over 800 billion messages per day which amounts to over 175 terabytes of data. Over 650 terabytes of messages are then consumed daily, which is why the ability of Kafka to handle multiple producers and multiple consumers for each topic is important. At the busiest times of day, linkedin is receiving over 13 million messages per second, or 2.75 gigabytes of data per second. To handle all these messages, LinkedIn runs over 1100 Kafka brokers organized into more than 60 clusters.
- 5. What is Apache Kafka Apache Kafka is a distributed publish-subscribe messaging system and a robust queue that can handle a high volume of data and enables you to pass messages from one end-point to another. Kafka is suitable for both offline and online message consumption. Kafka messages are persisted on the disk and replicated within the cluster to prevent data loss. Kafka supports low latency message delivery and gives guarantee for fault tolerance in the presence of machine failures. Kafka is very fast, performs 2 million writes/sec.
- 6. Kafka persists all data to the disk, which essentially means that all the writes go to the page cache of the OS (RAM). This makes it very efficient to transfer data from page cache to a network socket. Kafka is very fast and guarantees zero downtime and zero data loss Kafka is built on top of the ZooKeeper synchronization service. It integrates very well with Apache Storm and Spark for real-time streaming data analysis.
- 7. Features of Kafka Following are a few benefits of Kafka − ● Reliability − Kafka is distributed, partitioned, replicated and fault tolerance. ● Scalability − Kafka messaging system scales easily without down time.. ● Durability − Kafka uses distributed commit log which means messages persists on disk as fast as possible, hence it is durable. ● Performance − Kafka has high throughput for both publishing and subscribing messages. It maintains stable performance even many TB of messages are stored.
- 8. Components Of Kafka ● Topics: The categories in which Kafka maintains its feeds of messages. ● Producers: The processes that publish messages to a topic. ● Consumers: The processes that subscribe to topics so as to fetch the above published messages. ● Broker: The cluster consisting of one or more servers in Kafka. ● TCP Protocol: The client and server communicate using this protocol. ● ZooKeeper : distributed configuration and synchronization service
- 9. Uses of Zookeeper ● Each Kafka Broker can coordinate with other broker with the help of zookeeper ● Zookeeper serves as the coordination interface between the Kafka brokers and consumers. ● Kafka stores basic metadata in Zookeeper such as information about topics, brokers, consumer offsets (queue readers) and so on. ● The leader election between the Kafka broker is also done by using Zookeeper in the event of leader failure.
- 10. Kafka Cluster With Kafka we can create multiple types of cluster ● A single node—single broker cluster ● A single node—multiple broker clusters ● Multiple nodes—multiple broker clusters
- 11. A single node—multiple broker clusters
- 12. Partition and topics ● A topic may have many partitions thus enabling it to handle an arbitrary amount of data. Each partition is an ordered, immutable sequence of messages that is continually appended to a commit log. The messages in the partition are each assigned with a sequential id number called the offset, which uniquely identifies each message within the partition. ● The partitions of the log are distributed over the servers in the Kafka cluster with each server handling data and requests for a share of partitions. Each partition is replicated across a configurable number of servers. ● Kafka assigns each server with a leader and follower, which helps in the whole replication cycle of messages in the partitions. ● In a nutshell, Kafka partitions the incoming messages for a topic, and assigns these partitions to an available Kafka broker.
- 13. Architectural Flow for Pub-Sub Messaging Kafka offers a single consumer abstraction that generalizes both Queuing and Publish-Subscribe Following is the step wise workflow of the Pub-Sub Messaging − ● Producers send message to a topic at regular intervals. ● Kafka broker stores all messages in the partitions configured for that particular topic. It ensures the messages are equally shared between partitions. If the producer sends two messages and there are two partitions, Kafka will store one message in the first partition and the second message in the second partition. ● Consumer subscribes to a specific topic. ● Once the consumer subscribes to a topic, Kafka will provide the current offset of the topic to the consumer and also saves the offset in the Zookeeper ensemble. ● Consumer will request the Kafka in a regular interval (like 100 Ms) for new messages.
- 14. Architectural Flow ● Once Kafka receives the messages from producers, it forwards these messages to the consumers. ● Consumer will receive the message and process it. ● Once the messages are processed, consumer will send an acknowledgement to the Kafka broker. ● Once Kafka receives an acknowledgement, it changes the offset to the new value and updates it in the Zookeeper. Since offsets are maintained in the Zookeeper, the consumer can read next message correctly even during server outrages. ● This above flow will repeat until the consumer stops the request. ● Consumer has the option to rewind/skip to the desired offset of a topic at any time and read all the subsequent messages.
- 15. Using Apache Kafka ● Install Zookeeper sudo apt-get install zookeeperd ● Download and extract kafka in a directory ● Start the Kafka Server Sh ~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties
- 16. ● Create a producer with a topic sh ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 - --topic TutorialTopic ● Create a consumer with same topic ~/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic TutorialTopic --from-beginning
- 17. Who else uses Kafka Twitter - Twitter uses Storm-Kafka as a part of their stream processing infrastructure. Netflix - uses Kafka for real-time monitoring and event processing. Mozilla - Kafka will soon be replacing a part of Mozilla current production system to collect performance and usage data from the end-user’s browser for projects like Telemetry, Test Pilot, etc. https://cwiki.apache.org/confluence/display/KAFKA/Powered+By
- 18. Demo of Kafka ● This demo is a grails web application which uses kafka to send messages from producer to consumer ● The messages will include time spent on page , time when page visited . time when page left , current authenticated user and uri of page ● This will be received on consumer end and records are persisted in db
- 19. Common Use cases of Apache kafka ● Website activity tracking: The web application sends events such as page views and searches Kafka, where they become available for real-time processing, dashboards and offline analytics in Hadoop ● Operational metrics: Alerting and reporting on operational metrics ● Log aggregation: Kafka can be used across an organization to collect logs from multiple services and make them available in standard format to multiple consumers, including Hadoop and Apache Solr.
- 20. Common Use cases of Apache kafka Stream processing: A framework such as Spark Streaming reads data from a topic, processes it and writes processed data to a new topic where it becomes available for users and applications. Kafka’s strong durability is also very useful in the context of stream processing.
- 21. Comparison with other Messaging system In maturity and features Rabbit MQ outshines Kafka but when it comes to durability , high throughput and fault tolerance Apache Kafka stands as winner https://www.infoq.com/articles/apache-kafka
- 23. Thanks Project Demo url : https://github.com/ackhare/GrailsKafkaPageCounter Presented By - Chetan Khare