The Magic of Apache Kafka
Let us understand what is Apache Kafka and how it works with Spring Boot
Through this post, I would like to discuss about message streaming platform know as Apache Kafka. I would not bore you with the mundane details about Kafka and its benefits. Because this information can be easily found and read on Apache Kafka official website : https://kafka.apache.org/documentation/. Here, I would like to share my findings and few key points which I came across when I started learning about Kafka. This might help someone who wants to learn more about Kafka. Also, I would like to share how to use Spring Boot application with Apache Kafka Producer and Consumer. So, let’s us now understand what Kafka exactly does and how we can use it to solve our problems.
In simple terms, Apache Kafka is an open-source distributed publish-subscribe messaging platform that has been purpose-built to handle real-time streaming data. A kafka cluster consist of topics. Events are organized and durably stored in topics. In other words, a topic is similar to a folder in a filesystem, and the events are the files in that folder. Events in a topic are not deleted after consumption and can be retained according to a per-topic configuration setting. The older events are then discarded accordingly.
Configuring Apache Kafka Server
Let us configure Apache Kafka Server now. We need to downloaded the Apache Kafka version 2.8.0 from https://kafka.apache.org/downloads. Download the zip file and extract its content. Rename the folder and copy it to the drive as shown below :
Now, we need to create kafka-logs and zookeeper-data folders and then edit the directory paths in zookeeper.properties and server.properties as shown below. This is done to store log files otherwise they will get stored in default tmp folder.
Also, you need to add listeners=PLAINTEXT://localhost:9092 otherwise the connection will not be established and a warning will be displayed.
WARN Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
In this post, we are using .bat files to run the server on windows platform. You can use .sh files if you are running the server on linux/unix platform.
Spring Boot with Apache Kafka Producer
Now, we are going to learn how to use Spring Boot application to publish messages on Kafka topic. Open https://start.spring.io/ and add the configurations as mentioned in the image below. Here, we have added two dependencies Spring for Apache Kafka and Spring Web. Generate the jar to download the zip folder.
Extract the zip folder and import the maven project in Eclipse IDE or IntelliJ IDEA. We are using Eclipse IDE here. In the pom.xml file you can verify the dependencies.
In, application.properties set server.port=8080
Now, create a user resource to post the messages for kafka consumer. Here, we have created a user with Name, Department and ID. A simple string message can also be used.
We need to start the Apache Kafka server before triggering our Spring Boot application. Apache Kafka contains zookeeper. It serves as the coordination interface between the Kafka broker and consumers. Zookeeper keeps track of status of the Kafka cluster nodes and it also keeps track of Kafka topics, partitions etc. In production, you might need to setup zookeeper separately. But here, we are using the zookeeper which comes with Apache Kafka.
Run C:\kafka>.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Run C:\kafka>.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties on command prompt. This will start the zookeeper service on port 2181.
Open new command prompt terminal and run C:\kafka>.\bin\windows\kafka-server-start.bat .\config\server.properties to start Apache Kafka server.
Now, open another terminal to receive messages in kafka consumer console.
The below command creates a topic called Kafka_Example with single topic and only one replica :
C:\kafka>.\bin\windows\kafka-topics.bat — create — zookeeper localhost:2181 — replication-factor 1 — partitions 1 — topic Kafka_Example
Kafka also provides a command-line consumer client for message consumption. The following command is used to start the console-based consumer :
C:\kafka>.\bin\windows\kafka-console-consumer.bat — bootstrap-server localhost:9092 — topic Kafka_Example — from-beginning
Now publish the user name on localhost then you can observe that Kafka consumer console displaying the details of those user.
http://localhost:8080/kafka/publish/Maria
http://localhost:8080/kafka/publish/John
GitHub Link :
Spring Boot with Apache Kafka Consumer
Now, we will see how to use Spring Boot application to consume messages from a Kafka topic. We need to create another project for consumer. The same way we created for producer.
After creating and setting up the project in Eclipse. Here, I have created a sample Kafka configuration file which sends messages in String.
KafkaConsumer is used to read data from Kafka server.
Now, kick start your spring boot application and then start zookeeper and Kafka server in separate command prompt. As we have already created a Kafka topic Kafka_Example. We need not create this topic again. But if you are using a new Kafka topic then a new topic has to be created.
Kafka provides users with a command-line producer client that accepts inputs from the command line and publishes them as a message to the Kafka cluster. By default each new line entered is considered as a new message. The following command is used for starting the console-based producer for sending messages :
Now, after running Kafka console producer in separate command prompt, the messages can be published. These messages are then consumed in our spring boot application.
This is how Spring Boot application reads data from Apache Kafka.
GitHub Link :
This is a very simple example to understand the concepts of Apache Kafka Consumer and Producer with Spring Boot application. Hope this will help anyone who wants to learn more about Apache Kafka. I am just trying to share my experiences and providing solutions for the issues observed during my learning about Kafka. Please feel free to review or correct the details mentioned in this article which will help me to create more valuable contents for other developers. Happy Learning! :)
