Apache Kafka is a buzz word now and days and it lives upto its reputation. Kafka is a pub-sub messaging system in simple words, but differs a lot from traditional JMS systems.
This post a quick start guide on using Apache Kafka.
Following needs to be installed before you proceed further
- JDK (tested with Java 1.7)
- Kafka (choose a distribution)
- Apache Zookeeper (used ver 3.4.6)
The rest of the tutorial is organized as
- Starting Zookeeper
- Starting Apache Kafka
- Playing with topics
- Writing a message(s) to the topic
- Reading message(s) from the topic
Starting Zookeeper with default is simple. Move to the directory where Zookeeper is installed and execute the following
bash$ bin/zkServer.sh start
This assumes that you have the zoo.cfg in the conf directory.
Starting Apache Kafka
Move to the directory where Apache Kafka is installed and execute following commands
bash$ bin/kafka-server-start.sh config/server.properties
Kafka shall startup and display messages on console.
Now we have Kafka ready to be used.
Playing with Topics
Now that we have a fresh installation of Kafka, we need to create a topic before we can use it. Let’s create a topic with name testTopic. Let’s run the list command before and after we create the topic.
bash-3.2$ bin/kafka-topics.sh --list --zookeeper localhost:2181 bash-3.2$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testTopic Created topic "testTopic". bash-3.2$ bin/kafka-topics.sh --list --zookeeper localhost:2181 testTopic bash-3.2$
The first run of list won’t return anything, as we don’t have any topic yet. The next step is to create a topic and then execute the list command again.
Now it’s time to generate and consume messages using the testTopic. To keep things simple, we shall use the console utilities that ship with Kafka to produce and consume the messages.
Writing messages to Topic
Here we shall use the Console producer and push few messages to the topic. Execute following on Console
bash-3.2$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testTopic [2015-06-17 11:43:54,779] WARN Property topic is not valid (kafka.utils.VerifiableProperties) This message# 1 This is message #2 ^Cbash-3.2$
This pushed 2 messages to the Topic. We kill the process by Ctrl-C
Now let’s read these messages.
Reading messages from Topic
Again to read the messages we shall use the Console consumer utility. Execute following on the command prompt.
bash-3.2$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning--topic testTopic 'from-beginning--topic' is not a recognized option bash-3.2$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic testTopic This message# 1 This is message #2
Notice the difference here. Instead of broker list, we use the Zookeeper ensemble here and ask the client to read from the beginning.
This is it for now. In subsequent posts we shall see, using Producer and Consumer example in Java.