Apache Kafka – Quick Start Guide

Apache Kafka is a buzz word now and days and it lives upto its reputation. Kafka is a pub-sub messaging system in simple words, but differs a lot from traditional JMS systems.

This post a quick start guide on using Apache Kafka.


Following needs to be installed before you proceed further

  • JDK (tested with Java 1.7)
  • Kafka (choose a distribution)
  • Apache Zookeeper (used ver 3.4.6)

The rest of the tutorial is organized as

  • Starting Zookeeper
  • Starting Apache Kafka
  • Playing with topics
  • Writing a message(s) to the topic
  • Reading message(s) from the topic

Starting Zookeeper

Starting Zookeeper with default is simple. Move to the directory where Zookeeper is installed and execute the following

bash$ bin/zkServer.sh start

This assumes that you have the zoo.cfg in the conf directory.

Starting Apache Kafka

Move to the directory where Apache Kafka is installed and execute following commands

bash$ bin/kafka-server-start.sh config/server.properties

Kafka shall startup and display messages on console.

Now we have Kafka ready to be used.

Playing with Topics

Now that we have a fresh installation of Kafka, we need to create a topic before we can use it. Let’s create a topic with name testTopic. Let’s run the list command before and after we create the topic.

bash-3.2$ bin/kafka-topics.sh --list --zookeeper localhost:2181
bash-3.2$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testTopic
Created topic "testTopic".
bash-3.2$ bin/kafka-topics.sh --list --zookeeper localhost:2181

The first run of list won’t return anything, as we don’t have any topic yet. The next step is to create a topic and then execute the list command again.

Now it’s time to generate and consume messages using the testTopic. To keep things simple, we shall use the console utilities that ship with Kafka to produce and consume the messages.

Writing messages to Topic

Here we shall use the Console producer and push few messages to the topic. Execute following on Console

bash-3.2$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testTopic
[2015-06-17 11:43:54,779] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
This message# 1
This is message #2

This pushed 2 messages to the Topic. We kill the process by Ctrl-C

Now let’s read these messages.

Reading messages from Topic

Again to read the messages we shall use the Console consumer utility. Execute following on the command prompt.

bash-3.2$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning--topic testTopic
'from-beginning--topic' is not a recognized option
bash-3.2$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic testTopic
This message# 1
This is message #2

Notice the difference here. Instead of broker list, we use the Zookeeper ensemble here and ask the client to read from the beginning.

This is it for now. In subsequent posts we shall see, using Producer and Consumer example in Java.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.