[Flume Cookbook] Quick Start

Now that we have installed Flume, let's run a quick demo. We shall be using a single Node installation.

Pre-requisite
Java 6 or higher

The object of this quick start would be to send some messages to Flume and see them printed on console.

Create the configuration file with name single-node-demo.properties in FLUME_HOME/conf

a1.sources=src1
a1.sinks=sink1
a1.channels=ch1

# Configure the source
a1.sources.src1.type=netcat
a1.sources.src1.bind=localhost
a1.sources.src1.port=51000

# Sink Configuration
a1.sinks.sink1.type=logger

# Channel configuration
a1.channels.ch1.type=memory
a1.channels.ch1.capacity=1000
a1.channels.ch1.transactionCapacity=100

# Link stuff together
a1.sources.src1.channels=ch1
a1.sinks.sink1.channel=ch1

You can get the file from github

Now, we need to start the Agent with this config file. Go to FLUME_HOME, and execute the following command

$bin/flume-ng agent --conf-file conf/single-node-demo.properties --name a1 --conf ./conf/ -Dflume.root.logger=INFO,console

I use a simple script to eliminate the repetitive task

bin/flume-ng agent --conf-file $1 --name a1 --conf ./conf/ -Dflume.root.logger=INFO,console

Can get the script from here

Once the agent is start, open a new terminal window and execute nc command

$nc localhost 51000

Here is the image how it would look

nc command

nc command

After that type the message and press "Enter", the message shall be displayed on Flume Agent window, as shown in figure below.

Flume Agent Output

Flume Agent Output

We shall discuss the details in subsequent sections.

References

https://www.ashishpaliwal.com/blog/2013/06/flume-cookbook-playing-with-apache-flume/

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.