[Flume Cookbook] Understanding Config file format

In this post we shall take a look at Flume's configuration file.

Flume configuration tells an Agent how to configure Sources, Sinks, Channels and related components.

Flume's configuration can be viewed as a Tree (not a BTree), with Agent name as the root node.

Flume Config file

Flume Config file

The picture above gives a broad view of Flume configuration hierarchy.

The same can be translated into file as

The first level can be translated as

<agent-name>.sources=<source> <source>
<agent-name>.channels=<channel> <channel>
<agent-name>.sinks=<sink> <sink>

We can now add subsequent details like

 
<agent-name>.sources.<source>.<property-name>=<property-value>

And same is applicable for other configuration option.

Note: This is not a comprehensive list of all options supported by Flume. We shall add all the params in detail in related sections.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.