[Flume Cookbook] Zookeeper based Configuration

Update: The functionality is now officially part of Flume release.

Zookeeper based configuration is not officially available as part of Flume distribution. If you want to enable it, need to apply the patch from FLUME-1491

Please refer Understanding Config file format for details on how config file is organized

Applying the patch

Download the patch file with highest ending number, i.e. FLUME-1491-7.patch at the time of writing, and apply to the checkout version of choice.

NOTE: The patch may not apply and may need rebasing as per the version checked-out. The functionality is simple and the changes can be applied manually with ease.

Zookeeper Configuration

Zookeeper Configuration

Zookeeper Configuration

The current patch expects the agent configuration to be part of node, whose name equals Agent name. The configuration is part of the data portion of the node.

Uploading configuration to Zookeeper

You can use the following utility to upload config to Zookeeper

Zookeeper Uploader

The utility is still in beta, usage instruction provided as part of javadoc

Starting with Zookeeper support

Once the configuration is uploaded, start Flume agent with following parameters to pull configuration from Zookeeper. If the configuration is changed in Zookeeper, the Agent shall reload the configuration.

Following are the configuration options present

Configuration properties

  • basePath – Base Path where agent configuration needs to be stored. Defaults to /flume
  • zkConnectionString – Connection string to Zookeeper Ensemble (host:port,host1:port1)
  • agentName – Name of Agent for which configuration needs to be pulled

Following is a sample on how to start the same

$bin/flume-ng agent --name a1 -z localhost:2181

Feedback on the usage is appreciate, so that we can improve on the patch

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.