Update: The functionality is now officially part of Flume release.
Zookeeper based configuration is not officially available as part of Flume distribution. If you want to enable it, need to apply the patch from FLUME-1491
Please refer Understanding Config file format for details on how config file is organized
Applying the patch
Download the patch file with highest ending number, i.e. FLUME-1491-7.patch at the time of writing, and apply to the checkout version of choice.
NOTE: The patch may not apply and may need rebasing as per the version checked-out. The functionality is simple and the changes can be applied manually with ease.
The current patch expects the agent configuration to be part of node, whose name equals Agent name. The configuration is part of the data portion of the node.
Uploading configuration to Zookeeper
You can use the following utility to upload config to Zookeeper
The utility is still in beta, usage instruction provided as part of javadoc
Starting with Zookeeper support
Once the configuration is uploaded, start Flume agent with following parameters to pull configuration from Zookeeper. If the configuration is changed in Zookeeper, the Agent shall reload the configuration.
Following are the configuration options present
- basePath – Base Path where agent configuration needs to be stored. Defaults to /flume
- zkConnectionString – Connection string to Zookeeper Ensemble (host:port,host1:port1)
- agentName – Name of Agent for which configuration needs to be pulled
Following is a sample on how to start the same
$bin/flume-ng agent --name a1 -z localhost:2181
Feedback on the usage is appreciate, so that we can improve on the patch