[Flume Cookbook] Interceptors

Interceptor is the mechanism through which Flume provides ability to manipulate an Event entering Flume Agent. Using Interceptors one can modify Events, add Events or even drop the Events. All Flume Interceptors are implementation of Interceptor interface.

Flume Interceptors

Flume Interceptors

How it works

Once the Event is accepted by Source, it is run thorough the Interceptor chain, if configured, before being processed by the Channel. For actual code implementation, please refer ChannelProcessor.java

Interceptor Configuration

Interceptors need to be configured for a Source. Following is a sample taken from Flume User Guide

a1.sources = r1
a1.sinks = k1
a1.channels = c1
a1.sources.r1.interceptors = i1 i2
a1.sources.r1.interceptors.i1.type = org.apache.flume.interceptor.HostInterceptor$Builder
a1.sources.r1.interceptors.i1.preserveExisting = false
a1.sources.r1.interceptors.i1.hostHeader = hostname
a1.sources.r1.interceptors.i2.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
a1.sinks.k1.filePrefix = FlumeData.%{CollectorHost}.%Y-%m-%d
a1.sinks.k1.channel = c1

Here we configure two interceptors i1 and i2. The Interceptor chain is build by the sequence in which interceptors are defined in the config file.

Interceptors are initialized using the Interceptor.Builder. We need to provide an implementation of interface which shall be used by Flume to initialize the interceptor.

We shall at Interceptors provided by Flume and writing custom Interceptor in subsequent sections.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.