[Flume Cookbook] Channels

Lets look at the Channels in this post. Channels are the link between Source and Sink. We shall briefly look at Channel Lifecycle and how the message is passed from Source to Channel.

Channel Lifecycle

Life Cycle

Life Cycle

Channel's lifecycle is same as Source life cycle, as depicted in figure above.

Source to Channel message flow

Let's look at how message is passed from a Source to Channel

Message Flow

Message Flow

Flume provide reliable message delivery and this brings transactions into picture. For simplicity of discussion would take Transaction later. As a simple rule, once the next stage Acks receipt of message, then only current stage would remove the message from it's store. This ensures that messages are not lost.

Now let's look at main flow.

  • A message is received at Source
  • A Source can be associated with one or more mandatory or optional channels
  • Source has a ChannelProcessor associated with it, Event is passed to it
  • Channel processor process the Interceptor's configured
  • Channel processor uses the ChannelSelector to determine the Channels the Event needs to be sent to
  • The Event is pushed to all the Channels within a Transaction

That's a brief about Channels. We shall look at implementing Custom Channel next.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.