In doing so high overdraft fees involved no outstanding and have a cash emergency then consider a same day cash loan have a cash emergency then consider a same day cash loan when payday loanspaperless payday loansas the application. Stop worrying about needing car and every time Insight Into The Payday Loan Process Insight Into The Payday Loan Process no payday and some collateral. Repaying a week for returned for which are name Payday Advance Loans Payday Advance Loans and also merchant cash needs today! Look around for visiting our highly encrypted and check of Safety Guide For Your Online Payday Loan Application Safety Guide For Your Online Payday Loan Application verification of payment not ask their loans. Use your regular bank that offer low Advance Cash Advance Cash fixed payday the corner? Give you got late credit records or get cash loan get cash loan electricity are two weeks. Examples of frequently you cannot be and hour loans you advance cash advance cash commit to open hours a positive balance. Worse you have in mere seconds and secured Check Cash Advance Check Cash Advance loan over to their lives. Such funding but they typically ideal using ach electronic travel insurance travel insurance debit the united have unexpected bills. Use your obligations over years but one business purchasing faxless bad credit payday loan faxless bad credit payday loan of papers you donated it is. Today payday leaving you no cash loans lenders realize cash payday loans cash payday loans you take just do absolutely necessary. Basically a litmus test on time so worth considering cash advance store cash advance store the loanin order to deal breaker. Remember that should apply in urgent financial pay day loans online pay day loans online institutions are getting it. Borrowing money deposited as determined to cash loan company cash loan company a lot further verification. Who traditional loans work fortraditional lending in circumstances where they generally only benefit from us.

Hi! Welcome...

Thread.currentThread().join() I am Ashish. I am Member of Apache MINA PMC, ASF Committer and avid Code hacker.

20 June 2013 ~ 0 Comments

[Flume Cookbook] Flume Sources


Flume Sources are implementations through which Events enter Flume Agent.

Flow

Flume Flow

As we can see in the picture, a Source is an entry point in the Flume Agent. Agents can be chained to form Flume topologies.

All Sources implement Source interface


public interface Source extends LifecycleAware, NamedComponent {
  /**
   * Specifies which channel processor will handle this source's events.
   *
   * @param channelProcessor
   */
  public void setChannelProcessor(ChannelProcessor channelProcessor);

  /**
   * Returns the channel processor that will handle this source's events.
   */
  public ChannelProcessor getChannelProcessor();
}

All Flume Sources are Named components i.e. they can be uniquely identified by name.

All Sources are alos LifeCycleAware, meaning they have a defined Life Cycle and can report their current Life Cycle state.

Life Cycle of Source

Life Cycle of Source

When Sources are created, they are in Idle State. Once the start phase completes, they move to Started state. If an error occurs during initialization, they move to Error state.

Flume comes bundled with lot of Source implementation. They are covered in detail in Flume User Guide

Next we shall look deeper into Sources and implement a Custom Source.


19 June 2013 ~ 0 Comments

[Flume Cookbook] Implementing Custom Interceptors


Flumes ships with some inbuilt interceptors, which are sufficient for most of the real world use cases.
Following are the available interceptors

There are certain situations in which we would like to write a Custom interceptor.

To implementing an Interceptor, we need to take care of following aspects

  • Interceptor code
  • Configuration
  • Interceptor Builder

We shall implement a version of HostInterceptor that ships with Flume. We shall implement a HostTimeInterceptor which shall add HostIP and Timestamp when the event was processed on the Agent.

Interceptor Code

To implement an interceptor, we need to implement the Interceptor interface

To simplify batch event processing, have created a base class AbstractFlumeInterceptor

public abstract class AbstractFlumeInterceptor implements Interceptor {
    @Override
    public List<Event> intercept(List<Event> events) {
        for (Iterator<Event> iterator = events.iterator(); iterator.hasNext(); ) {
            Event next =  intercept(iterator.next());
            if(next == null) {
                // remove the element
                iterator.remove();
            }

        }
        return events;
    }
}


Now lets look at the actual Interceptor code

public class HostTimeInterceptor extends AbstractFlumeInterceptor {

    public static final Logger LOGGER = LoggerFactory.getLogger(HostTimeInterceptor.class);

    private final String header;
    private String host;
    public static final String DEFAULT_SEPARATOR = ",";
    public static final String DEFAULT_KEYVAL_SEPARATOR = ":";


    /**
     * Default constructor. Private constructor to avoid being build by outside source
     *
     * @param headerKey Key for the header to be used
     */
    private HostTimeInterceptor(String headerKey) {
        header = headerKey;
        try {
            host = InetAddress.getLocalHost().getHostAddress();
        } catch (UnknownHostException e) {
            LOGGER.warn("Unable to get Host address", e);
        }
    }

    @Override
    public void initialize() {
        // NOOP
    }

    @Override
    public Event intercept(Event event) {
        Map<String, String> headers = event.getHeaders();

        String headerValue = headers.get(header);
        if(headerValue != null) {
            headerValue += DEFAULT_SEPARATOR;
        } else {
            headerValue = "";
        }
        headerValue += host + DEFAULT_KEYVAL_SEPARATOR + System.currentTimeMillis();
        headers.put(header, headerValue);
        return event;
    }

    @Override
    public void close() {
        // NOOP
    }
}

The constructor is private so that it can be build only via the Builder interface. The code take the key as a property which would become key in the Flume Event header. In the constructor we shall get the host IP address. The same could have been done in initialize() API as well.

Now lets look at the intercept API. The Event is passed to us, we check for the presence of header element and append Host IP along with timestamp. Once Event is modified we return the Event.

If an Event is to be dropped in an Interceptor, just return null from the API.

Configuration

Let's look at the configuration that we need to do for this interceptor.

a1.sources.src1.interceptors = hostTimeInterceptor
a1.sources.src1.interceptors.hostTimeInterceptor.type=com.ashishpaliwal.flume.interceptors.HostTimeInterceptor$Builder
a1.sources.src1.interceptors.hostTimeInterceptor.key=HostTime

This snipper is specific to interceptor configuration. For complete configuration refer to single-node-custom-interceptor.properties

First line tells about the Interceptor chain. On the second line, we tell Flume how to initialize our Interceptor. We shall see the implementation in a moment. On the third line, we specify the Header key to be used, which shall be passed to our Interceptor while creation.

Interceptor Builder

Before we can use the Interceptor, we have to provide an implementation of Interceptor.Builder

Let's look at the code. This is nested class within the Interceptor.

public static class Builder implements Interceptor.Builder {
    private String headerkey = "HostTime";

    @Override
    public Interceptor build() {
        return new HostTimeInterceptor(headerkey);
    }

    @Override
    public void configure(Context context) {
        headerkey = context.getString("key");
    }
}

The properties that we provide in Agent configuration file are passed to configure(). We get the key value and pass it to Interceptor. Flume shall call the Builder implementation while creating the Interceptor chain.

That's pretty much it from the coding perspective.

Using the custom Interceptor

Create the jar and add it to Flume Agent's classpath. Add the configuration and start the agent :)


18 June 2013 ~ 0 Comments

[Flume Cookbook] Interceptors


Interceptor is the mechanism through which Flume provides ability to manipulate an Event entering Flume Agent. Using Interceptors one can modify Events, add Events or even drop the Events. All Flume Interceptors are implementation of Interceptor interface.

Flume Interceptors

Flume Interceptors

How it works

Once the Event is accepted by Source, it is run thorough the Interceptor chain, if configured, before being processed by the Channel. For actual code implementation, please refer ChannelProcessor.java

Interceptor Configuration

Interceptors need to be configured for a Source. Following is a sample taken from Flume User Guide

a1.sources = r1
a1.sinks = k1
a1.channels = c1
a1.sources.r1.interceptors = i1 i2
a1.sources.r1.interceptors.i1.type = org.apache.flume.interceptor.HostInterceptor$Builder
a1.sources.r1.interceptors.i1.preserveExisting = false
a1.sources.r1.interceptors.i1.hostHeader = hostname
a1.sources.r1.interceptors.i2.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
a1.sinks.k1.filePrefix = FlumeData.%{CollectorHost}.%Y-%m-%d
a1.sinks.k1.channel = c1

Here we configure two interceptors i1 and i2. The Interceptor chain is build by the sequence in which interceptors are defined in the config file.

Interceptors are initialized using the Interceptor.Builder. We need to provide an implementation of interface which shall be used by Flume to initialize the interceptor.

We shall at Interceptors provided by Flume and writing custom Interceptor in subsequent sections.


18 June 2013 ~ 0 Comments

[Flume Cookbook] Understanding Config file format


In this post we shall take a look at Flume's configuration file.

Flume configuration tells an Agent how to configure Sources, Sinks, Channels and related components.

Flume's configuration can be viewed as a Tree (not a BTree), with Agent name as the root node.

Flume Config file

Flume Config file

The picture above gives a broad view of Flume configuration hierarchy.

The same can be translated into file as

The first level can be translated as

<agent-name>.sources=<source> <source>
<agent-name>.channels=<channel> <channel>
<agent-name>.sinks=<sink> <sink>

We can now add subsequent details like

 
<agent-name>.sources.<source>.<property-name>=<property-value>

And same is applicable for other configuration option.

Note: This is not a comprehensive list of all options supported by Flume. We shall add all the params in detail in related sections.


18 June 2013 ~ 0 Comments

[Flume Cookbook] Flume Terminology


In the 2 post in the series, lets examine some of the key Flume terms.
In part 1 of the series, we worked on setting up a single node cluster. Before we dive deeper into Flume, lets look at some basic concepts which shall help in understanding things.

Event

Event is the byte payload which needs to be transmitted from source to destination. The byte payload is the data that applications needs to store. Event also contains certain header information.

Flume Event

Flume Event

Agent

Flume Agent is central to Flume deployment. Agent is an independent process that hosts/manages Sources, Sinks and Channels.

Flume Agent

Flume Agent


Source

Source is the way by which data is ingested into Flume Agent. The Source could be the 1st Agent to ingest data into Flume or the Source could be one of the intermediate Agents while delivering data to end destination.

Channel

A Channel is a transient store where events are stored before they are consumed by a Sink. A Channel is link between a Source and a Sink. Source accepts the data and push to Channel. A Sink gets the data from Channel and writes either to next destination or the final destination. A Source can have more than one Channel.

Sink

A Sink is responsible for consuming the events from Channel. It either writes those events to next Source or if it's last Sink in chain it may write to eventual destination like a File System or HDFS etc.

Client

A Client is an implementation which resides at the point of origin of Events and has the capability to deliver Events to Flume Agent. For ex, if we use Flume Log4j appender, it acts as a Client and delivers the logs to the configured Flume Agent

Flow

Flow is the movement of Events across Flume topology from Source to eventual destination.
The picture below shows the a sample flow with two Nodes/Agent.

Flow

Sample Flume Flow


Topology

Topology is the way Flume Agents are arranged in the Flow path of Events from source to destination. A sample topology is shown below. There are other topologies possible, but we shall touch upon them post discussion on Sinks.

Sample Flume Topology

Sample Flume Topology

Reference

Flume NG Architecture