Overview

FEP-1 describes a method of implementing Flume NG (1.x.x) backward compatibility with Flume OG (0.9.x).

The goal of FEP-1 is to allow existing Flume installations running the 0.9.x branch (Flume OG) to interoperate with the 1.x.x branch (Flume NG). It describes a series of sources and sinks that act as compatibility bridges, mimicking the behavior of 0.9.x within 1.x.

Authors

E. Sammer - esammer@cloudera.com

Status

This is a Draft

Description

With the release of Flume NG (1.x) users who currently run Flume OG (0.9.x) have started asking about backward compatibility. FEP-1 is about describing a method of integration between 0.9.x and 1.x.

First, this FEP makes a few assumptions.

• No code in 0.9.x can be changed. If it could, users could simply upgrade to 1.x rather than redeploying an updated 0.9.x.
• Integration can be constrained to the RPC mechanism. In other words, the way one connects 0.9.x agents to 1.x agents is via RPC.
• The contents of 0.9.x write ahead logs will either be drained to a 1.x agent or discarded.

What's not compatible between 0.9.x and 1.x?

• Events are represented differently and have a different interface
• Source and sink APIs are different
• The RPC mechanisms are different

The proposal is to create a 1.x source that mimics the 0.9.x RPC thrift server. This would allow existing 0.9.x clients to connect to 1.x agents if they were configured with the compatibility source. The event received by the compatibility source by the 1.x agent would have to be transformed into a 1.x format event although this isn't very different than receiving an event from syslog or any other 3rd party system.

With respect to 0.9.x's reliability modes, there are some caveats. When configured to use end-to-end reliability (0.9.x), events are normally retained until they reach the final destination. In the proposed scenario, 1.x wouldn't be able to provide the ACKs when the event was written. Instead, the event would immediately be ACKed. The expectation is that the 1.x agent would be configured with a durable channel implementation maintaining the delivery guarantee. This allows the 0.9.x agent to believe the event has been delivered and discard the event from its WAL.

Clients that speak 0.9.x's native thrift RPC protocol can, in theory, speak directly to the compatibility source. This should satisfy most, if not all, client cases.

Note that the inverse deployment loses some of the delivery guarantees and is purposefully not included in this proposal (because it's a bad idea). This would be the case where 1.x is sending events to a 0.9.x agent. The reason the delivery guarantee is lost is because 1.x requires that once it receives a successful ACK from the next hop in a delivery flow, it is permitted to release the event from its channel. This means that, unlike 0.9.x, an event may not make it to its final destination. All of that said, it would be trivial to implement a 1.x sink that speaks 0.9.x's RPC protocol.

References

TODO