Research and development roadmap for Flink Streaming
 

Fault tolerance guarantees for streaming
 

There has been a lot of development throughout the years towards fault tolerant stateful streaming. It is true that fault tolerance mechanisms can be very expensive in stream processing applications both in storage and latency requirements. Part of our research on Flink is to minimize these costs by taking advantage of semantical properties such as window pre-aggregates as well as investigating different state management approaches. The ultimate goal of fault tolerance in Flink will be to offer exactly-once processing semantics while sacrificing as few resources as possible.

...

• Identification and implementation of a “lambda architecture” kernel of binary operators between DataSets and DataStreams.

• Create a unique context and environment for such executions

• Unify fault tolerance mechanisms efficiently

API integrations 

Currently we only provide a Java api for Flink streaming. In the near future we would like to make the same functionality accessible through a Scala (and later Python) api like for the batch processing functionality. Implementing the Scala api should be fairly straightforward because most of the features can be reused from the batch api.

...