Status

Discussion threadhttps://lists.apache.org/thread/9pftp3x10602nzr9c0ln4h9zt8hg2rjk
Vote threadhttps://lists.apache.org/thread/ln44lns9nqxo31n7bx0tg1mww8t232j5
JIRA

FLINK-15571 - Getting issue details... STATUS

Release

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Redis Streams is an append-only data structure that helps in various streaming use cases like real-time messaging (RTM), message broker, etc. There was a Jira ticket created in January 2020, which shows quite a high number of people watching progress on that ticket. With the introduction of externalizing connectors from Flink to their own repository, it also makes it interesting to add a Redis Streams connector in its own external connector repository. 

Public Interfaces

The Redis Streams connector will consist of both a Source and Sink. The following interfaces will be used: 

Proposed Changes

The new connector will be based on the Unified Source Interface (FLIP-27) for the Source and the Async Sink (FLIP-171) for the Sink, support both Bounded (Batch) and Unbounded (Streaming) and both DataStream and Table API/SQL.

The Flink community will create a new connector repo, similar to ElasticSearch [1]. The repository name will be:

  • flink-connector-redis-streams


[1] https://github.com/apache/flink-connector-elasticsearch

Versioning Strategy

The flink-connector-redis-streams version will be independent of Flink. We will follow the same versioning strategy as Flink in terms of feature freeze windows, release candidates and branching/tagging. We will publish a Flink support matrix in the connector README and also update Flink documentation to reference supported connectors. The initial release of flink-connector-redis-streams will target 3.0.0 and support Flink 1.16.x and upwards. 

Compatibility, Deprecation, and Migration Plan

  • With respect to Flink, this is a new feature, no compatibility, deprecation, and migration plan is expected.

Test Plan

We will add the following tests:

  • Unit tests
  • Integration tests that perform end to end tests against a Redis Streams testcontainer

Rejected Alternatives

N/A