Status
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Redis Streams is an append-only data structure that helps in various streaming use cases like real-time messaging (RTM), message broker, etc. There was a Jira ticket created in January 2020, which shows quite a high number of people watching progress on that ticket. With the introduction of externalizing connectors from Flink to their own repository, it also makes it interesting to add a Redis Streams connector in its own external connector repository.
Public Interfaces
The Redis Streams connector will consist of both a Source and Sink. The following interfaces will be used:
- Source:
- Sink:
- FLIP-171: Async Sink to provide at-least once guarantees
Proposed Changes
The new connector will be based on the Unified Source Interface (FLIP-27) for the Source and the Async Sink (FLIP-171) for the Sink, support both Bounded (Batch) and Unbounded (Streaming) and both DataStream and Table API/SQL.
The Flink community will create a new connector repo, similar to ElasticSearch [1]. The repository name will be:
flink-connector-redis-streams
[1] https://github.com/apache/flink-connector-elasticsearch
Versioning Strategy
The flink-connector-redis-streams version will be independent of Flink. We will follow the same versioning strategy as Flink in terms of feature freeze windows, release candidates and branching/tagging. We will publish a Flink support matrix in the connector README and also update Flink documentation to reference supported connectors. The initial release of flink-connector-redis-streams will target 3.0.0 and support Flink 1.16.x and upwards.
Compatibility, Deprecation, and Migration Plan
- With respect to Flink, this is a new feature, no compatibility, deprecation, and migration plan is expected.
Test Plan
We will add the following tests:
- Unit tests
- Integration tests that perform end to end tests against a Redis Streams testcontainer
Rejected Alternatives
N/A