Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

MongoDB is a non-relational document database that provides support for JSON-like storage that helps store complex structures easily. There was a Jira ticket created in May 2017, which shows quite a high number of people watching progress on that ticket. With the introduction of externalizing connectors from Flink to their own repository, it also makes it interesting to add a MongoDB connector in its own external connector repository. 

Public Interfaces

The MongoDB connector will consist of both a Source and Sink. The following interfaces will be used: 

• Source:
  • FLIP-27: Refactor Source Interface
  • FLIP-221: Abstraction for lookup source cache and metric
• Sink: 
  
  • FLIP-177: Extend Sink API to provide at-least once guarantees
• Table: 
  • FLIP-95: New TableSource and TableSink interfaces

Proposed Changes

The new connector will be based on the Unified Source Interface (FLIP-27) for the Source and the Unified Sink (FLIP-143) for the Sink, support both Bounded (Batch) and Unbounded (Streaming) and both DataStream and Table API/SQL.

The Flink community will create a new connector repo, similar to ElasticSearch [1]. The repository name will be:

• flink-connector-mongodb

[1] https://github.com/apache/flink-connector-elasticsearch

Versioning Strategy

The flink-connector-mongodb version will be independent of Flink. We will follow the same versioning strategy as Flink in terms of feature freeze windows, release candidates and branching/tagging. We will publish a Flink support matrix in the connector README and also update Flink documentation to reference supported connectors. The initial release of flink-connector-mongodb will target 1.0.0 and support Flink 1.16.x and upwards. 

Compatibility, Deprecation, and Migration Plan

The connectors are compatible with MongoDB. With respect to Flink, this is a new feature, no compatibility, deprecation, and migration plan is expected.

Test Plan

We will add the following tests:

• Unit test
• Integration tests that perform end to end tests against a MongoDB testcontainer

Rejected Alternatives

There are currently no rejected alternatives.