Status
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
POC: https://github.com/apache/flink/pull/18541
Motivation
Since the time Opensearch got forked from Elasticsearch a few things got changed. The projects evolve in different directions, the Elasticsearch clients up to 7.13.x were able to connect to Opensearch clusters, but since 7.14 - not anymore (Elastic continues to harden their clients to connect to Elasticsearch clusters only, https://github.com/elastic/elasticsearch/pull/73910). Arguably, the best way to proceed in this situation is to provide dedicated Opensearch connectors, on par with Elasticsearch ones.
Public Interfaces
The Opensearch connectors will support:
- Integrate with Flink Sink API (FLIP-143)
- Integrate with Flink new Source API (FLIP-27)
- Build upon Flink New DynamicTableSink and DynamicTableSinkFactory interfaces (FLIP-95)
- Sink streaming results with checkpointing enabled (at-least-once delivery semantics)
- Integrate with Table API
The Opensearch connectors are largely based on already existing Elasticsearch connectors (more specifically, Elasticsearch 7 flavor) and APIs-wise provide the identical feature set.
Proposed Changes
We propose to introduce dedicated Opensearch connectors (for 1.x and 2.x release lines) in Flink, based on existing Elasticsearch 7 connectors (https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/elasticsearch/).
Compatibility, Deprecation, and Migration Plan
The connectors are compatible with Opensearch 1.x and 2.x release lines. With respect to Flink, this is new feature, no compatibility, deprecation, and migration plan is expected.
Test Plan
The integration test cases along with end-2-end test suites have been implemented, on par with Elasticsearch connectors test coverage.
Rejected Alternatives
Keep using existing Elasticsearch to connect to Opensearch cluster would not work.