Status

Discussion threadhttps://lists.apache.org/thread/j5gywt7hoo8gr3myxh1ctdxdjlq92hcf
Vote threadhttps://lists.apache.org/thread/57sj2zzrrcohk77mczrzcth7wsvrjrqk
JIRA

FLINK-25756 - Getting issue details... STATUS

Release1.17

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

POC: https://github.com/apache/flink/pull/18541

Motivation

Since the time Opensearch got forked from Elasticsearch a few things got changed. The projects evolve in different directions, the Elasticsearch clients up to 7.13.x were able to connect to Opensearch clusters, but since 7.14 - not anymore (Elastic continues to harden their clients to connect to Elasticsearch clusters only, https://github.com/elastic/elasticsearch/pull/73910). Arguably, the best way to proceed in this situation is to provide dedicated Opensearch connectors, on par with Elasticsearch ones.

Public Interfaces

The  Opensearch connectors will support:

  • Integrate with Flink Sink API (FLIP-143)
  • Integrate with Flink new Source API (FLIP-27)
  • Build upon Flink New DynamicTableSink and DynamicTableSinkFactory interfaces (FLIP-95)
  • Sink streaming results with checkpointing enabled (at-least-once delivery semantics)
  • Integrate with Table API

The Opensearch connectors are largely based on already existing Elasticsearch connectors (more specifically, Elasticsearch 7 flavor) and APIs-wise provide the identical feature set.

Proposed Changes

We propose to introduce dedicated Opensearch connectors (for 1.x and 2.x release lines) in Flink, based on existing Elasticsearch 7 connectors (https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/elasticsearch/).

Compatibility, Deprecation, and Migration Plan

The connectors are compatible with Opensearch 1.x and 2.x release lines. With respect to Flink, this is new feature, no compatibility, deprecation, and migration plan is expected.

Test Plan

The integration test cases along with end-2-end test suites have been implemented, on par with Elasticsearch connectors test coverage.

Rejected Alternatives

Keep using existing Elasticsearch to connect to Opensearch cluster would not work.