Status

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

There has been a requirement in the community for a Flink HTTP connector, but this work has not been done yet. For example Flip-233 [2] has been closed due to lack of capacity. 

I and others have been working on enhancing the Flink getIndata HTTP connector[1]. This connector is widely used. In the last year this connector has been substantially improved. We feel that this HTTP connector is now mature enough to be a Flink connector.  Currently lookup joins and sinks are supported. The initial discussion has already occurred on the dev list [3], where the community showed support, this included Krzysztof Zarzycki, CTO of Getindata (soon to be Xebia) who indicates his support of this donation.

An overview of the capabilities of the current HTTP connector:

 * works with SQL and datastream APIs
 * manages a thread pool to help with scaling under load
 * provides lookup join and sink capabilities
 * ability to customize
 * 90% junit coverage

 More recent enhancements include:

 * a retry mechanism ( that was asked for in Flips previous discussions) [4]
 * a new query creator, that allows json rest calls to map query param, body, path request information from table columns; without needing to write your own java, [5]
 * standard Flink caching using the same mechanism as JDBC [6]
 * supports OIDC [7]


 The above capabilities are useful for many scenarios.
 One omission is that it is not currently a scan source. Though this is being discussed [8].


 [1]
 https://github.com/getindata/flink-http-connector
 [2]
 https://cwiki.apache.org/confluence/display/FLINK/FLIP-233%3A+Introduce+HTTP+Connector
 [3]

https://lists.apache.org/list?dev@flink.apache.org:lte=1M:http%20connector

 [4]
 https://github.com/getindata/flink-http-connector/pull/148
 [5]
 https://github.com/getindata/flink-http-connector/pull/149
 [6]
 https://github.com/getindata/flink-http-connector/pull/94
 [7]
 https://github.com/getindata/flink-http-connector?tab=readme-ov-file#oidc-bearer-authentication
 [8]
> https://github.com/getindata/flink-http-connector/issues/41

Public Interfaces

Proposed Changes

• This is a new connector based on the getindata HTTP connector
• Some changes will be required to bring it into Flink including:
  • follow the usual process to bring in external repos. 
  • use the standard Flink code formatter, licence checker etc
  • change the package names to be the expected Flink ones
  • change the configuration key names to replace gid to flink an habe fallback keys as the gid keys  
  • Move the readme content into a documentation folder and produce a connector page in the usual style 
  • Bring the docs in the main Flink docs  

Compatibility, Deprecation, and Migration Plan

• There will be no impact on existing Flink users
• Exiting gid connector users will need to move to the Flink version

Test Plan

The GetIndata connector comes with unit tests that have 90% coverage.

Rejected Alternatives

• Flip-233
• extend the existing Flink http sample.