Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Apache Flink offers APIs for building your Flink application using the DataStream and Table API. These are offered in Java and Scala. The Python APIs use the Java APIs under the hood. Over the course of time, the primary focus of the Flink community has shifted towards the Java API and the Scala support in Flink is not up-to-par with the Java API. A couple of examples:
- Flink still only supports Scala 2.12.7 and can only upgrade to a later version of Scala 2.12 by breaking compatibility - FLINK-20969Getting issue details... STATUS
- Flink doesn't support Scala 2.13 even though the initial ticket was created in July 2019 - FLINK-13414Getting issue details... STATUS .
- The DataStream API in Java has more features available compared to the DataStream API in Scala. For example - FLINK-29498Getting issue details... STATUS
Prior to Flink 1.15, there was tight dependency on Scala (Scala was always on the classpath). As of Flink 1.15, there is an option to use the Java API from any Scala version. The changes in Flink 1.15 are explained in-depth in a blogpost from the project website, which you can find at https://flink.apache.org/2022/02/22/scala-free.html
The current state of Scala in Flink can be summarised as:
- Most of the Scala-related contributions were/are focussed on unblocking users to use the Java API with any Scala version
- The Flink community is lacking maintainers with Scala knowledge who can help forward the Scala support directly in Flink
- New API interfaces that are added to Flink are created first for Java and are not or have limited support in Scala
- The open source community (outside of the ASF Flink project) are creating initiatives to help with Scala support in Flink, such as https://github.com/ariskk/flink4s (wraps Java APIs for Scala 3 users) and https://github.com/findify/flink-adt (which replaces the `TypeInformation` derivation mechanism in flink-scala)
Given the current state of Scala, this FLIP proposes to deprecate all Scala APIs in Flink 1.17 and remove all Scala APIs from Flink 2.0.
One of the primary parts of Flink that's using Scala is the table-planner. This is out of scope for deprecation since these are
@Internal interfaces. Removing Scala from the table-planner is still considered future work as first outlined in FLIP-32.
- Deprecate all customer-facing Scala APIs in the next Flink 1.x version, meaning:
- We annotate all
@Public, @PublicEvolving and @Experimental Scala APIs as @Deprecated.
- We remove all Scala API code examples from the codebase
- We remove all Scala API documentation
- We annotate all
- Remove these APIs in the next Flink major (2.x) version.
Compatibility, Deprecation, and Migration Plan
- Users who rely on the Scala APIs can and should consider porting to their equivalent from the Java APIs
- Otherwise, Flink 1.x will be the last version that the Flink Scala users can use.