Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Currently, the connector contains both core and database-related code within the same module, resulting in increased complexity in development and maintenance.
The proposed reorganization aims to alleviate this complexity by segregating the core functionality and individual database components into separate modules.
Some initial groundwork has been laid with testing.
Some pain points
We generally use derby/h2 for testing, but about 10% of tests use arbitrary databases.
When old functionality is changed, sometimes database tests fail and we have a hard time finding which database is failing (tests that never end).
Sometimes we need to change CI/CD configurations to the most restrictive database.
Some of the code is not extensible to allow externalization of database code. For example, catalogs are maintained with the if else statement.
Public Interfaces
Introducing new public interfaces may not be necessary. Instead, we'll explore options such as moving interfaces to new packages or renaming them within the new modules while ensuring backward compatibility with the old module.
Proposed Changes
The primary change involves creating new modules while repurposing the existing one to serve as a shaded module encompassing the newly created ones.
This approach aims to maintain a consistent user experience across releases while allowing for the deprecation of outdated public interfaces.
Modules to be created (inside same repository)
- flink-connector-jdbc-core: Responsible for maintaining core code.
- flink-connector-jdbc-derby: Dedicated to Derby database-related functionalities, also utilized as the primary database for testing and validations.
- flink-connector-jdbc: The current module, which will be transformed to shade all other modules and maintain backward compatibility.
- flink-connector-jdbc-[database-name]: Modules for databases like CrateDB, DB2, MySQL, OceanBase, Oracle, PostgreSQL, SQL Server, and Trino.
Bonus work
- flink-connector-jdbc-python: This additional module will integrate Python code into the repository, offering Python users seamless interaction with the connector. Leveraging existing progress made for other connectors here.
- Revamped JDBC Documentation: With these changes, we'll update the JDBC documentation to reflect the updated architecture and functionalities. Each supported database will have a dedicated page covering configuration settings, supported operations, and usage examples.
Compatibility, Deprecation, and Migration Plan
- Move all code to the flink-connector-jdbc-core and transform the existing flink-connector-jdbc module into a shaded module of the flink-connector-jdbc-core.
- Transition database-specific code to their respective modules while maintaining the shading of the flink-connector-jdbc module.
- Refactor code within the flink-connector-jdbc-core to establish a new structure and simplify where possible.
- Deprecate any necessary public interfaces within the flink-connector-jdbc-core, reintroducing them to flink-connector-jdbc and extending the new public interface on flink-connector-jdbc-core.
This phased approach enables the gradual deprecation of old interfaces, giving users time to migrate to the new ones.
Deprecated code can be removed from flink-connector-jdbc in future releases, and we could consider to maintaining it as a fat-jar or removing it as a module.
Test Plan
Open to suggestions, but considering creating a test module implementing sources and sinks using flink-connector-jdbc to test retro-compatibility. However, minimizing module creation is preferred.
Rejected Alternatives
No alternatives have been rejected at this stage.