Status
...
Page properties | |
---|---|
|
...
|
JIRA: https://issues.apache.org/jira/browse/FLINK-14686
...
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation:
In flink 1.9, we have introduced some awesome features such as complete catalog support[1] and sql ddl support[2]. These features have been a critical integration for Flink to be able to manage data and metadata like a classic RDBMS and make developers more easy to construct their real-time/off-line warehouse or sth similar base on flink.
...
- Don’t support show/describe/use catalog(s)
- Don’t support create/drop/alter/use/show/desc database(s)
- Don’t support show/desc table(s) and create table as select
- Don’t support create/drop/show/desc view(s)
Goals:
We would like to achieve the following goals in this FLIP.
- Add Catalog DDL enhancement support
- Add Database DDL enhancement support
- Add Table DDL enhancement support
Note:
The following issues are out of the scope of this FLIP. They will be taken care of by other teams/committers.
- DDL related to View(s)
- DDL related to Partition(s)
- DDL related to Function(s) (another seperate FLIP-79 to solve this issure)
- Unify the way of SQL CLI and TableEnvironment parse sql through common Flink SQL Parser module
Proposed DDL Changes:
There are a few DDLs following that is already supported in SQL client. We want them should be migrated using the Flink Sql Parser approach properly.
Catalog DDL:
We propose to add the following DDLs related to catalog operations.
showCatalogsStatement
SHOW CATALOGS |
Return all catalogs in the current catalog.
note: already support in sql client but need support in TableEnvironment.
describeCatalogStatement
DESCRIBE CATALOG catalogName |
Return the default database name and expalnCatalog(newly added method in the catalog) content of an existing catalogName
Newly added method in Catalog.java:
/** |
use eg:
Flink SQL> describe catalog hiveCatalog; |
useCatalogStatement
USE CATALOG catalogName |
Set the current catalog.
note: already support in sql client but need support in tEnv.sqlUpdate();
Database DDL:
We propose to add the following DDLs related to database operations.
createDatabaseStatement:
CREATE DATABASE [ IF NOT EXISTS ] [ catalogName.] dataBaseName |
IF NOT EXISTS:
If a database with the same name already exists, nothing will happen.
dropDatabaseStatement:
DROP DATABASE [ IF EXISTS ] [ catalogName.] dataBaseName |
IF EXISTS
If the database to drop does not exist, nothing happens.
RESTRICT
Dropping a non-empty database triggers an exception. Enabled by default.
CASCADE
Dropping a non-empty database also drops all associated tables and functions.
...
/** * @param cascade Flag to specify behavior when the database is not empty |
alterDatabaseStatement:
ALTER DATABASE [ catalogName.] dataBaseName SET |
...
Set one or more properties in the specified database. If a particular property is already set in the database, override the old value with the new one.
useDatabaseStatement:
USE [ catalogName.] dataBaseName |
...
note: already support in sql client but need support in TableEnvironment;
showDatabasesStatement:
SHOW DATABASES |
Return all databases in the current catalog.
note: already support in sql client but need support in TableEnvironment;
descDatabaseStatement:
DESCRIBE DATABASE [ EXTENDED] [ catalogName.] dataBasesName |
Return the metadata of an existing database (name, comment).
EXTENDED:
Return the metadata of an existing database (name, comment, and properties).
...
FLINK SQL> describe database hive.default; |
Table DDL:
We propose to add the following DDLs related to table operations.
showTablesStatement:
SHOW TABLES |
Return all tables in the current database.
note: already support in sql client but need support in TableEnvironment.
descTableStatement:
DESCRIBE [ EXTENDED] [[catalogName.] dataBasesName].tableName |
...
Return the metadata of an existing table (column names, data types, and comments).
EXTENDED
Display detailed information about the table, including table type, properties and so on.
note: already support in sql client but need support in TableEnvironment.
alterTableStatement:
ALTER TABLE [[catalogName.] dataBasesName].tableName |
...
Set the properties of an existing table
Proposed TableEnvironment SQL API Changes:
Notice:
Not the scope of this flip anymore, we have another dedicated flip FLIP-84 to discuss the API changes. So I just removed this partition, anyone interested can refer to the archive of table api change .
Implementation Changes:
Support these ddl in the TableEnvironment, and there a few DDL implemented by SQL Client that should be migrated to using Calcite parser.
Implementation Plan:
- Implement all parser logic in the flink-sql-parser module
- After FLIP-84 completed, add DDLs support one by one through the `executeString(String sql)` or something similar interface.
- After step1,2 finished, SQL CLI can very easily support correspondingly ddl by just calling TableEnvironment’s execute sql method.
Compatibility, Deprecation, and Migration Plan
- What impact (if any) will there be on existing users?
...
- When will we remove the existing behavior?
Not Applicable.
Test Plan
- New syntax add to calcite parser can be tested using FlinkSqlParserImplTest by adding test cases
- DDLs supported in TableEnvironment can be verified by unit test.
Rejected Alternatives
No rejected alternatives.
Reference:
- https://issues.apache.org/jira/browse/FLINK-11275
- https://issues.apache.org/jira/browse/FLINK-10232
- hive ddl https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL
- spark ddl https://docs.databricks.com/spark/latest/spark-sql/index.html
- flink sql ddl design https://docs.google.com/document/d/1TTP-GCC8wSsibJaSUyFZ_5NBAHYEB1FVmPpP7RgDGBA/edit
...