...
Jira server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key STORM-1443 Jira server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key STORM-1446 Jira server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key STORM-2073
Not prioritized yet
Expand supporting external components
...
How this milestone will help users to play with Storm SQL?
Done
- Kafka as Input / Output
- Redis as Output
Remaining works
- Users can run 'explain' to see query plan for DML statement before submitting Trident topology.
- Users can use GROUP BY and JOIN statement in DML statement
- Note that these feature follows the Trident semantic of aggregation and join
- Aggregation is done within batch
- SQE does stateful aggregation, so Storm SQL might follow later
- Join is done with each of batches
- But respecting Streaming SQL semantic seems correct, which means that we could get rid of these features again and wait for Streaming SQL
- This is what Flink is doing, but Flink have Table API so Flink can support window aggregation and join without Calcite support
- Users can run Storm SQL Runner without copying dependencies to extlib.
- This is resolved from
Jira server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key STORM-
- This is resolved from
...
but this issue was originally came from2016
Jira server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key STORM-
.1435
- Users can use user defined aggregate function in Trident mode.
- Users can specify parallelism of input data source which will be used unless repartitioning is made
How this milestone will improve Storm SQL internally?
will reduce redundant multiple Trident steps into one.Jira server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key STORM-21022073 Jira server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key STORM-2103 - And more
- Any external modules which support Trident state can be candidates.
Consideration
...
will do some query optimizations, and open the way to address next works, like automatic parallelism, pushdown, and so on.1446
Next works
Below works can be done without waiting some other works. We might want to pick several works from each category and create next milestone.
When building milestone, it would be better to clarify the goal - "How this milestone helps the users to play with Storm SQL?".
...
Automatic parallelism for input data source with metadata
JIRA link
Jira | ||||||||
---|---|---|---|---|---|---|---|---|
|
...
Schema support on input format and output format
JIRA link
Jira | ||||||||
---|---|---|---|---|---|---|---|---|
|
Things to do (not filed to issues yet)
- CSV
- Avro
- TSV
- Schema Registry
- And more
Supports more functions (scalar and aggregation)
JIRA link
None yet
Things to do (not filed to issues yet)
- DATE / TIMESTAMP related functions
- Calcite lacks here, so may need to refer one of RDBMS - MySQL / Oracle / PostgreSQL
- Functions which SQE supports now
- And more
Expand supporting external components
JIRA link
Jira | ||||||||
---|---|---|---|---|---|---|---|---|
|
Done
- Kafka as Input / Output
- Redis as Output
Remaining works
Jira server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key STORM-2082 Jira server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key STORM-2102 Jira server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key STORM-2103 - And more
- Any external modules which support Trident state can be candidates.
Consideration
- They should be rewritten if we replaces the backend of Storm SQL to higher-level core API
- Need to determine 'Widely used' data sources and only provides them for now
Projection / Filter pushdown to data source
...