Link: Unresolved issues in storm-sql
Current milestone
Storm SQL Phase II
JIRA link
STORM-1433 - Getting issue details... STATUS
Remaining works
- STORM-1443 - Getting issue details... STATUS
- STORM-1446 - Getting issue details... STATUS
- STORM-2073 - Getting issue details... STATUS
Not prioritized yet
Expand supporting external components
JIRA link
STORM-2075 - Getting issue details... STATUS
Done
- Kafka as Input / Output
- Redis as Output
Remaining works
- STORM-2082 - Getting issue details... STATUS
- STORM-2102 - Getting issue details... STATUS
- STORM-2103 - Getting issue details... STATUS
- And more
- Any external modules which support Trident state can be candidates.
Consideration
- They should be rewritten if we replaces the backend of Storm SQL to higher-level core API
- Need to determine 'Widely used' data sources and only provides them for now
Automatic parallelism for input data source with metadata
JIRA link
STORM-2147 - Getting issue details... STATUS
Things to do (not filed to issue yet)
- Apply this to Kafka input data source (maybe handled from STORM-2147)
- Do we want to add more input data sources? Then they need to be considered as well.
Schema support on input format and output format
JIRA link
STORM-2149 - Getting issue details... STATUS
Things to do (not filed to issues yet)
- CSV
- Avro
- TSV
- Schema Registry
- And more
Supports more functions (scalar and aggregation)
JIRA link
None yet
Things to do (not filed to issues yet)
- DATE / TIMESTAMP related functions
- Calcite lacks here, so may need to refer one of RDBMS - MySQL / Oracle / PostgreSQL
- Functions which SQE supports now
- And more
Projection / Filter pushdown to data source
JIRA link
None yet
Note
- Not sure it helps stream data source
- It may help but we should make sure that Spout supports projection (maybe only column referring) / filter
- It definitely helps with input data sources which accepts query (for example, JDBC)
Depends on other works (Future work)
Change backend of SQL to higher-level core API (get rid of Trident)
JIRA link
None yet
Precondition
- Apache Storm adopts higher-level core API
- JIRA issue: STORM-1843 - Getting issue details... STATUS
- Higher-level core API supports exactly-once
Note
- Storm SQL may go back to basic feature: no aggregation, no join, no sort
- because higher-level core API cannot support aggregation and join without window
- current Storm SQL's aggregation and join semantic are very different from Streaming SQL
Support Streaming SQL
JIRA link
None yet
Precondition
- Calcite supports Streaming SQL : https://calcite.apache.org/docs/stream.html
- For now this page states that Streaming SQL features are NOT IMPLEMENTED yet
- This still seems to be a design phase.
- Julian Hyde had various talks in various place, but in parallel he posted to Calcite dev mailing list for design doc about join recent days.
Note
- group by window
- join between stream and table (without support temporal table)
- join between stream and stream
- join between stream and table (with support temporal table or similar)