Goals
- Release stable version 1.0 of Apache StreamPipes
- Ensure that post-1.0 versions are backwards compatible
- Ensure that the feature set users expect from a 1.0 release is complete
- Ensure that no critical bugs are present
- Have the code base in a state which allows for future improvements
Non-Goals
- Identify features we see as future improvements, but not critical for the 1.0 release
Features
Are there any features which are missing and critical for an 1.0 release?
| ID | Description/Reason | 1.0 relevancy/comments |
|---|---|---|
| F.1 Improved start page | The StreamPipes start page is too much centered arount technical terms. Users should find resources based on physical assets such as machines. Links to directly create dashboards/pipelines should be present on the main entry page. | |
| F.2 Improved asset management | Users should be able to better link StreamPipes resources to physical assets (e.g., machines). Users should be able to filter resources such as pipelines and dashboards by their linked asset. | |
| F.3 Details for connected data sources | Currently, after creating an adapter, there is no way to have an overview of the connected data stream. There should be a details page for each adapter showing the schema, live preview of the data and transformation rules. | |
| F.4 OPC-UA | Feature-completeness of the OPC-UA connectors. Currently, we are missing authentication options. | |
| F.5 MQTT | Feature-completeness of the MQTT connectors. | |
| F.6 Logging | Support various logging options besides console logger, e.g., write rolling log files, use Prometheus endpoints for logging as well | |
| F.7 Data Explorer | Support the most important configuration options in the data explorer → clarify with users | |
| F.8 Database Retention Time | Automatically delete/aggregate persisted data streams | |
| F.9 Paging | UI should use paging in all views to avoid performance problems | |
| F.10 Backup | Feature to perform a complete system backup & restore |
Bugs
Are there any critical bugs we need to address before releasing 1.0?
| ID | Description/Reason | 1.0 relevancy/comments |
|---|---|---|
| B.1 Monitoring | Check adapter and pipeline health monitoring and metrics. It seems these are not always present/correct | |
| B.2 File upload | File upload only works for admin users | |
| B.3 User management | User management not completely supported in all views |
Usability
Are there any usability painpoints we need to address before releasing 1.0?
User Experience (Usability of the UI)
| ID | Description/Reason | 1.0 relevancy/comments |
|---|---|---|
| U.U.1 Clean up resource overview | Overviews (dashboards, pipelines, adapters, measurements) get bloated | |
| U.U.2 Measurements | Clarify terminology between data lake/measurements/adapters. Not clear to users → see asset management features | |
Developer Experience (Usability of SDK/Extensions)
| ID | Description/Reason | 1.0 relevancy/comments |
|---|---|---|
| U.D.1 Fluent API for adapter and pipeline element configurations | The current SDK to create configurations methods are hardly understandable for new developers. E.g., instead of freeTextStaticProperty, we should provide SDK methods such as textInput | |
| U.D.2 Improve migration process | Clarify how to better test migration of pipeline elements and adapters (dry run) |
User Acceptance
Are there any issues which negatively impact user acceptance? E.g., frustrations users experience when trying the product for the first time or in the long run.
| ID | Description/Reason | 1.0 relevancy/comments |
|---|---|---|
| UA.1 Unclear modeling | Create best practice catalog/concept how to model adapters/pipelines | |
| UA.2 More lightweight default system | Do we need a more lightweight default system? |
Code Quality and Maintenance
Where do we need to improve the code quality to make sure that future backwards compatibility is guaranteed?
| ID | Description/Reason | 1.0 relevancy/comments |
|---|---|---|
| CQ.1 CouchDB serialization | Currently, we use an outdated CouchDB library which depends on Gson, while we use Jackson in the code everywhere else. We should find a way to eliminate the Gson dependency and rely on a single serializer only. This also requires migrations for existing CouchDB documents. | |
| CQ.2 Model simplifications | The internal model to represent data streams, adapters, data processors and sinks is too complicated. The model still suffers from limitations we had when we were representing the model as RDF back in the days. We should embrace composition over inheritance (e.g., having an object which represents labels with name/description/id). This will pave the way to use language-agnostic serialization e.g. by using protobufs, reducing the maintenance effort for our multi-language clients. | |
| CQ.3 Database | Migration from InfluxDB to IoTDB or other TS database | |
| CQ.4 Architectural Review | Is our architecture ready for 1.0? Do we need multi-protocol support for message brokers? What is the future architecture we aim at? | |
| CQ.5 Influx v1 volume | Clarify required migration in case of removal of the old Influx v1 volumes (in k8s and docker) | |
| CQ.6 Harmonize naming | Harmonize naming of StreamPipes concepts (e.g., data processor/sepa) in the codebase, semantic types/domain properties | |
| CQ.7 Testing framework Data Explorer | We should be able to automatically test data explorer features | |
| CQ.8 Automatic testing of upgrades | Our test framework should include upgrade tests from one version to the next |
Next steps
Collect more feedback
Any other issues
Please add here all other thoughts on the 1.0 release
- Clarify role of dashboard
- Clarify role of asset dashboard
- Clarify post-1.0 release train