StreamPipes Services
- Core (1, streampipes-backend)
- Pipeline Element Containers (n, streampipes-processors-*, streampipes-sinks-*)
- Adapter Worker (n, streampipes-connect-adapters-**)
- Consul (1)
- Nginx (1)
Service Discovery
- Should be independent from SD implementation
- Current standard implementation is Consul
- Create new module streampipes-service-discovery
- Move streampipes-config to streampipes-service-discovery-consul for consul-specific implementation
Requirements
- All services register at Consul with an identifying tag
- No fixed hostnames are stored in any graph (DataProcessorInvocation, DataSinkInvocation)
- For each pipeline element, the specific target pipeline element container is resolved at pipeline start
- The hostname of a service is resolved automatically and can be overwritten by an env variable