Data Lake Sink

Hi,

here are some comments from my side.

Data Lake Controller Class

• The configurations batch size and flush duration should at least be configurable via the container configs

DataLakeResourceV3 

• the endpoints have no authentication because they are called from the DataLakeSink and of course the user/password is not known there. In order to introduce authentication, a concept with technical users would probably have to be introduced.

Data Lake Controller Class

• Here we use a lot of path parameters, which are query parameters. E.g.

  •  /data/{index}/{startdate}/{enddate}/grouping/{groupingTag} → /data/{index}?stardate=...&enddate=..&groupby=...

  • /data/delete/all → /data?index=... (No index ^= delete all)

General - Saving and Reading Images and Coco Files

• It would also be good to think about the file sharing part. The way we do it now, it only works because we use volume sharing. Maybe we should use blob storage for this kind of data? 

Hi Daniel,

thanks for coming up with this!

Regarding authentication, I'm currently working on improving the communication between pipeline elements and core services. The idea is to provide an instance of the StreamPipes client within the onInvocation method so that pipeline elements requesting anything from the core APIs can directly interact with these using the aforementioned service token. In this case, the data lake sink would just call the platform service API in order to create a new measure or do anything else in the core (e.g., sending emails or creating live visualizations). For that, we need to extend our Auth concept with a service user. I'm also trying to get rid of the Consul config provider and integrate this directly into the description of a pipeline element.

After that, we can simply move the data lake API to the platform services module and make the endpoint authenticated.

In the meanwhile, we can work on improving the REST interface itself, e.g., the comment from Johannes concerning the usage of query parameters instead of path parameters seems very important to me. So discussing how to improve the DataLakeRestResourceV3 (let's say the way towards v4) should be the next step IMO.

From a user perspective, it should be easily understandable how to get data, filter and group data, upload/download/list images and so on. Do you have any first suggestion on how to redesign this endpoint? Otherwise, I can provide some first input.

Dominik

Hi,
I have created a first draft for the revised endpoint definitions (see table below). Thereby, I have tried to map as suggested as much as possible via query parameters. However, the user should have for example the possibility to specify multiple grouping tags for the "groupBy" parameter.

The file sharing part for the image / coco files I kept very prototypical and assumed a simple object store (key-value store). Here we can also discuss the solution mentioned by Johannes using a blob storage and adapt the draft accordingly.

I am interested in your comments and suggestions for adjustments.

Suggestion for the revised endpoint definitions

Looks good!

A few comments/questions from my side:

• Can we integrate the paged endpoint into the standard endpoint? E.g., by having offset and limit parameters?
• Multiple grouping tags in the groupingTag query parameter can be comma-separated
• Can we merge the download feature into the standard endpoint? E.g., by providing a query parameter action=download and another parameter format=csv 
• Do we already have or foresee an endpoint to upload an image file? That would be something I'd currently love to have 

I'm not sure whether the multiple query endpoints for paged, grouped and non-grouped queries return the same schema - that would be something we would need to harmonize in case there is a single endpoint.

Regarding the comment from Johannes, what is the difference between measurement and index?

And regarding naming of resources as raised by Johannes, a common best practice is to have an endpoint such as /measurements (using plural) where a GET would return a list of all measurements and /measurements/{measurementId} (maybe this would be index) that returns the content for the specific source. In that case, to delete a specific measurement, a DELETE call to /measurements/{measurementId} could be available. Hope this helps!

And another, more high-level comment regarding the roadmap: In the near future, I'd really like to simplify some StreamPipes concepts. One idea here is that everything in StreamPipes is either a Data Set or a Data Stream, which can be configured to be persisted (without manually creating a pipeline). For pipelines that transform data streams that should be stored in the data lake, there would be two "Virtual Data Stream/Set" sinks instead of the data lake/dashboard sink, which itself represent a stream/set in the UI that can be configured to be persisted. In this case, any resource in the data lake could be discovered by the (e.g., appId or some other identifier) of the underlying data stream/set. Maybe we can already foresee that in the naming of resources for this API redesign.

Thanks for your helpful input. I have incorporated your comments into the draft (see table below).

As suggested, I have left the file sharing /storage part out for now. However, this is also an exciting and important topic that we should address.

@Dominik: As far as I know, the image upload is currently done via the image connect adapter (set: org.apache.streampipes.connect.adapters.image.set, stream: org.apache.streampipes.connect.adapters.image.stream). I have not worked with it yet. But it would certainly be nice to have an endpoint for it.

Regarding the high-level roadmap you mentioned, we can foresee the use of an UUID (or similar) as measurementID. However, imo it would then make sense to write additional metadata about the measurement series to the data lake when creating a persistent adapter (e.g. user-assigned plaintext name), for user convenience purposes. But I think, with InfluxDB only measurement-level metadata is supported in the form of tags, which would generate too much overhead if this additional information had to be written each time. But please correct me here if I am wrong. But does the resource naming fit then or do you have a concrete suggestion for the renaming?

Hi Daniel,

thanks, looks very good to me!

I'd say we can keep the user-defined index name as the measurementId by now and switch it to the unique ID of the corresponding data stream once we've integrated data streams and their persistence. In that case, the metadata would directly come from the data stream description and users would only interact with the name of the data stream in the data explorer and dashboard without the need to explicitly assign/know the index name.

So from my point of view, this API redesign is ready for entering the implementation phase