Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.


By building a dashboard, with ability to filter on hoodie dataset, ingestion run, host information, stage bottlenecks and host/executor outliers can be identified.

Sample dashboard:

Image Removed

Rollout/Adoption Plan

Once the proposed solution is implemented, the observability metrics collected from Hudi, using the above described framework, would be available as a Graphite dashboard and will also be published to a Kafka stream. When a pipeline is setup to ingest the metrics available in Kafka into a Hudi table, the data will be available, for search and visualization, capturing insights specific to a dataset as well as insights across all datasets ingested into Hudi.