...
By building a dashboard, with ability to filter on hoodie dataset, ingestion run, host information, stage bottlenecks and host/executor outliers can be identified.
Sample dashboard:
Rollout/Adoption Plan
Once the proposed solution is implemented, the observability metrics collected from Hudi, using the above described framework, would be available as a Graphite dashboard and will also be published to a Kafka stream. When a pipeline is setup to ingest the metrics available in Kafka into a Hudi table, the data will be available, for search and visualization, capturing insights specific to a dataset as well as insights across all datasets ingested into Hudi.
...