- Motivation:
- describing dataset in schema has many benefits like assuring data quality while moving around data, AS IS data delivery, etc...
- currently Apache Gobblin
...
- takes schema as part of configuration and does not integrate with schema registry service so that it can enforce latest schema for validations or evolution.
- Proposed Change:
- Add ability to integrate with Hortonworks ( HDP ) schema registry : https://github.com/hortonworks/registry
- New or Changed Public Interfaces: TBD
- Migration Plan and Compatibility: TBD
- Rejected Alternatives:
- we should not use confluent schema registry for various reasons, some are listed here as part of the comparison: https://registry-project.readthedocs.io/en/latest/competition.html