The Apache Atlas bridges provide an asynchronous mechanism for data platforms/services and data movement engines to notify Apache Atlas of changes in the data landscape. It is through these mechanisms that Apache Atlas is able to maintain an up-to-date perspective on where data is stored and its lineage.
Apache Atlas may also be configured to trigger automated metadata discovery for new and changed data sources, creating a rich definition of the data source in its metadata repository.
There are two main mechanisms:
- A bridge is a one time load of metadata from a data platform/service/engine. This is used to perform the initial load of metadata into Atlas for the current state of the data resources.
- A hook is an on-going trickle feed of updates to Atlas as the state of the data resources changes in the data platform/service/engine.
The bridges and hooks implementation are located in the addons module of the code base. They run outside of the Atlas runtime. The intg module provides the integration capability within Atlas that receives and sends messages to the hooks.
<more to come ... just organizing pages>