ATLAS-1773 - Getting issue details... STATUS

Overview

The Open Metadata Repository Services (OMRS) provide the means for different metadata repositories to exchange metadata.  It breaks the mould of traditional metadata repositories by assuming that metadata is going to be distributed amongst a number of metadata repositories.  These repositories may all be instances of Apache Atlas, or it may include a mixture of repositories from different vendors.

The philosophy is that metadata should be managed as close to its source as possible.  This means new metadata is created in the repository that is connected to the tools a person uses, the engine processing data or a specific group of data sources.  So for example, an organization may have:

• a metadata repository and tools for its governance team.  This metadata repository may host the canonical glossary, and the governance policies, rules and classifications.
• a metadata repository for its data lake.  This metadata repository has the details of the data repositories in the data lake and the movement of data between them.
• a metadata repository for its data integration tools that continuously extract data from the operational systems and sends them to the data lake.

Whichever repository is used to create a metadata entity, it has the master copy and all updates to this metadata should be done through this repository. So part of the responsibility of the OMRS is to ensure updates happen in the right metadata repository.

With the metadata distributed across multiple repositories, an enterprise will want to be able to query across all of the repositories together and to be able to link metadata together from different repositories.  Both of these capabilities are part of the OMRS.

The integration between metadata repositories needs to be flexible to support different non-functional requirements.  As such the OMRS covers integration either:

• Through a call interface which is provided by an OMRS connector
• Using notifications that broadcast changes to metadata in a repository that other repositories can subscribe to in order to maintain reference copies of specific metadata entities
• Via linked data URLs that enable a metadata entity to have a relationship with a metadata entity in a different repository.

OMRS Connectors

The OMRS Connector API is a standard interface for a connector to a metadata repository.  We plan 4 implementations of this API:

• Local Atlas OMRS Connector – this is the connector to a local Apache Atlas metadata repository.
• OMRS REST Connector – this is a connector to a remote Apache Atlas repository (or any other metadata repository that supports the OMRS REST APIs).
• IGC OMRS Connector – this is the connector for IBM’s Information Governance Catalog
• Enterprise OMRS Connector – this connector can federate multiple metadata repositories by aggregating the results of calls to their OMRS connectors.

OMAS APIs will use the OMRS Connector API to access metadata.  The name of the metadata repository connection they use (set in the OMAS Scope) will determine which implementation of the connector is used and hence which metadata repository/ repositories are called.

Figure 2 illustrates these OMRS connectors in action:

The notes below correspond to the numbers on the diagram in figure 2.

1. There are many OMAS APIs, each designed for a different type of consumer.  Each OMAS API has a simple, bean-like Java interface for applications to call (or the application may use the OMAS REST API directly).
2. The OMAS REST APIs provide JSON versions of the objects supported in the OMAS APIs.  The OMAS Java Server classes implement the REST APIs.  Each Java Server Class uses the connector broker to acquire an OMRS Connector.
3. The Local Atlas OMRS Connector provides access to the local metadata repository.
4. This local repository includes a graphDB accessed through TinkerPop.
5. The Enterprise OMRS is able to aggregate metadata from multiple metadata repositories in response to a single request for metadata.  It does this by replicating the request across multiple downstream OMRS Connectors and then aggregating and correlating the results.
6. The Enterprise OMRS Connector is typically configured to call the local Atlas Repository for creates.  Updates go to the repository where the metadata entity was created.
7. In addition, the Enterprise OMRS may be configured to call other metadata repositories using OMRS Connector implementations that have been built into the Atlas runtime.  In this example,  (7) shows an OMRS connector that can call IBM’s Information Governance Catalog (IGC).
8. The IGC OMRS Connector will translate OMRS calls into IGC REST calls.
9. For calls to remote Atlas Servers, and other servers that have OMRS connectors that are not included into the Atlas runtime, the Enterprise OMRS Connector will use the OMRS REST Connector.  This connector translates OMRS connector requests into remote OMRS REST calls. 
10. The OMRS REST Connector may also be called from the OMAS Java Server classes to pass-through metadata requests to a remote metadata server.
11. Every Atlas runtime supports the OMRS REST APIs.
12. When an Atlas runtime received an OMRS REST call, it is passed to the local Atlas OMRS connector and is executed against the local metadata repository.
13. The OMRS REST Connector can be used to connect to an adapter for other types of metadata repositories.  
14. The adapter would host an OMRS Connector to the metadata repository.  This provides a solution for OMRS Connector implementations that are not (yet) integrated into the Atlas build.  The downside of this approach is the additional network hop that the adapter introduces.
15. An OMRS REST Connector can be incorporated into other metadata repositories to enable them to query Atlas metadata.  Typically the repository will use the Enterprise OMRS Connector to ensure it is reaching as much metadata as possible
16. This includes IBM’s Information Governance Catalog (IGC).

OMRS Notifications

OMRS Linking

TBD