You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 15 Next »

 

This page aims to catalogue and describe the various public facing APIs exposed by Hive in order to inform developers wishing to integrate their applications and frameworks with the Hive ecosystem. To date the following APIs have been identified in the Hive project that are either considered public, or widely used in the public domain:

HCatClient (Java)

This is a Java API that presents a number of DDL type operations, however it is not as comprehensive as the Metastore API. The HCatClient was intended to be the Java based entry point to WebHCat HCatalog API although this was never realised. Currently HCatClientHMSImpl is the only concrete implementation of the API; it integrates directly with the Metastore using the Metastore API and does not utilise WebHCat whatsoever despite being packaged inside the WebHCat project. The HCatClientHMSImpl was originally provided as a reference implementation but it has over time gained traction as a public client. Anecdotally, it is now the preferred API for issuing DDL type operations from external programs; and feature contributions are encouraged. There is some minimal documentation on the HCatalog wiki in the form of a design document describing the interface but not the implementation.

HCatalog Storage Handlers (Java)

This is well documented on the wiki.

TODO

Requires overview.

HiveServer2 API

TODO

Describe.

HCatalog CLI (Command Line)

This is well documented on the wiki.

TODO

Requires overview. Describe differences with hive CLI.

Metastore (Java)

A Thrift based API with Java bindings, described by the IMetaStoreClient interface. The API decouples the metastore storage layer from other Hive internals. Because Hive itself uses this internally, it is required to implement a comprehensive feature set which makes it attractive to developers who might find the other APIs lacking. It was not originally intended to be a public API although it became public in version 1.0.0 (HIVE-3280) and there is a proposal that it be documented more fully (HIVE-9363). Anecdotally, its use outside of the Hive project is not currently recommended.

TODO: API usage

There are numerous ways of instantiating the metastore API including: HCatUtil.getHiveMetastoreClient(), new HiveMetaStoreClient.HiveMetaStoreClient(...). It may be useful to make some recommendations on the preferred approach.

Hive (Java)

Refers to the org.apache.hadoop.hive.ql.metadata.Hive class. Appears to be a distinct concrete implementation of a variation of the metastore API. Delegates to the metastore API but does not directly extend/implement it.

TODO

I suspect its use is not encouraged. Seeking clarification on the motivations behind this class and thoughts on its use outside of Hive.

Driver (Java)

Refers to the org.apache.hadoop.hive.ql.Driver class.

TODO

Describe the role of Driver, when to use it, etc.

WebHCat (REST)

WebHCat is a REST API for HCatalog. This is well documented on the wiki.

TODO

Is this API actively used? Is its use encouraged? When should one use it?

Streaming Data Ingest (Java)

A Java API focused on the writing of continuous streams of data into transactional tables using Hive’s ACID feature. New data is inserted into tables using small batches and short-lived transactions. Documented on the wiki and has package level Javadoc. Introduced in Hive version 0.13.0 (HIVE-5687).

Streaming Mutation (Java)

A Java API focused on mutating (insert/update/delete) records into transactional tables using Hive’s ACID feature. Large volumes of mutations are applied atomically in a single long-lived transaction. Documented with package level Javadoc. Scheduled for release in Hive version 2.0.0 (HIVE-10165).

hive-jdbc (JDBC)

TODO

Seeking documentation. Is this just for reading data? writing data? does it support DDL operations?

  • No labels