You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 11 Next »

 

Guide for contributors to the metastore on hbase development work. Umbrella JIRA - HIVE-9452

 

Setup for running hive against hbase metastore - 

Once you’ve built the code from the HBase metastore patch, here’s how to make it run against HBase:

 

  1. Install HBase, preferably HBase 0.98 as that’s what is being used for testing.
  2. Copy following jars into $HBASE_HOME/lib

    1. hive-common-1.*.jar

    2. hive-metastore-1.*.jar

    3. hive-serde-1.*.jar

  3. Setup HBase, http://hbase.apache.org/book.html#quickstart  I run it in stand alone mode, so you have to set a couple of values in hbase-site.xml for this to work.
  4. Set HADOOP_HOME if you’re not in a cluster where hadoop is already on your path.
  5. Start HBase: $HBASE_HOME/bin/start-hbase.sh
  6. Set it up so that HBase jars and conf file are picked up by Hive
    1. export HIVE_AUX_JARS_PATH=$HBASE_HOME/lib/
    2. export AUX_CLASSPATH=$HBASE_HOME/conf
  7. Create the metastore tables in HBase: hive --service hbaseschematool install
  8. Configure Hive to use HBase as its metastore, in hive-site.xml:
<property>
    <name>hive.metastore.rawstore.impl</name>
    <value>org.apache.hadoop.hive.metastore.hbase.HBaseStore</value>
  </property>
  <property>
    <name>hive.metastore.fastpath</name>
    <value>true</value>
  </property>

 

Now start Hive as normal, all should just work.

Notes

  • Currently (as of Apr 8 2015) we have not tested the HBase metastore with the metastore service.  We don't know if it works or not.  We have tested it with the command line client and HiveServer2.
  • Not all Hive operations have been tested.  Insert, select, create table, drop table, create database, add partition, drop partition have been tested.  Other features may not work.  

 

Importing metadata from rdbms to hbase 

Set hive-site.xml so that it has necessary jdo properties for rdbms, and setup Hive to use hbase for metadata storage as documented above

The following command will import the metadata from the rdbms to hbase:

hive --service hbaseimport 


Design Docs

Overall Approach

Hbase execution plans for RawStore partition filter condition

 

  • No labels