Hive Developer FAQ
Mavenization is complete
The hive build has been mavenized and trunk uses maven. 0.12 still uses ant thus we need to keep the ant commands below.
Building
How do I rerun precommit tests over the same patch?
Upload the exact same patch again to the JIRA.
Maven settings
You might have to set the following maven options on certain systems to get build working. Check out the suggested value for MAVEN_OPTS under the testing section.
How to build all source?
MVN:
mvn clean install -DskipTests cd itests mvn clean install -DskipTests
How do I import into eclipse?
Build and generate eclipse files (the conservative method):
$ mkdir workspace $ cd workspace $ git clone https://github.com/apache/hive.git $ cd hive $ mvn clean install -DskipTests $ mvn eclipse:clean $ mvn eclipse:eclipse -DdownloadSources -DdownloadJavadocs $ cd itests $ mvn clean install -DskipTests $ mvn eclipse:clean $ mvn eclipse:eclipse -DdownloadSources -DdownloadJavadocs
In eclipse define M2_REPO in Preferences -> Java -> Build Path -> Classpath Variables to either:
Mac Example:
/Users/$USER/.m2/repository
Linux Example:
/home/$USER/.m2/repository
Windows Example:
C:/users/$USER/.m2/repository
Then import the workspaces. If you an error about "restricted use of Signal" for beeline and cli follow the following instructions.
Note that if you use the hive git base dir as eclipse workspace, then it does not pick the right project names (eg picks 'ant' instead of 'hive-ant'). Therefore it's recommended to have the workspace directory one up from the git directory. For example workspaces/hive-workspace/hive where hive-workspace is the eclipse workspace and hive is the git base directory.
How to generate tarball?
MVN:
mvn clean package -DskipTests -Pdist
It will then be located in packaging/target/
How to generate protobuf code?
MVN:
cd ql mvn clean install -DskipTests -Phadoop-1,protobuf
How to generate thrift code?
MVN:
mvn clean install -Phadoop-1,thriftif -DskipTests -Dthrift.home=/usr/local
How to compile ODBC?
MVN:
cd odbc mvn compile -Podbc -Dthrift.home=/usr/local -Dboost.home=/usr/local
How do I publish hive artifacts to my local maven repository?
ant package ant -Dmvn.publish.repo=local maven-build ant -Dmvn.publish.repo=local maven-publish
MVN:
mvn clean install -DskipTests cd itests mvn clean install -DskipTests
Testing
How do I run a single test?
ITests
Note that any test in the itests directory needs to be executed from with the itests directory. The pom is disconnected from the parent project for technical reasons.
Single test class:
mvn test -Dtest=ClassName
Single test method:
mvn test -Dtest=ClassName#methodName
How do I debug into a single test in Eclipse?
You can debug into a single JUnit test in Eclipse by first making sure you've built the Eclipse files and imported the project into Eclipse as described here. Then set one or more breakpoints, highlight the method name of the JUnit test method you want to debug into, and do Run->Debug
.
A test fails with a NullPointerException in MiniDFSCluster
If any test fails with the error below it means you have an inappoprate umask setting. It should be set to 0022.
java.lang.NullPointerException: null at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:426) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:284) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124)
How do I run all of the unit tests?
Make sure that your JAVA_HOME is appropriately set (some tests need this), and set ANT_OPTS to increase the size allocated to the Permanent Generation as per the following:
export ANT_OPTS="-XX:MaxPermSize=512m" export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=256M"
Then, for a clean build, run
ant clean package test
MVN:
mvn test cd itests mvn test
Note that running ant test
will not work; ant package
does some setup work that is required for the testcases to run successfully.
MVN: Note that you need to have previously built and installed the jars:
mvn clean install -DskipTests cd itests mvn clean install -DskipTests
How do update the output of a CliDriver testcase?
ant test -Dtestcase=TestCliDriver -Dqfile=alter1.q -Doverwrite=true
MVN:
cd itests/qtest mvn test -Dtest=TestCliDriver -Dqfile=alter1.q -Dtest.output.overwrite=true
As of Hive 0.11.0+ you can cut this time in half by specifying that only the ql module needs to rebuild
ant test -Dmodule=ql -Dtestcase=TestCliDriver -Dqfile=alter1.q -Doverwrite=true
How do I run the clientpositive/clientnegative unit tests?
All of the below require that you have previously run ant package
.
To run clientpositive tests
ant -Dtestcase=TestCliDriver test
MVN:
cd itests/qtest mvn test -Dtest=TestCliDriver
To run a single clientnegative test alter1.q
ant -Dtestcase=TestNegativeCliDriver -Dqfile=alter1.q test
MVN:
cd itests/qtest mvn test -Dtest=TestNegativeCliDriver -Dqfile=alter1.q
To run all of the clientpositive tests that match a regex, e.g. the partition_wise_fileformat tests
ant -Dtestcase=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* test
MVN:
cd itests/qtest mvn test -Dtest=TestCliDriver -Dqfile_regex=partition_wise_fileformat.*
To run a single contrib test alter1.q and overwrite the result file
ant -Dtestcase=TestContribCliDriver -Dqfile=alter1.q -Doverwrite=true test
MVN:
cd itests/qtest mvn test -Dtest=TestContribCliDriver -Dqfile=alter1.q -Dtest.output.overwrite=true
To run a single test groupby1.q and output detailed information during execution
ant -Dtestcase=TestCliDriver -Dqfile=groupby1.q -Dtest.silent=false test
As of Hive 0.11.0+ you can cut down the total build time by specifying that only the ql module needs to rebuild. e.g. run all the partition_wise_fileformat tests
ant -Dmodule=ql -Dtestcase=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* test