You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 45 Next »

Hive Developer FAQ

Maven

Hive is using Maven as its build tool. Versions prior to 0.13 were using Ant.

Developing

How do I add a new MiniMR test?

See MiniMR and PTest2

How do I move some files?

Post a patch for testing purposes which simply does add and deletes. SVN will not understand these patches are actually moves, therefore you should actually upload the following, in order so the last upload is the patch for testing purposes:

  1. A patch which has only the non-move changes for commit e.g. HIVE-XXX-for-commit.patch
  2. A script of of commands required to make the moves HIVE-XXX-moves.sh
  3. A patch for testing purposes HIVE-XXX.patch

The script should be a set of svn mv commands along with any perl commands required for find/replace. E.g.:

$ svn mv MyCLass.java MyClass.java
$ perl -i -pe 's<at:var at:name="MyCLass" />MyClass@g' MyClass.java

Building

Why isn't the itests pom connected to the root pom?

The qfile tests in itests require the packaging phase. The maven test phase is after compile and before packaging. We could change the qfile tests to run during the integration-test phase using the "failsafe" plugin but the "failsafe" plugin is different than surefire and IMO is hard to use. If you'd like to give that a try, by all means, go ahead.

How do I rerun precommit tests over the same patch?

Upload the exact same patch again to the JIRA.

Maven settings

You might have to set the following maven options on certain systems to get build working. Check out the suggested value for MAVEN_OPTS under the testing section.

How to build all source?

MVN:

mvn clean install -DskipTests -Phadoop-1
cd itests 
mvn clean install -DskipTests -Phadoop-1

How to specify the Hadoop version?

In mvn commands, use -Phadoop-1 or -Phadoop-2 to specify the Hadoop version. Several examples are shown in these build instructions.

How do I import into eclipse?

Build and generate eclipse files (the conservative method):

$ mkdir workspace
$ cd workspace
$ git clone https://github.com/apache/hive.git
$ cd hive
$ mvn clean install -DskipTests -Phadoop-2
$ mvn eclipse:clean
$ mvn eclipse:eclipse -DdownloadSources -DdownloadJavadocs -Phadoop-2
$ cd itests
$ mvn clean install -DskipTests -Phadoop-2
$ mvn eclipse:clean
$ mvn eclipse:eclipse -DdownloadSources -DdownloadJavadocs -Phadoop-2

In eclipse define M2_REPO in Preferences -> Java -> Build Path -> Classpath Variables to either:

Mac Example:

/Users/$USER/.m2/repository

Linux Example:

/home/$USER/.m2/repository

Windows Example:

C:/users/$USER/.m2/repository

Then import the workspaces. If you an error about "restricted use of Signal" for beeline and cli follow the following instructions.

Note that if you use the hive git base dir as eclipse workspace, then it does not pick the right project names (eg picks 'ant' instead of 'hive-ant'). Therefore it's recommended to have the workspace directory one up from the git directory. For example workspaces/hive-workspace/hive where hive-workspace is the eclipse workspace and hive is the git base directory.

How to generate tarball?

MVN:

mvn clean package -DskipTests -Phadoop-1 -Pdist

It will then be located in packaging/target/

How to generate protobuf code?

MVN:

cd ql
mvn clean install -DskipTests -Phadoop-1,protobuf

How to generate thrift code?

MVN:

mvn clean install -Phadoop-1,thriftif -DskipTests -Dthrift.home=/usr/local

How to compile ODBC?

MVN:

cd odbc
mvn compile -Phadoop-1,odbc -Dthrift.home=/usr/local -Dboost.home=/usr/local

How do I publish hive artifacts to my local maven repository?

ant package
ant -Dmvn.publish.repo=local maven-build
ant -Dmvn.publish.repo=local maven-publish

MVN:

mvn clean install -DskipTests -Phadoop-1
cd itests 
mvn clean install -DskipTests -Phadoop-1

Testing

Where is the log output of a test?

Logs are put in a couple locations:

  • From the root of the source tree: find . -name hive.log
  • /tmp/$USER/ (Linux) or $TMPDIR/$USER/ (MacOS)

How do I run a single test?

ITests

Note that any test in the itests directory needs to be executed from with the itests directory. The pom is disconnected from the parent project for technical reasons.

Single test class:

mvn test -Dtest=ClassName -Phadoop-1

Single test method:

mvn test -Dtest=ClassName#methodName -Phadoop-1

How do I debug into a single test in Eclipse?

You can debug into a single JUnit test in Eclipse by first making sure you've built the Eclipse files and imported the project into Eclipse as described here. Then set one or more breakpoints, highlight the method name of the JUnit test method you want to debug into, and do Run->Debug.

A test fails with a NullPointerException in MiniDFSCluster

If any test fails with the error below it means you have an inappropriate umask setting. It should be set to 0022.

java.lang.NullPointerException: null
    at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:426)
    at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:284)
    at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124)

How do I run all of the unit tests?

Make sure that your JAVA_HOME is appropriately set (some tests need this), and set ANT_OPTS to increase the size allocated to the Permanent Generation as per the following:

export ANT_OPTS="-XX:MaxPermSize=512m"
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=256M"

Then, for a clean build, run

ant clean package test

MVN:

mvn test -Phadoop-1
cd itests 
mvn test -Phadoop-1

Note that running ant test will not work; ant package does some setup work that is required for the testcases to run successfully.

MVN: Note that you need to have previously built and installed the jars:

mvn clean install -DskipTests -Phadoop-1
cd itests 
mvn clean install -DskipTests -Phadoop-1

How do update the output of a CliDriver testcase?

ant test -Dtestcase=TestCliDriver -Dqfile=alter1.q -Doverwrite=true

MVN:

cd itests/qtest
mvn test -Dtest=TestCliDriver -Dqfile=alter1.q -Dtest.output.overwrite=true -Phadoop-1

As of Hive 0.11.0+ you can cut this time in half by specifying that only the ql module needs to rebuild

ant test -Dmodule=ql -Dtestcase=TestCliDriver -Dqfile=alter1.q -Doverwrite=true

How do I run the clientpositive/clientnegative unit tests?

All of the below require that you have previously run ant package.

To run clientpositive tests

ant -Dtestcase=TestCliDriver test

MVN:

cd itests/qtest
mvn test -Dtest=TestCliDriver -Phadoop-1

To run a single clientnegative test alter1.q

ant -Dtestcase=TestNegativeCliDriver -Dqfile=alter1.q test

MVN:

cd itests/qtest
mvn test -Dtest=TestNegativeCliDriver -Dqfile=alter1.q -Phadoop-1

To run all of the clientpositive tests that match a regex, e.g. the partition_wise_fileformat tests

ant -Dtestcase=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* test

MVN:

cd itests/qtest
mvn test -Dtest=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* -Phadoop-1

To run a single contrib test alter1.q and overwrite the result file

ant -Dtestcase=TestContribCliDriver -Dqfile=alter1.q -Doverwrite=true test

MVN:

cd itests/qtest
mvn test -Dtest=TestContribCliDriver -Dqfile=alter1.q -Dtest.output.overwrite=true -Phadoop-1

To run a single test groupby1.q and output detailed information during execution

ant -Dtestcase=TestCliDriver -Dqfile=groupby1.q -Dtest.silent=false test

As of Hive 0.11.0+ you can cut down the total build time by specifying that only the ql module needs to rebuild. e.g. run all the partition_wise_fileformat tests

ant -Dmodule=ql -Dtestcase=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* test
  • No labels