Page History

...

...

In master, only Hadoop 2.x is supported, thus there is no need to specify a Maven profile for most build operations.

In master, MVN:

mvn clean install -DskipTests
cd itests 
mvn clean install -DskipTests

...

Note that if you use the Hive git base directory as the Eclipse workspace, then it does not pick the right project names (for example, picks 'ant' instead of 'hive-ant'). Therefore it's recommended to have the workspace directory one up from the git directory. For example workspaces/hive-workspace/hive where hive-workspace is the Eclipse workspace and hive is the git base directory.

How to generate tarball?

MVN:

mvn clean package -DskipTests -Pdist

...

How to generate protobuf code?

MVN:

cd ql
mvn clean install -DskipTests -Pprotobuf

How to generate Thrift code?

MVN:

mvn clean install -Pthriftif -DskipTests -Dthrift.home=/usr/local

How to run findbugs after a change?

mvn site -Pfindbugs 

How to run findbugs after a change?

mvn site -Pfindbugs 

Note: Note:  Available in Hive 1.1.0 onward (see HIVE-8327).

How to compile ODBC?

MVN:

cd odbc
mvn compile -Podbc -Dthrift.home=/usr/local -Dboost.home=/usr/local

How do I publish Hive artifacts to my local Maven repository?

ant package
ant -Dmvn.publish.repo=local maven-build
ant -Dmvn.publish.repo=local maven-publish

MVN:

mvnmvn clean install -DskipTests
cd itests 
mvn clean install -DskipTests
cd itests 
mvn clean install -DskipTests

Testing

For general information, see Unit Tests and Debugging in the Developer Guide.

How do I run precommit tests on a patch?

Hive precommit testing is triggered automatically when a file is uploaded to the JIRA ticket:

1. Attach the patch file to a JIRA ticket: in the ticket's "More" tab, select "Attach Files" and use "Choose File" to upload the file, then add a descriptive comment.
2. Put the patch in the review queue: click the "Submit Patch" button. The button name will change to "Cancel Patch" and the ticket status will change to Patch Available.

...

How do I rerun precommit tests over the same patch?

For patch updates, our convention is to number them like HIVE-1856.1.patch, HIVE-1856.2.patch, etc. And then click the "Submit Patch" button again when a new one is uploaded; this makes sure it gets back into the review queue.

How do I run a single test?

Single test class:

mvn test -Dtest=ClassName

...

mvn test -Dtest=ClassName#methodName

Note that a pattern can also be supplied to -Dtests to run multiple tests matching the pattern:

mvn test -Dtest='org.apache.hive.beeline.*' 

For more usage see the documentation for the Maven Surefire Plugin.

How do I run all of the unit tests?

...

mvn clean install -DskipTests 
cd itests 
mvn clean install -DskipTests 

How do I run all of the unit tests except for a certain few tests?

Similar to running all tests, but define test.excludes.additional to specify a test/pattern to exclude from the test run. For example the following will run all tests except for the CliDriver tests:

Then, for a clean build, run

export ANT_OPTS="-XX:MaxPermSize=512m"

ant clean package test

cd itests 
mvn test -Dtest.excludes.additional='**/Test*CliDriver.java' 

How do I run

...

the

...

clientpositive/clientnegative unit tests?

...

All of the below require that you have previously run ant package.

To run clientpositive tests

cd itests/qtest

Similar to running all tests, but define test.excludes.additional to specify a test/pattern to exclude from the test run. For example the following will run all tests except for the CliDriver tests:

cd itests 
mvn test -Dtest.excludes.additional='**/Test*CliDriver.java' 

How do I run the clientpositive/clientnegative unit tests? 

All of the below require that you have previously run ant package.

To run clientpositive tests

ant -Dtestcase=TestCliDriver test

MVN:

=TestCliDriver 

To run a single clientnegative test alter1.q

cd itests/qtest
mvn test -Dtest=TestCliDriver 

To run a single clientnegative test alter1.q

ant -Dtestcase=TestNegativeCliDriver -Dqfile=alter1.q test

MVN:

cd itests/qtest
mvn test -Dtest=TestNegativeCliDriver -Dqfile=alter1.q 

To To run all of the clientpositive tests that match a regex, for example the partition_wise_fileformat tests

...

MVN:

cd itests/qtest
mvn test -Dtest=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* 

To run a single contrib test alter1.q and overwrite the result file

ant -Dtestcase=TestContribCliDriver -Dqfile=alter1.q -Doverwrite=true test

MVN:

cd itests/qtest

# Alternatively, you can specify comma separated list with "-Dqfile" argument
mvn test -Dtest=TestContribCliDriverTestMiniLlapLocalCliDriver -Dqfile=alter1'vectorization_0.q -Dtest.output.overwrite=true ,vectorization_17.q,vectorization_8.q'

To run a single contrib test groupby1alter1.q and output detailed information during executionoverwrite the result file

antcd itests/qtest
mvn test -DtestcaseDtest=TestCliDriverTestContribCliDriver -Dqfile=groupby1alter1.q -Dtest.output.silentoverwrite=falsetrue test

As of Hive 0.11.0+ you can cut down the total build time by specifying that only the ql module needs to rebuild. For example, run all the partition_wise_fileformat tests

ant -Dmodule=ql -Dtestcase=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* test

How do I modify the init script when testing?

The option to skip the init script or supply a custom init script was added in Hive 2.0 (see HIVE-11538).

To skip initialization:

How do I run with Postgre/MySQL/Oracle?

To run test test with a specified DB it is possible by adding "-Dtest.metastore.db" parameter like in the following commands:

mvn test -Pitests -pl itests/qtest -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q -Dtest.metastore.db=postgres

mvn test -Pitests -pl itests/qtest

mvn test -Dtest=TestCliDriver -Phadoop-2 -Dqfile=testpartition_toparams_runpostgres.q  -DinitScript=

To supply a custom script:

Dtest.metastore.db=mssql
mvn test -Pitests -pl itests/qtest -Dtest=TestCliDriver -Phadoop-2Dqfile=partition_params_postgres.q -Dtest.outputmetastore.overwritedb=true -Dqfile=test_to_run.q  -DinitScript=custom_script.sql

How do I update the output of a CliDriver testcase?

ant test -Dtestcasemysql
mvn test -Pitests -pl itests/qtest -Dtest=TestCliDriver -Dqfile=alter1partition_params_postgres.q -Doverwrite=true

...

Dtest.metastore.db=oracle -Ditest.jdbc.jars=/path/to/your/god/damn/oracle/jdbc/driver/ojdbc6.jar

Without specifying -Dqfile it will run all .q files .

How do I remote debug a qtest?

cd itests/qtest
mvn test -Dtest=TestCliDriver -Dqfile=alter1.q -Dtest.output.overwrite=true 
-Dmaven.surefire.debug="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 -Xnoagent -Djava.compiler=NONE" test -Dtest=TestCliDriver -Dqfile=<test>.q

How do I modify the init script when testing?

The option to skip the init script or supply a custom init script was added in Hive 2.0 (see HIVE-11538).

To skip initialization:As of Hive 0.11.0+ you can cut this time in half by specifying that only the ql module needs to rebuild

antmvn test -DmoduleDtest=qlTestCliDriver -Dtestcase=TestCliDriverPhadoop-2 -Dqfile=alter1test_to_run.q  -DoverwriteDinitScript=true

How do I update the results of many test cases?

To supply a custom script:Assume that you have a file like below which you'd like to re-generate output files for. Such a file could be created by copying the output from the precommit tests.

headmvn test -2 /tmp/failed-TestCliDriver-file-tests
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join

...

Dtest=TestCliDriver -Phadoop-2 -Dtest.output.overwrite=true -Dqfile=test_to_run.q  -DinitScript=custom_script.sql

How do I update the output of a CliDriver testcase?

egrep 'TestCliDriver' /tmp/failed-TestCliDriver-file-tests | perl -pe 's@.*testCliDriver_@@g' | awk '{print $1 ".q"}' | xargs -n 30 | perl -pe 's@ @,@g' | xargs -I{} mvn test -Dtest=TestCliDriver -Dtest.output.overwrite=true -Dqfile={}

To do the same from the output of a precommit result, with multiple drivers, you can do

cd itests/qtest
mvn test -Dtest=TestCliDriver -Dqfile=alter1.q -Dtest.output.overwrite=true 

How do I update the results of many test cases?

Assume that you have a file like below which you'd like to re-generate output files for. Such a file could be created by copying the output from the precommit tests.

head -2 /tmp/failed-TestCliDriver-file-tests

import re
from itertools import groupby
s = """
org.apache.hadoop.hive.cli.TestBeeLineDriverTestCliDriver.testCliDriver[drop_allcolref_with_concurrency] (batchId=231)in_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join

You can re-generate all those output files in batches of 20 with the command below

egrep 'TestCliDriver' /tmp/failed-TestCliDriver-file-tests | perl -pe 's@.*testCliDriver_@@g' | awk '{print $1 ".q"}' | xargs -n 30 | perl -pe 's@ @,@g' | xargs -I{} mvn test -Dtest=TestCliDriver -Dtest.output.overwrite=true -Dqfile={}

To do the same from the output of a precommit result, with multiple drivers, you can do

import re
from itertools import groupby
s = """
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency] (batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=141)
"""
PAT = re.compile([comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=141)
"""
PAT = re.compile("org.apache.hadoop.hive.cli.([^\.]*).*\[([^\]]*).*")
l = [PAT.match(x.strip()) for x in s.split("\n") if x.strip()]
for driver,q in groupby(sorted([a.groups() for a in l if a]), key=lambda a:a[0]):
	print """mvn clean test -Dtest=%s '-Dqfile=%s' -Dtest.output.overwrite=true""" % (driver, ",".join(["%s.q" % a[1] for a in q]))

...

• From the root of the source tree: find . -name hive.log

• /tmp/$USER/ (Linux) or $TMPDIR/$USER/ (MacOS)

See Hive Logging for details about log files, including alternative configurations.

How do I add a test case?

First, add the test case to the qfile test suite:

• name hive.log

• /tmp/$USER/ (Linux) or $TMPDIR/$USER/ (MacOS)

See Hive Logging for details about log files, including alternative configurations.

How do I add a test case?

First, add the test case to the qfile test suite:

• Copy the test to a new file under ql/src/test/queries/clientpositive/<filename>.q (or /clientnegative if it is a negative test).
  • If the new test creates any table, view, function, etc., make sure that the name is unique across tests. For instance, name a table in the test file foo.q, foo_t1 instead of simply t1. This will help reduce flakiness in the test runs, since Jenkins will run tests and batches, and currently it does not restore to former state after running each of the q files.
  • If there is any interaction with file system, use unique folders for the test to avoid any collision with other tests.
  Copy the test to a new file under ql/src/test/queries/clientpositive/<filename>.q (or /clientnegative if it is a negative test).
• Add the <filename.q> to itests/src/test/resources/testconfiguration.properties to the appropriate variable (ex. minimr.query.files).

...

• Run the test and generate the output file using the appropriate -Dtest (ex. TestMinimrCliDriver TestCliDriver; see itests/qtest/pom.xml for the names of other test suites):

  Code Block
  language text
  cd qtest mvn test -Dtest=TestMinimrCliDriverTestCliDriver -Dqfile=<filename>.q -Dtest.output.overwrite=true

  
  

...

At this point, attach the remote debugger as mentioned before to start debugging your queries.