How to Contribute to Apache Hive
This page describes the mechanics of how to contribute software to Apache Hive. For ideas about what you might contribute, please see open tickets in Jira.
Getting the source code
First of all, you need the Hive source code.
Get the source code on your local drive using SVN. Most development is done on the "trunk":
svn checkout http://svn.apache.org/repos/asf/hive/trunk hive-trunk
You also have the option of using one of the Git mirrors of the SVN repository:
git clone git://git.apache.org/hive.git
or
git clone http://github.com/apache/hive.git
Setting up Eclipse Development Environment (Optional)
This is an optional step. Eclipse has a lot of advanced features for Java development, and it makes the life much easier for Hive developers as well.
How to set up Eclipse for Hive development
Making Changes
Before you start, send a message to the Hive developer mailing list, or file a bug report in JIRA. Describe your proposed changes and check that they fit in with what others are doing and have planned for the project. Be patient, it may take folks a while to understand your requirements.
Modify the source code and add some (very) nice features using your favorite IDE.
Coding Convention
Please take care about the following points
- All public classes and methods should have informative Javadoc comments.
- Do not use @author tags.
- Code should be formatted according to Sun's conventions, with one exception:
- Indent two (2) spaces per level, not four (4).
- Line length limit is 100 chars, instead of 80 chars.
- Contributions should not introduce new Checkstyle violations.
- Check for new Checkstyle violations by running
ant checkstyle
, and then inspect the results in thebuild/checkstyle
directory. - If you use Eclipse you should install the eclipse-cs Checkstyle plugin. This plugin highlights violations in your code and is also able to automatically correct some types of violations.
- Check for new Checkstyle violations by running
- Contributions should pass existing unit tests.
- New unit tests should be provided to demonstrate bugs and fixes. JUnit is our test framework:
- You must implement a class that extends
junit.framework.TestCase
and whose class name starts withTest
. - Define methods within your class whose names begin with
test
, and call JUnit's many assert methods to verify conditions; these methods will be executed when you runant test
. - You can run all the unit test with the command
ant test
, or you can run a specific unit test with the commandant -Dtestcase=<class name without package prefix> test
(for exampleant -Dtestcase=TestFileSystem test
)Understanding Ant
Hive is built by Ant, a Java building tool.
- You must implement a class that extends
- Good Ant tutorial: http://i-proving.ca/space/Technologies/Ant+Tutorial
Hadoop Dependencies
The Hive build downloads a number of different Hadoop versions via ivy in order to compile "shims" which allow for compatibility with these Hadoop versions. However, by default, the rest of Hive is only built and tested against a single Hadoop version (0.20.1 as of this writing, but check build.properties for the latest).
You can specify a different Hadoop version with -Dhadoop.version="<your-hadoop-version>". By default, Hadoop tarballs are pulled from http://mirror.facebook.net/facebook/hive-deps, which contains Hadoop 0.17.2.1, 0.18.3, 0.19.0, 0.20.0, 0.20.1, and 0.20S (aka 0.20.3-CDH3-SNAPSHOT). If the version you want is not here, then you'll need to set hadoop.mirror to a different source. For 0.19.2 and 0.20.2, you can use http://mirror.facebook.net/apache
or any other Apache mirror. For other versions, you'll need to use http://archive.apache.org/dist
(but don't use this unless you have to, since it's an overloaded server).
Trunk builds of Hive require Hadoop version at least 0.20.1; older versions are no longer supported.
Unit Tests
Please make sure that all unit tests succeed before and after applying your patch and that no new javac compiler warnings are introduced by your patch. Also see the information in the previous section about testing with different Hadoop versions if you want to verify compatibility with something other than the default Hadoop version.
> cd hive-trunk > ant clean package test tar -logfile ant.log
After a while, if you see
BUILD SUCCESSFUL
all is ok, but if you see
BUILD FAILED
then you should fix things before proceeding. Running
> ant testreport
and examining the HTML report in build/test
might be helpful (also build/ql/tmp/hive.log
).
Add a Unit Test
There are two kinds of unit tests in Hive:
- Normal unit test: These are used by testing a particular component of Hive.
- We just need to add a new class (name must start with "Test") in */src/test directory.
- We can run "ant test -Dtestcase=TestAbc" where TestAbc is the name of the new class. This will test only the new testcase, which will be faster than "ant test" which tests all testcases.
- A new query: If the new feature can be tested using Hive command line, we just need to add a new *.q file and a new *.q.out file:
- If the feature is added in ql
- Add a new XXXXXX.q file in ql/src/test/queries/clientpositive
- Run "ant test -Dtestcase=TestCliDriver -Dqfile=XXXXXX.q -Doverwrite=true -Dtest.silent=false". This will generate a new XXXXXX.q.out file in ql/src/test/results/clientpositive.
- If the feature is added in contrib
- Do the steps above, replacing "ql" with "contrib", and "TestCliDriver" with "TestContribCliDriver".
Debugging
- Do the steps above, replacing "ql" with "contrib", and "TestCliDriver" with "TestContribCliDriver".
- If the feature is added in ql
Please see Debugging Hive code in Development Guide.
Creating a patch
Check to see what files you have modified with:
svn stat
Add any new files with:
svn add .../MyNewClass.java svn add .../TestMyNewClass.java svn add .../XXXXXX.q svn add .../XXXXXX.q.out
In order to create a patch, type (from the base directory of hive):
svn diff > HIVE-1234.1.patch.txt
This will report all modifications done on Hive sources on your local disk and save them into the HIVE-1234.1.patch.txt file. Read the patch file. Make sure it includes ONLY the modifications required to fix a single issue.
If you are using Git instead of Subversion, it's important that you generate your patch using the following command:
git diff --no-prefix <commit> > HIVE-1234.1.patch.txt
Please do not:
- reformat code unrelated to the bug being fixed: formatting changes should be separate patches/commits.
- comment out code that is now obsolete: just remove it.
- insert comments around each change, marking the change: folks can use subversion to figure out what's changed and by whom.
- make things public which are not required by end users.
Please do:
- try to adhere to the coding style of files you edit;
- comment code whose function or rationale is not obvious;
- update documentation (e.g., package.html files, this wiki, etc.)
If you need to rename files in your patch:
- Write a shell script that uses 'svn mv' to rename the original files.
- Edit files as needed (e.g., to change package names).
- Create a patch file with 'svn diff --no-diff-deleted --notice-ancestry'.
- Submit both the shell script and the patch file.
This way other developers can preview your change by running the script and then applying the patch.
Updating a patch
For patch updates, our convention is to number them like HIVE-1856.1.patch.txt, HIVE-1856.2.patch.txt, etc. And then click the "Submit Patch" button again when a new one is uploaded; this makes sure it gets back into the review queue. Appending '.txt' to the patch file name makes it easy to quickly view the contents of the patch in a web browser.
Applying a patch
To apply a patch either you generated or found from JIRA, you can issue
patch -p0 < cool_patch.patch
if you just want to check whether the patch applies you can run patch with --dry-run option
patch -p0 --dry-run < cool_patch.patch
If you are an Eclipse user, you can apply a patch by : 1. Right click project name in Package Explorer , 2. Team -> Apply Patch
Review Process
If your patch contains more than 2 files that are changed and not yet had a review, please submit to Review Board. Some helpful hints for Review Board:
- When creating a review request you should select the "hive" repository and enter "trunk" in the "Base Directory" field (note that this value is interpreted relative to "http://svn.apache.org/repos/asf/hive/").
- If you're using Git instead of svn, you will need to select the
hive-git
repository.
- If you're using Git instead of svn, you will need to select the
- Add the JIRA ID to the Bugs field, e.g. "HIVE-1212".
- Add "hive" to the "Groups" field.
- This ensures that Review Board will send a notification email to the hive-dev list.
- Use JIRA name and the short title for the "Summary" Field (eg: HIVE-1234: fix X Y Z)
- You will need to click on the Publish button after you make a change.
- Please add a comment to the JIRA ticket with a link to your review request.
- Use the code review checklist as a guide when doing reviews
- Reviewers can approve the patch by clicking on the "Ship it" option.
- Should your patch earn a -1 review, set the issue status to 'Resume Progress', upload a patch to Review Board with necessary fixes and then set the status to 'Submit Patch' again.
NOTE: there was previously a rule about not uploading patches to JIRA until they had passed review board. We may adopt this eventually, but for now, keep uploading all patch versions to JIRA.
Contributing your work
Finally, patches should be attached to an issue report in JIRA via the Attach File link on the issue's JIRA. Please add a comment that asks for a code review following our code review checklist. Please note that the attachment should be granted license to ASF for inclusion in ASF works (as per the Apache License §5).
When you believe that your patch is ready to be committed, select the Submit Patch link on the issue's JIRA.
Folks should run ant clean package test
before selecting Submit Patch. Tests should all pass. If your patch involves performance optimizations, they should be validated by benchmarks that demonstrate an improvement.
If your patch creates an incompatibility with the latest major release, then you must set the Incompatible change flag on the issue's JIRA 'and' fill in the Release Note field with an explanation of the impact of the incompatibility and the necessary steps users must take.
If your patch implements a major feature or improvement, then you must fill in the Release Note field on the issue's JIRA with an explanation of the feature that will be comprehensible by the end user.
A committer should evaluate the patch within a few days and either: commit it; or reject it with an explanation.
Please be patient. Committers are busy people too. If no one responds to your patch after a few days, please make friendly reminders. Please incorporate other's suggestions into your patch if you think they're reasonable. Finally, remember that even a patch that is not committed is useful to the community.
Should your patch receive a "-1" select the Resume Progress on the issue's JIRA, upload a new patch with necessary fixes, and then select the Submit Patch link again.
Committers: for non-trivial changes, it is best to get another committer to review your patches before commit. Use Submit Patch link like other contributors, and then wait for a "+1" from another committer before committing. Please also try to frequently review things in the patch queue.
JIRA Guidelines
Please comment on issues in JIRA, making their concerns known. Please also vote for issues that are a high priority for you.
Please refrain from editing descriptions and comments if possible, as edits spam the mailing list and clutter JIRA's "All" display, which is otherwise very useful. Instead, preview descriptions and comments using the preview button (on the right) before posting them. Keep descriptions brief and save more elaborate proposals for comments, since descriptions are included in JIRA's automatically sent messages. If you change your mind, note this in a new comment, rather than editing an older comment. The issue should preserve this history of the discussion.
Generating Code
Some portions of the Hive code are generated by thrift. For most Hive changes, you don't need to worry about this, but if you end up needing to change the file metastore/if/hive_metastore.thrift
and/or service/if/hive_service.thrift
, then you'll also need to regenerate these files and submit their updated versions as part of your patch.
Here are the steps relevant to hive_metastore.thrift
:
- Don't make any changes to
hive_metastore.thrift
until instructed below. - Use the approved version of thrift. This is currently
thrift-0.5.0
, which you can obtain from http://thrift.apache.org/.
- Build the thrift compiler from its sources, then install it:
cd /path/to/thrift-0.5.0
./configure --without-csharp --without-ruby
make
sudo make install
- Before proceeding, verify that
which thrift
returns the build of thrift you just installed (typically/usr/local/bin
on Linux); if not, edit your PATH and repeat the verification. Also verify that the command 'thrift -version' returns the expected version number of Thrift. - Now you can run the ant 'thriftif' target to generate the Thrift code:
cd /path/to/hive-trunk/metastore
ant thriftif -Dthrift.home=/path/to/thrift-0.5.0
- Use
svn status
to verify that the code generation was a no-op, which should be the case if you have the correct thrift version and everyone has been following these instructions. If you can't figure out what is going wrong, ask for help from a committer. - Now make your changes to
hive_metastore.thrift
, and then run the compiler again: ant thriftif
- Now use
svn status
andsvn diff
to verify that the regenerated code corresponds only to the changes you made tohive_metastore.thrift
. You may also needsvn add
if new files were generated (andsvn remove
if files have been obsoleted). cd /path/to/hive-trunk
ant clean package
- Verify that hive is still working correctly with both embedded and remote metastore configurations.
If you are changing hive_service.thrift
, the steps should be similar. In both cases, you may also need to deal with recompiling C++ clients as well.
Metastore JDO Changes
For some changes, you may need to update the persistence specification in hive/metastore/src/model/package.jdo
, together with the various Java data access classes which it references. If you get this wrong (e.g. adding a new field in JDO without adding corresponding getters and setters in the Java class), you'll get an error like this at runtime:
FAILED: Error in metadata: javax.jdo.JDOFatalUserException: Class org.apache.hadoop.hive.metastore.model.MTable has field org.apache.hadoop.hive.metastore.model.MTable.someNewField declared in MetaData, but this field doesnt exist in the class!
You can quickly find all such mistakes as follows:
cd /path/to/hive-trunk/metastore ant model-enhance
Unfortunately, this ant task (which runs as part of the main build) does NOT cause the build to fail when an error is encountered, so you have to read through the output to verify that there are no errors remaining.
Note that persistence changes can have impact on the upgradability of existing deployed metastores, so you should undertake them carefully and discuss them on the mailing lists.
Stay involved
Contributors should join the Hive mailing lists. In particular the dev list (to join discussions of changes) and the user list (to help others).