Skip to end of metadata
Go to start of metadata

How to Contribute to Apache Hive

This page describes the mechanics of how to contribute software to Apache Hive. For ideas about what you might contribute, please see open tickets in Jira.

Mavenization is complete

Icon

Hive now uses Maven as a build tool as opposed to ant.

Getting the Source Code

First of all, you need the Hive source code.

Get the source code on your local drive using SVN. Most development is done on the "trunk":

You also have the option of using one of the Git mirrors of the SVN repository:

or

Setting Up Eclipse Development Environment (Optional)

This is an optional step. Eclipse has a lot of advanced features for Java development, and it makes the life much easier for Hive developers as well.

How do I import into eclipse?

Becoming a Contributor

This checklist tells you how to create accounts and obtain permissions needed by Hive contributors.  See the Hive website for additional information.

  • Create an Apache Software Foundation JIRA account if you do not already have one:  Sign Up for JIRA.
    • The ASF JIRA system dashboard is here.
    • The Hive JIRA is here.
  • Send a message to the Hive developer mailing list (dev@hive.apache.org) asking to be included as a Hive contributor, and provide your JIRA username.  This makes it possible to assign JIRA tickets to you.
  • To review patches for JIRA tickets, use the Review Board.  If you need an account, register here.  See Review Process below for more information.
    • All Hive patches posted for review are listed here.
    • Individual JIRA tickets provide a link to the issue on the review board when a review request has been made.
    • For simple reviews, you can just read the patch attached to the JIRA ticket and post a comment.
  • To contribute to the Hive wiki, follow the instructions in About This Wiki.
  • To edit the Hive website, follow the instructions in How to edit the website.
  • Join the Hive mailing lists to receive email about issues and discussions.

Making Changes

Before you start, send a message to the Hive developer mailing list, or file a bug report in JIRA. Describe your proposed changes and check that they fit in with what others are doing and have planned for the project. Be patient, it may take folks a while to understand your requirements.

Modify the source code and add some features using your favorite IDE.

Coding Conventions

Please take care about the following points.

  • All public classes and methods should have informative Javadoc comments.
    • Do not use @author tags.
  • Code should be formatted according to Sun's conventions, with two exceptions:
    • Indent two (2) spaces per level, not four (4).
    • Line length limit is 100 chars, instead of 80 chars.
  • Contributions should not introduce new Checkstyle violations.
    • Check for new Checkstyle violations by running mvn checkstyle:checkstyle-aggregate, and then inspect the results in the target/site directory.
    • If you use Eclipse you should install the eclipse-cs Checkstyle plugin. This plugin highlights violations in your code and is also able to automatically correct some types of violations.
  • Contributions should pass existing unit tests.
  • New unit tests should be provided to demonstrate bugs and fixes. JUnit is our test framework:
    • You must implement a class that extends junit.framework.TestCase and whose class name starts with Test.
    • Define methods within your class whose names begin with test, and call JUnit's many assert methods to verify conditions; these methods will be executed when you run mvn test.
    • You can run all the unit tests with the command mvn test -Phadoop-1, or you can run a specific unit test with the command mvn test -Phadoop-1 -Dtest=<class name without package prefix> (for example:  mvn test -Phadoop-1 -Dtest=TestFileSystem).

Understanding Maven

Hive is a multi-module Maven project. If you are new to Maven, the articles below may be of interest:

Additionally, Hive actually has two projects, "core" and "itests". The reason that itests is not connected to the core reactor is that itests requires the packages to be built.

The actual Maven commands you will need are discussed on the HiveDeveloperFAQ page.

Hadoop Dependencies

The Hive build downloads a number of different Hadoop versions via Maven in order to compile "shims" which allow for compatibility with these Hadoop versions. However, by default, the rest of Hive is only built and tested against a single Hadoop version (1.2.1 as of this writing, but check pom.xml for the latest).

The Maven build has two profiles, one for Hadoop 1 (0.20 and 1.X) and one for Hadoop 2 (2.X). By default the hadoop-1 profile is used; to use the hadoop-2 profile just specify "-Phadoop-2".

Trunk builds of Hive require Hadoop version at least 0.20.1; older versions are no longer supported.

Unit Tests

Please make sure that all unit tests succeed before and after applying your patch and that no new javac compiler warnings are introduced by your patch. Also see the information in the previous section about testing with different Hadoop versions if you want to verify compatibility with something other than the default Hadoop version.

When submitting a patch it's highly recommended you execute tests locally which you believe will be impacted in addition to any new tests. The full test suite can be executed by Hive PreCommit Patch Testing. Hive Developer FAQ describes how to execute a specific set of tests.

After a while, if you see

all is ok, but if you see

then you should fix things before proceeding.

Unit tests take a long time (several hours) to run sequentially even on a very fast machine; for information on how to run them in parallel, see Hive PreCommit Patch Testing.

Add a Unit Test

There are two kinds of unit tests in Hive:

  • Normal unit test: These are used by testing a particular component of Hive.
    • We just need to add a new class (name must start with "Test") in */src/test directory.
    • We can run "mvn test -Dtest=TestAbc -Phadoop-1" where TestAbc is the name of the new class. This will test only the new testcase, which will be faster than "mvn test -Phadoop-1" which tests all testcases.
  • A new query: If the new feature can be tested using Hive command line, we just need to add a new *.q file and a new *.q.out file:
    • If the feature is added in ql
      • Add a new XXXXXX.q file in ql/src/test/queries/clientpositive.
      • Run "mvn test -Phadoop-1 -Dtest=TestCliDriver -Dqfile=XXXXXX.q -Dtest.output.overwrite=true". This will generate a new XXXXXX.q.out file in ql/src/test/results/clientpositive.
        • If you want to run multiple .q files in the test run, you can specify comma separated .q files, for example -Dqfile="X1.q,X2.q". You can also specify a Java regex, for example -Dqfile_regex='join.*'. (Note that it takes Java regex, i.e., 'join.*' and not 'join*'). The regex match first removes the .q from the file name before matching regex, so specifying "join*.q" will not work.
      • If you are using hive-0.11.0 or later, you can specify "-Dmodule=ql".
    • If the feature is added in contrib
      • Do the steps above, replacing "ql" with "contrib", and "TestCliDriver" with "TestContribCliDriver".
      • If you are using hive-0.11.0 or later, you can specify "-Dmodule=contrib".

Debugging

Please see Debugging Hive code in Development Guide.

Creating a Patch

Check to see what files you have modified with:

Add any new files with:

In order to create a patch, type (from the base directory of Hive):

This will report all modifications done on Hive sources on your local disk and save them into the HIVE-1234.1.patch file. Read the patch file. Make sure it includes ONLY the modifications required to fix a single issue.

If you are using Git instead of Subversion, it's important that you generate your patch using the following command:

Please do not:

  • reformat code unrelated to the bug being fixed: formatting changes should be separate patches/commits.
  • comment out code that is now obsolete: just remove it.
  • insert comments around each change, marking the change: folks can use subversion to figure out what's changed and by whom.
  • make things public which are not required by end users.

Please do:

  • try to adhere to the coding style of files you edit;
  • comment code whose function or rationale is not obvious;
  • update documentation (e.g., package.html files, hive-default.xml.template, this wiki, etc.)

If you need to rename files in your patch:

  1. Write a shell script that uses 'svn mv' to rename the original files.
  2. Edit files as needed (e.g., to change package names).
  3. Create a patch file with 'svn diff --no-diff-deleted --notice-ancestry'.
  4. Submit both the shell script and the patch file.

This way other developers can preview your change by running the script and then applying the patch.

Attaching and Submitting a Patch

This section only gives the basic procedures for attaching and submitting a patch.  See Contributing Your Work below for more information.

  1. Attach the patch file to a JIRA ticket: in the ticket's "More" tab, select "Attach Files" and use "Choose File" to upload the file, then add a descriptive comment.
  2. Put the patch in the review queue: click the "Submit Patch" button. The button name will change to "Cancel Patch" and the ticket status will change to Patch Available.

Updating a Patch

For patch updates, our convention is to number them like HIVE-1856.1.patch, HIVE-1856.2.patch, etc. And then click the "Submit Patch" button again when a new one is uploaded; this makes sure it gets back into the review queue.

Applying a Patch

To apply a patch that you either generated or found from JIRA, you can issue:

If you just want to check whether the patch applies you can run patch with --dry-run option:

If you are an Eclipse user, you can apply a patch by:

  1. Right click project name in Package Explorer.
  2. Team -> Apply Patch.

Review Process

See Review Board for instructions.

  • Use Hadoop's code review checklist as a rough guide when doing reviews.
  • In JIRA, use 'Submit Patch' to get your review request into the queue.
  • If a committer requests changes, set the issue status to 'Resume Progress', then once you're ready, submit an updated patch with necessary fixes and then request another round of review with 'Submit Patch' again.
  • Once your patch is accepted, be sure to upload a final version which grants rights to the ASF.

Contributing Your Work

Finally, patches should be attached to an issue report in JIRA via the Attach File link on the issue's JIRA. Please add a comment that asks for a code review. Please note that the attachment should be granted license to ASF for inclusion in ASF works (as per the Apache License).

When you believe that your patch is ready to be committed, select the Submit Patch link on the issue's JIRA. Unit tests will run automatically if the file is named according to the naming standards. See Hive PreCommit Patch TestingTests should all pass. If your patch involves performance optimizations, they should be validated by benchmarks that demonstrate an improvement.

If your patch creates an incompatibility with the latest major release, then you must set the Incompatible change flag on the issue's JIRA and fill in the Release Note field with an explanation of the impact of the incompatibility and the necessary steps users must take.

If your patch implements a major feature or improvement, then you must fill in the Release Note field on the issue's JIRA with an explanation of the feature that will be comprehensible by the end user.

The Release Note field can also document changes in the user interface (such as new HiveQL syntax or configuration parameters) prior to inclusion in the wiki documentation.

A committer should evaluate the patch within a few days and either: commit it; or reject it with an explanation.

Please be patient. Committers are busy people too. If no one responds to your patch after a few days, please make friendly reminders. Please incorporate others' suggestions into your patch if you think they're reasonable. Finally, remember that even a patch that is not committed is useful to the community.

Should your patch receive a "-1" select Resume Progress on the issue's JIRA, upload a new patch with necessary fixes, and then select the Submit Patch link again.

Committers: for non-trivial changes, it is best to get another committer to review your patches before commit. Use the Submit Patch link like other contributors, and then wait for a "+1" from another committer before committing. Please also try to frequently review things in the patch queue.

JIRA Guidelines

Please comment on issues in JIRA, making your concerns known. Please also vote for issues that are a high priority for you.

Please refrain from editing descriptions and comments if possible, as edits spam the mailing list and clutter JIRA's "All" display, which is otherwise very useful. Instead, preview descriptions and comments using the preview button (icon below the comment box) before posting them. Keep descriptions brief and save more elaborate proposals for comments, since descriptions are included in JIRA's automatically sent messages. If you change your mind, note this in a new comment, rather than editing an older comment. The issue should preserve this history of the discussion.

Generating Thrift Code

Some portions of the Hive code are generated by Thrift. For most Hive changes, you don't need to worry about this, but if you modify any of the Thrift IDL files (e.g. metastore/if/hive_metastore.thrift and service/if/hive_service.thrift), then you'll also need to regenerate these files and submit their updated versions as part of your patch.

Here are the steps relevant to hive_metastore.thrift:

  1. Don't make any changes to hive_metastore.thrift until instructed below.
  2. Use the approved version of Thrift. This is currently thrift-0.9.0, which you can obtain from http://thrift.apache.org/.
  3. Build the Thrift compiler from its sources, then install it:
    1. cd /path/to/thrift-0.9.0
    2. ./configure --without-csharp --without-ruby
    3. make
    4. sudo make install
  4. Before proceeding, verify that which thrift returns the build of Thrift you just installed (typically /usr/local/bin on Linux); if not, edit your PATH and repeat the verification. Also verify that the command 'thrift -version' returns the expected version number of Thrift.
  5. Now you can run the Maven 'thriftif' profile to generate the Thrift code:
    1. cd /path/to/hive-trunk/
    2. mvn clean install -Phadoop-1,thriftif -DskipTests -Dthrift.home=/usr/local
    3. If you see an error about fb303.thrift not being found, copy it to the appropriate directory and run above command again. On centOS/RHEL:
      cp /path/to/thrift-0.9.0/contrib/fb303/if/fb303.thrift /usr/local/share/fb303/if/fb303.thrift
  6. Use svn status to verify that the code generation was a no-op, which should be the case if you have the correct Thrift version and everyone has been following these instructions. If you can't figure out what is going wrong, ask for help from a committer.
  7. Now make your changes to hive_metastore.thrift, and then run the compiler again:
    1. mvn clean install -Phadoop-1,thriftif -DskipTests -Dthrift.home=/usr/local
  8. Now use svn status and svn diff to verify that the regenerated code corresponds only to the changes you made to hive_metastore.thrift. You may also need svn add if new files were generated (and svn remove if files have been obsoleted).
  9. cd /path/to/hive-trunk
  10. ant clean package
  11. Verify that Hive is still working correctly with both embedded and remote metastore configurations.

Stay Involved

Contributors should join the Hive mailing lists. In particular the dev list (to join discussions of changes) and the user list (to help others).

See Also

Labels
  • No labels