Page History

...

• All public classes and methods should have informative Javadoc comments.
  • Do not use @author tags.
• Code should be formatted according to Sun's conventions, with two exceptions:
  • Indent two (2) spaces per level, not four (4).
  • Line length limit is 100 chars, instead of 80 chars.
• Contributions should not introduce new Checkstyle violations.
  • Check for new Checkstyle violations by running mvn checkstyle:checkstyle-aggregate, and then inspect the results in the target/site directory.
  • If you use Eclipse you should install the eclipse-cs Checkstyle plugin. This plugin highlights violations in your code and is also able to automatically correct some types of violations.
• Contributions should pass existing unit tests.
• New unit tests should be provided to demonstrate bugs and fixes. JUnit is our test framework:
  • You must implement a class that extends junit.framework.TestCase and whose class name starts with Test.
  • Define methods within your class whose names begin with test, and call JUnit's many assert methods to verify conditions; these methods will be executed when you run mvn test.
  • You can run all the unit tests with the command mvn test -Phadoop-1, or you can run a specific unit test with the command mvn test -Phadoop-1 -Dtest=<class name without package prefix> (for example:  mvn test -Phadoop-1 -Dtest=TestFileSystem).

Understanding Maven

...

For general information about Hive branches, see Hive Versions and Branches.

Hadoop Dependencies

Hadoop dependencies are handled differently in master and branch-1.

branch-1

In branch-1 both Hadoop 1.x and 2.x are supported.  The The Hive build downloads a number of different Hadoop versions via Maven in order to compile "shims" which allow for compatibility with these Hadoop versions. However, by default, the rest of Hive is only built and tested against a single Hadoop version (1.2.1 as of this writing, but check pom.xml for the latest).

The Maven build has two profiles, one for Hadoop 1 (0.20 and 1.X) and one for Hadoop 2 (2.X). By default the hadoop-1 profile is used; to use the hadoop-2 profile just specify "-Phadoop-2".

...

.

The Maven build has two profiles, hadoop-1 for Hadoop 1.x and hadoop-2 for Hadoop 2.x. When building, you must specify which profile (via maven's -P command line option) you wish to use.

master

Hadoop 1.x is no longer supported in Hive's master branch.  There is no need to specify a profile for most maven commands, as Hadoop 2.x will always be chosen.

Unit Tests

Please make sure that all unit tests succeed before and after applying your patch and that no new javac compiler warnings are introduced by your patch. Also see the information in the previous section about testing with different Hadoop versions if you want to verify compatibility with something other than the default Hadoop version.

...

> cd hive-trunk
> mvn clean install -DskipTests -Phadoop-1
> mvn test -Dtest=SomeTest -Phadoop-1

After a while, if you see

...

• Normal unit test: These are used by testing a particular component of Hive.
  • We just need to add a new class (name must start with "Test") in */src/test directory.
  • We can run "mvn test -Dtest=TestAbc -Phadoop-1" where TestAbc is the name of the new class. This will test only the new testcase, which will be faster than "mvn test -Phadoop-1" which tests all testcases.
• A new query: If the new feature can be tested using Hive command line, we just need to add a new *.q file and a new *.q.out file:
  • If the feature is added in ql
    • Add a new XXXXXX.q file in ql/src/test/queries/clientpositive.
    • Run "mvn test -Phadoop-1 -Dtest=TestCliDriver -Dqfile=XXXXXX.q -Dtest.output.overwrite=true". This will generate a new XXXXXX.q.out file in ql/src/test/results/clientpositive.
      • If you want to run multiple .q files in the test run, you can specify comma separated .q files, for example -Dqfile="X1.q,X2.q". You can also specify a Java regex, for example -Dqfile_regex='join.*'. (Note that it takes Java regex, i.e., 'join.*' and not 'join*'). The regex match first removes the .q from the file name before matching regex, so specifying "join*.q" will not work.
    • If you are using hive-0.11.0 or later, you can specify "-Dmodule=ql".
  • If the feature is added in contrib
    • Do the steps above, replacing "ql" with "contrib", and "TestCliDriver" with "TestContribCliDriver".
    • If you are using hive-0.11.0 or later, you can specify "-Dmodule=contrib".

...

1. Don't make any changes to hive_metastore.thrift until instructed below.
2. Use the approved version of Thrift. This is currently thrift-0.9.0, which you can obtain from http://thrift.apache.org/.
3. Build the Thrift compiler from its sources, then install it:
   1. cd /path/to/thrift-0.9.0
   2. ./configure --without-csharp --without-ruby
   3. make
   4. sudo make install
4. Before proceeding, verify that which thrift returns the build of Thrift you just installed (typically /usr/local/bin on Linux); if not, edit your PATH and repeat the verification. Also verify that the command 'thrift -version' returns the expected version number of Thrift.
5. Now you can run the Maven 'thriftif' profile to generate the Thrift code:
   1. cd /path/to/hive-trunk/

   2. mvn clean install -Phadoop-1,thriftifPthriftif -DskipTests -Dthrift.home=/usr/local

   3. If you see an error about fb303.thrift not being found, copy it to the appropriate directory and run above command again. On centOS/RHEL:
      cp /path/to/thrift-0.9.0/contrib/fb303/if/fb303.thrift /usr/local/share/fb303/if/fb303.thrift
6. Use svn status to verify that the code generation was a no-op, which should be the case if you have the correct Thrift version and everyone has been following these instructions. If you can't figure out what is going wrong, ask for help from a committer.
7. Now make your changes to hive_metastore.thrift, and then run the compiler again:

   1. mvn clean install -Phadoop-1,thriftifPthriftif -DskipTests -Dthrift.home=/usr/local

8. Now use svn status and svn diff to verify that the regenerated code corresponds only to the changes you made to hive_metastore.thrift. You may also need svn add if new files were generated (and svn remove if files have been obsoleted).
9. cd /path/to/hive-trunk
10. ant clean package
11. Verify that Hive is still working correctly with both embedded and remote metastore configurations.

...