Page History

...

Please refrain from editing descriptions and comments if possible, as edits spam the mailing list and clutter JIRA's "All" display, which is otherwise very useful. Instead, preview descriptions and comments using the preview button (on the right) before posting them. Keep descriptions brief and save more elaborate proposals for comments, since descriptions are included in JIRA's automatically sent messages. If you change your mind, note this in a new comment, rather than editing an older comment. The issue should preserve this history of the discussion.

Generating Thrift Code

Some portions of the Hive code are generated by thrift. For most Hive changes, you don't need to worry about this, but if you end up needing to change the file modify any of the Thrift IDL files (e.g. metastore/if/hive_metastore.thrift and /or service/if/hive_service.thrift), then you'll also need to regenerate these files and submit their updated versions as part of your patch.

...

1. Don't make any changes to hive_metastore.thrift until instructed below.
2. Use the approved version of thrift. This is currently thrift-0.59.0, which you can obtain from http://thrift.apache.org/.
3. Build the thrift compiler from its sources, then install it:
4. cd /path/to/thrift-0.59.0
5. ./configure --without-csharp --without-ruby
6. make
7. sudo make install
8. Before proceeding, verify that which thrift returns the build of thrift you just installed (typically /usr/local/bin on Linux); if not, edit your PATH and repeat the verification. Also verify that the command 'thrift -version' returns the expected version number of Thrift.
9. Now you can run the ant 'thriftif' target to generate the Thrift code:
10. cd /path/to/hive-trunk/metastore
11. ant thriftif -Dthrift.home=/path/to/thrift-0.59.0
12. Use svn status to verify that the code generation was a no-op, which should be the case if you have the correct thrift version and everyone has been following these instructions. If you can't figure out what is going wrong, ask for help from a committer.
13. Now make your changes to hive_metastore.thrift, and then run the compiler again:
14. ant thriftif
15. Now use svn status and svn diff to verify that the regenerated code corresponds only to the changes you made to hive_metastore.thrift. You may also need svn add if new files were generated (and svn remove if files have been obsoleted).
16. cd /path/to/hive-trunk
17. ant clean package
18. Verify that hive is still working correctly with both embedded and remote metastore configurations.

If you are changing hive_service.thrift, the steps should be similar. In both cases, you may also need to deal with recompiling C++ clients as well.

Generate thrift code in Mac OS

You can follow the instructions to generate thrift code in Mac OS. For example, you change metastore/if/hive_metastore.thrift and want to regenerate code.

1. Install boost which is used to compile thrift in Mac OS
   1. You can get the boost sources here: http://sourceforge.net/projects/boost/files/
   2. Download the bzip2 of boost and unpack it
   3. Go to the unpacked directory
   4. ./bootstrap.sh
   5. ./bjam
   6. ./bjam install
2. Install thrift
   1. Get approved thrift version, thrift-0.7.0, from http://incubator.apache.org/thrift/download/.
   2. Unpack the tgz file and change to the parent of thrift directory
   3. chmod -R 777 thrift-0.7.0
   4. cd thrift-0.7.0
   5. ./configure
   6. make
   7. make install
   8. Download http://svn.apache.org/repos/asf/thrift/branches/0.7.x/contrib/fb303/if/fb303.thrift to /usr/local/share/fb303/if
   9. cd <hive-trunk>/metastore/
   10. ant thriftif -Dthrift.home=/usr/local

Notes:

1. Some of the commands may need "sudo" mode. Please use it accordingly.
2. It assumes thrift is installed in /usr/local/bin/thrift. Otherwise, please change it accordingly.
3. MACPORTS can install thrift easily in Mac OS. But its thrift version is 0.6.0 as June 01 2012 so that we can't use it. Once it supports 0.7.0, MACPORTS is a good choice.

Metastore JDO Changes

For some changes, you may need to update the persistence specification in hive/metastore/src/model/package.jdo, together with the various Java data access classes which it references. If you get this wrong (e.g. adding a new field in JDO without adding corresponding getters and setters in the Java class), you'll get an error like this at runtime:

FAILED: Error in metadata: javax.jdo.JDOFatalUserException: Class org.apache.hadoop.hive.metastore.model.MTable has field org.apache.hadoop.hive.metastore.model.MTable.someNewField declared in MetaData, but this field doesnt exist in the class!

You can quickly find all such mistakes as follows:

cd /path/to/hive-trunk/metastore
ant model-enhance

Unfortunately, this ant task (which runs as part of the main build) does NOT cause the build to fail when an error is encountered, so you have to read through the output to verify that there are no errors remaining.

...

1. .

Stay involved

Contributors should join the Hive mailing lists. In particular the dev list (to join discussions of changes) and the user list (to help others).

...