Page History

...

Some portions of the Hive code are generated by Thrift. For most Hive changes, you don't need to worry about this, but if you modify any of the Thrift IDL files (e.g., standalone-metastore/if/metastore-common/src/main/thrift/hive_metastore.thrift and service-rpc/if/hive_serviceTCLIService.thrift), then you'll also need to regenerate these files and submit their updated versions as part of your patch.

...

1. Don't make any changes to hive_metastore.thrift until instructed below.
2. Use the approved version of Thrift. This is currently thrift-0.914.31, which you can obtain from http://thrift.apache.org/. (Or on Mac: brew install thrift@0.9, and no need to build, skip to step 4.)
3. Build the Thrift compiler from its sources, then install it:
   1. cd /path/to/thrift-0.9.3
   2. ./configure --without-csharp --without-ruby
   3. make
   4. sudo make install
4. Before proceeding, verify that which thrift returns the build of Thrift you just installed (typically /usr/local/bin on Linux); if not, edit your PATH and repeat the verification. Also verify that the command 'thrift -version' returns the expected version number of Thrift.

5. 1. For Mac via Homebrew (since the version we need is not available by default):

      Code Block
      language bash collapse true
      brew tap-new $USER/local-tap brew extract --version='0.14.1' thrift $USER/local-tap brew install thrift@0.14.1 mkdir -p /usr/local/share/fb303/if cp /usr/local/Cellar/thrift@0.14.1/0.14.1/share/fb303/if/fb303.thrift /usr/local/share/fb303/if

      
      

   2. For Mac, building from sources:

      Code Block
      language bash collapse true
      wget http://archive.apache.org/dist/thrift/0.14.1/thrift-0.14.1.tar.gz tar xzf thrift-0.14.1.tar.gz #If configure fails with "syntax error near unexpected token `QT5", then run "brew install pkg-config" ./bootstrap.sh sudo ./configure --with-openssl=/usr/local/Cellar/openssl@1.1/1.1.1j --without-erlang --without-nodejs --without-python --without-py3 --without-perl --without-php --without-php_extension --without-ruby --without-haskell --without-go --without-swift --without-dotnetcore --without-qt5 brew install openssl sudo ln -s /usr/local/opt/openssl/include/openssl/ /usr/local/include/ sudo make sudo make install mkdir -p /usr/local/share/fb303/if cp path/to/thrift-0.14.1/contrib/fb303/if/fb303.thrift /usr/local/share/fb303/if/fb303.thrift # or alternatively the following command curl -o /usr/local/share/fb303/if/fb303.thrift https://raw.githubusercontent.com/apache/thrift/master/contrib/fb303/if/fb303.thrift

      
      

   3. For Linux (not tested recently):

      Code Block
      language bash
      cd /path/to/thrift-0.14.1 ./configure --without-csharp --without-ruby make sudo make install cp

   Now you can run the Maven 'thriftif' profile to generate the Thrift code:
   1. cd /path/to/hive-trunk/

   2. mvn clean install -Pthriftif -DskipTests -Dthrift.home=/usr/local  -Phadoop-2

   3. If you see an error about fb303.thrift not being found, copy it to the appropriate directory and run above command again.On centOS/RHEL:
      cp

      /path/to/thrift-0.

      9

      14.

      3

      1/contrib/fb303/if/fb303.thrift /usr/local/share/fb303/if/fb303.thrift

      
      

6. Before proceeding, verify that which thrift returns the build of Thrift you just installed (typically /usr/local/bin on Linux); if not, edit your PATH and repeat the verification. Also verify that the command 'thrift -version' returns the expected version number of Thrift.

7. Now you can run the Maven 'thriftif' profile to generate the Thrift code:
   1. cd /path/to/hive/

   2. mvn clean install -Pthriftif -DskipTests -Dthrift.home=/usr/local

      share/fb303/if/fb303.thriftOn Mac with homebrew:
      $ cd /usr/local/Cellar/thrift@0.9/0.9.3 && mkdir -p share/fb303/if && cd share/fb303/if && curl -o fb303.thrift  https://raw.githubusercontent.com/apache/thrift/master/contrib/fb303/if/fb303.thrift

8. Verify that the code generation was a no-op, which should be the case if you have the correct Thrift version and everyone has been following these instructions. You may use git status or svn status for for the same. If you can't figure out what is going wrong, ask for help from a committer.
9. Now make your changes to hive_metastore.thrift, and then run the compiler again, from /path/to/hive-trunk/<hive_metastore.thrift's module>:

   1. mvn clean install -Pthriftif -DskipTests -Dthrift.home=/usr/local  -Phadoop-2

10. Now use svn status and svn diff or git status and git diff to verify that the regenerated code corresponds only to the changes you made to hive_metastore.thrift. You may also need svn add or need git add if new files were generated (and svn remove or git rm if some files have been are now obsoleted).
11. cd /path/to/hive-trunk
12. ant mvn clean package -DskiptTests (at the time of writing also "-Dmaven.javadoc.skip" is needed)
13. Verify that Hive is still working correctly with both embedded and remote metastore configurations.

...