This page documents how to build https://github.com/cloudera/native-toolchain from scratch and use it with Impala.

This is for advanced use only. It is usually easier to use prebuilt binary packages that are downloaded by Impala's bootstrap_toolchain.py script. This instructions are useful in a few scenarios:

  • You want to build Impala purely from source without relying on prebuilt binaries.
  • You are trying to build Impala on a platform for which prebuilt binaries are not available.

  • You want to add or modify a dependency in native-toolchain. E.g. build Impala with a newer version of LLVM, or with different configuration options.

Prerequisites

A number of prerequisites are required to build the full toolchain.

Below I have documented the requirements for Ubuntu 16.04 along with commands to install it. Package names and other details may vary between operating systems and versions. Also, if a package fails to build and you take steps to fix it, you might want to remove the old built directory in "$SOURCE_DIR/source/<package-name>/" before you retry building it again.

# General build requirements
sudo apt-get install build-essential git
sudo apt-get install texinfo

# Requirements for specific packages
sudo apt-get install bison # For binutils
sudo apt-get install autoconf automake libtool # For libevent
sudo apt-get install libz-dev # For OpenSSL
sudo apt-get install libssl-dev # For Thrift
sudo apt-get install libncurses-dev # For GDB
sudo apt-get install libsasl2-dev libkrb5-dev # For Kudu

Building

# Checkout the toolchain
git clone https://github.com/cloudera/native-toolchain.git
cd native-toolchain

# This may take a long time! See check/ for detailed logs.
./buildall.sh

Using custom toolchain with Impala

Starting with Impala 4.3.0, you can use a custom native toolchain with

# Set a path for the toolchain. If this directory doesn't exist, Impala's buildall.sh script will download and build 
# native-toolchain according to details in impala-config.sh.
export NATIVE_TOOLCHAIN_HOME=/path/to/native-toolchain

# Start the Impala build. This will (re)build native-toolchain and use the results in ${NATIVE_TOOLCHAIN_HOME}/build.
cd ${IMPALA_HOME}
. bin/impala-config.sh
./buildall.sh <args>

For Impala checkouts prior to 4.3.0, use

# Assume that ${IMPALA_HOME} has the Impala source checkout and ${NATIVE_TOOLCHAIN_HOME} has the native-toolchain source checkout
cd ${IMPALA_HOME}
# Symlink the built packages into the toolchain/ subdirectory
(mkdir -p toolchain && cd toolchain && ln -s ${NATIVE_TOOLCHAIN_HOME}/build/* .)

# Disable toolchain bootstrapping, which could replace our binaries with precompiled ones.
# Adding this to impala-config-local.sh makes it permanent for this Impala checkout.
echo "export SKIP_TOOLCHAIN_BOOTSTRAP=true" >> bin/impala-config-local.sh
. bin/impala-config.sh
# Optional: ${IMPALA_HOME}/toolchain is the default value for IMPALA_TOOLCHAIN. You may want to add this to your .bashrc file.
export IMPALA_TOOLCHAIN=${IMPALA_HOME}/toolchain

 # Build Impala - this will pick up the new toolchain.
./buildall.sh <args>
  • No labels