Running the release verification script on each major platform

Linux and macOS

First, you must make sure your machine is configured as indicated below. Then, run the verification script either for the source release or the binary artifacts:

# this will create and automatically clean up a temporary directory for the verification environment and will run the source verification
TEST_DEFAULT=0 TEST_SOURCE=1 verify-release-candidate.sh $VERSION $RC_NUM

# this will create and automatically clean up a temporary directory for the verification environment and will run the binary verification
TEST_DEFAULT=0 TEST_BINARIES=1 dev/release/verify-release-candidate.sh 13.0.0 3

# to verify only certain implementations use the TEST_DEFAULT=0 and TEST_* variables
# here are a couple of examples, but see the source code for the available options
TEST_DEFAULT=0 TEST_SOURCE=1 TEST_CPP=1 verify-release-candidate.sh $VERSION $RC_NUM  # only C++ tests
TEST_DEFAULT=0 TEST_SOURCE=1 TEST_RUST=1 verify-release-candidate.sh $VERSION $RC_NUM  # only Rust tests
TEST_DEFAULT=0 TEST_SOURCE=1 TEST_PYTHON=1 verify-release-candidate.sh $VERSION $RC_NUM  # C++ and Python tests
TEST_DEFAULT=0 TEST_SOURCE=1 TEST_INTEGRATION_CPP=1 TEST_INTEGRATION_JAVA=1 verify-release-candidate.sh $VERSION $RC_NUM  # C++ and Java integration tests

# to verify certain binaries use the TEST_* variables as:
TEST_DEFAULT=0 TEST_WHEELS=1 verify-release-candidate.sh $VERSION $RC_NUM  # only Wheels
TEST_DEFAULT=0 TEST_APT=1 verify-release-candidate.sh $VERSION $RC_NUM  # only APT packages
TEST_DEFAULT=0 TEST_YUM=1 verify-release-candidate.sh $VERSION $RC_NUM  # only YUM packages
TEST_DEFAULT=0 TEST_JARS=1 verify-release-candidate.sh $VERSION $RC_NUM  # only JARS


Testing GPU Support

If you have an NVIDIA GPU and CUDA support configured, make nvidia-smi --list-gpus workable or set the environment variable ARROW_CUDA=ON to also run CUDA unit tests when running the release verification script.

Windows 10 (64-bit)

Configure Cmder (not necessary, but useful)

Control-click in Cmder and select "Settings". Then click on "Tasks". Create a new Task in Cmder to launch a command prompt configured with the correct Visual Studio compilation environment. The task command should look like:

cmd /k "%ConEmuDir%\..\init.bat"  -new_console 
cmd /k "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\Tools\VsDevCmd.bat" -arch=amd64

Another approach you may prefer is to add this to the Cmder startup, by adding the following to %CMDER_ROOT%\vendor\init.bat

"C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\Tools\VsDevCmd.bat" -arch=amd64

Run the release verification script

Download the source tarball from the SVN dist system that you wish to verify . 

dev\release\verify-release-candidate.bat %VERSION% %RC_NUM%


System Configuration Instructions

Ubuntu Linux (14.04 and higher)

Verifying on Ubuntu has a number of system dependencies

  • cmake 3.2 or higher
  • Misc apt-get dependencies (see below)
  • JDK8
  • Recent Apache Maven (3.3.9 and higher)
  • Recent Ruby (2.5.x will do), see instructions below on building from source
  • Bundler gem
  • NodeJS 11.12 and higher

Confirm that the following are in your path after following below steps and work:

  • mvn
  • npm
  • ruby
  • bundle
  • cmake

Install apt dependencies

sudo apt-get install -y cmake  # Do not use on Ubuntu 14.04
sudo apt-get install -y wget curl libboost-all-dev jq


# GLib build dependencies
sudo apt-get install -y -q autoconf-archive gtk-doc-tools libgirepository1.0-dev

On Ubuntu 14.04, it is necessary to install a newer CMake than what is in apt

CMAKE_PATH=cmake-3.12.0-Linux-x86_64.sh
wget https://cmake.org/files/v3.12/$CMAKE_PATH
sudo sh $CMAKE_PATH -- --skip-license --prefix=/usr
rm $CMAKE_PATH


Installing Oracle JDK8 and setting JAVA_HOME

The following will perform a headless install on Ubuntu 14.04 and higher:

sudo add-apt-repository ppa:webupd8team/java -y
sudo apt-get update -qq
echo "oracle-java8-installer shared/accepted-oracle-license-v1-1 select true" | sudo debconf-set-selections
sudo apt-get install -y oracle-java8-installer
sudo apt-get install -u oracle-java8-set-default

Installing Apache Maven

Download binary zip from https://maven.apache.org/download.cgi. This must be installed somewhere and added to your $PATH. For example:

MAVEN_VERSION=3.6.0
unzip apache-maven-$MAVEN_VERSION-bin.zip
mkdir -p $HOME/java
mv apache-maven-$MAVEN_VERSION $HOME/java
export PATH=$HOME/java/apache-maven-$MAVEN_VERSION/bin:$PATH

Install NodeJS

One easy way is to use nvm

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.34.0/install.sh | bash

Add the following to your bash profile

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # This loads nvm bash_completion

Install NodeJS 11

nvm install 11

Install Ruby

# Build and install Ruby from source
sudo apt-get build-dep -y ruby-defaults
RUBY_VERSION=2.6.2
wget https://cache.ruby-lang.org/pub/ruby/$(echo $RUBY_VERSION | sed -e 's/\.[0-9]*$//')/ruby-$RUBY_VERSION.tar.gz
tar xvf ruby-$RUBY_VERSION.tar.gz
pushd ruby-$RUBY_VERSION
CC=gcc ./configure --prefix=$HOME/ruby
CC=gcc make -j8
make install
popd
rm -rf ruby-$RUBY_VERSION
rm -rf ruby-$RUBY_VERSION.tar.gz

Remember to add this Ruby to your PATH in your Bash profile

# Add Ruby to PATH
export PATH=$HOME/ruby/bin:$PATH

macOS

Release verification requires at least one dependency you wouldn't otherwise need for development:

brew install gpg
# llvm@7 is not automatically pulled in thirdparty dependencies and the bison coming with macOS is too old
brew install bison llvm@7
  • No labels