This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Panel


Tabs Container
idbuild-tabs
directionhorizontal


Tabs Page
idmacos
titleMacOS (10.10+)

Install dependencies on MAC (with xcode installed)

Make sure you have done: xcode-select --install to install developer tools

Code Block
languagebash
brew install protobuf protobuf-c Gsasl boost json-c ccache snappy libyaml libevent python cmake lcov apr iproute2mac

brew install thrift
# make sure thrift version = 0.9, otherwise must backport the higher version to 0.9 by the following command:
# brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/9d524e4850651cfedd64bc0740f1379b533f607d/Formula/thrift.rb

# make sure json-c version <= 0.12.1, otherwise must backport the higher version to 0.12.1 by the following command:
# brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/4a0b27e3556edec55a432b1ecbe1aaa18fc1fccd/Formula/json-c.rb

brew install openssl
brew link openssl apr --force
# set openssl path, make sure version >=1.0.2
export DEPENDENCY_INSTALL_PREFIX=/usr/local/opt/openssl
 
brew cask install java
# make sure jdk version is 7 or 8, otherwise must get java by the following website:
# http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

brew install maven
# need tomcat6 if enable-rps
brew install tomcat@6
 
sudo easy_install pip
sudo pip install pycrypto

brew install cpanm
sudo cpanm install JSON

# install libesmtp manually
# get libesmtp from http://linuxfromscratch.org/blfs/view/svn/general/libesmtp.html
tar jxvf libesmtp-1.0.6.tar.bz2
cd libesmtp-1.0.6
./configure && make
sudo make install

Please refer to section below titled Running catalog tidycat perl modules for installing perl-JSON module on MAC/

Info
titleNote for Installing Dependencies
  • some dependencies require brew install <packagename> --universal, if you see HAWQ complains about an already installed package is required, try that.
  • El Capitan issues: boost cannot be installed as --universal (command shell will hang), you need follow manul steps http://www.boost.org/doc/libs/1_61_0/more/getting_started/unix-variants.html#easy-build-and-install
  • bison version on local MAC is too high.  try bison --version to look at the version, if it is later than 3.0, try install bison with a lower version, such as 2.3. And rebuild HAWQ.
  • If you see openssl error, try:

    brew link --force openssl

OS requirement

Use a text editor to edit the /etc/sysctl.conf file. Add or edit each of the following parameter definitions to set the required value. 

       kern.sysv.shmmax=2147483648

       kern.sysv.shmmin=1

        kern.sysv.shmmni=64

        kern.sysv.shmseg=16

        kern.sysv.shmall=524288

        kern.maxfiles=65535

        kern.maxfilesperproc=65536

  • Reboot to apply the change.

Install Xcode and command line tools

After install/update xcode, please run ‘xcode-select --install’ to install command line tools, and then open xcode to make sure you have already installed it. 

MUST: Turning Off Rootless System Integrity Protection in OS X El Capitan 10.11+

If not do this, you may encounter some tricky LIBRARY_PATH problems. e.g. HAWQ-513

Following below instructions: ( refer to http://osxdaily.com/2015/10/05/disable-rootless-system-integrity-protection-mac-os-x )

  1. Reboot the Mac and hold down Command + R keys simultaneously after you hear the startup chime, this will boot OS X into Recovery Mode
  2. When the “OS X Utilities” screen appears, pull down the ‘Utilities’ menu at the top of the screen instead, and choose “Terminal”
  3. Type the following command into the terminal then hit return: csrutil disable; reboot


Tabs Page
idtab-yum
titleRed Hat/CentOS 7.X

Install dependencies on Rad Hat/CentOS 7.X

Dependencies

Code Block
languagebash
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# For CentOs 7 the link is https://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-9.noarch.rpm
rpm -ivh epel-release-latest-7.noarch.rpm
yum makecache
# On redhat7, make sure enabled rhel-7-server-extras-rpms and rhel-7-server-optional-rpms channel in /etc/yum.repos.d/redhat.repo
# Otherwise yum will prompt some packages(e.g. gperf) not be found
yum install -y man passwd sudo tar which git mlocate links make bzip2 net-tools \
  autoconf automake libtool m4 gcc gcc-c++ gdb bison flex gperf maven indent \
  libuuid-devel krb5-devel libgsasl-devel expat-devel libxml2-devel \
  perl-ExtUtils-Embed pam-devel python-devel libcurl-devel snappy-devel \
  thrift-devel libyaml-devel libevent-devel bzip2-devel openssl-devel \
  openldap-devel protobuf-devel readline-devel net-snmp-devel apr-devel \
  libesmtp-devel python-pip json-c-devel \
  java-1.7.0-openjdk-devel lcov cmake3 \
  openssh-clients openssh-server perl-JSON perl-Env

# need tomcat6 if enable-rps
# download from http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.44/

ln -s /usr/bin/cmake3 /usr/bin/cmake
pip --retries=50 --timeout=300 install pycrypto

 

OS requirement

  • use a text editor to edit the /etc/sysctl.conf file. Add or edit each of the following parameter definitions to set the required value. 

     kernel.shmmax = 1000000000
     kernel.shmmni = 4096
     kernel.shmall = 4000000000
     kernel.sem = 250 512000 100 2048
     kernel.sysrq = 1
     kernel.core_uses_pid = 1
     kernel.msgmnb = 65536
     kernel.msgmax = 65536
     kernel.msgmni = 2048
     net.ipv4.tcp_syncookies = 0
     net.ipv4.conf.default.accept_source_route = 0
     net.ipv4.tcp_tw_recycle = 1
     net.ipv4.tcp_max_syn_backlog = 200000
     net.ipv4.conf.all.arp_filter = 1
     net.ipv4.ip_local_port_range = 1281 65535
     net.core.netdev_max_backlog = 200000
     vm.overcommit_memory = 2
     fs.nr_open = 3000000
     kernel.threads-max = 798720
     kernel.pid_max = 798720
     # increase network
     net.core.rmem_max=2097152
     net.core.wmem_max=2097152
  • Execute the following command to apply your updated /etc/sysctl.conf file to the operating system configuration:
    sysctl -p
  • Use a text editor to edit the /etc/security/limits.conf file. Add the following definitions in the exact order that they are listed
    ( Please make sure fs.nr_open = 3000000 is applied before edit limits.conf, else you may not able to ssh to your instance.)
     * soft nofile 2900000
     * hard nofile 2900000
     * soft nproc 131072
     * hard nproc 131072
  • Bison version: run bison --version to look at the version, if it is later than 3.0, try install bison with a lower version, such as 2.3. And rebuild HAWQ.


Tabs Page
titleRed Hat/CentOS 6.X
classtabs-compile-dependencies

Build dependencies yourself ( tested on Red Hat 6.X).

Dependencies

There are several dependencies (see the following table) you must install before building HAWQ. To build Apache HAWQ, gcc and some dependencies are needed. The libraries are tested on the given versions. Most of the dependencies can be installed through yum. Other dependencies should be installed through the source tarball. Typically you can use "./configure && make && make install" to install from source tarball.

Libraries that must be installed using source tarball.


 You might need to run "ldconfig -p <LIBRARY_INSTALL_PATH>" after installing them.

For thrift build, you might need "--without-tests" for configure.

Install maven:
sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo
sudo yum install -y apache-maven

Install pip:

wget https://bootstrap.pypa.io/get-pip.py

python get-pip.py

 

pip --retries=50 --timeout=300 install pycrypto
 

Libraries that can be installed through yum.

NameVersion
epel-release6-8
make
3.81
gperf
3.0.4
snappy-devel
1.1.3
bzip2-devel 
1.0.6
python-devel 
2.6.2
libevent-devel
1.4.6
krb5-devel
1.11.3
libuuid-devel
2.26.2
libgsasl-devel
1.8.0
libxml2-devel 
2.7.8
zlib-devel
1.2.3
readline-devel
6
openssl-devel
>=1.0.2
bison
1.875
apr-devel
1.2.12
libyaml-devel
0.1.1
flex
>2.5.4
lcov1.12
libesmtp-devel1.0.4
perl-JSON2.15
tomcat6.0.44

Default version of gcc in Red Hat/CentOS 6.X is 4.4.7 or lower, you can quickly upgrade gcc following instructions below:

Code Block
languagebash
cd /etc/yum.repos.d
# make sure you have root permission
wget -O /etc/yum.repos.d/slc6-devtoolset.repo http://linuxsoft.cern.ch/cern/devtoolset/slc6-devtoolset.repo
# install higher version using devtoolset-2
yum install devtoolset-2-gcc devtoolset-2-binutils devtoolset-2-gcc-c++
# Start using software collections
scl enable devtoolset-2 bash

You will need to install python packages same as those which are required for Red Hat/CentOS 7.

OS requirement

  • use a text editor to edit the /etc/sysctl.conf file. Add or edit each of the following parameter definitions to set the required value. 

     kernel.shmmax = 1000000000
     kernel.shmmni = 4096
     kernel.shmall = 4000000000
     kernel.sem = 250 512000 100 2048
     kernel.sysrq = 1
     kernel.core_uses_pid = 1
     kernel.msgmnb = 65536
     kernel.msgmax = 65536
     kernel.msgmni = 2048
     net.ipv4.tcp_syncookies = 0
     net.ipv4.conf.default.accept_source_route = 0
     net.ipv4.tcp_tw_recycle = 1
     net.ipv4.tcp_max_syn_backlog = 200000
     net.ipv4.conf.all.arp_filter = 1
     net.ipv4.ip_local_port_range = 1281 65535
     net.core.netdev_max_backlog = 200000
     vm.overcommit_memory = 2
     fs.nr_open = 3000000
     kernel.threads-max = 798720
     kernel.pid_max = 798720
     # increase network
     net.core.rmem_max=2097152
     net.core.wmem_max=2097152
  • Execute the following command to apply your updated /etc/sysctl.conf file to the operating system configuration:
    sysctl -p
  • Use a text editor to edit the /etc/security/limits.conf file. Add the following definitions in the exact order that they are listed
     * soft nofile 2900000
     * hard nofile 2900000
     * soft nproc 131072
     * hard nproc 131072


Tabs Page
iddocker-tab
titleDocker (Red Hat/CentOS 7.X)

Build with Prebuilt Docker Image

Apache HAWQ source code contains the Dockerfiles to help developers to setup building and testing environment with docker.

To use the docker image follow the steps on: https://github.com/apache/incubator-hawq/tree/master/contrib/hawq-docker



Build optional extension modules

...

Code Block
languagebash
# The Apache HAWQ source code can be obtained from the the following link: 
# Apache Repo: https://git-wip-us.apache.org/repos/asf/incubator-hawq.git or 
# GitHub Mirror: https://github.com/apache/incubator-hawq. 
git clone https://git-wip-us.apache.org/repos/asf/incubator-hawq.git
 
# The code directory is incubator-hawq.
CODE_BASE=`pwd`/incubator-hawq
 
cd $CODE_BASE
 
# Run command to generate makefile.
./configure

# Or you could use --prefix=/hawq/install/path to change the Apache HAWQ install path, 
# and you can also add some optional components using options (--with-python --with-perl)
# For El Capitan (Mac OS 10.11), you may need to do: export CPPFLAGS="-I/usr/local/include -L/usr/local/lib" if the configure cannot find some components
./configure --prefix=/hawq/install/path --with-python --with-perl

# If you need to Enable RPS for Ranger Integration
export CATALINA_HOME=/tomcat/install/path
./configure --prefix=/hawq/install/path --enable-rps

# You can also run the command with --help for more configuration.
./configure --help


# Run command to build and install
# To build concurrently , run make with -j option. For example, make -j8
# On Linux system without large memory, you will probably encounter errors like
# "Error occurred during initialization of VM" and/or "Could not reserve enough space for object heap"
# and/or "out of memory", try to set vm.overcommit_memory = 1 temporarily, and/or avoid "-j" build,
# and/or add more memory and then rebuild.
# On mac os, you will probably see this error: "'openssl/ssl.h' file not found". 
# "brew link openssl --force" should be able to solve the issue.
make -j8
 
# Install HAWQ
make install

...