|
Extension | How to enable | Pre-build steps on Mac |
---|---|---|
PL/R | ./configure --with-r | #install R before build brew tap homebrew/science brew install r |
PL/Python | ./configure --with-python | |
PL/Java | ./configure --with-java | |
PL/PERL | ./configure --with-perl | |
pgcrypto | ./configure --with-pgcrypto --with-openssl | |
gporca | ./configure --enable-orca |
Please follow the steps here: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html
Note you might need to build hadoop from source on Red Hat/CentOS 6.x if the downloaded hadoop package has higher
glibc version requirement. When that happens, you will probably see the warning below when running start-dfs.h.
" WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform"
You will also need to set the port for fs.defaultFS to 8020 in etc/hadoop/core-site.xml (The example above set it as 9000.)
HDFS is a must, but YARN is optional. YARN is only needed when you want to use YARN as the global resource manager.
Your need to verify your HDFS works.
# start HDFS start-dfs.sh # Do some basic tests to make sure HDFS works hadoop fs -lr / hadoop fs -mkdir /test hadoop fs -put ./testfile / hadoop fs -get /testfile . |
Once you have an environment with the necessary dependencies installed and Hadoop is ready, the next step is to get the code and build HAWQ
# The Apache HAWQ source code can be obtained from the the following link: # Apache Repo: https://git-wip-us.apache.org/repos/asf/incubator-hawq.git or # GitHub Mirror: https://github.com/apache/incubator-hawq. git clone https://git-wip-us.apache.org/repos/asf/incubator-hawq.git # The code directory is incubator-hawq. CODE_BASE=`pwd`/incubator-hawq cd $CODE_BASE # Run command to generate makefile. ./configure # Or you could use --prefix=/hawq/install/path to change the Apache HAWQ install path, # and you can also add some optional components using options (--with-python --with-perl) # For El Capitan (Mac OS 10.11), you may need to do: export CPPFLAGS="-I/usr/local/include -L/usr/local/lib" if the configure cannot find some components ./configure --prefix=/hawq/install/path --with-python --with-perl # You can also run the command with --help for more configuration. ./configure --help # Run command to build and install # To build concurrently , run make with -j option. For example, make -j8 # On Linux system without large memory, you will probably encounter errors like # "Error occurred during initialization of VM" and/or "Could not reserve enough space for object heap" # and/or "out of memory", try to set vm.overcommit_memory = 1 temporarily, and/or avoid "-j" build, # and/or add more memory and then rebuild. # On mac os, you will probably see this error: "'openssl/ssl.h' file not found". # "brew link openssl --force" should be able to solve the issue. make -j8 # Install HAWQ make install |
# Before initializing HAWQ, you need to install HDFS and make sure it works. # Besides you need to set password-less ssh on the systems. source /hawq/install/path/greenplum_path.sh hawq init cluster # after initialization, HAWQ is started by default # Now you can stop/restart/start the cluster by using: hawq stop/restart/start cluster # HAWQ master and segments are completely decoupled. So you can also init, start or stop the master and segments separately. # For example, to init: hawq init master, then hawq init segment # to stop: hawq stop master, then hawq stop segment # to start: hawq start master, then hawq start segment |
psql -d postgres create table t ( i int ); insert into t values(1); insert into t select generate_series(1,10000); select count(*) from t; |
# Unit test. To do unit test, go to the src/backend and run unittest. cd $CODE_BASE/src/backend make unittest-check # Code coverage cd $CODE_BASE ./configure --enable-coverage --enable-debug (for debug build), or ./configure --enable-coverage (for opt build) make -j8 make install run some test to exercise hawq (i.e., unit test, install check, feature test, etc) make coverage-show to see summary code coverage information in console, and detailed code coverage information in CodeCoverageReport (html format) make coverage-show filter="./src/backend/executor/nodeAgg.c -d ./src/backend/commands" to see code coverage for specific files or directories make coverage-reset to clear code coverage statistics # Installcheck-good test. After installing HAWQ, please ensure HDFS work before initializing HAWQ. source /install/dir/greenplum_path.sh hawq init cluster make installcheck-good # Feature test cd $CODE_BASE make feature-test ./feature-test to run all feature test, or ./feature-test --gtest_filter=TestCommonLib.TestSqlUtil to run test suite TestCommonLib.TestSqlUtil |