Excerpt |
---|
Explains how to build the Trafodion source. |
...
...
...
Note |
---|
|
You MUST complete the steps in Create Build Environment before you build the Trafodion source since those steps install required Linux packages as well as set up important environmental variables. |
Download and Install Trafodion Source Tree
If you haven’t done so already, then download and install the Trafodion source tree. Refer to the Source Download instructions in the How-To chapter.
Set Up Environmental Variables
Start a new ssh
session. Use the following commands to set up the Trafodion environmental variables.
Info |
---|
|
Choose correct version to build
By default, the source code tree is at the HEAD of main branch of git. So you will build the latest code from Trafodion community. If one want to build a specific version, one need to checkout correct branch or tag.
List all branches and tags
Code Block |
---|
language | bash |
---|
title | Set Default (Debug) Environment |
---|
|
cd <Trafodion source directory>
git branch -a
git tag |
Set correct tag or branch before the build
You should have set up the TOOLSDIR environmental variable in your .bashrc or .profile already. as part of installing the Trafodion build tools. If you have not, please do so now. Refer to Install Required Build Tools. Code Block |
---|
language | bash |
---|
title | Set Default (Debug) Environment |
---|
|
cd <Trafodion source directory>
source ./env.sh |
Example
git checkout <the tag>
git checkout <the branch> |
Set Up Environmental Variables
Use the following commands to set up the Trafodion environmental variables.
Info |
---|
|
You should have set up the TOOLSDIR environment variable in your .bashrc or .profile already. as part of installing the Trafodion build tools. If you have not, please do so now. Refer to Install Required Build Tools. If you need to over-ride a value set in the core/sqf/sqenvcom.sh file, you may set the variables in ~/.trafodion. See core/sqf/LocalSettingsTemplate.sh file for more info. |
Trafodion can be built in one of two flavors, debug or release. The default flavor is debug.
Code Block |
---|
language | bash |
---|
title | Set Default (Debug) Environment |
---|
|
cd <Trafodion source directory>
source ./env.sh |
Code Block |
---|
language | bash |
---|
title | Set Release Build Environment |
---|
|
cd <Trafodion source directory>/core/sqf
source ./sqenvr.sh
cd ../.. |
Info |
---|
|
Test to make sure environment variables are set afterwards. |
Example
Code Block |
---|
language | bash |
---|
title | Set Up Environmental Variables |
---|
|
$ cd mysource/trafodion
$ source ./env.sh
If you are ready to build Trafodion, perform one of the following options:
make all (Build Trafodion, DCS, and REST) OR
make package (Build Trafodion, DCS, REST, and Client drivers) OR
make package-all (Build Trafodion, DCS, REST, Client drivers, and Tests)
If Trafodion has been built and you want test:
|
Code Block |
---|
language | bash |
---|
title | Set Up Environmental Variables |
---|
|
$ cd mysource/incubator-trafodion
$ source ./env.sh
If you are ready to build Trafodion, perform one of the following options:
make all (Build Trafodion, DCS, and REST) OR
make package (Build Trafodion, DCS, REST, and Client drivers) OR
make package-all (Build Trafodion, DCS, REST, Client drivers, and Tests)
If Trafodion has been built and you want test:
Execute the install_local_hadoop script which performs a single node
install using a popular Hadoop distribution
cd /home/trafdeveloper/mysource/incubator-trafodion/core/sqf/sql/scripts
install_local_hadoop [-p <port option>]
install_traf_components
Execute the install_local_hadoop configurescript Trafodionwhich andperforms starta thesingle processesnode
install using a popular HAVEHadoop FUN!distribution
You can also choose to cd /home/trafdeveloper/mysource/trafodion/core/sqf/sql/scripts
install_local_hadoop before[-p building<port Trafodion.
$ |
Anchor |
---|
option>]
install_traf_components
configure Trafodion and start the processes
HAVE FUN!
You can also choose to install_local_hadoop before building Trafodion.
$ env | grep LLVM
LLVM=/root/trafodion-build-tools/dest-llvm-3.2 |
Anchor |
---|
| build-source-build-trafodion-components |
---|
| build-source-build-trafodion-components |
---|
|
Build Trafodion Components
All make options sets up your environment to run a debug version of the Trafodion server-side code on the server where you are building the code. This is refereed to as "Trafodion" in the table below.
build-source-build-trafodion-components | build-source-build-trafodion-components | Build Trafodion Components
All make options sets up your environment to run a debug version of the Trafodion server-side code on the server where you are building the code. This is refereed to as "Trafodion" in the table below.
Info |
---|
You use the generated tar files in case you want to install Trafodion on a different development environment. (To be documented later.) |
Command | What It Builds |
---|
make all | Trafodion, DCS, and REST. |
make package | Trafodion, DCS, REST, and Client Drivers. |
Often referred to as "Trafodion binaries".It does not build RPMs which are needed for an Ambari install |
make package-all | Trafodion, DCS, REST, Client Drivers, RPMs and tests for all components. This builds the complete set of binaries needed. |
Info |
---|
|
If the build fails, you might want to rerun the make step. Trafodion downloads many dependencies and sometimes one of the download operations fail. Rerunning the build generally works. If the build continues to fail after a few tries, then rerun the make with the -j 1 option to run the build in non-parallel mode, which makes it easier to locate errors. |
Anchor |
---|
| build-make-all |
---|
| build-make-all |
---|
|
make all
Build Trafodion plus DCS and REST tar files.
Code Block |
---|
language | bash |
---|
title | Example: make all |
---|
|
$ make all
Building all Trafodion components
cd core && make all
make[1]: Entering directory `/home/trafdeveloper/mysource/incubator-trafodion/core'
./bldenvchk.sh;
cd sqf && make genverhdr
make[2]: Entering directory `/home/trafdeveloper/mysource/incubator-trafodion/core/sqf'
Building Apache Trafodion Version 1.3.0 debug
*** Warning: QT_TOOLKIT does not point to an existing directory.
*** Warning: SQL Compiler Debugger will not be built.
cd sqf && make setupdir
make[2]: Entering directory `/home/trafdeveloper/mysource/incubator-trafodion/core/sqf'
mkdir -p export/bin64d
mkdir -p export/lib64d
mkdir -p export/include/sql
mkdir -p export/include
cd monitor/linux; make setup
.
.
.
[INFO] Nothing to compile - all classes are up to date ##(DCS)
[INFO] Building jar: /home/trafdeveloper/mysource/incubator-trafodion/dcs/target/dcs-1.3.0.jar ##(DCS)------------------------------------------------------------------------
[INFO] Building tar: /home/trafdeveloper/mysource/incubator-trafodion/dcs/target/dcs-1.3.0.tar.gz ##(DCS)
[INFO] BUILD SUCCESS ##(DCS)
mkdir -p ../distribution ##(DCS)
mv target/dcs-1.3.0.tar.gz ../distribution ##(DCS)
rm -f trafodion-dcs.jar.versions ##(DCS)
make[2]: Leaving directory `/home/trafdeveloper/mysource/incubator-trafodion/dcs' ##(DCS)
make[1]: Leaving directory `/home/trafdeveloper/mysource/incubator-trafodion/core'
$ # Check that distribution files were created
$ ls distribution
dcs-1.3.0.tar.gz rest-1.3.0.tar.gz
$ |
Anchor |
---|
BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10.412 s
[INFO] Finished at: 2017-11-07T21:32:31-08:00
[INFO] Final Memory: 24M/57M
[INFO] ------------------------------------------------------------------------
mkdir -p /home/wangxz/Trafodion/src/trafodion/core/sqf/export/lib
/bin/cp -f target/lib_mgmt-2.3.0.jar /home/wangxz/Trafodion/src/trafodion/core/sqf/expor\
t/lib/lib_mgmt.jar
make[2]: Leaving directory `/home/wangxz/Trafodion/src/trafodion/core/sql/lib_mgmt'
make[1]: Leaving directory `/home/wangxz/Trafodion/src/trafodion/core'
$ |
Anchor |
---|
| build-make-package |
---|
| build-make-package |
---|
|
make package
Build Trafodion plus tar files for DCS, REST, and Client Drivers. (If you need to package windows drivers see special instructions below "Build/update windows drivers")
Code Block |
---|
language | bash |
---|
title | Example: make package |
---|
|
$ cd mysource/incubator-trafodion
$ source ./env.sh
$ make package
Packaging Trafodion components
cd core && make package
make[1]: Entering |
build-make-package | build-make-package | make package
Build Trafodion plus tar files for DCS, REST, and Client Drivers.
Code Block |
---|
language | bash |
---|
title | Example: make package |
---|
|
$ cd mysource/incubator-trafodion
$ source ./env.sh
$ make package
Packaging Trafodion components
cd core && make package
make[1]: Entering directory `/home/trafdeveloper/mysource/incubator-trafodion/core'
./bldenvchk.sh;
cd sqf && make genverhdr
make[2]: Entering directory `/home/trafdeveloper/mysource/incubator-trafodion/core/sqf'
Building Apache Trafodion Version 1.3.0 debug
.
.
.
rm -f "trafodion_debug.tar" pkglist-files pkglist-symlinks tmp/sqenv.sh ##(Package)
make[2]: Leaving directory `/home/trafdeveloper/mysource/incubator-trafodion/core/sqf' ##(Package)
make[1]: Leaving directory `/home/trafdeveloper/mysource/incubator-trafodion/core'
$ ls distribution/
dcs-
./bldenvchk.sh;
cd sqf && make genverhdr
make[2]: Entering directory `/home/trafdeveloper/mysource/trafodion/core/sqf'
Building Apache Trafodion Version 1.3.0.tar.gz rest-1.3.0.tar.gz trafodion_clients-1.3.0-debug.tgz trafodion_server-1.3.0-debug.tgz |
Anchor |
---|
build-make-package-all | build-make-package-all | make package-all
Build Trafodion plus tar files for DCS, REST, Client Drivers, and tests for all components.
Code Block |
---|
language | bash |
---|
title | Example: make package-all |
---|
|
Packaging all Trafodion components
cd core && make package-all
make[1]: Entering directory `/home/trafdeveloper/mysource/incubator-trafodion/core'
./bldenvchk.sh;
cd sqf && make genverhdr
make[2]: Entering directory `/home/trafdeveloper/mysource/incubator-trafodion/core/sqf'
Building Apache Trafodion Version 1.3.0 debug
.
.
.
cd tmp; tar --mode o-rwx -rf ../"trafodion_debug.tar" sqenv.sh ##(Package)
gzip -c "trafodion_debug.tar" > "../../distribution/trafodion_"server-1.3.0-debug".tgz" ##(Package)
rm -f "trafodion_debug.tar" pkglist-files pkglist-symlinks tmp/sqenv.sh ##(Package)
make[2]: Leaving directory `/home/trafdeveloper/mysource/incubator-trafodion/core/sqf' debug
.
.
.
+ umask 022 ##(Package installer)
+ cd /home/wangxz/Trafodion/src/incubator-trafodion/install/ambari-installer/RPMROOT/BUILD ##(Package installer)
+ cd traf_ambari ##(Package installer)
+ /bin/rm -rf /home/wangxz/Trafodion/src/trafodion/install/ambari-installer/RPMROOT/BUILDROOT/traf_ambari-2.3.0-1.x86_64 ##(Package installer)
+ exit 0 ##(Package installer)
mkdir -p ../../distribution ##(Package installer)
mv -f /home/wangxz/Trafodion/src/trafodion/install/ambari-installer/RPMROOT/RPMS/noarch/traf_ambari*.rpm ../../distribution/ ##(Package installer)
make[3]: Leaving directory `/home/wangxz/Trafodion/src/trafodion/install/ambari-installer' ##(Package installer)
make[2]: Leaving directory `/home/wangxz/Trafodion/src/trafodion/install' ##(Package installer)
make[1]: Leaving directory `/home/trafdeveloper/mysource/incubator-trafodion/core'
$ ls distribution/
dcs-1.3.0.tar.gz phoenix-tests.tgz trafodion_clients-1.3.0-debug.tgz trafodion_server-1.3.0-debug.tgz
dcs-tests.tgz rest-1.3.0.tar.gz trafodion-regress.tgzwangxz/Trafodion/src/trafodion/core'
$ ls distribution/
dcs-1.3.0.tar.gz rest-1.3.0.tar.gz trafodion_clients-1.3.0-debug.tgz trafodion_server-1.3.0-debug.tgz |
Anchor |
---|
| build-make-package-all |
---|
| build-make-package-all |
---|
|
make package-all
Build Trafodion plus tar files for DCS, REST, Client Drivers, and tests for all components.
Code Block |
---|
language | bash |
---|
title | Example: make package-all |
---|
|
Packaging all Trafodion components
cd core && make package-all
make[1]: Entering directory `/home/trafdeveloper/mysource/trafodion/core'
./bldenvchk.sh;
cd sqf && make genverhdr
make[2]: Entering directory `/home/trafdeveloper/mysource/trafodion/core/sqf'
Building Apache Trafodion Version 1.3.0 debug
.
.
.
cd tmp; tar --mode o-rwx -rf ../"trafodion_debug.tar" sqenv.sh ##(Package)
gzip -c "trafodion_debug.tar" > "../../distribution/trafodion_"server-1.3.0-debug".tgz" ##(Package)
rm -f "trafodion_debug.tar" pkglist-files pkglist-symlinks tmp/sqenv.sh ##(Package)
make[2]: Leaving directory `/home/trafdeveloper/mysource/trafodion/core/sqf' ##(Package)
make[1]: Leaving directory `/home/trafdeveloper/mysource/trafodion/core'
$ ls distribution/
dcs-1.3.0.tar.gz phoenix-tests.tgz trafodion_clients-1.3.0-debug.tgz trafodion_server-1.3.0-debug.tgz
dcs-tests.tgz rest-1.3.0.tar.gz trafodion-regress.tgz |
Make/update windows drivers
Note : This needs to be done only by Release Managers before a release or whenever someone makes a change to the drivers . The process is manual since it's built on Windows. The nightly Jenkins build picks up this the executable and packages it everytime.
Build instructions for Build Machine Windows 8.1 (Windows 10 has not been tested)
- First create C:\Build\win-odbc64\inc, redist directory
- Download Active perl for windows and install https://www.perl.org/get.html
- Download and install Visual Studio 2013 community edition The latest 2013 installs vc12 version. So, you might need to use vc12 instead vc11 in the following steps.
- Download, build and install openSSL version(openssl-1.0.1p.tar.gz) ftp://ftp.openssl.org/source/ Follow instructions in “Build openssl” section below
- Download zlib for Windows(https://github.com/madler/zlib/)Follow instructions in “Build zlib “ section below
- Download and install Unicode version of Inno software for packaging (http://www.jrsoftware.org/isdl.php)
- Download the C++ Redistributable runtime library http://www.microsoft.com/en-us/download/details.aspx?id=40784 and copy it to C:\Build\winodbc64\redist folder (Note: if the folder does not exist please create that folder)
- Download github ( http://git-scm.com/downloads) if you need to contribute
- Download a copy of Trafodion source to build Windows ODBC driver (git clone git@github.com:apache/trafodion);Follow instructions in “Build Trafodion ODBC driver
Build openssl static libraries for Windows 64 bit
- Un-tar the openssl-1.0.1p.tar.gz file to c:\Traf-Windows-BldTools folder
- Launch Visual Studio 2013 tool x64 Cross Tools Command promp
- It is available as a shortcut within Tools directory of Visual Studio 2013 launch folder. You might need to double click on "C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools\shortcuts\VS2013 x64 Cross Tools Command Prompt" to run the commands shown below.
- Configure for the target directory to install openssl using the command
- perl Configure VC-WIN64A --prefix=c:/openssl-1.0.1p.
- Run the following:
- ms\do_nasm
- ms\do_win64a
- nmake –f ms\nt.mak clean ? To clean
- nmake -f ms\nt.mak install ? To build static libraries and install it to the location specified during configure step
- Copy the file “applink.c” to the installed openssl include location. This file can be found in ms directory of the openssl source directory
- copy c:\Traf-WinBld-Tools\openss-1.0.1p\ms\applink.c c:\openssl-1.0.1p\include)
Build zlib libraries for Windows 64 bit
- Launch Visual Studio 2013 tool x64 Cross Tools Command prompt
- Change to the directory where you have un-zipped zlib source location (Ex: cd c:\Traf-WinBld-Tools\zlib-master)
- Open the Visual Studio solution (zlibvc.sln) located in zlib-master\contrib\vstudio\vc11. Then do the following:
- Change the Solution Configuration to “Release” “x64” for the solution
- Then do Build ? Clean Solution
- And Build? Build Soution
- Following error will be seen
- error LNK1118: syntax error in 'VERSION' statement C:\Traf-Windows-bldTools\zlib-master\contrib\vstudio\vc11\zlibvc.def
- To resolve the above error, Double click on the error when it opens in the editor change the Version from 1.2.8 to 1.28
- VERSION 1.2.8 to
- VERSION 1.28
- Click Save and Build the solution again
- Build will succeed
- Close the Visual studio solution.
Build Trafodion windows ODBC driver for Windows 64 bit
- Copy trafsqlext.h from c:\Trafodion-Workspace\trafodion\win-odbc64\odbcclient\inc to C:\Build\trafodion\win-odbc64\inc folder
- Launch Visual Studio 2013 tool x64 Cross Tools Command prompt
- Cd c:\Trafodion-Workspace\trafodion\win-odbc64\odbcclient
- Edit build_os.bat and update the following
- set ZLIB_INCLUDE_PATH=C:\zlib-master\include
- set ZLIB_LIB_PATH= C:\Traf-Windows-bldTools\zlib-master\contrib\vstudio\vc11\x64\ZlibStatRelease
- set OPENSSL_INCLUDE_PATH= C:\openssl-1.0.1p\include
- set OPENSSL_LIB_PATH= C:\openssl-1.0.1p\lib
- Run the build script by executing, build_os.bat file
- drvAdm35 build failed. This problem was resolved by installing Multibyte MFC library for Visual Studio 2013.
ODBC driver exe file will be created in c:\Build\win-odbc64 folder.This exe should be moved to linux to the core/conn/clients directory, so that it will be packaged along with other clients when we do the step "Make package-all"
Verify Build
Local Binary Files
Check that local Trafodion binary files were created in the build.
Debug build objects will be found in $TRAF_HOME/export/bin64d.
Release objects will be found in $TRAF_HOME/export/bin64. For example :
Code Block |
---|
language | bash |
---|
title | Example: Check Local File Creation |
---|
|
$ # Check that local binary files were created
$ ls $MY$TRAF_SQROOTHOME/export/bin64dbin64
dtmci ldapcheck monitor mpirun mxosrvr odb64luo sqid sqvers tdm_arkesp tmshutdown
hydra_pmi_proxy ldapconfigcheck monmemlog mxcierrors.cat mxsscp pstartd sqinfo sqwatchdog tdm_udrserv
idtmsrv mondump mpicc mxlobsrvr mxssmp shell sqlci tdm_arkcmp tm |
sqvers
Use sqvers
-u to verify the build.
Code Block |
---|
language | bash |
---|
title | Example: sqvers -u |
---|
|
$ sqvers -u
MYTRAF_SQROOTHOME=/home/trafdeveloper/mysource/incubator-trafodion/core/sqf
who@host=trafdeveloper@box01
JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64
SQ_MBTYPE=64d (64-debug)
linux=2.6.32-504.el6.x86_64
redhat=6.6
NO patches
Most common Apache_Trafodion Release 1.3.0 (Build debug [trafdeveloper], branch -, date 25Jan16)
UTT count is 1
[6] Release 1.3.0 (Build debug [trafdeveloper], branch -, date 25Jan16)
export/lib/hbase-trx-cdh5_3-1.3.0.jar
export/lib/hbase-trx-hbase_98_4-1.3.0.jar
export/lib/hbase-trx-hdp2_2-1.3.0.jar
export/lib/sqmanvers.jar
export/lib/trafodion-dtm-1.3.0.jar
export/lib/trafodion-sql-1.3.0.jar
$ |
The output from the sqvers -u
commands should show several jar files. The number of files differs based on the version of Trafodion you downloaded.
Next Steps
Do one of the following once the build has completed:
- Create Test Environment: Set up the Trafodion development test environment on the server where you did the build. You'll install a Hadoop environment, start Trafodion, and run the Trafodion Test Libraries.
- Install Trafodion Binaries: Using the packages you created (make package or make package-all), install the Trafodion Binaries using the steps described in the Trafodion documentation.