Sqoop2 Manual Setup
Skip to end of metadata
Go to start of metadata

This document outlines how to manually build and setup a Sqoop2 installation. It is expected that this setup will eventually be replaced by the assembly setup of Sqoop2 which will automate these steps.

Building Sqoop2

Checkout sources:

$ svn co https://svn.apache.org/repos/asf/sqoop/branches/sqoop2

And build them by doing the following:

$ cd sqoop2
$ mvn install

For the rest of this document we will refer to the workspace directory as sqoop workspace where necessary.

Setup Install Structure

Create another directory outside of Sqoop2 workspace where you will install the system.

$ cd ..
$ mkdir s2test
$ cd s2test

Within this directory, create subdirectories as noted below

$ mkdir bin conf logs repository 
$ ls
$ ls -l
total 0
drwxr-xr-x 2 arvind staff 68 May  9 00:26 bin
drwxr-xr-x 2 arvind staff 68 May  9 00:26 conf
drwxr-xr-x 2 arvind staff 68 May  9 00:26 logs
drwxr-xr-x 2 arvind staff 68 May  9 00:26 repository

Add Tomcat Server

Download Tomcat version 7.0.X to a convenient location, and untar it in the s2test directory and rename the expanded directory to server.

$ wget http://apache.mirrors.hoobly.com/tomcat/tomcat-7/v7.0.27/bin/apache-tomcat-7.0.27.tar.gz
$ tar zxvf apache-tomcat-7.0.27.tar.gz
$ rm apache-tomcat-7.0.27.tar.gz
$ mv apache-tomcat-7.0.27 server

Now configure the Tomcat server to setup the configuration directory for Sqoop2. This can be done by creating a file called setenv.sh under server/bin directory. This file should contain the setting for Sqoop2 configuratiohn directory as follows:

$ # Working directory is s2test created earlier
$ echo "export JAVA_OPTS=-Dsqoop.config.dir=/path/to/s2test/conf" >> server/bin/setenv.sh
$ chmod +x server/bin/setenv.sh

Note: You must specify the correct path to the s2test directory in the above command.

Setup Configuration

Under the s2test/conf directory, setup the bootstrap configuration sqoop_bootstrap.properties as follows:

# File s2test/conf/sqoop_bootstrap.properties

Now setup the main configuration sqoop.properties as follows:

# File s2test/conf/sqoop.properties
# Log4J system
org.apache.sqoop.log4j.appender.file.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} [%l] %m%n

org.apache.sqoop.log4j.rootCategory=WARN, file

# Repository

Note: You must specify the correct path to the s2test directory in the above configuration entries where necessary.

Deploy Sqoop2

To deploy Sqoop2 to the Tomcat, simply copy over the sqoop.war file from sqoop2/server/target/ to the s2test/server/webapps directory. It is recommend that when you do this the Tomcat server should not be running and that you delete any exploded webapp directory for sqoop2 if it exists in the Tomcat server.

$ cp /path/to/sqoop2/server/target/sqoop.war /path/to/s2test/server/webapps/
$ rm -rf /path/to/s2test/server/webapps/sqoop2/

Start Sqoop2

To start Sqoop2 now, simply start the Tomcat server.

$ /path/to/s2test/server/bin/catalina.sh start

If everything is fine, you will see logfiles by the name derbyrepo.log and sqoop.log created under /path/to/s2test/logs directory. The contents of the sqoop.log should indicate successful startup of the system.

By default the Tomcat server listens on port 8080 and you can access Sqoop2 by going to the URL http://localhost:8080/sqoop/.

You can try accessing the version resource by entering the following address in your browser: http://localhost:8080/sqoop/version

The output may be something like the following:

{"revision":"1238736","protocols":["1"],"date":"Wed May  9 00:23:18 PDT 2012","user":"arvind","url":"https:\/\/svn.apache.org\/repos\/asf\/sqoop\/branches\/sqoop2\/common","version":"2.0.0-SNAPSHOT"}

To stop Sqoop2, simply stop the Tomcat server. To update Sqoop2 to test out the changes you are working on, you should stop the server and redeploy the freshly generated war file as noted above.

  • No labels