You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 135 Next »

DUCC Project Documentation

Tutorials

  1. Quick Start
  2. Install WebServer Demo
  3. Check out 1.0.0

Quick Start

Audience: DUCC Users/Administrators

Instructions on how to fetch, build and install DUCC 1.0 for single host system (my laptop!)
See DUCC Project Documentation for further details.

The following steps were performed on Red Hat Enterprise Linux Workstation release 6.4 (Santiago)

Prerequisites

  • Install svn
  • Install mvn
  • Install jdk (1.6 or better)
    export JAVA_HOME=~/apps/java/java-home
  • Install latex
  • Install ant

Start sshd

[degenaro]$ sudo su root
[root]# chkconfig sshd on
[root]# /sbin/service sshd restart
[root]# exit
[degenaro]$

Create user ducc

[degenaro]$ sudo adduser ducc

Permit universal read+execute access to home directory for users degenaro and ducc

[degenaro]$ sudo su root
[root]# cd /home
[root]# chmod 755 degenaro
[root]# chmod 755 ducc
[root]# exit
[degenaro]$

Enable passwordless ssh for user ducc

 
[degenaro]$ sudo su ducc 
[ducc]$ cd
[ducc]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ducc/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/ducc/.ssh/id_rsa.
Your public key has been saved in /home/ducc/.ssh/id_rsa.pub.
The key fingerprint is:
4f:4a:81:60:c9:a5:d8:88:80:f3:6c:a1:ef:31:8b:8e ducc@hal9000.apache.com
The key's randomart image is:
+--[ RSA 2048]----+
|o  .oo.          |
|+..=oo .         |
|.=o.o . .        |
|. +      .       |
| o      S .      |
|  +    . +       |
| o +    . .      |
|o o              |
|E.               |
+-----------------+
[ducc]$ cd .ssh
[ducc]$ cat id_rsa.pub >> authorized_keys2
[ducc]$ chmod 600 *
[ducc]$ exit
[degenaro]$

Insure JAVA_HOME is set, and PATH contains LaTeX

Sample code to set JAVA_HOME

.bashrc
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

# User specific aliases and functions

export JAVA_HOME=~/apps/java/java-home
export PATH=$PATH:/usr/local/tex

Fetch DUCC from svn

[degenaro]$ svn co https://svn.apache.org/repos/asf/uima/sandbox/uima-ducc/trunk ~/svn/ducc/trunk

Build DUCC jars

[degenaro]$ cd ~/svn/ducc/trunk
[degenaro]$ mvn clean install -Pbuild-duccdocs

Install

[degenaro]$ cd target
[degenaro]$ sudo su ducc
[ducc]$ tar -zxf uima-ducc-1.0.0-SNAPSHOT-bin.tar.gz -C ~ducc
[ducc]$ cd ~
[ducc]$ ln -s apache-uima-ducc-1.0.0-SNAPSHOT ducc_install 
[ducc]$ cd ~ducc/ducc_install/admin
[ducc]$ ./ducc_post_install

Check

[ducc]$ cd ~ducc/ducc_install/admin
[ducc]$ ./check_ducc -s
[ducc]$ exit
[degenaro]$

Optional: enable ducc_ling and login to Web-server via Linux authentication

[degenaro]$ sudo su root
[root]$ mkdir /local
[root]$ mkdir /local/ducc
[root]$ mkdir /local/ducc/bin
[root]$ chown ducc.ducc /local/ducc
[root]$ chown ducc.ducc /local/ducc/bin
[root]$ chmod 700 /local/ducc
[root]$ chmod 700 /local/ducc/bin
[root]$ cp ~ducc/ducc_install/duccling/src/ducc_ling /local/ducc/bin
[root]$ chown root.ducc /local/ducc/bin/ducc_ling
[root]$ chmod 4750 /local/ducc/bin/ducc_ling
[root]$ exit
[degenaro]$

[degenaro]$ sudo su ducc
[ducc]$ cd ~ducc/ducc_install/resources
1. edit ducc.properties
2. locate: ducc.ws.login.enabled = false
3. modify: ducc.ws.login.enabled = true
4. locate: ducc.agent.launcher.ducc_spawn_path=${DUCC_HOME}/admin/ducc_ling
5. modify: ducc.agent.launcher.ducc_spawn_path=/local/ducc/bin/ducc_ling
6. save

Start

[ducc]$ cd ~ducc/ducc_install/admin
[ducc]$ ./start_ducc -s
[ducc]$ exit
[degenaro]$

ducc-mon System Daemons

Start your favorite browser (firefox, chrome, opera) and visit the ducc-mon system daemons page.

http://localhost:42133/system.daemons.jsp

Submit a test Job

[degenaro]$ cd ~ducc/ducc_install/bin
[degenaro]$ export PATH=$PATH:$PWD
[degenaro]$ cd ~
[degenaro]$ mkdir ducc
[degenaro]$ cd ducc
[degenaro]$ cp -pR ~ducc/ducc_install/examples .
[degenaro]$ cd examples/simple
[degenaro]$ ducc_submit -f 1.job
Job 1 submitted
[degenaro]$

ducc-mon Jobs

Visit the ducc-mon jobs page.

http://localhost:42133/jobs.jsp

End.

Install WebServer Demo

 

Audience: DUCC Developers

From local workstation, copy DUCC binary tar-ball to installation host

[degenaro]$ scp uima-ducc-1.1.0-bin.tar.gz degenaro@uima-ducc-vm.apache.org:~degenaro

Login to installation host

Note: Developers who are presently unable to login should request access via uima-dev mailing list

[degenaro@oc0883280806 Downloads]$ ssh degenaro@uima-ducc-vm.apache.org
...
degenaro@uima-ducc-vm:~$

Stop demo (if running)

degenaro@uima-ducc-vm:~$ killall -9 python

Stop ducc (if running)

degenaro@uima-ducc-vm:~$ killall -9 java

Stop custom service and/or sleepers (if running)

degenaro@uima-ducc-vm:~$ killall -9 service

Install new code

degenaro@uima-ducc-vm:~$ cd ~/ducc/versions
degenaro@uima-ducc-vm:~/ducc/versions$ rm -fr trash/*
degenaro@uima-ducc-vm:~/ducc/versions$ mv u* trash
degenaro@uima-ducc-vm:~/ducc/versions$ mv ~/uima-ducc-1.1.0-bin.tar.gz .
degenaro@uima-ducc-vm:~/ducc/versions$ tar -zxf *gz
degenaro@uima-ducc-vm:~/ducc/versions$ ls
apache-uima-ducc-1.1.0  trash  uima-ducc-1.1.0-bin.tar.gz
degenaro@uima-ducc-vm:~$ cd ~/ducc
degenaro@uima-ducc-vm:~/ducc$ rm release
degenaro@uima-ducc-vm:~/ducc$ rm 1.0.1
degenaro@uima-ducc-vm:~/ducc$ ln -s versions/apache-uima-ducc-1.1.0 1.1.0
degenaro@uima-ducc-vm:~/ducc$ ln -s 1.1.0 release
degenaro@uima-ducc-vm:~/ducc$ ls -atl
drwxr-xr-x  3 degenaro degenaro 4096 Oct  7 13:06 .
lrwxrwxrwx  1 degenaro degenaro    5 Oct  7 13:00 release -> 1.1.0
lrwxrwxrwx  1 degenaro degenaro   31 Oct  7 12:59 1.1.0 -> versions/apache-uima-ducc-1.1.0
drwxr-xr-x  4 degenaro degenaro 4096 Oct  7 12:55 versions
drwxr-xr-x 19 degenaro degenaro 4096 Oct  7 12:54 ..
lrwxrwxrwx  1 degenaro degenaro   28 Aug 27 14:44 logs -> /home/degenaro/tmp/ducc/logs
lrwxrwxrwx  1 degenaro degenaro    7 Mar 29  2014 install -> release

Run ducc_post_install

degenaro@uima-ducc-vm:~$ cd ~/ducc/install
degenaro@uima-ducc-vm:~/ducc/install$ ls
activemq  admin  bin  docs  duccling  examples  lib  LICENSE  NOTICE  README  RELEASE_NOTES.html  resources  state  webserver
degenaro@uima-ducc-vm:~/ducc/install$ cd ~/ducc/install/admin/
degenaro@uima-ducc-vm:~/ducc/install/admin$ ./ducc_post_install 
Python version:
2.7.3 (default, Feb 27 2014, 19:58:35) 
[GCC 4.6.3]
Ducc head is configured as uima-ducc-vm 

Initial /home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/resources/ducc.nodes created.
Initial nodepool /home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/resources/jobdriver.nodes created.

ActiveMQ is automanaged on node  tcp://uima-ducc-vm:61617?wireFormat.maxInactivityDuration=0&jms.useCompression=true&closeAsync=false
Initialized property "ducc.jvm" to /home/degenaro/local/sun/jdk1.7.0_45/bin/java
JAVA: java version "1.7.0_45"
JAVA: Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
JAVA: Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
Java version: "1.7.0_45"
Java is verified.

WARNING
WARNING The default ssl pw of  quackquack  is configured. For security, change the property 
	ducc.ws.port.ssl.pw to a secure password in ducc.properties and rerun ducc_post_install.
WARNING

None
keystore created.

Web server keystore generated from ducc.properties

DUCC book installed into webserver root

cc -g    ducc_ling.c   -o ducc_ling
Initial ducc_ling is installed.  See the installation guide for multi-user setup.

Install local modifications

degenaro@uima-ducc-vm:~$ cd ~/ducc/install/examples/uima-ducc-vm/driver/
degenaro@uima-ducc-vm:~/ducc/install/examples/uima-ducc-vm/driver$ ./installer
installed: /home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/resources/ducc.properties
installed: /home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/resources/ducc.administrators
installed: /home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/examples/systemtest/ducc.nodes

Install patches (if any)

degenaro@uima-ducc-vm:~$ cd ~/ducc/install/examples/uima-ducc-vm/driver/
degenaro@uima-ducc-vm:~/ducc/install/examples/uima-ducc-vm/driver$ ./patches
saved: ...
installed: ...

Launch DUCC (with "sim" nodes)

degenaro@uima-ducc-vm:~$ cd ~/ducc/install/examples/systemtest
degenaro@uima-ducc-vm:~/ducc/install/examples/systemtest$ ./start_sim --nothreading -n ducc.nodes -c all
Running as 574
Set ducc_ling version from uima-ducc-vm : /home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/admin/ducc_ling -v >/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/state/duccling.version
ducc_ling OK
broker host localhost
Waiting for broker 0
Starting from nodes in ducc.nodes
Starting agent on uima-ducc-vm instance 1 as pseudo-node uima-ducc-vm-1 IP 192.168.4.1 memory 15
  ... Started,  PID 1344
Starting agent on uima-ducc-vm instance 2 as pseudo-node uima-ducc-vm-2 IP 192.168.4.2 memory 15
  ... Started,  PID 1549
Starting agent on uima-ducc-vm instance 3 as pseudo-node uima-ducc-vm-3 IP 192.168.4.3 memory 15
  ... Started,  PID 1771
Starting agent on uima-ducc-vm instance 4 as pseudo-node uima-ducc-vm-4 IP 192.168.4.4 memory 15
  ... Started,  PID 1990
Starting agent on uima-ducc-vm instance 5 as pseudo-node uima-ducc-vm-5 IP 192.168.4.5 memory 30
  ... Started,  PID 2204
Starting agent on uima-ducc-vm instance 6 as pseudo-node uima-ducc-vm-6 IP 192.168.4.6 memory 30
  ... Started,  PID 2409
Starting agent on uima-ducc-vm instance 7 as pseudo-node uima-ducc-vm-7 IP 192.168.4.7 memory 30
  ... Started,  PID 2629
Starting agent on uima-ducc-vm instance 8 as pseudo-node uima-ducc-vm-8 IP 192.168.4.8 memory 45
  ... Started,  PID 2841
Starting agent on uima-ducc-vm instance 9 as pseudo-node uima-ducc-vm-9 IP 192.168.4.9 memory 45
  ... Started,  PID 3053
Start db on uima-ducc-vm
    PID 3233
Start ws on uima-ducc-vm
    PID 3407
Start sm on uima-ducc-vm
    PID 3591
Start rm on uima-ducc-vm
    PID 3766
Start or on uima-ducc-vm
    PID 3943
Start pm on uima-ducc-vm
    PID 4123
All Work completed

Create user logs directory

degenaro@uima-ducc-vm:~$ cd /tmp
degenaro@uima-ducc-vm:/tmp$ mkdir degenaro
degenaro@uima-ducc-vm:/tmp$ mkdir degenaro/ducc
degenaro@uima-ducc-vm:/tmp$ mkdir degenaro/ducc/logs

degenaro@uima-ducc-vm:~$ cd ~/ducc
degenaro@uima-ducc-vm:~/ducc$ ln -s /tmp/degenaro/ducc/logs/ logs
degenaro@uima-ducc-vm:~/ducc$

Wait for all DUCC daemons to be running

Visit uima-ducc-vm.apache.org:42133/system.daemons.jsp

Create services

degenaro@uima-ducc-vm:~$ cd ~/ducc/install/examples/uima-ducc-vm/driver
degenaro@uima-ducc-vm:~/ducc/install/examples/uima-ducc-vm/driver$ ./prep
/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/bin/ducc_services --register --instances 1 --description 'Test Service 1' --process_jvm_args '-DdefaultBrokerURL=tcp://localhost:61617' --classpath '/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/lib/uima-ducc/examples/*:/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/lib/uima/*:/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/examples/simple/resources/service' --service_ping_arguments 'broker-jmx-port=9999' --environment 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0' --process_memory_size '15' --working_directory '/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/examples/uima-ducc-vm/driver' --process_DD '/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/examples/simple/resources/service/Service_FixedSleep_1.xml' --scheduling_class 'fixed'
Service Register succeeded - Registered. - UIMA-AS:FixedSleepAE_1:tcp://localhost:61617 ID[0]
registered service: 1
/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/bin/ducc_services --register --instances 1 --description 'Test Service 2' --process_jvm_args '-DdefaultBrokerURL=tcp://localhost:61617' --classpath '/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/lib/uima-ducc/examples/*:/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/lib/uima/*:/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/examples/simple/resources/service' --service_ping_arguments 'broker-jmx-port=9999' --environment 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0' --process_memory_size '15' --working_directory '/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/examples/uima-ducc-vm/driver' --process_DD '/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/examples/simple/resources/service/Service_FixedSleep_2.xml' --scheduling_class 'fixed'
Service Register succeeded - Registered. - UIMA-AS:FixedSleepAE_2:tcp://localhost:61617 ID[1]
registered service: 2
/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/bin/ducc_services --register --instances 1 --description 'Test Service 3' --process_jvm_args '-DdefaultBrokerURL=tcp://localhost:61617' --classpath '/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/lib/uima-ducc/examples/*:/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/lib/uima/*:/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/examples/simple/resources/service' --service_ping_arguments 'broker-jmx-port=9999' --environment 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0' --process_memory_size '15' --working_directory '/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/examples/uima-ducc-vm/driver' --process_DD '/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/examples/simple/resources/service/Service_FixedSleep_3.xml' --scheduling_class 'fixed'
Service Register succeeded - Registered. - UIMA-AS:FixedSleepAE_3:tcp://localhost:61617 ID[2]
registered service: 3
/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/bin/ducc_services --register --instances 1 --description 'Test Service 4' --process_jvm_args '-DdefaultBrokerURL=tcp://localhost:61617' --classpath '/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/lib/uima-ducc/examples/*:/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/lib/uima/*:/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/examples/simple/resources/service' --service_ping_arguments 'broker-jmx-port=9999' --environment 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0' --process_memory_size '15' --working_directory '/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/examples/uima-ducc-vm/driver' --process_DD '/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/examples/simple/resources/service/Service_FixedSleep_4.xml' --scheduling_class 'fixed'
Service Register succeeded - Registered. - UIMA-AS:FixedSleepAE_4:tcp://localhost:61617 ID[3]
registered service: 4
/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/bin/ducc_services --register --instances 1 --description 'Test Service 5' --process_jvm_args '-DdefaultBrokerURL=tcp://localhost:61617' --classpath '/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/lib/uima-ducc/examples/*:/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/lib/uima/*:/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/examples/simple/resources/service' --service_ping_arguments 'broker-jmx-port=9999' --environment 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0' --process_memory_size '15' --working_directory '/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/examples/uima-ducc-vm/driver' --process_DD '/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/examples/simple/resources/service/Service_FixedSleep_5.xml' --scheduling_class 'fixed'
Service Register succeeded - Registered. - UIMA-AS:FixedSleepAE_5:tcp://localhost:61617 ID[4]
registered service: 5
Service Modify succeeded - Service 1 modify request accepted. - UIMA-AS:FixedSleepAE_2:tcp://localhost:61617 ID[1]
autostart service: 1

Create Custom services

degenaro@uima-ducc-vm:~$ cd ~/ducc/install/examples/uima-ducc-vm/custom
degenaro@uima-ducc-vm:~/ducc/install/examples/uima-ducc-vm/custom$ make
cc -g    service.c   -o service
javac -classpath ~/ducc/install/lib/uima-ducc-cli.jar CustomPing.java
degenaro@uima-ducc-vm:~/ducc/install/examples/uima-ducc-vm/custom$ ./customsvc
Service setup using
   project_home  : /home/degenaro/ducc/versions/apache-uima-ducc-1.0.1-SNAPSHOT/examples/uima-ducc-vm
   DUCC_HOME     : /home/degenaro/ducc/install
   autostart     : True
   CLASSPATH:
     /home/degenaro/ducc/versions/apache-uima-ducc-1.0.1-SNAPSHOT/examples/uima-ducc-vm/custom
Register new service
/home/degenaro/ducc/install/bin/ducc_services --register --process_executable 'service' --service_request_endpoint 'CUSTOM:localhost:7175' --service_ping_class 'CustomPing' --description 'Custom Pinger' --autostart 'True' --service_ping_timeout '10000' --service_ping_classpath '/home/degenaro/ducc/versions/apache-uima-ducc-1.0.1-SNAPSHOT/examples/uima-ducc-vm/custom' --service_ping_dolog 'false' --process_executable_args '7175' 
Service Register succeeded - Registered. - CUSTOM:localhost:7175 ID[5]

Start driver

degenaro@uima-ducc-vm:~$ cd ~/ducc/install/examples/uima-ducc-vm/driver
degenaro@uima-ducc-vm:~/ducc/install/examples/uima-ducc-vm/driver$ ./driver_start
degenaro@uima-ducc-vm:~/ducc/install/examples/uima-ducc-vm/driver$ nohup: appending output to `nohup.out'

degenaro@uima-ducc-vm:~/ducc/install/examples/uima-ducc-vm/driver$ tail -n 20 -f nohup.out 
driver.start
init:1
init:2
run.start:1
Thread-1 seconds to sleep: 124
init:3
run.start:2
Thread-2 seconds to sleep: 90
init:4
run.start:3
Thread-3 seconds to sleep: 22
init:5
run.start:4
Thread-4 seconds to sleep: 243
run.start:5
Thread-5 seconds to sleep: 225
dodo
['/home/degenaro/ducc/versions/apache-uima-ducc-1.0.1-SNAPSHOT/bin/ducc_submit', '--wait_for_completion', '--scheduling_class', 'normal', '--process_memory_size', '30', '--process_thread_count', '6', '--log_directory', '/tmp/ducc/driver/dodo/ducc/logs', '--working_directory', '/tmp/ducc/driver/dodo/ducc/work/3', '-f', '/home/degenaro/ducc/versions/apache-uima-ducc-1.0.1-SNAPSHOT/examples/uima-ducc-vm/jobs/2.job']
...

Check for Jobs to be running

Visit uima-ducc-vm.apache.org:42133/jobs.jsp


End.

Check out 1.0.0

[degenaro]$ svn co https://svn.apache.org/repos/asf/uima/sandbox/uima-ducc/branches/uima-ducc-1.0.0

 

 

  • No labels