Apache Airavata
Tutorial at the SDSC 2012 Summer Institute - SDSC-SI
Tutorial Title: Developing Science Gateways using Apache Airavata API
Date: Thursday August 8th 2012
Venue: University of California at San Diego, SDSC Buillding
Software Requirements: Java, SSH Shell, Web Browser
Prerequisite Knowledge: General understanding of Cluster and Grid computing, Batch Job Submission, Science Portals/Gateways. Knowledge of Java & Web services is preferable but not required.
Description:
Science Gateways simplify access to XSEDE by providing easy to use familiar interfaces. Gateways abstract the complexity of various grid middleware that in turn absorb the heterogeneity of the underlying batch management systems. Gateway developers are challenge in building and maintaining gateways and are often overwhelmed in choosing middleware. Often these middleware solutions are black boxes to the developers that make them expend valuable man hours in debugging real-time problems. This tutorial will provide by means of number of hands on tutorials practical insights into gateway building aspects. Alongside the tutorial will provide insights into aspects of open community software development practices to not just use open software but how to contribute, get recognized and also have a say in future directions.
The tutorial will be based on Apache Airavata, a software toolkit to build science gateways. Airavata providing features to compose, manage, execute, and monitor small to large-scale applications and workflows on computational resources ranging from local clusters to national grids like XSDE and computing clouds like Amazon Elastic Compute Cloud. Airavata builds on general concepts of service oriented computing, distributed messaging, workflow composition and orchestration. Airavata suite include tools for workflow composition and monitoring. The stand out feature of workflow engine allows to interpret at each step providing dynamic interactive capabilities. The core capabilities provide ability to wrap command line-driven science applications and make them into robust, network- accessible services. Airavata Registry provides persistent data store. The gateway building toolkit also includes publish-subscribe based messaging system with features to incorporate clients behind firewalls and overcomes network glitches.
Target Audience and take home lessons:
The tutorial is targeted for Gateway developers who want to build a new gateway from scratch; for advanced developers who are investigating to improve a particular capability; and Campus Champions who can learn easier mechanisms to move data and execute jobs on XSEDE. The take home lessons include skills to execute compute jobs and workflows on XSEDE resources through grid interfaces; Wrap command line applications turning them into web accessible programmable interfaces and learn about open community development process to contribute to existing software with proper contribution governance models.
Download XBaya - JNLP Link
or
Use the binary distribution (Mac/Linux):
Select menu,
Registry -> Setup Airavata Registry...
Provide the following,
URL
http://demo.airavata.org:8090/jackrabbit/rmi
or
http://gf9.ucs.indiana.edu:8090/jackrabbit-webapp-2.4.0/rmi
Username
admin
Password
admin
Click "OK"
To register a new application select menu,
XBaya -> Register Application menu.
Provide the following Application details
Application name:
list_files (or any other unique name)
Application Parameters:
I/O |
Name |
Type |
Input |
path |
String |
Input |
options |
String |
Output |
result |
StdOut |
Add Deployments
Click on the “New deployment” button. “New Application Deployment” form will appear.
Provide the following details
Application host
ranger
Executable path
/bin/ls
Scratch working directory
/scratch/01437/ogce/test
Click on the “HPC Configuration...” button. The HPC Configuration Options form will appear. Provide the following details,
Job Type
serial
Project Account Number
TG-STA110014S
Queue Type
normal
Click “Update”
Click “Add”
Click “Register”. You will see a Successfully saved confirmation message if the registration is successful.
After registering the Application successfully you will see your application in Left hand side Application Services list. Create a workflow using that service and run the workflow as if you did in previous workflow run.
To add a new workflow select menu,
XBaya -> New Workflow.
A new Workflow editor will open along with the Workflow Properties dialog for that workflow. Enter the following details in the dialog
Name
retrieve_files_names
Click “OK”
From the left hand side “Component List” drag & drop the following to the Workflow editor.
2 x System Components -> Input
1 x System Components -> Output
1 x Application Services -> list_files
Connect output of “Input” components to inputs of “list_files_invoke”
Connect output of “list_files_invoke” component to input of “Output”
You may save the workflow by menu item
XBaya -> Save (in order to save it in the local file system)
or
XBaya -> Export -> Workflow to Registry... (in order to save it in the Airavata Registry)
If the workflow is not already opened in XBaya, select menu,
XBaya -> Open Workflow
or
XBaya -> Import -> Workflow from Registry
based on whether the workflow was saved in the file system or Airavata Registry.
Once the workflow is loaded on to XBaya select menu,
Run -> Run on Interpreter Server...
Provide the following as values for the parameters
path
/bin
options
-al
Experiment name
Executables in ranger (or any name you want to call it)
Click “Run”
The monitoring panel docked at the bottom of XBaya should get focus and show the notifications of the execution of the workflow.
The workflow loaded in XBaya should indicate the progress of the workflow execution.
In XBaya select menu,
View -> Airavata Registry
This will open a panel in XBaya docked at the left hand side. This will show the saved content in the Airavata Registry.
Expand the “Experiments” node to view the details of the experiment.
If you reached this step that means you have followed the instructions given here carefully and successfully launched a job in Ranger.
So now you might think that you want to run your own application installed on Ranger. In that case you wont be able to use this same XBaya instance (which was started through JNLP) to invoke your application. That is because, this demo uses a community account associated with the Science Gateway Group to submit the jobs to Ranger. Since you have installed your application through your Ranger account, you should configure backend Airavata server to talk to Ranger properly with your XSEDE credentials.
Please refer the blog post to configure your Airavata Server properly to submit jobs to Ranger.
If you run into any issues please contact airavata-user mailing list and post your question.