Apache Airavata

Short description: GenApp is a modular framework for multiscale science computations. Apache Airavata is a software framework for executing and managing computational jobs and workflows on distributed computing resources. The primary goal of this project is to integrate GenApp with Airavata to be able to exexcute long running computational jobs.GenApp also needs to be integrated with Airavata to enable composition and execution of workflows. GenApp needs to be integrated with Airavata's Messaging System as well.

Public URL: http://www.google-melange.com/gsoc/proposal/public/google/gsoc2014/nadee...

1. Introduction

1.1 About GenApp

GenApp is a modular framework for multiscale science computations. It enables one to generate web & gui applications from same source. 
The basic framework of GenApp can be seen below:  
   
       
 
The Salient features of GenApp include:
  • The target languages for the user interface side are totally independent of the modules
  • Different target applications or deployed systems can be generated from the specifications. There include HTML5 with PHP, Qt3 with C++, Qt4 with C++. Java, Android and iOS version will also be available shortly
  • It can generate multiple targets simultaneously

The benefits of GenApp include:

  • can add features, create interfaces etc. with minimal effort
  • changes to target code can be done without effecting the "base" logic
    • i.e. if we generate Qt4 GUI code and want to move to Qt6, our underlying modules don't need change
    • text files allow "mouse-free" generation
      • this means that further automation is easy
    • can later target an application managing the application itself

 1.2 About Apache Airavata

Apache Airavata is a software framework for executing and managing computational jobs and workflows on distributed computing resources including local clusters, supercomputers, national grids, academic and commercial clouds.
The components of Airavata include:
  • XBaya Workflow Suite - includes a GUI for workflow composition and monitoring
  • GFac - an application wrapper service 
  • WS-Messenger - a publish-subscribe based message broker
  • Registry-API - A thick client registry API for Airavata to put and get documents.
  • Thrift-API - allows building Airavata clients in different language interfaces
  • Workflow Interpreter
  • Workflow Model

1.3 Overview of Integration

The Integration of GenApp with Apache Airavata needs to be achieved at three levels:
  • Integration to allow GenApp the ability to execute long running jobs on various clusters
  • Integration to allow GenApp the ability for composition and execution of workflows
  • Integration of Airavata Messaging System with GenApp Messaging System

 2. Benefits for Airavata and GenApp

This section explains why this project will be beneficial to both Airavata and GenApp.

2.1 Benefits to Airavata

Airavata will have the following benefits from this project:

  • GenApp provides the capability to generate user interfaces quickly. Besides, that it provides the option to generate these interfaces in multiple languages. Airavata will benefit by having the option to generate these interfaces in multiple languages.
  • Airavata will also benefit from the various bits of codes and modules that GenApp provides. These modules can be chained together to create more complex modules.

 2.2 Benefits to GenApp

GenApp will have the following benefits from this project:

  • The integration with Airavata will provide GenApp the capability to run long running, non-interactive jobs on XSEDE. 
  • GenApp will be able to harness distributed computing resources including local clusters, supercomputers, national grids, academic and commercial clouds 
  • It will provide GenApp the ability to compose and execute workflows using Airavata's workflow engine. GenApp will then have the ability to chain its modules and to create more complex modules
  • GenApp will also benefit from Airavata's Messaging System, which provides a layer of messages from jobs submitted to cluster resources.
 
3. Detailed Description and Approach

 3.1. Integration of GenApp with \Airavata to allow execution of long running jobs on clusters

  • Genapp provides interfaces in various programming languages that need to be connected with Airavata to run various computational jobs
  • The following interfaces of GenApp will be integrated with Airavata:

            o   PHP/HTML5 

            o   C++ with Qt

            o   Java

  • Integration will be achieved using Airavata Thrift API, which has language interfaces in php, c++ and java. This will enable integration of GenApp interfaces with Airavata for executing long running computational jobs on clusters
  • This will involve creating calls in the wrapper based on certain variable settings in input json for targeting execution to Airavata
  • PHP integration can be achieved by using the Airavata PHP sdk, which allows connection with Airavata to get a job launched
  • Registration Issues: Apache Thrift API assumes that documents are already registered in the Airavata registry. Thus the registration needs to be performed before launching a job. The registration requires information on the needs, hosts, applications interfaces, among others
  • To address the registration issue, a registry file will be generated during the generation phase that will perform the registration
  • The GenApp codebase will be extended to perform registration with Airavata as well during the registration phase, using the Thrift API

 3.2. WorkFlow Integration

  • Airavata provides interfaces for composition, execution and monitoring of workflows
  • The GenApp modules can be used as building blocks to create more complex execution models
  • Each GenApp Module has associated inputs and outputs which can be chained with those of other modules into a workflow
  • GenApp Modules will be integrated with Airavata to allow composition, execution and monitoring of complex execution models using the GenApp modules as building blocks
  • Integration needs to be done with Airavata's Xbaya or Workflow Interpreter to achieve this
  • Since neither Xbaya nor Workflow Interpreter is exposed in the current API, a good design document will be worked upon to achieve this integration. The design document is the main deliverable for this section
  • Depending upon discussion with the community, a new "XBaya-lite" composition tool (based on Airavata API) could be implemented for composing workflows in GenApp

3.3 Messaging Integration

  • GenApp's Html/Php Language Implementation has a Messaging system based on WebSockets
  • GenApp's messaging system allows an executable to directly forward state information (UDP to not interrupt execution) to a websocket server which forwards onto the html5 clients for live status updates.
  • Apache Airavata includes WS Messenger, a web services based messaging system which guarantees scalable, reliable and efficient message delivery
  • Airavata messaging provides a different layer of messages from jobs submitted to cluster resources (i.e. there is information available even when the executable is not running - such as 'failed to start' or 'waiting in queue to run' or others).
  • GenApp messaging needs include getting updates back on .GenApp, messages sent by the executable to the users for live stauts updates, job submission messaging, among others
  • GenApp messaging systems will be integrated with the Airavata Messaging system to address the above mentioned needs
  • An implementation-Independent Component Programming Interface will be defined and this will be exposed through the API to GenApp

4. Delivarables

The deliverables to the community from this project include the following:
  • GenApp PHP/HTML5 interface integrated with Aiavata for executing long computational jobs
  • GenApp C++/Qt interface integrated with Aiavata for executing long computational jobs
  • GenApp Java interface integrated with Aiavata for executing long computational jobs
  • Design Document and/or implementation for Workflow Integration with GenApp
  • GenApp Messaging System  Integrated with Airavata Messaging System

5. Timeline

Community bonding period

April 21 - April 27 (1 Week)         : Understanding and playing around with GenApp codebase

April 28 - May 4 (1 Week)            : Understanding and playing around with Airavata and Thrift codebase

May 5 – May 11 (1 Week)            : Deciding the implementation model to integrate  GenApp PHP/HTML5 with Airavata to execute long running jobs on various clusters

May 12 – May 18 (1 Week)          : Deciding the implementation model to integrate  GenApp C++/Qt and Java versions with Airavata to execute long running jobs on various clusters

Coding period

May 19 – June 1 (2 Weeks)          : Integration of GenApp’s HTML5/PHP interface with Airatava

June 2 – June 15 (2 Weeks)         : Integration of GenApp’s Qt/C++ interface with Airatava

June 16 – June 29 (2 Weeks)       : Integration of GenApp’s Java Interface with Airavata

June 30 – July 20 (3 Weeks)        : Design Document and/or implementation of Workflow Integration

July 21 – August 10 (3 Weeks)     : Integration of GenApp with Airavata’s Messaging System

August 11 – August 18 (1 Week)  : Documentation and Testing

 

6. Milestones

  • Sprint 1: Bonding with community, learning more about GenApp and Airavata
  • Sprint 2: Finalizing implementation models for Integration of GenApp with Airavata for excuting long runnning jobs on various clusters.
  • Sprint 3: Integration of GenApp’s HTML5/PHP interface with Airatava
  • Sprint 4: Integration of GenApp’s Qt/C++ interface with Airatava
  • Sprint 5: Integration of GenApp’s Java Interface with Airavata
  • Sprint 6: Design Documenttation and/or implementation for GenApp -Airavata Workflow Integration.
  • Sprint 7: Integration of GenApp with Airavata’s Messaging System
  • Sprint 8: Testing, Documentation and Evaluation

7. Involvement with Apache Software Foundation Community

 7.1 Apache Airavata

I have investigated the Airavata's XBaya issue [1] reported by Heejoon related to security concerns with jre(51). My response can be seen at this thread [2] 

[1]: http://markmail.org/thread/c6exit64mmhhpew7

[2]: https://www.mail-archive.com/users@airavata.apache.org/msg00196.html

Besides I have been in constant touch with Dr. Suresh Marru, Dr. Emre Brookes and the mailing list regarding this project

7.2 Apache Spatial Information Systems

I was a GSoC 2013 participant with Apache SIS and worked on Agent Based Modeling based geo-profiling of criminology projects, which can be found at [1]. The related JIRA issue can be found at [2]. Even after GSoC 2013, I have continued contributing to the criminology project till date. 

[1]: http://www.google-melange.com/gsoc/project/google/gsoc2013/nadeemanjum/6591984525377536

[2]: https://issues.apache.org/jira/browse/SIS-97 

8. Post GSoC

After GSoc too, I will like to continue contributing to Apache Airavata

9. Personal Information and Background 

Name: Nadeem Anjum
University: Indian Institute of Technology (IIT), Kharagpur 
Major: Pursuing B.Tech in Computer Science and Engineering (Fourth Year Student)
Expected Graduation: May 2014
Homepage: http://cse.iitkgp.ac.in/~nanjum

Technical Skills: Well-versed in Java, C, C++, Pyhton, PHP, Javascript, Jquery, CSS, HTML, XML, MySQL

Experience: I have been involved in the following open source and development and research projects:

Contact Information:

  • Skype ID: nadeem.anjum.iitkgp

10. Availability

I am available full time during the entire GSoC 2014 period. I have no other commitments during this period
 
Typical Working Hours:
10 am to 1 pm, 2pm  to 7 pm IST
(6 days a week, amounting to 48 hours per week. If required, I can work 7 days a week too).
 
Other Commitments: None 
 
Any time in Summer where you will be away for more than 2 days: None

11. References

 1. http://people.apache.org/~smarru/papers/airavata-gce11.pdf

 2. http://gw105.iu.xsede.org:8000/genapp/wiki/preliminaryoverview

 

  • No labels

For an overview of Apache Airavata visit the Website - https://airavata.apache.org/