SlideShare a Scribd company logo

WS-VLAM workflow

•Download as PPT, PDF•
•
G
guest6295d0

- The presentation introduces WS-VLAM, a workflow management system that aims to enable end-users to define, execute, and monitor e-science applications in a location-independent way. - WS-VLAM adopts a service-oriented approach, implementing the workflow engine and repository as WSRF services using Globus Toolkit 4. This allows for interoperability with other workflow systems. - Current work involves testing on rapid prototyping environments and planned integration with Taverna and Kepler to allow executing predefined VLAM workflows from within those systems.

Related slideshows

Machine Learning At Speed: Operationalizing ML For Real-Time Data Streams
Machine Learning At Speed: Operationalizing ML For Real-Time Data StreamsMachine Learning At Speed: Operationalizing ML For Real-Time Data Streams
Machine Learning At Speed: Operationalizing ML For Real-Time Data Streams
 
How To Build, Integrate, and Deploy Real-Time Streaming Pipelines On Kubernetes
How To Build, Integrate, and Deploy Real-Time Streaming Pipelines On KubernetesHow To Build, Integrate, and Deploy Real-Time Streaming Pipelines On Kubernetes
How To Build, Integrate, and Deploy Real-Time Streaming Pipelines On Kubernetes
 
Lessons from Building Large-Scale, Multi-Cloud, SaaS Software at Databricks
Lessons from Building Large-Scale, Multi-Cloud, SaaS Software at DatabricksLessons from Building Large-Scale, Multi-Cloud, SaaS Software at Databricks
Lessons from Building Large-Scale, Multi-Cloud, SaaS Software at Databricks
 
1 of 21
WS-VLAM Introduction Systems and Network Engineering group Institute of informatics University of Amsterdam
Introduction This presentation gives a quick introduction to the WS-VLAM workflow management system It helps the reader understand the technology push which motivated our design, describes the current state and present the ongoing activities and planned ones . NOTE : We are currently still working on the User’s and developer’s Guide, they are not available at this time . The presentations do not replace the documentation it just give a quick introduction. Do not hesitate to contact us for further information Thanks for taking some of your time to read about the WS-VLAM work, any comments are more than welcome WS-VLAM developers [email_address]
Needed background … To be able to follow the content of this presentation you will need to know about: WSRF = Web Service Reference Framework GT4 = Globus Toolkit version 4 More detailed description of the presented work can be found in the section of publication of the WS-VLAM home page
Objective of this presentation… Why you should look at this presentations You want to know how WS-VLAM works and interacts with the Grid middleware. Why you don’t have to worry if you do not understand the content on this presentation: You are an end-user: it will be enough for you to read the User’s guide or the WS-VLAM composer presentation You are an application developer: it will be enough for you to read the Developer’s guide & the WS-VLAM developers presentation
Outline Introduction Architecture of the WS-VLAM Hierarchical workflow compositions & execution Interoperability with other WMS Conclusions
GVLAM: Enable end-users (scientists) to define , execute , and monitor e-science applications. Location independent experimentation. Application Domain portal. Provide application developers : Abstract view of the Grid (shields the Grid details). Easy way to port applications to the Grid.
The Virtual Laboratory for e-Science … Data intensive science Generic e -science framework layer Application layer Bioinformatics ASP Grid layer Dutch telescience Medical diagnosis Bio diversity Food Informatics
Before and after OGSA/WSRF Process Management Stack Network & Computing Resources Grid Middleware: Process & resource management Application Templates User Applications Workflow Management system Web Service Interface Data Management Stack Network & storage Resources Grid Middleware: Data management Web Service Interface
Workflow Management Systems Process Management Stack Data Management Stack Network & Computing Resources Grid Middleware: Process & resource management Application Templates User Applications Workflow execution Workflow composition Network & storage Resources Grid Middleware: Data management Workflow Management system Web Service Interface Web Service Interface
Grid Service & OGSA-DAI Data Management Stack Process Management Stack Network & Computing Resources Network & storage Resources Grid Middleware: Process & resource management Grid Middleware: Data management Application Users Workflow execution OGSA DAI Application Templates User Applications Workflow composition Workflow Management system Web Service Interface Web Service Interface Grid Services Grid Services Grid Services
First Application developers … Data Management Stack Process Management Stack Network & Computing Resources Network & storage Resources Grid Middleware: Process & resource management Grid Middleware: Data management Workflow execution Grid Services Application web service Grid Services Grid Services Application Templates User Applications Application Templates Web service Workflow composition Web Service Interface Web Service Interface OGSA DAI Workflow Management system
Second the Workflow developers … Process Management Stack Data Management Stack Network & Computing Resources Network & storage Resources Grid Middleware: Process & resource management Grid Middleware: Data management Workflow execution Workflow composition Workflow Management system Grid Services Grid Services Grid Services Application Templates Web service Web Service Interface OGSA DAI Workflow Engine Web service Workflow Engine Web service Web Service Interface Application web service Distributed Workflow
WS-VLAM WMS… Goals Create a WMS which adopt a Service Oriented Paradigm Approach Workflow Engine (RTSM) Implemented as a WSRF service using Globus 4 (GT4) Workflow Composer (GUI) Thin client Repository for workflow components Implemented also as WSRF service.
Conclusions WSRF compliant middleware such as GT4 offers useful basic services to develop a truly grid enabled workflow engine: Delegation service and ws-notification mechanisms proved to be very useful Separating the workflow composition from execution suits better the Grid environments and improve interoperability OGSA-WG is collecting cases and input to decide which services should be standardized
http://www.vl-e.nl/
New workflow Components ( Under development ) The workflow components developed the last year, can be still used and will be deployed on the shared workflow component repository on the rapid prototyping environment Ridge-O-Grammar modules from (SP 1.5) Kansk Toolbox from (SP 1.4) AID web services (SP 2.2) Virtual file system Web service (SP 2.4) Generic workflow components (SP 2.5) File Reader/Writer, universal File Reader/Writer (SRB/GFTP/...) Demos: MRIscanner, Histogram, Floatingball We will contact other SPs to integrate their components Legacy code, or fully integrated code (java, C++, Python)
Current deployment & validation Current tests are performed on the rapid prototyping environment RTSM Service and Repository service are deployed on ds2a.das2.nikhef.nl ws-VLAM Client (3 MB) can be used only on ds2.das2.nikhef (because GT4 is only available on this node) ws-VLAM client can also be used a host where Poc R1 is installed Tests will be performed on the validation environment for PoC R2
Planned tasks Access to the ws-vlam engine from Taverna This will be developed in the context of a use case together with SP1.5 Access to the ws-vlam engine from Kepler Parameter sharing among workflow components Job farming Support for more than one job submission mechanisms (GT4-GRAM, Direct, etc) Fault tolerance Control flow Data provenance Semantic Service discovery
References and download VLAM client can be downloaded from www.science.uva.nl/~gvlam/wsvlam/ Documentation can be downloaded form ( TO BE DONE VERY SOON ) www.science.uva.nl/~gvlam/wsvlam/doc/
Workflow interoperability: Execute a VLAM workflow from Kepler/Taverna A predefined Application workflow developed in VLAM can be executed as a single step in Kepler/Taverna (no need to recompose graphically the whole workflow). The predefined Application workflow will be executed on any remote computing resource where the VLAM-RTSM GT4 Web service is installed. Advantages: Compose workflow where sub-workflows ( which require grid resources ) are executed on grid-enabled resources, while the rest of the workflow is either executed using other Kepler actors or taverna processors It is also more efficient , since it avoid the overhead which will result by wrapping every workflow component as a separated web service or a separate remote grid-execution.
Workflow interoperability: Execute a VLAM workflow from Kepler/Taverna GRAM services GT4 Java Container RTSM Factory Delegation service Worker nodes pre-ws-GRAM RTSM Instance Workflow components Kepler/Taverna workbench RTSM Client Workfow Description (XML) Das2 or PoC facilities. Kepler/Taverna users can have access to some of the parameters of the Application workflow to change the default values Kepler/Taverna users have to specify the location of the input data file as URL and will get back a URL if the Sigwin generate data files Graphical output of the Application workflow are handled automatically by the VLAM Taverna processor /Kepler actor. RTSM-GT4 Web service VLAM Actor or Taverna processor (1) Proxy Delegate (2) Service Invocation

More Related Content

WS-VLAM workflow

  • 1. WS-VLAM Introduction Systems and Network Engineering group Institute of informatics University of Amsterdam
  • 2. Introduction This presentation gives a quick introduction to the WS-VLAM workflow management system It helps the reader understand the technology push which motivated our design, describes the current state and present the ongoing activities and planned ones . NOTE : We are currently still working on the User’s and developer’s Guide, they are not available at this time . The presentations do not replace the documentation it just give a quick introduction. Do not hesitate to contact us for further information Thanks for taking some of your time to read about the WS-VLAM work, any comments are more than welcome WS-VLAM developers [email_address]
  • 3. Needed background … To be able to follow the content of this presentation you will need to know about: WSRF = Web Service Reference Framework GT4 = Globus Toolkit version 4 More detailed description of the presented work can be found in the section of publication of the WS-VLAM home page
  • 4. Objective of this presentation… Why you should look at this presentations You want to know how WS-VLAM works and interacts with the Grid middleware. Why you don’t have to worry if you do not understand the content on this presentation: You are an end-user: it will be enough for you to read the User’s guide or the WS-VLAM composer presentation You are an application developer: it will be enough for you to read the Developer’s guide & the WS-VLAM developers presentation
  • 5. Outline Introduction Architecture of the WS-VLAM Hierarchical workflow compositions & execution Interoperability with other WMS Conclusions
  • 6. GVLAM: Enable end-users (scientists) to define , execute , and monitor e-science applications. Location independent experimentation. Application Domain portal. Provide application developers : Abstract view of the Grid (shields the Grid details). Easy way to port applications to the Grid.
  • 7. The Virtual Laboratory for e-Science … Data intensive science Generic e -science framework layer Application layer Bioinformatics ASP Grid layer Dutch telescience Medical diagnosis Bio diversity Food Informatics
  • 8. Before and after OGSA/WSRF Process Management Stack Network & Computing Resources Grid Middleware: Process & resource management Application Templates User Applications Workflow Management system Web Service Interface Data Management Stack Network & storage Resources Grid Middleware: Data management Web Service Interface
  • 9. Workflow Management Systems Process Management Stack Data Management Stack Network & Computing Resources Grid Middleware: Process & resource management Application Templates User Applications Workflow execution Workflow composition Network & storage Resources Grid Middleware: Data management Workflow Management system Web Service Interface Web Service Interface
  • 10. Grid Service & OGSA-DAI Data Management Stack Process Management Stack Network & Computing Resources Network & storage Resources Grid Middleware: Process & resource management Grid Middleware: Data management Application Users Workflow execution OGSA DAI Application Templates User Applications Workflow composition Workflow Management system Web Service Interface Web Service Interface Grid Services Grid Services Grid Services
  • 11. First Application developers … Data Management Stack Process Management Stack Network & Computing Resources Network & storage Resources Grid Middleware: Process & resource management Grid Middleware: Data management Workflow execution Grid Services Application web service Grid Services Grid Services Application Templates User Applications Application Templates Web service Workflow composition Web Service Interface Web Service Interface OGSA DAI Workflow Management system
  • 12. Second the Workflow developers … Process Management Stack Data Management Stack Network & Computing Resources Network & storage Resources Grid Middleware: Process & resource management Grid Middleware: Data management Workflow execution Workflow composition Workflow Management system Grid Services Grid Services Grid Services Application Templates Web service Web Service Interface OGSA DAI Workflow Engine Web service Workflow Engine Web service Web Service Interface Application web service Distributed Workflow
  • 13. WS-VLAM WMS… Goals Create a WMS which adopt a Service Oriented Paradigm Approach Workflow Engine (RTSM) Implemented as a WSRF service using Globus 4 (GT4) Workflow Composer (GUI) Thin client Repository for workflow components Implemented also as WSRF service.
  • 14. Conclusions WSRF compliant middleware such as GT4 offers useful basic services to develop a truly grid enabled workflow engine: Delegation service and ws-notification mechanisms proved to be very useful Separating the workflow composition from execution suits better the Grid environments and improve interoperability OGSA-WG is collecting cases and input to decide which services should be standardized
  • 16. New workflow Components ( Under development ) The workflow components developed the last year, can be still used and will be deployed on the shared workflow component repository on the rapid prototyping environment Ridge-O-Grammar modules from (SP 1.5) Kansk Toolbox from (SP 1.4) AID web services (SP 2.2) Virtual file system Web service (SP 2.4) Generic workflow components (SP 2.5) File Reader/Writer, universal File Reader/Writer (SRB/GFTP/...) Demos: MRIscanner, Histogram, Floatingball We will contact other SPs to integrate their components Legacy code, or fully integrated code (java, C++, Python)
  • 17. Current deployment & validation Current tests are performed on the rapid prototyping environment RTSM Service and Repository service are deployed on ds2a.das2.nikhef.nl ws-VLAM Client (3 MB) can be used only on ds2.das2.nikhef (because GT4 is only available on this node) ws-VLAM client can also be used a host where Poc R1 is installed Tests will be performed on the validation environment for PoC R2
  • 18. Planned tasks Access to the ws-vlam engine from Taverna This will be developed in the context of a use case together with SP1.5 Access to the ws-vlam engine from Kepler Parameter sharing among workflow components Job farming Support for more than one job submission mechanisms (GT4-GRAM, Direct, etc) Fault tolerance Control flow Data provenance Semantic Service discovery
  • 19. References and download VLAM client can be downloaded from www.science.uva.nl/~gvlam/wsvlam/ Documentation can be downloaded form ( TO BE DONE VERY SOON ) www.science.uva.nl/~gvlam/wsvlam/doc/
  • 20. Workflow interoperability: Execute a VLAM workflow from Kepler/Taverna A predefined Application workflow developed in VLAM can be executed as a single step in Kepler/Taverna (no need to recompose graphically the whole workflow). The predefined Application workflow will be executed on any remote computing resource where the VLAM-RTSM GT4 Web service is installed. Advantages: Compose workflow where sub-workflows ( which require grid resources ) are executed on grid-enabled resources, while the rest of the workflow is either executed using other Kepler actors or taverna processors It is also more efficient , since it avoid the overhead which will result by wrapping every workflow component as a separated web service or a separate remote grid-execution.
  • 21. Workflow interoperability: Execute a VLAM workflow from Kepler/Taverna GRAM services GT4 Java Container RTSM Factory Delegation service Worker nodes pre-ws-GRAM RTSM Instance Workflow components Kepler/Taverna workbench RTSM Client Workfow Description (XML) Das2 or PoC facilities. Kepler/Taverna users can have access to some of the parameters of the Application workflow to change the default values Kepler/Taverna users have to specify the location of the input data file as URL and will get back a URL if the Sigwin generate data files Graphical output of the Application workflow are handled automatically by the VLAM Taverna processor /Kepler actor. RTSM-GT4 Web service VLAM Actor or Taverna processor (1) Proxy Delegate (2) Service Invocation