IWSG2014: Developing Science Gateways Using Apache Airavata
•Download as PPTX, PDF•
0 likes•481 views
Presentation on using Apache Airavata to build science gateways at the International Workshop on Science Gateways (IWSG) 2014 in Dublin, Ireland.
Report
Share
IWSG2014: Developing Science Gateways Using Apache Airavata
- 1. Developing Computational Science Gateways using Apache Airavata Marlon Pierce Science Gateway Group, Indiana University marpierc@iu.edu
- 2. 1. We Are Hiring Apply: http://goo.gl/LbiXNO
- 4. 3. We’ll get to this one later…. http://www.bayoubelles.com/2011/08/night-before-big-day-was-not-what-was.html
- 5. Airavata’s Philosophy • There a lots of ways to build Web interfaces for Science Gateways. – By Hand: PHP, Twitter Bootstrap, AngularJS, … – Turnkey Frameworks: Liferay, Drupal, Plone, Joomla, … – Science Gateway Frameworks: the SDSC Workbench, HUBzero, gUse/WS-PGrade • Gateway developers should concentrate on building interfaces that serve their community. • And outsource the general purpose services to Airavata.
- 6. Airavata Role and Goals: Improve sustainability by converging on a single set of hosted infrastructure services
- 7. Apache Airavata Components Component Description Airavata API Server Apache Thrift-generated server skeletons of the API and data models; directs traffic to appropriate components Registry Insert and access application, host machine, workflow, and provenance data. Orchestrator Handles experiment execution request validation, scheduling, and decision making; selects GFAC instances that can fulfill a given request GFAC Manages the execution and monitoring of an individual application. Workflow Interpreter Execute the workflow on one or more resources. Messaging System WS-Notification and WS-Eventing compliant publish/subscribe messaging system for workflow events
- 9. Airavata’s Future Directions: SciGaP • SciGaP: Airavata as a multi-tenanted Gateway Platform as a Service • Goal: We run Airavata so you don’t have to. – Scalable support • Challenges: –Centralize system state –Make Airavata more cloud friendly, elastic http://scigap.org
- 10. Some Contribution Opportunities Component Research Opportunities Registry Better support for Thrift-generated objects; NoSQL and other backend data stores; fault tolerance Orchestrator Pluggable scheduling; load balancing and elasticity GFAC ZooKeeper-like strategies for configuring and managing. Messenger Investigate AMQP, Kafka, and other newer messaging systems Workflow Interpreter Alternative workflow processing engines. Overall Message-based rather than direct CPI calls. Airavata components expose Component Programming Interfaces (CPIs) that allow you to switch out implementations. GFAC is also designed to be pluggable.
- 11. Airavata and the Apache Software Foundation Experiences with Open Governance
- 12. Cyberinfrastructure: How open is open source? • What’s missing? – Open source licensing – Open Standards – Open Code (GitHub, SourceForge, Google Code, etc) We also need Open Governance
- 13. Open Source Software and Governance • Open source projects need diversity, governance. – Sustainability • Incentives for projects to diversify their developer base. • Govern – Software releases – Contributions – Credit sharing. – Members are added – Project direction decisions. – IP, legal issues • Our approach: Apache Software Foundation Collaborate Compete
- 14. “Apache” Means “Open” Join the Airavata developer or architecture mailing lists, get involved, submit patches, contribute. Use Give Back
- 15. More Information • Contact Us: – marpierc@iu.edu, smarru@iu.edu – Join dev@airavata.apache.org, users@airavata.apache.org, architecture@airavata.apache.org • Websites: – Apache Airavata: http://airavata.apache.org – SciGaP Project: http://scigap.org – Science Gateway Institute: http://sciencegateways.org
- 16. Airavata Audiences Audience Description Gateway Developers Use the Airavata API through and SDK in their favorite programming language. Airavata Developers Want to change Airavata components, experiment with different implementations. Middleware Developers Want to extend Airavata to talk to their middleware clients. Resource Providers Want to configure Airavata to work with their middleware.
- 17. Apache Contributions Aren’t Just Software • Apache committers and PMC members aren’t just code writers. • Successful communities also include – Important users – Project evangelists – Content providers: documentation, tutorials – Testers, requirements providers, architects, and constructive complainers • Using Jira and mailing lists – Anything else that needs doing.
- 18. Getting Involved, Contributing Back • Airavata is open source, open community software. • Open Community: you can contribute back – Patches, suggestions, wiki documentation, etc • We reward contributors – Committers: write access to master Git repo – Project Management Committee members: full, binding voting rights
- 19. Airavata Audiences Audience Description Gateway Developers Use the Airavata API through and SDK in their favorite programming language. Airavata Developers Want to change Airavata components, experiment with different implementations. Middleware Developers Want to extend Airavata to talk to their middleware clients. Resource Providers Want to configure Airavata to work with their middleware.
- 20. The Apache Software Foundation • Apache software powers 65% of web sites worldwide • 501(c)3 non-profit foundation • Reasons for creating ASF – Create legal entity – Protect contributors from liability – Protect Apache assets • Membership: individual • Apache Incubator • Governance and Staffing – Board of Directors – Project Management Committees – ASF Members – Committers – Contributors • Funding – All-volunteer staffing/development resources – Donations – Corporate investment
- 21. Where Is Airavata 1.0? • Airavata 1.0 will be the stable version of the API. • Version 1.0 determined by developer community vote. • Semantic versioning
- 22. Airavata Thrift Definitions airavata/airavata-api/thrift-interface-descriptions Thrift File Contents airavataAPI Definitions of services; top level file for other Thrift files. airavataErrors Definitions of exceptions thrown by the API services. airavataDataModel Top level container for data models. workspaceModel Definitions of the structs (Project, User, Gateway, Group); includes experimentModel experimentModel Definitions of data structures that constitute the experimentModel. applicationCatalogAPI App catalog service definitions. applicationCatalogDataModel App catalog data model definitions.
- 23. Airavata API and Apache Thrift • We use Apache Thrift to define the API. • Advantages of Thrift – Supports well-defined, typed messages. – Custom defined errors, exceptions – Generators for many different programming languages. – Some shielding from API versioning problems. • Downsides of Thrift – No message headers, so everything must be explicitly defined in the API.
- 24. A Few Observations on Successful Gateways • Support familiar community applications. • Make HPC systems easy for new user communities who need HPC. • Keep it simple. • Have champions who build and support the community. • Have a lot of common features.
- 25. Airavata API and Apache Thrift • We use Apache Thrift to define the API. – TCP/IP typically – Not WS, not REST • Advantages of Thrift – Supports well-defined, typed messages. – Custom defined errors, exceptions – Generators for many different programming languages. – Some shielding from API versioning problems. • Downsides of Thrift – No message headers, so everything must be explicitly defined in the API.
- 26. Airavata API Thrift Definitions airavata/airavata-api/thrift-interface-descriptions Thrift File Contents airavataAPI Definitions of services; top level file for other Thrift files. airavataErrors Definitions of exceptions thrown by the API services. airavataDataModel Top level container for data models. workspaceModel Definitions of the structs (Project, User, Gateway, Group); includes experimentModel experimentModel Definitions of data structures that constitute the experimentModel. applicationCatalogAPI App catalog service definitions. applicationCatalogDataModel App catalog data model definitions.