SlideShare a Scribd company logo

SFScon19 - Alexander Jacob - openEO

•
•
South Tyrol Free Software Conference
South Tyrol Free Software Conference

an API for standardised access to Big Earth Observation data in a landscape of a growing number of EO cloud providers 11:40 16/11/2019 With the opening of global archives of Earth Observation data streams from satellites we have arrived at a richness of operationally available observations over the whole globe, starting from the Landsat series of satellites and now the plethora of available data coming through Copernicus and its series of Sentinel satellites, that has never been available before. This created huge opportunities for research and businesses, being able to exploit the temporal domain of those observations in a powerful manner, but also poses challenges in terms of data management and processing capacities. As a consequence, a growing number of cloud services and customized solutions in various research centres have been developed, leading to processing workflows optimized for specific system architectures and back-end infrastructures. As such this is limiting portability and reproducibility of workflows across backends, both for science and business applications. (...)

Related slideshows

Peering Inside the Black Box: A Case for Observability
Peering Inside the Black Box: A Case for ObservabilityPeering Inside the Black Box: A Case for Observability
Peering Inside the Black Box: A Case for Observability
 
OpenACC and Hackathons Monthly Highlights
OpenACC and Hackathons Monthly HighlightsOpenACC and Hackathons Monthly Highlights
OpenACC and Hackathons Monthly Highlights
 
OpenACC Monthly Highlights September 2020
OpenACC Monthly Highlights September 2020OpenACC Monthly Highlights September 2020
OpenACC Monthly Highlights September 2020
 
1 of 49
Download to read offline
Authors: Alexander Jacob, Jeroen Dries , Matthias Mohr , Luca Foresta , Markus Neteler, Edzer Pebesma, Prateek Budhwar, Simone Tritini, Peter Zellner, Armin Costa, Matthias Schramm Contact: alexander.jacob@eurac.edu, openEO@list.tuwien.ac.at An API for Standardised Access to Big Earth Observation Data in a Landscape of a Growing Number of EO Cloud Providers
Why do we need openEO? 1openEO – SFSCon – 2019-11-16 Eurac Research 2019, contains modified Copernicus Sentinel data [2016], processed by ESA.
Why do we need openEO? 1openEO – SFSCon – 2019-11-16 Eurac Research 2019, contains modified Copernicus Sentinel data [2016], processed by ESA.
Why do we need openEO? 1openEO – SFSCon – 2019-11-16 Eurac Research 2019, contains modified Copernicus Sentinel data [2016], processed by ESA.
Why do we need openEO? 1openEO – SFSCon – 2019-11-16 Eurac Research 2019, contains modified Copernicus Sentinel data [2016], processed by ESA.
Why do we need openEO? 1openEO – SFSCon – 2019-11-16 Eurac Research 2019, contains modified Copernicus Sentinel data [2016], processed by ESA.
2 Why do we need openEO? CEPH_FS CEPH_RBD Hardware File Systems Cluster Orchestration & Virtual Environments Applications Data Models & Data Bases 1.4 PB 2 x 40 Gb/s 336 Cores 3 TB RAM ~50 Cores ~200 MB RAM e.g. pre - processing of S1 / S2 WCS WCPS SOS openEO – SFSCon – 2019-11-16
2 Why do we need openEO? CEPH_FS CEPH_RBD Hardware File Systems Cluster Orchestration & Virtual Environments Applications Data Models & Data Bases 1.4 PB 2 x 40 Gb/s 336 Cores 3 TB RAM ~50 Cores ~200 MB RAM e.g. pre - processing of S1 / S2 WCS WCPS SOS openEO – SFSCon – 2019-11-16
2 Why do we need openEO? CEPH_FS CEPH_RBD Hardware File Systems Cluster Orchestration & Virtual Environments Applications Data Models & Data Bases 1.4 PB 2 x 40 Gb/s 336 Cores 3 TB RAM ~50 Cores ~200 MB RAM e.g. pre - processing of S1 / S2 WCS WCPS SOS openEO – SFSCon – 2019-11-16
3 Why do we need openEO? CEPH_FS CEPH_RBD Hardware File Systems Cluster Orchestration & Virtual Environments Applications Data Models & Data Bases 1.4 PB 2 x 40 Gb/s 336 Cores 3 TB RAM ~50 Cores ~200 MB RAM e.g. pre - processing of S1 / S2 WCS WCPS SOS openEO – SFSCon – 2019-11-16
3 Why do we need openEO? openEO – SFSCon – 2019-11-16
3 Why do we need openEO? openEO – SFSCon – 2019-11-16
3 Why do we need openEO? openEO – SFSCon – 2019-11-16
3 Why do we need openEO? openEO – SFSCon – 2019-11-16
3 Why do we need openEO? openEO – SFSCon – 2019-11-16
3 Why do we need openEO? openEO – SFSCon – 2019-11-16
3 Why do we need openEO? openEO – SFSCon – 2019-11-16
3 Why do we need openEO? openEO – SFSCon – 2019-11-16
4 Why do we need openEO? API openEO – SFSCon – 2019-11-16
5 Why do we need openEO? API openEO – SFSCon – 2019-11-16
6 Getting started… • First check for existing drivers @ https://github.com/Open-EO https://open-eo.github.io/openeo-api/gettingstarted-backends/ RESTful implementation using OpenAPI Specification Version 3.0.1 openEO – SFSCon – 2019-11-16
6 Getting started… • First check for existing drivers @ https://github.com/Open-EO https://open-eo.github.io/openeo-api/gettingstarted-backends/ RESTful implementation using OpenAPI Specification Version 3.0.1 openEO – SFSCon – 2019-11-16
6 Getting started… • First check for existing drivers @ https://github.com/Open-EO • If an own implementation is needed: • You can still rely on some base functionality in the existing implementations • Or start with openAPI code generator @ https://github.com/OpenAPITools/openapi- generator • Start with implementing the essential endpoints https://open-eo.github.io/openeo-api/gettingstarted-backends/ RESTful implementation using OpenAPI Specification Version 3.0.1 openEO – SFSCon – 2019-11-16
7 The Endpoints of openEO root slash for capabilities and well-known for versioning/.well-known/openeo / /output_formats openEO – SFSCon – 2019-11-16 RESTful implementation using OpenAPI Specification Version 3.0.1
7 The Endpoints of openEO openEO strives for compatibility with STAC and OGC API as far as possible for data discovery. root slash for capabilities and well-known for versioning /collections /.well-known/openeo / /output_formats /collections/{collectionid} openEO – SFSCon – 2019-11-16 RESTful implementation using OpenAPI Specification Version 3.0.1
7 The Endpoints of openEO openEO strives for compatibility with STAC and OGC API as far as possible for data discovery. The basis for all computation are processes root slash for capabilities and well-known for versioning /collections /processes /.well-known/openeo / /output_formats /collections/{collectionid} openEO – SFSCon – 2019-11-16 RESTful implementation using OpenAPI Specification Version 3.0.1
7 The Endpoints of openEO openEO strives for compatibility with STAC and OGC API as far as possible for data discovery. The basis for all computation are processes Processes can be chained into process graphs root slash for capabilities and well-known for versioning /collections /processes /.well-known/openeo / /process_graphs /process_graphs/{graphID} /output_formats /collections/{collectionid} openEO – SFSCon – 2019-11-16 RESTful implementation using OpenAPI Specification Version 3.0.1
7 The Endpoints of openEO openEO strives for compatibility with STAC and OGC API as far as possible for data discovery. The basis for all computation are processes Processes can be chained into process graphs Results can be processed and downloaded synchronously Process graphs can be submitted as batch- jobs, queued for processing and results can be download upon completion root slash for capabilities and well-known for versioning /collections /processes /jobs/{jobid} /results /jobs/{jobid}/results /.well-known/openeo / /jobs /process_graphs /process_graphs/{graphID} /output_formats /collections/{collectionid} openEO – SFSCon – 2019-11-16 RESTful implementation using OpenAPI Specification Version 3.0.1
7 The Endpoints of openEO openEO strives for compatibility with STAC and OGC API as far as possible for data discovery. The basis for all computation are processes Processes can be chained into process graphs Results can be processed and downloaded synchronously Process graphs can be submitted as batch- jobs, queued for processing and results can be download upon completion Handling of user authentication and billing. root slash for capabilities and well-known for versioning /collections /processes /jobs/{jobid} /credentials/oidc /results /jobs/{jobid}/results /.well-known/openeo / /credentials/basic /jobs /process_graphs /process_graphs/{graphID} /output_formats /collections/{collectionid} openEO – SFSCon – 2019-11-16 RESTful implementation using OpenAPI Specification Version 3.0.1
7 The Endpoints of openEO openEO strives for compatibility with STAC and OGC API as far as possible for data discovery. The basis for all computation are processes Processes can be chained into process graphs Results can be processed and downloaded synchronously Process graphs can be submitted as batch- jobs, queued for processing and results can be download upon completion User can upload own files Handling of user authentication and billing. root slash for capabilities and well-known for versioning /collections /processes /jobs/{jobid} /files/{userid} /credentials/oidc /results /jobs/{jobid}/results /.well-known/openeo / /credentials/basic /jobs /process_graphs /process_graphs/{graphID} /output_formats /collections/{collectionid} openEO – SFSCon – 2019-11-16 RESTful implementation using OpenAPI Specification Version 3.0.1
7 The Endpoints of openEO openEO strives for compatibility with STAC and OGC API as far as possible for data discovery. The basis for all computation are processes Processes can be chained into process graphs Results can be processed and downloaded synchronously Process graphs can be submitted as batch- jobs, queued for processing and results can be download upon completion User can upload own files Handling of user authentication and billing. root slash for capabilities and well-known for versioning /collections /processes /jobs/{jobid} /files/{userid} /credentials/oidc /results /jobs/{jobid}/results /service_types/{serviceid} /.well-known/openeo / Consume results as secondary web services (e.g. WMS, XYZ, WCS) /credentials/basic /jobs /process_graphs /process_graphs/{graphID} /output_formats /collections/{collectionid} openEO – SFSCon – 2019-11-16 RESTful implementation using OpenAPI Specification Version 3.0.1
7 The Endpoints of openEO openEO strives for compatibility with STAC and OGC API as far as possible for data discovery. The basis for all computation are processes Processes can be chained into process graphs Results can be processed and downloaded synchronously Process graphs can be submitted as batch- jobs, queued for processing and results can be download upon completion User can create and integrate user defined functions into process graphs User can upload own files Handling of user authentication and billing. root slash for capabilities and well-known for versioning /collections /processes /jobs/{jobid} /files/{userid} /credentials/oidc /results /jobs/{jobid}/results /service_types/{serviceid} /.well-known/openeo / Consume results as secondary web services (e.g. WMS, XYZ, WCS) /credentials/basic /jobs /process_graphs /process_graphs/{graphID} /output_formats /udf_runtimes /collections/{collectionid} openEO – SFSCon – 2019-11-16 RESTful implementation using OpenAPI Specification Version 3.0.1
8 The Architecture of openEO everything is JSON openEO – SFSCon – 2019-11-16
9 Useful resources http://docs.openeo.org https://open-eo.github.io/openeo-api/apireference/ openEO – SFSCon – 2019-11-16
9 Useful resources http://docs.openeo.org https://open-eo.github.io/openeo-api/apireference/ http://processes.openeo.org openEO – SFSCon – 2019-11-16
10 The Implementations of openEO Back-end Implementations Client Implementations openEO – SFSCon – 2019-11-16
11 Implementations: http://hub.openeo.org openEO – SFSCon – 2019-11-16
11 Backends: http://hub.openeo.org openEO – SFSCon – 2019-11-16
12 WCPS backend (EURAC): https://openeo.eurac.edu CEPH_FS CEPH_RBD Hardware File Systems Cluster Orchestration & Virtual Environments Applications Data Models & Data Bases 1.4 PB 2 x 40 Gb/s 336 Cores 3 TB RAM ~50 Cores ~200 MB RAM e.g. pre - processing of S1 / S2 WCS WCPS SOS https://github.com/Open-EO/openeo-wcps-driver openEO – SFSCon – 2019-11-16
12 WCPS backend (EURAC) : https://openeo.eurac.edu CEPH_FS CEPH_RBD Hardware File Systems Cluster Orchestration & Virtual Environments Applications Data Models & Data Bases 1.4 PB 2 x 40 Gb/s 336 Cores 3 TB RAM ~50 Cores ~200 MB RAM e.g. pre - processing of S1 / S2 WCS WCPS SOS https://github.com/Open-EO/openeo-wcps-driver openEO – SFSCon – 2019-11-16
13 WCPS backend (EURAC) https://github.com/Open-EO/openeo-wcps-driver Based on swagger-jersey2-jaxrs for rest API implementation. Sqlite for openEO related DB => Batch job management, storing of process graphs GDAL for image operations and coordinate transformations JJWT for openID connect implementation => Linked to Microsoft azure for authentication Packaged as web archive using maven => Deployable on any java capable web container (e.g. tomcat or jetty) openEO – SFSCon – 2019-11-16
13 WCPS backend (EURAC) https://github.com/Open-EO/openeo-wcps-driver Based on swagger-jersey2-jaxrs for rest API implementation. Sqlite for openEO related DB => Batch job management, storing of process graphs GDAL for image operations and coordinate transformations JJWT for openID connect implementation => Linked to Microsoft azure for authentication Packaged as web archive using maven => Deployable on any java capable web container (e.g. tomcat or jetty) Configuration: • Properties File • WCPS endpoint • openEO endpoint • Authentication endpoint (for oidc) • DB location • TMP location • Session timings (auth expiry, tmp duration, etc.) openEO – SFSCon – 2019-11-16
13 WCPS backend (EURAC) https://github.com/Open-EO/openeo-wcps-driver Based on swagger-jersey2-jaxrs for rest API implementation. Sqlite for openEO related DB => Batch job management, storing of process graphs GDAL for image operations and coordinate transformations JJWT for openID connect implementation => Linked to Microsoft azure for authentication Packaged as web archive using maven => Deployable on any java capable web container (e.g. tomcat or jetty) Configuration: • Properties File • WCPS endpoint • openEO endpoint • Authentication endpoint (for oidc) • DB location • TMP location • Session timings (auth expiry, tmp duration, etc.) • Setup of Host Environment • Centos 7 or Ubuntu 18.04 • Install Tomcat 7 or later • Configure for https • Install sqlite (v3) & GDAL (v2.4) • Deploy openEO.war • Setup of proxy server for public access openEO – SFSCon – 2019-11-16
14 Client Examples (Python) openEO – SFSCon – 2019-11-16
15 Client Examples (Python) openEO – SFSCon – 2019-11-16
15 Client Examples (Python) openEO – SFSCon – 2019-11-16
16 Client Examples (JavaScript): https://editor.openeo.org openEO – SFSCon – 2019-11-16
17 Conclusions • A RESTful API has been defined • Interface human and machine readable (JSON) • Process catalogue • Extendable by user defined functions • Standardized process graph for abstract description of EO computational workflows • A number of reference implementations are currently in development • Based on different programming languages • Python • Java • R • Java-script • Based on different existing hard and software infrastructure • Can be used as starting point for own implementation • Together with extensive documentation • And guides • All of this is open source openEO – SFSCon – 2019-11-16
Thank you for your attention! H2020 openEO EO-2-2017: EO Big Data Shift Grant agreement No 776242 Contact: http://openeo.org/ openEO@list.tuwien.ac.at https://github.com/Open-EO @open_EO https://www.youtube.com/channel/UCMJQil8 j9sHBQkcSlSaEsvQ https://www.researchgate.net/project/openEO https://openeo-chat.eodc.eu/channel/public https://zenodo.org/communities/openeo alexander.jacob@eurac.edu openEO – SFSCon – 2019-11-16

More Related Content

SFScon19 - Alexander Jacob - openEO

  • 1. Authors: Alexander Jacob, Jeroen Dries , Matthias Mohr , Luca Foresta , Markus Neteler, Edzer Pebesma, Prateek Budhwar, Simone Tritini, Peter Zellner, Armin Costa, Matthias Schramm Contact: alexander.jacob@eurac.edu, openEO@list.tuwien.ac.at An API for Standardised Access to Big Earth Observation Data in a Landscape of a Growing Number of EO Cloud Providers
  • 2. Why do we need openEO? 1openEO – SFSCon – 2019-11-16 Eurac Research 2019, contains modified Copernicus Sentinel data [2016], processed by ESA.
  • 3. Why do we need openEO? 1openEO – SFSCon – 2019-11-16 Eurac Research 2019, contains modified Copernicus Sentinel data [2016], processed by ESA.
  • 4. Why do we need openEO? 1openEO – SFSCon – 2019-11-16 Eurac Research 2019, contains modified Copernicus Sentinel data [2016], processed by ESA.
  • 5. Why do we need openEO? 1openEO – SFSCon – 2019-11-16 Eurac Research 2019, contains modified Copernicus Sentinel data [2016], processed by ESA.
  • 6. Why do we need openEO? 1openEO – SFSCon – 2019-11-16 Eurac Research 2019, contains modified Copernicus Sentinel data [2016], processed by ESA.
  • 7. 2 Why do we need openEO? CEPH_FS CEPH_RBD Hardware File Systems Cluster Orchestration & Virtual Environments Applications Data Models & Data Bases 1.4 PB 2 x 40 Gb/s 336 Cores 3 TB RAM ~50 Cores ~200 MB RAM e.g. pre - processing of S1 / S2 WCS WCPS SOS openEO – SFSCon – 2019-11-16
  • 8. 2 Why do we need openEO? CEPH_FS CEPH_RBD Hardware File Systems Cluster Orchestration & Virtual Environments Applications Data Models & Data Bases 1.4 PB 2 x 40 Gb/s 336 Cores 3 TB RAM ~50 Cores ~200 MB RAM e.g. pre - processing of S1 / S2 WCS WCPS SOS openEO – SFSCon – 2019-11-16
  • 9. 2 Why do we need openEO? CEPH_FS CEPH_RBD Hardware File Systems Cluster Orchestration & Virtual Environments Applications Data Models & Data Bases 1.4 PB 2 x 40 Gb/s 336 Cores 3 TB RAM ~50 Cores ~200 MB RAM e.g. pre - processing of S1 / S2 WCS WCPS SOS openEO – SFSCon – 2019-11-16
  • 10. 3 Why do we need openEO? CEPH_FS CEPH_RBD Hardware File Systems Cluster Orchestration & Virtual Environments Applications Data Models & Data Bases 1.4 PB 2 x 40 Gb/s 336 Cores 3 TB RAM ~50 Cores ~200 MB RAM e.g. pre - processing of S1 / S2 WCS WCPS SOS openEO – SFSCon – 2019-11-16
  • 11. 3 Why do we need openEO? openEO – SFSCon – 2019-11-16
  • 12. 3 Why do we need openEO? openEO – SFSCon – 2019-11-16
  • 13. 3 Why do we need openEO? openEO – SFSCon – 2019-11-16
  • 14. 3 Why do we need openEO? openEO – SFSCon – 2019-11-16
  • 15. 3 Why do we need openEO? openEO – SFSCon – 2019-11-16
  • 16. 3 Why do we need openEO? openEO – SFSCon – 2019-11-16
  • 17. 3 Why do we need openEO? openEO – SFSCon – 2019-11-16
  • 18. 3 Why do we need openEO? openEO – SFSCon – 2019-11-16
  • 19. 4 Why do we need openEO? API openEO – SFSCon – 2019-11-16
  • 20. 5 Why do we need openEO? API openEO – SFSCon – 2019-11-16
  • 21. 6 Getting started… • First check for existing drivers @ https://github.com/Open-EO https://open-eo.github.io/openeo-api/gettingstarted-backends/ RESTful implementation using OpenAPI Specification Version 3.0.1 openEO – SFSCon – 2019-11-16
  • 22. 6 Getting started… • First check for existing drivers @ https://github.com/Open-EO https://open-eo.github.io/openeo-api/gettingstarted-backends/ RESTful implementation using OpenAPI Specification Version 3.0.1 openEO – SFSCon – 2019-11-16
  • 23. 6 Getting started… • First check for existing drivers @ https://github.com/Open-EO • If an own implementation is needed: • You can still rely on some base functionality in the existing implementations • Or start with openAPI code generator @ https://github.com/OpenAPITools/openapi- generator • Start with implementing the essential endpoints https://open-eo.github.io/openeo-api/gettingstarted-backends/ RESTful implementation using OpenAPI Specification Version 3.0.1 openEO – SFSCon – 2019-11-16
  • 24. 7 The Endpoints of openEO root slash for capabilities and well-known for versioning/.well-known/openeo / /output_formats openEO – SFSCon – 2019-11-16 RESTful implementation using OpenAPI Specification Version 3.0.1
  • 25. 7 The Endpoints of openEO openEO strives for compatibility with STAC and OGC API as far as possible for data discovery. root slash for capabilities and well-known for versioning /collections /.well-known/openeo / /output_formats /collections/{collectionid} openEO – SFSCon – 2019-11-16 RESTful implementation using OpenAPI Specification Version 3.0.1
  • 26. 7 The Endpoints of openEO openEO strives for compatibility with STAC and OGC API as far as possible for data discovery. The basis for all computation are processes root slash for capabilities and well-known for versioning /collections /processes /.well-known/openeo / /output_formats /collections/{collectionid} openEO – SFSCon – 2019-11-16 RESTful implementation using OpenAPI Specification Version 3.0.1
  • 27. 7 The Endpoints of openEO openEO strives for compatibility with STAC and OGC API as far as possible for data discovery. The basis for all computation are processes Processes can be chained into process graphs root slash for capabilities and well-known for versioning /collections /processes /.well-known/openeo / /process_graphs /process_graphs/{graphID} /output_formats /collections/{collectionid} openEO – SFSCon – 2019-11-16 RESTful implementation using OpenAPI Specification Version 3.0.1
  • 28. 7 The Endpoints of openEO openEO strives for compatibility with STAC and OGC API as far as possible for data discovery. The basis for all computation are processes Processes can be chained into process graphs Results can be processed and downloaded synchronously Process graphs can be submitted as batch- jobs, queued for processing and results can be download upon completion root slash for capabilities and well-known for versioning /collections /processes /jobs/{jobid} /results /jobs/{jobid}/results /.well-known/openeo / /jobs /process_graphs /process_graphs/{graphID} /output_formats /collections/{collectionid} openEO – SFSCon – 2019-11-16 RESTful implementation using OpenAPI Specification Version 3.0.1
  • 29. 7 The Endpoints of openEO openEO strives for compatibility with STAC and OGC API as far as possible for data discovery. The basis for all computation are processes Processes can be chained into process graphs Results can be processed and downloaded synchronously Process graphs can be submitted as batch- jobs, queued for processing and results can be download upon completion Handling of user authentication and billing. root slash for capabilities and well-known for versioning /collections /processes /jobs/{jobid} /credentials/oidc /results /jobs/{jobid}/results /.well-known/openeo / /credentials/basic /jobs /process_graphs /process_graphs/{graphID} /output_formats /collections/{collectionid} openEO – SFSCon – 2019-11-16 RESTful implementation using OpenAPI Specification Version 3.0.1
  • 30. 7 The Endpoints of openEO openEO strives for compatibility with STAC and OGC API as far as possible for data discovery. The basis for all computation are processes Processes can be chained into process graphs Results can be processed and downloaded synchronously Process graphs can be submitted as batch- jobs, queued for processing and results can be download upon completion User can upload own files Handling of user authentication and billing. root slash for capabilities and well-known for versioning /collections /processes /jobs/{jobid} /files/{userid} /credentials/oidc /results /jobs/{jobid}/results /.well-known/openeo / /credentials/basic /jobs /process_graphs /process_graphs/{graphID} /output_formats /collections/{collectionid} openEO – SFSCon – 2019-11-16 RESTful implementation using OpenAPI Specification Version 3.0.1
  • 31. 7 The Endpoints of openEO openEO strives for compatibility with STAC and OGC API as far as possible for data discovery. The basis for all computation are processes Processes can be chained into process graphs Results can be processed and downloaded synchronously Process graphs can be submitted as batch- jobs, queued for processing and results can be download upon completion User can upload own files Handling of user authentication and billing. root slash for capabilities and well-known for versioning /collections /processes /jobs/{jobid} /files/{userid} /credentials/oidc /results /jobs/{jobid}/results /service_types/{serviceid} /.well-known/openeo / Consume results as secondary web services (e.g. WMS, XYZ, WCS) /credentials/basic /jobs /process_graphs /process_graphs/{graphID} /output_formats /collections/{collectionid} openEO – SFSCon – 2019-11-16 RESTful implementation using OpenAPI Specification Version 3.0.1
  • 32. 7 The Endpoints of openEO openEO strives for compatibility with STAC and OGC API as far as possible for data discovery. The basis for all computation are processes Processes can be chained into process graphs Results can be processed and downloaded synchronously Process graphs can be submitted as batch- jobs, queued for processing and results can be download upon completion User can create and integrate user defined functions into process graphs User can upload own files Handling of user authentication and billing. root slash for capabilities and well-known for versioning /collections /processes /jobs/{jobid} /files/{userid} /credentials/oidc /results /jobs/{jobid}/results /service_types/{serviceid} /.well-known/openeo / Consume results as secondary web services (e.g. WMS, XYZ, WCS) /credentials/basic /jobs /process_graphs /process_graphs/{graphID} /output_formats /udf_runtimes /collections/{collectionid} openEO – SFSCon – 2019-11-16 RESTful implementation using OpenAPI Specification Version 3.0.1
  • 33. 8 The Architecture of openEO everything is JSON openEO – SFSCon – 2019-11-16
  • 35. 9 Useful resources http://docs.openeo.org https://open-eo.github.io/openeo-api/apireference/ http://processes.openeo.org openEO – SFSCon – 2019-11-16
  • 36. 10 The Implementations of openEO Back-end Implementations Client Implementations openEO – SFSCon – 2019-11-16
  • 39. 12 WCPS backend (EURAC): https://openeo.eurac.edu CEPH_FS CEPH_RBD Hardware File Systems Cluster Orchestration & Virtual Environments Applications Data Models & Data Bases 1.4 PB 2 x 40 Gb/s 336 Cores 3 TB RAM ~50 Cores ~200 MB RAM e.g. pre - processing of S1 / S2 WCS WCPS SOS https://github.com/Open-EO/openeo-wcps-driver openEO – SFSCon – 2019-11-16
  • 40. 12 WCPS backend (EURAC) : https://openeo.eurac.edu CEPH_FS CEPH_RBD Hardware File Systems Cluster Orchestration & Virtual Environments Applications Data Models & Data Bases 1.4 PB 2 x 40 Gb/s 336 Cores 3 TB RAM ~50 Cores ~200 MB RAM e.g. pre - processing of S1 / S2 WCS WCPS SOS https://github.com/Open-EO/openeo-wcps-driver openEO – SFSCon – 2019-11-16
  • 41. 13 WCPS backend (EURAC) https://github.com/Open-EO/openeo-wcps-driver Based on swagger-jersey2-jaxrs for rest API implementation. Sqlite for openEO related DB => Batch job management, storing of process graphs GDAL for image operations and coordinate transformations JJWT for openID connect implementation => Linked to Microsoft azure for authentication Packaged as web archive using maven => Deployable on any java capable web container (e.g. tomcat or jetty) openEO – SFSCon – 2019-11-16
  • 42. 13 WCPS backend (EURAC) https://github.com/Open-EO/openeo-wcps-driver Based on swagger-jersey2-jaxrs for rest API implementation. Sqlite for openEO related DB => Batch job management, storing of process graphs GDAL for image operations and coordinate transformations JJWT for openID connect implementation => Linked to Microsoft azure for authentication Packaged as web archive using maven => Deployable on any java capable web container (e.g. tomcat or jetty) Configuration: • Properties File • WCPS endpoint • openEO endpoint • Authentication endpoint (for oidc) • DB location • TMP location • Session timings (auth expiry, tmp duration, etc.) openEO – SFSCon – 2019-11-16
  • 43. 13 WCPS backend (EURAC) https://github.com/Open-EO/openeo-wcps-driver Based on swagger-jersey2-jaxrs for rest API implementation. Sqlite for openEO related DB => Batch job management, storing of process graphs GDAL for image operations and coordinate transformations JJWT for openID connect implementation => Linked to Microsoft azure for authentication Packaged as web archive using maven => Deployable on any java capable web container (e.g. tomcat or jetty) Configuration: • Properties File • WCPS endpoint • openEO endpoint • Authentication endpoint (for oidc) • DB location • TMP location • Session timings (auth expiry, tmp duration, etc.) • Setup of Host Environment • Centos 7 or Ubuntu 18.04 • Install Tomcat 7 or later • Configure for https • Install sqlite (v3) & GDAL (v2.4) • Deploy openEO.war • Setup of proxy server for public access openEO – SFSCon – 2019-11-16
  • 44. 14 Client Examples (Python) openEO – SFSCon – 2019-11-16
  • 45. 15 Client Examples (Python) openEO – SFSCon – 2019-11-16
  • 46. 15 Client Examples (Python) openEO – SFSCon – 2019-11-16
  • 47. 16 Client Examples (JavaScript): https://editor.openeo.org openEO – SFSCon – 2019-11-16
  • 48. 17 Conclusions • A RESTful API has been defined • Interface human and machine readable (JSON) • Process catalogue • Extendable by user defined functions • Standardized process graph for abstract description of EO computational workflows • A number of reference implementations are currently in development • Based on different programming languages • Python • Java • R • Java-script • Based on different existing hard and software infrastructure • Can be used as starting point for own implementation • Together with extensive documentation • And guides • All of this is open source openEO – SFSCon – 2019-11-16
  • 49. Thank you for your attention! H2020 openEO EO-2-2017: EO Big Data Shift Grant agreement No 776242 Contact: http://openeo.org/ openEO@list.tuwien.ac.at https://github.com/Open-EO @open_EO https://www.youtube.com/channel/UCMJQil8 j9sHBQkcSlSaEsvQ https://www.researchgate.net/project/openEO https://openeo-chat.eodc.eu/channel/public https://zenodo.org/communities/openeo alexander.jacob@eurac.edu openEO – SFSCon – 2019-11-16