EAS Data Flow lessons learnt

Editor's Notes

1. HelloAccess to EAS from IAL. lessons learnt from IAL mockup integrationOutstanding issues
2. Brief overview of termsDRMS: scheduler, sometimes used as overall term for computing infrastructure.Submission Host: typically SSH connectionExecution Host: computing nodes. In some setups the submission host is an execution host at the same time.Job. Computing entity, Task: Euclid job.
3. IAL Infrastructure abstraction layer and not interface abstraction layer
4. Red, green and yellow shapes are file storages (binary files)Two SDCs:SDR-FR only sketched. EAS file archive is used to access binary files.SDC-CH consists of the IAL and the computing infrastructureThe computing infrastructure is divided into submission host and execution hostIAL connects on submission host to submit jobs. The Task Scheduler provides an implementation independent abstraction.Submission host/Task scheduler distributes jobs to multiple execution hosts.Task is executed and creates results that IAL needs to register in the Euclid Metadata Archive System.EMA is in the left side.Let’s look into the data flowLet’s assume: data becomes available at SDC-FR, someone from the science community can access it through http/ftp/….IAL on SDC-CH needs the data local and triggers a replicate jobData moves to SDC-EASNow the science community can get the data from two places, once from SDC-FR and once from SDC-CH.Next IAL makes the data available to the Computing Infrastructure. For performance reasons IAL should use some local protocol (here file://), rather than HTTP to read the file. Currently, the submission host is writable through sftp.Now the file is copied to a dedicated workspace on the Submission Host Storage.Once the data is available IAL starts the Task Scheduler which triggers to the Task Executor.The Task executor may have to copy the files over to the Execution Host (Grid setup)Currently we support the file protocol, only. This might not be sufficient.File is copiedPipeline Task may need another protocol or path to get to the data.Seven different URLs!Three copy jobs
5. Red, green and yellow shapes are file storages (binary files)Two SDCs:SDR-FR only sketched. EAS file archive is used to access binary files.SDC-CH consists of the IAL and the computing infrastructureThe computing infrastructure is divided into submission host and execution hostIAL connects on submission host to submit jobs. The Task Scheduler provides an implementation independent abstraction.Submission host/Task scheduler distributes jobs to multiple execution hosts.Task is executed and creates results that IAL needs to register in the Euclid Metadata Archive System.EMA is in the left side.Let’s look into the data flowLet’s assume: data becomes available at SDC-FR, someone from the science community can access it through http/ftp/….IAL on SDC-CH needs the data local and triggers a replicate jobData moves to SDC-EASNow the science community can get the data from two places, once from SDC-FR and once from SDC-CH.Next IAL makes the data available to the Computing Infrastructure. For performance reasons IAL should use some local protocol (here file://), rather than HTTP to read the file. Currently, the submission host is writable through sftp.Now the file is copied to a dedicated workspace on the Submission Host Storage.Once the data is available IAL starts the Task Scheduler which triggers to the Task Executor.The Task executor may have to copy the files over to the Execution Host (Grid setup)Currently we support the file protocol, only. This might not be sufficient.File is copiedPipeline Task may need another protocol or path to get to the data.Seven different URLs!Three copy jobs
6. Red, green and yellow shapes are file storages (binary files)Two SDCs:SDR-FR only sketched. EAS file archive is used to access binary files.SDC-CH consists of the IAL and the computing infrastructureThe computing infrastructure is divided into submission host and execution hostIAL connects on submission host to submit jobs. The Task Scheduler provides an implementation independent abstraction.Submission host/Task scheduler distributes jobs to multiple execution hosts.Task is executed and creates results that IAL needs to register in the Euclid Metadata Archive System.EMA is in the left side.Let’s look into the data flowLet’s assume: data becomes available at SDC-FR, someone from the science community can access it through http/ftp/….IAL on SDC-CH needs the data local and triggers a replicate jobData moves to SDC-EASNow the science community can get the data from two places, once from SDC-FR and once from SDC-CH.Next IAL makes the data available to the Computing Infrastructure. For performance reasons IAL should use some local protocol (here file://), rather than HTTP to read the file. Currently, the submission host is writable through sftp.Now the file is copied to a dedicated workspace on the Submission Host Storage.Once the data is available IAL starts the Task Scheduler which triggers to the Task Executor.The Task executor may have to copy the files over to the Execution Host (Grid setup)Currently we support the file protocol, only. This might not be sufficient.File is copiedPipeline Task may need another protocol or path to get to the data.Seven different URLs!Three copy jobs
7. Red, green and yellow shapes are file storages (binary files)Two SDCs:SDR-FR only sketched. EAS file archive is used to access binary files.SDC-CH consists of the IAL and the computing infrastructureThe computing infrastructure is divided into submission host and execution hostIAL connects on submission host to submit jobs. The Task Scheduler provides an implementation independent abstraction.Submission host/Task scheduler distributes jobs to multiple execution hosts.Task is executed and creates results that IAL needs to register in the Euclid Metadata Archive System.EMA is in the left side.Let’s look into the data flowLet’s assume: data becomes available at SDC-FR, someone from the science community can access it through http/ftp/….IAL on SDC-CH needs the data local and triggers a replicate jobData moves to SDC-EASNow the science community can get the data from two places, once from SDC-FR and once from SDC-CH.Next IAL makes the data available to the Computing Infrastructure. For performance reasons IAL should use some local protocol (here file://), rather than HTTP to read the file. Currently, the submission host is writable through sftp.Now the file is copied to a dedicated workspace on the Submission Host Storage.Once the data is available IAL starts the Task Scheduler which triggers to the Task Executor.The Task executor may have to copy the files over to the Execution Host (Grid setup)Currently we support the file protocol, only. This might not be sufficient.File is copiedPipeline Task may need another protocol or path to get to the data.Seven different URLs!Three copy jobs
8. Red, green and yellow shapes are file storages (binary files)Two SDCs:SDR-FR only sketched. EAS file archive is used to access binary files.SDC-CH consists of the IAL and the computing infrastructureThe computing infrastructure is divided into submission host and execution hostIAL connects on submission host to submit jobs. The Task Scheduler provides an implementation independent abstraction.Submission host/Task scheduler distributes jobs to multiple execution hosts.Task is executed and creates results that IAL needs to register in the Euclid Metadata Archive System.EMA is in the left side.Let’s look into the data flowLet’s assume: data becomes available at SDC-FR, someone from the science community can access it through http/ftp/….IAL on SDC-CH needs the data local and triggers a replicate jobData moves to SDC-EASNow the science community can get the data from two places, once from SDC-FR and once from SDC-CH.Next IAL makes the data available to the Computing Infrastructure. For performance reasons IAL should use some local protocol (here file://), rather than HTTP to read the file. Currently, the submission host is writable through sftp.Now the file is copied to a dedicated workspace on the Submission Host Storage.Once the data is available IAL starts the Task Scheduler which triggers to the Task Executor.The Task executor may have to copy the files over to the Execution Host (Grid setup)Currently we support the file protocol, only. This might not be sufficient.File is copiedPipeline Task may need another protocol or path to get to the data.Seven different URLs!Three copy jobs
9. 1b.Our XML data model contains references to binary data in quite some places. These references need to be changed or added all along the processing chain.
10. Rearrangement of previous drawing: colored boxes at the bottomClients of data storages at the topNew layer (Euclid File Access Service) in between (hopefully based on some existing software)File Access Service needs to provide some functionality (by far not exhaustive).
11. How to reduce the number of copies: Merge the file storages.Red circle: Merge SDC-EAS and Submission host. IAL would not have to copy files anymore. Requires a fast connection from the computing infrastructure to the archive system.Green circle:Merge Submission and Execution host storage, Task Executor does not need to copy files aroundRequires parallel write file system and fast network connection.Purple circle:Merge SDC-EAS, Submission and Execution Host StorageBest solution regarding performance. Puts high demands on Archive system:Support for parallel writes, support for access over the internet, fast network connection between computing infrastructure and archive, petabytes of storage
12. How to reduce the number of copies: Merge the file storages.Red circle: Merge SDC-EAS and Submission host. IAL would not have to copy files anymore. Requires a fast connection from the computing infrastructure to the archive system.Green circle:Merge Submission and Execution host storage, Task Executor does not need to copy files aroundRequires parallel write file system and fast network connection.Purple circle:Merge SDC-EAS, Submission and Execution Host StorageBest solution regarding performance. Puts high demands on Archive system:Support for parallel writes, support for access over the internet, fast network connection between computing infrastructure and archive, petabytes of storage
13. How to reduce the number of copies: Merge the file storages.Red circle: Merge SDC-EAS and Submission host. IAL would not have to copy files anymore. Requires a fast connection from the computing infrastructure to the archive system.Green circle:Merge Submission and Execution host storage, Task Executor does not need to copy files aroundRequires parallel write file system and fast network connection.Purple circle:Merge SDC-EAS, Submission and Execution Host StorageBest solution regarding performance. Puts high demands on Archive system:Support for parallel writes, support for access over the internet, fast network connection between computing infrastructure and archive, petabytes of storage
14. How to reduce the number of copies: Merge the file storages.Red circle: Merge SDC-EAS and Submission host. IAL would not have to copy files anymore. Requires a fast connection from the computing infrastructure to the archive system.Green circle:Merge Submission and Execution host storage, Task Executor does not need to copy files aroundRequires parallel write file system and fast network connection.Purple circle:Merge SDC-EAS, Submission and Execution Host StorageBest solution regarding performance. Puts high demands on Archive system:Support for parallel writes, support for access over the internet, fast network connection between computing infrastructure and archive, petabytes of storage