Computing Services API

Cross-science platforms execution project

The GWS WG is going to start a project to identify the main changes and new standards to allow remote execution on different astronomical science platforms.

1. discover computing services, answering the question "what computing services are available and suitable to run my task?"
2. access computing services, answering the question "how can I run my task on the selected computing service?"

The preliminary items to be analysed are:

Execution planner

Currently, a note has been prepared at
https://github.com/ivoa/ExecutionPlannerNote

This note should be either converted into a standard or, at least, upgraded making it a more powerful interface. To do that, it is necessary to standardise:
Characterisation of software (in terms not only of software dependencies but, also, dependencies that could affect execution like memory, requirements on computing architecture (e.g. CPUs and GPUs), data location, etc. Software should be exposed in a portable format
Metadata characterisation of the science platform node is also a requirement

Access to this metadata should be also characterised as per the definition of a software discovery service and a science platform resources discovery service. The execution planner could use metadata of both types to decide the best place to execute a certain workflow.

For the software characterisation and discovery, other CI/CD solutions could be expanded and already have the connections between code and containers. For the science platform node service, we need to have a dynamic service to capture the real-time status of the resources.

The execution planner and the science platform discovery service could be integrated into the same service and the software discovery looks to be a different one.

Endorsement of a standard for workflow serialisation

Identify one standard for workflow serialisation (e.g. CWL) and endorse it for the IVOA. This format should have software to parse and decompose a workflow in smaller executable elements in different languages if possible.

Execution interface

Identify an execution interface that could abstract the internal execution interfaces of the different science platforms. Probably something based on IVOA UWS could be used to control execution

Credentials Delegation

Review of related standards for authentication/authorisation combined with the execution interface. This could be a problem for a general approach as some computing centres have their local access policies

For this exercise, SKA Regional Centre Network members are available to collaborate, lead and implement although it would be also needed to have members of other projects that are implementing (or already have) running science platforms to create another independent implementation (candidates Vera Rubin, ESA, CADC, Gaia UK,...)