TWiki> IVOA Web>IvoaGridAndWebServices>GWSTelecon20230920 (2023-09-25, JesusSalgado) EditAttach

Minutes: GWS Telecon 2023-09-20

Time: 20-09-2023; 13:00 UTC

VideoConf link: https://skatelescope.zoom.us/j/4371396863?pwd=bVl4ZThNeko4S3RSQnJxd0plVUkxZz09

Online notes: https://yopad.eu/p/IVOA_GWS_Science_Platforms_Federated_Execution

Background information

Computing Services: https://wiki.ivoa.net/twiki/bin/view/IVOA/ComputingServicesAPI

Execution Planner: https://github.com/Zarquan/ExecutionPlanner

Main Topic. Science Platforms Federated Execution: Agenda

Attendees: 22

  • Jesus Salgado - JS
  • Sara Bertocco - SB
  • Dave Morris - DM
  • Brian Major
  • Claudio Gheller
  • David Aikema
  • Gerard Lemson
  • Rob Simmonds
  • Chenzhou Cui
  • Paul Harrison
  • Susana Sanchez
  • Vicente Navarro
  • Zhijun Xu
  • Tess Jaffe
  • Severin Gaudet
  • Renaud Savalle
  • David Aikema
  • James Collison
  • Sara Nieto
  • Rajesh Tamhane
  • Andreas Wicenec
  • Rohini Joshi
  • Iulia Cimpan

Agenda

  • Introduction and presentation of concept - Jesus Salgado 15’
  • Execution Planner status and missing points - Dave Morris 15’
  • Open Discussion - All 15’
  • Identification of collaborators and Future plans 15’

Introduction and presentation of concept - Jesus Salgado

https://docs.google.com/presentation/d/1zow9L1hAAn3oLo3BRb_EDHmXQkLQ8zxO5DWXDQ_d9JA/

Connect sci plat on different centres.
IVOA focus
1 Discover data from different missions with different interfaces.
2 connect services
3 allow users to connect to services with interfaces
We need to execute code as close as possible to the data.
SRC net already provides some capabilities.
The idea is to connect different platforms with different missions.
A general picture of sci-plat
Federated authentication, federated data lake,
how to execute remotely (today's focus)
standardize service to access data (not the main thing for today)
How to use the software repository
Some possible data mesh services, a sort of SODA approach. Define a set of operations on the data and expand soda to this set of operations.
ExecutionPlanner description: run software on different platforms. Are you able to run this?
A more complex approach uses an information system. The IVOA approach is more pragmatic.
ExecPlanner: standardise the workflow description, and extend the UWS.
Summary: federate auth protocol, maybe improve data access, simplify federated execution, and characterize workflow and hardware resources.
Today's objectives: agree roadmap and collect interest and collaborators.
Q(Simmonds): are there teams doing similar things?
A(Jesus): It is some split what can be standard in IVOA and what can be standardized locally
Q(Simmonds):: reimplement the grid?
A(Jesus): we need a common solution in the current more heterogeneous environment (cloud, grid, ...)

Execution Planner status and missing points - Dave Morris

Dave will describe the common web interface to have in common over whatever implementation. A federated system of different technologies.
The document is based on a couple of years of work (presentations and collecting feedback) and is still a work in progress.
Scope: Collect a group working on a prototype and developing a common understanding.
Concept: "executable thing" can be a Python program (requires Python env) or a docker container (requires a container execution environment). The executable thing is described (see the document). Discoverable URL.
Service interaction. For data in IVOA, we have services to discover cutouts of data. Execution Planner does not care about service discovery.
Services interaction: Dave describes the exchange between client and service. The client makes a request with the "executable thing description", and the service answers with an offer.
...
Sara: Is the user or the execPlanner that calculates how long the job is?
Dave: The user.

Open Discussion - All

Gheller: need to think about the data model description

Dave: Dave is thinking about it. It is a type, which is a URI and then the spec, the specification with type-specific details. The structure comes from Kubernetes. Will try to have a vocabulary as simple as possible.

Jesus: should it be inside the execution planner standard or separate?
Dave: by now a simple data model is proposed in ExecPlanner. More complex definitions need to go out of the ExecPlanner (see the GPU example in the document).
Simmonds:
Dave:
Jesus: The data location is missing. Do you need a standard to do that?
Simmonds: a lot of stuff has been done before (for data reconstruction).
It seems to introduce complexity
Dave: it seems because there are a lot of details. The implementation is really simple, it depends on the level of detail.
Rohini:
Dave: we have the distinction between batch and interactive. Everything is asynchronous. also when data has to be staged, the time needed is kept into account.
Vincente Navarro: Developing a distributed pipeline implementation is difficult also in the same organization if you add multiple nodes. Provocative: Have you considered the possibility of using what already exists and putting it on top of an orchestration layer? We use CWL, airflow is becoming a standard. Why do not use an approach using these existing technologies?
Dave: in a large federated environment someone has new tech, while others have pretty old systems, i.e. docker run, docker-compose, Kubernetes cluster... our interface does not care about details, only answer yes, I can run your container.
Jesus: could we have a software stack to simplify this? The answer is no. Asked multiple partners.
Vincente: S3 is the technology, not the standard. Is an industrial de facto standard and can help us to data access speeding up the process.
Dave: totally agree that standardize....
Vincente: there is plenty of work if you need, for example, to replicate the data volume.
Severin: what do we need more on the specification to start with the prototyping?
Dave: now we can start. what we publish now is an initial draft of the standard. The implementation of a really simple prototype could provide precious feedback to improve the standard.
Jesus: will send an e-mail to understand if we are able to endorse the standard and find people to collaborate in prototyping. At IVOA Interop we will have a session to collaborate
Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf GWS_-_Federated_Computing.pdf r1 manage 2551.0 K 2023-09-20 - 10:35 JesusSalgado  
Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r6 - 2023-09-25 - JesusSalgado
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback