International Virtual Observatory Alliance

Difference: GWSTelecon20230920 (1 vs. 6)

Revision 62023-09-25 - JesusSalgado

  META TOPICPARENT 
  name="IvoaGridAndWebServices"  

 Minutes: GWS Telecon 2023-09-20 

Time: 20-09-2023; 13:00 UTC

VideoConf link: https://skatelescope.zoom.us/j/4371396863?pwd=bVl4ZThNeko4S3RSQnJxd0plVUkxZz09

Online notes: https://yopad.eu/p/IVOA_GWS_Science_Platforms_Federated_Execution
 Background information 

Computing Services: https://wiki.ivoa.net/twiki/bin/view/IVOA/ComputingServicesAPI

Execution Planner: https://github.com/Zarquan/ExecutionPlanner
 Main Topic. Science Platforms Federated Execution: Agenda 
 Attendees: 22
- META TOPICPARENT
+  name="IvoaGridAndWebServices"
->
>
  Jesus Salgado - JS
 
  
 Sara Bertocco - SB
 
  
 Dave Morris - DM
 
  
 Brian Major
 
  
 Claudio Gheller
 
  
 David Aikema
 
  
 Gerard Lemson
 
  
 Rob Simmonds
 
  
 Chenzhou Cui
 
  
 Paul Harrison
 
  
 Susana Sanchez
 
  
 Vicente Navarro
 
  
 Zhijun Xu
 
  
 Tess Jaffe
 
  
 Severin Gaudet
 
  
 Renaud Savalle
 
  
 David Aikema
 
  
 James Collison
 
  
 Sara Nieto
 
  
 Rajesh Tamhane
 
  
 Andreas Wicenec
 
  
 Rohini Joshi
->
>
+ Iulia Cimpan
->
>
  Agenda 
 

 Introduction and presentation of concept - Jesus Salgado 15’
  Execution Planner status and missing points - Dave Morris 15’
  Open Discussion - All 15’
  Identification of collaborators and Future plans 15’
 
 Introduction and presentation of concept - Jesus Salgado 

https://docs.google.com/presentation/d/1zow9L1hAAn3oLo3BRb_EDHmXQkLQ8zxO5DWXDQ_d9JA/
-<
<
+Connect sci plat on different centres.
 IVOA focus
 1 Discover data from different missions with different interfaces.
 2 connect services
 3 allow users to connect to services with interfaces
 We need to execute code as close as possible to the data.
 SRC net already provides some capabilities.
 The idea is to connect different platforms with different missions.
 A general picture of sci-plat
 Federated authentication, federated data lake,
 how to execute remotely (today's focus)
 standardize service to access data (not the main thing for today)
 How to use the software repository
 Some possible data mesh services, a sort of SODA approach. Define a set of operations on the data and expand soda to this set of operations.
 ExecutionPlanner description: run software on different platforms. Are you able to run this? 
 A more complex approach uses an information system. The IVOA approach is more pragmatic.
 ExecPlanner: standardise the workflow description, and extend the UWS.
 Summary: federate auth protocol, maybe improve data access, simplify federated execution, and characterize workflow and hardware resources.
 Today's objectives: agree roadmap and collect interest and collaborators.
    Q(Simmonds): are there teams doing similar things?
    A(Jesus): It is some split what can be standard in IVOA and what can be standardized locally
    Q(Simmonds):: reimplement the grid?
    A(Jesus): we need a common solution in the current more heterogeneous environment (cloud, grid, ...)
->
>
+Connect sci plat on different centres.
 IVOA focus
 1 Discover data from different missions with different interfaces.
 2 connect services
 3 allow users to connect to services with interfaces
 We need to execute code as close as possible to the data.
 SRC net already provides some capabilities.
 The idea is to connect different platforms with different missions.
 A general picture of sci-plat
 Federated authentication, federated data lake,
 how to execute remotely (today's focus)
 standardize service to access data (not the main thing for today)
 How to use the software repository
 Some possible data mesh services, a sort of SODA approach. Define a set of operations on the data and expand soda to this set of operations.
 ExecutionPlanner description: run software on different platforms. Are you able to run this?
 A more complex approach uses an information system. The IVOA approach is more pragmatic.
 ExecPlanner: standardise the workflow description, and extend the UWS.
 Summary: federate auth protocol, maybe improve data access, simplify federated execution, and characterize workflow and hardware resources.
 Today's objectives: agree roadmap and collect interest and collaborators.
 Q(Simmonds): are there teams doing similar things?
 A(Jesus): It is some split what can be standard in IVOA and what can be standardized locally
 Q(Simmonds):: reimplement the grid?
 A(Jesus): we need a common solution in the current more heterogeneous environment (cloud, grid, ...)
  Execution Planner status and missing points - Dave Morris
-<
<
+Dave will describe the common web interface to have in common over whatever implementation. A federated system of different technologies.
 The document is based on a couple of years of work (presentations and collecting feedback) and is still a work in progress.
 Scope: Collect a group working on a prototype and developing a common understanding.
 Concept: "executable thing" can be a Python program (requires Python env) or a docker container (requires a container execution environment). The executable thing is described (see the document). Discoverable URL. 
 Service interaction. For data in IVOA, we have services to discover cutouts of data. Execution Planner does not care about service discovery.
 Services interaction: Dave describes the exchange between client and service. The client makes a request with the "executable thing description", and the service answers with an offer.
 ...
 Sara: Is the user or the execPlanner that calculates how long the job is?
 Dave: The user.
->
>
->
>
+Dave will describe the common web interface to have in common over whatever implementation. A federated system of different technologies.
 The document is based on a couple of years of work (presentations and collecting feedback) and is still a work in progress.
 Scope: Collect a group working on a prototype and developing a common understanding.
 Concept: "executable thing" can be a Python program (requires Python env) or a docker container (requires a container execution environment). The executable thing is described (see the document). Discoverable URL.
 Service interaction. For data in IVOA, we have services to discover cutouts of data. Execution Planner does not care about service discovery.
 Services interaction: Dave describes the exchange between client and service. The client makes a request with the "executable thing description", and the service answers with an offer.
 ...
 Sara: Is the user or the execPlanner that calculates how long the job is?
 Dave: The user.
  Open Discussion - All 

Gheller: need to think about the data model description

Dave: Dave is thinking about it. It is a type, which is a URI and then the spec, the specification with type-specific details. The structure comes from Kubernetes. Will try to have a vocabulary as simple as possible.
-<
<
+Jesus: should it be inside the execution planner standard or separate?
 Dave: by now a simple data model is proposed in ExecPlanner. More complex definitions need to go out of the ExecPlanner (see the GPU example in the document).
 Simmonds:
 Dave:
 Jesus: The data location is missing. Do you need a standard to do that?
 Simmonds: a lot of stuff has been done before (for data reconstruction).
 It seems to introduce complexity
 Dave: it seems because there are a lot of details. The implementation is really simple, it depends on the level of detail.    
 Rohini: 
 Dave: we have the distinction between batch and interactive. Everything is asynchronous. also when data has to be staged, the time needed is kept into account.
 Vincente Navarro: Developing a distributed pipeline implementation is difficult also in the same organization if you add multiple nodes. Provocative: Have you considered the possibility of using what already exists and putting it on top of an orchestration layer? We use CWL, airflow is becoming a standard. Why do not use an approach using these existing technologies?
 Dave: in a large federated environment someone has new tech, while others have pretty old systems, i.e. docker run, docker-compose, Kubernetes cluster... our interface does not care about details, only answer yes, I can run your container.
 Jesus: could we have a software stack to simplify this? The answer is no. Asked multiple partners.
 Vincente: S3 is the technology, not the standard. Is an industrial de facto standard and can help us to data access speeding up the process.
 Dave: totally agree that standardize....
 Vincente: there is plenty of work if you need, for example, to replicate the data volume.
 Severin: what do we need more on the specification to start with the prototyping?
 Dave: now we can start. what we publish now is an initial draft of the standard. The implementation of a really simple prototype could provide precious feedback to improve the standard.
 Jesus: will send an e-mail to understand if we are able to endorse the standard and find people to collaborate in prototyping. At IVOA Interop we will have a session to collaborate
->
>
+Jesus: should it be inside the execution planner standard or separate?
 Dave: by now a simple data model is proposed in ExecPlanner. More complex definitions need to go out of the ExecPlanner (see the GPU example in the document).
 Simmonds:
 Dave:
 Jesus: The data location is missing. Do you need a standard to do that?
 Simmonds: a lot of stuff has been done before (for data reconstruction).
 It seems to introduce complexity
 Dave: it seems because there are a lot of details. The implementation is really simple, it depends on the level of detail.
 Rohini:
 Dave: we have the distinction between batch and interactive. Everything is asynchronous. also when data has to be staged, the time needed is kept into account.
 Vincente Navarro: Developing a distributed pipeline implementation is difficult also in the same organization if you add multiple nodes. Provocative: Have you considered the possibility of using what already exists and putting it on top of an orchestration layer? We use CWL, airflow is becoming a standard. Why do not use an approach using these existing technologies?
 Dave: in a large federated environment someone has new tech, while others have pretty old systems, i.e. docker run, docker-compose, Kubernetes cluster... our interface does not care about details, only answer yes, I can run your container.
 Jesus: could we have a software stack to simplify this? The answer is no. Asked multiple partners.
 Vincente: S3 is the technology, not the standard. Is an industrial de facto standard and can help us to data access speeding up the process.
 Dave: totally agree that standardize....
 Vincente: there is plenty of work if you need, for example, to replicate the data volume.
 Severin: what do we need more on the specification to start with the prototyping?
 Dave: now we can start. what we publish now is an initial draft of the standard. The implementation of a really simple prototype could provide precious feedback to improve the standard.
 Jesus: will send an e-mail to understand if we are able to endorse the standard and find people to collaborate in prototyping. At IVOA Interop we will have a session to collaborate
  META FILEATTACHMENT 
  attachment="GWS_-_Federated_Computing.pdf" attr="" comment="" date="1695206136" name="GWS_-_Federated_Computing.pdf" path="GWS_-_Federated_Computing.pdf" size="2612183" user="JesusSalgado" version="1"
- META FILEATTACHMENT
+  attachment="GWS_-_Federated_Computing.pdf" attr="" comment="" date="1695206136" name="GWS_-_Federated_Computing.pdf" path="GWS_-_Federated_Computing.pdf" size="2612183" user="JesusSalgado" version="1"

Revision 52023-09-21 - JesusSalgado

  META TOPICPARENT 
  name="IvoaGridAndWebServices"  

 Minutes: GWS Telecon 2023-09-20 

Time: 20-09-2023; 13:00 UTC

VideoConf link: https://skatelescope.zoom.us/j/4371396863?pwd=bVl4ZThNeko4S3RSQnJxd0plVUkxZz09

Online notes: https://yopad.eu/p/IVOA_GWS_Science_Platforms_Federated_Execution
 Background information 

Computing Services: https://wiki.ivoa.net/twiki/bin/view/IVOA/ComputingServicesAPI

Execution Planner: https://github.com/Zarquan/ExecutionPlanner
 Main Topic. Science Platforms Federated Execution: Agenda
- META TOPICPARENT
+  name="IvoaGridAndWebServices"
-<
<
+ Attendees: 
 
 Jesus Salgado
  Sara Bertocco
  Dave Morris
  ......
->
>
+ Attendees: 22 
 
 Jesus Salgado - JS
 
  
 Sara Bertocco - SB
->
>
+ Dave Morris - DM
 
  
 Brian Major
 
  
 Claudio Gheller
 
  
 David Aikema
 
  
 Gerard Lemson
 
  
 Rob Simmonds
 
  
 Chenzhou Cui
 
  
 Paul Harrison
 
  
 Susana Sanchez
 
  
 Vicente Navarro
 
  
 Zhijun Xu
 
  
 Tess Jaffe
 
  
 Severin Gaudet
 
  
 Renaud Savalle
 
  
 David Aikema
 
  
 James Collison
 
  
 Sara Nieto
 
  
 Rajesh Tamhane
 
  
 Andreas Wicenec
 
  
 Rohini Joshi
  Agenda 
 

 Introduction and presentation of concept - Jesus Salgado 15’
  Execution Planner status and missing points - Dave Morris 15’
  Open Discussion - All 15’
  Identification of collaborators and Future plans 15’
 
 Introduction and presentation of concept - Jesus Salgado 

https://docs.google.com/presentation/d/1zow9L1hAAn3oLo3BRb_EDHmXQkLQ8zxO5DWXDQ_d9JA/
->
>
+Connect sci plat on different centres.
 IVOA focus
 1 Discover data from different missions with different interfaces.
 2 connect services
 3 allow users to connect to services with interfaces
 We need to execute code as close as possible to the data.
 SRC net already provides some capabilities.
 The idea is to connect different platforms with different missions.
 A general picture of sci-plat
 Federated authentication, federated data lake,
 how to execute remotely (today's focus)
 standardize service to access data (not the main thing for today)
 How to use the software repository
 Some possible data mesh services, a sort of SODA approach. Define a set of operations on the data and expand soda to this set of operations.
 ExecutionPlanner description: run software on different platforms. Are you able to run this? 
 A more complex approach uses an information system. The IVOA approach is more pragmatic.
 ExecPlanner: standardise the workflow description, and extend the UWS.
 Summary: federate auth protocol, maybe improve data access, simplify federated execution, and characterize workflow and hardware resources.
 Today's objectives: agree roadmap and collect interest and collaborators.
    Q(Simmonds): are there teams doing similar things?
    A(Jesus): It is some split what can be standard in IVOA and what can be standardized locally
    Q(Simmonds):: reimplement the grid?
    A(Jesus): we need a common solution in the current more heterogeneous environment (cloud, grid, ...)
  Execution Planner status and missing points - Dave Morris
->
>
+Dave will describe the common web interface to have in common over whatever implementation. A federated system of different technologies.
 The document is based on a couple of years of work (presentations and collecting feedback) and is still a work in progress.
 Scope: Collect a group working on a prototype and developing a common understanding.
 Concept: "executable thing" can be a Python program (requires Python env) or a docker container (requires a container execution environment). The executable thing is described (see the document). Discoverable URL. 
 Service interaction. For data in IVOA, we have services to discover cutouts of data. Execution Planner does not care about service discovery.
 Services interaction: Dave describes the exchange between client and service. The client makes a request with the "executable thing description", and the service answers with an offer.
 ...
 Sara: Is the user or the execPlanner that calculates how long the job is?
 Dave: The user.
  Open Discussion - All
-<
<
+ Identification of collaborators and Future plans
->
>
->
>
+Gheller: need to think about the data model description

Dave: Dave is thinking about it. It is a type, which is a URI and then the spec, the specification with type-specific details. The structure comes from Kubernetes. Will try to have a vocabulary as simple as possible.
Jesus: should it be inside the execution planner standard or separate?
 Dave: by now a simple data model is proposed in ExecPlanner. More complex definitions need to go out of the ExecPlanner (see the GPU example in the document).
 Simmonds:
 Dave:
 Jesus: The data location is missing. Do you need a standard to do that?
 Simmonds: a lot of stuff has been done before (for data reconstruction).
 It seems to introduce complexity
 Dave: it seems because there are a lot of details. The implementation is really simple, it depends on the level of detail.    
 Rohini: 
 Dave: we have the distinction between batch and interactive. Everything is asynchronous. also when data has to be staged, the time needed is kept into account.
 Vincente Navarro: Developing a distributed pipeline implementation is difficult also in the same organization if you add multiple nodes. Provocative: Have you considered the possibility of using what already exists and putting it on top of an orchestration layer? We use CWL, airflow is becoming a standard. Why do not use an approach using these existing technologies?
 Dave: in a large federated environment someone has new tech, while others have pretty old systems, i.e. docker run, docker-compose, Kubernetes cluster... our interface does not care about details, only answer yes, I can run your container.
 Jesus: could we have a software stack to simplify this? The answer is no. Asked multiple partners.
 Vincente: S3 is the technology, not the standard. Is an industrial de facto standard and can help us to data access speeding up the process.
 Dave: totally agree that standardize....
 Vincente: there is plenty of work if you need, for example, to replicate the data volume.
 Severin: what do we need more on the specification to start with the prototyping?
 Dave: now we can start. what we publish now is an initial draft of the standard. The implementation of a really simple prototype could provide precious feedback to improve the standard.
 Jesus: will send an e-mail to understand if we are able to endorse the standard and find people to collaborate in prototyping. At IVOA Interop we will have a session to collaborate
  META FILEATTACHMENT 
  attachment="GWS_-_Federated_Computing.pdf" attr="" comment="" date="1695206136" name="GWS_-_Federated_Computing.pdf" path="GWS_-_Federated_Computing.pdf" size="2612183" user="JesusSalgado" version="1"
- META FILEATTACHMENT
+  attachment="GWS_-_Federated_Computing.pdf" attr="" comment="" date="1695206136" name="GWS_-_Federated_Computing.pdf" path="GWS_-_Federated_Computing.pdf" size="2612183" user="JesusSalgado" version="1"

Revision 42023-09-20 - JesusSalgado

META TOPICPARENT	name="IvoaGridAndWebServices"

Minutes: GWS Telecon 2023-09-20

Time: 20-09-2023; 13:00 UTC

VideoConf link: https://skatelescope.zoom.us/j/4371396863?pwd=bVl4ZThNeko4S3RSQnJxd0plVUkxZz09

Online notes: https://yopad.eu/p/IVOA_GWS_Science_Platforms_Federated_Execution

Background information

Computing Services: https://wiki.ivoa.net/twiki/bin/view/IVOA/ComputingServicesAPI

Execution Planner: https://github.com/Zarquan/ExecutionPlanner

Main Topic. Science Platforms Federated Execution: Agenda

Attendees:

Jesus Salgado
Sara Bertocco
Dave Morris
......

Agenda

Introduction and presentation of concept - Jesus Salgado 15’
Execution Planner status and missing points - Dave Morris 15’
Open Discussion - All 15’
Identification of collaborators and Future plans 15’

Introduction and presentation of concept - Jesus Salgado

Added:

>
>

https://docs.google.com/presentation/d/1zow9L1hAAn3oLo3BRb_EDHmXQkLQ8zxO5DWXDQ_d9JA/

Execution Planner status and missing points - Dave Morris

Open Discussion - All

Identification of collaborators and Future plans

Added:

>
>

META FILEATTACHMENT	attachment="GWS_-_Federated_Computing.pdf" attr="" comment="" date="1695206136" name="GWS_-_Federated_Computing.pdf" path="GWS_-_Federated_Computing.pdf" size="2612183" user="JesusSalgado" version="1"