Provenance meeting in Paris, France, 2018 August 28th to August 30thsupported by OV-France, GAVO, Paris Data Center and Asterics ProjectFollow_up_meeting | |||||||||
Changed: | |||||||||
< < | hosted at Observatoire de Paris, France
All details provided at the indico page at https://indico.obspm.fr/event/59/ | ||||||||
> > | Hosted at Observatoire de Paris, France
Agenda details and talks provided at the indico page at https://indico.obspm.fr/event/59/ | ||||||||
Finalising the proposed recommendation for Fall interoperability meeting | |||||||||
Changed: | |||||||||
< < | Minutes / ML to be completed | ||||||||
> > | Minutes of the meetings/ ML to be completed | ||||||||
Changed: | |||||||||
< < | side discussion before start: | ||||||||
> > | Side discussion before start: Poster for Adass conference ??? | ||||||||
Changed: | |||||||||
< < | Poster for Adass conference ??? | ||||||||
> > | planned: | ||||||||
Deleted: | |||||||||
< < | |||||||||
| |||||||||
Changed: | |||||||||
< < | The ProvDM draft update we plan to validate the next days:
| ||||||||
> > | The ProvDM draft update we plan to validate the next days:
| ||||||||
Deleted: | |||||||||
< < | Prov SAP : New draft updated by Kristin. Output format: in a multi layer graph, iDs should be unique for merging several graphs ( ?) Do you plan graph merging ? Kristin first attempt : All nodes are present only once | ||||||||
What does DEPTH= mean : | |||||||||
Changed: | |||||||||
< < | 1- ? Generation levels for entities 2- ? Relations between 2 classes of Prov Core : then we need to subclass the kind of relations entity/entity or entity/activity/entity? | ||||||||
> > |
Presentations around the table —Tour de table - participants
| ||||||||
Deleted: | |||||||||
< < | Presentations around the table —Tour de table - participants *Mathieu Servillat , Luth | ||||||||
| |||||||||
Deleted: | |||||||||
< < | |||||||||
| |||||||||
Changed: | |||||||||
< < | Parameter discussion : | ||||||||
> > | Parameter discussion : are there context where they are different from entities . Simulation ? A result is computed from an activity : is it an entity ? A parameter? Proposal : wasDerivedFrom is the relation to express a parameter stems from an Entity ? -->still to discuss | ||||||||
Deleted: | |||||||||
< < | are there context where they are different from entities . Simulation ? A result is computed from an activity : is it an entity ? A parameter? Proposal : WasDerivedFrom is the relation to express a parameter stems from an Entity ? still to discuss | ||||||||
Scope for the draft ?
| |||||||||
Changed: | |||||||||
< < | had context —> was influenced by | ||||||||
> > |
| ||||||||
Added: | |||||||||
> > |
| ||||||||
Changed: | |||||||||
< < | had config —> was influenced by | ||||||||
> > | Should we derive new visualisation methods for the ivoa serialisation? Enrich the Southampton Provenance Suite for instance … | ||||||||
Added: | |||||||||
> > | CTA pipe - MS | ||||||||
Changed: | |||||||||
< < | But then it is not easy to read for the user . Looses the fine classification for applications sorting the provenance info . Should we derive new visualisation methods for the ivoa? Enrich the Provenance suite for instance … | ||||||||
> > | !CTA pipe : how calibration provenance is simply recorded. | ||||||||
Changed: | |||||||||
< < |
| ||||||||
> > | Input output context config —> trace what happens. This is to cover the CTA pipeline execution . you can see it as a workflow tracking system | ||||||||
Deleted: | |||||||||
< < | CtA pipe : how calibration provenance is simply recorded. | ||||||||
Changed: | |||||||||
< < | Input output context config —> trace what happens This is to cover the CTA pipeline execution ML: I see it as a workflow tracking system | ||||||||
> > | Opus . MS | ||||||||
Changed: | |||||||||
< < |
| ||||||||
> > | Job management : PDAC service at observatoire de Paris. The description of the step is taken from the ActivityDescription definitions. A form can be filled by the job subscriptor user: | ||||||||
Deleted: | |||||||||
< < | Job management : PDAC service at observatoire de <Paris The description of the step is taken from the ActivityDescription definitions A form can be filled by the job subscriptor user: | ||||||||
Changed: | |||||||||
< < | Describe the ActivityDescription record with Param names , UCD , datatypes Used , generated section Includes defaults parameter values When the job is run, values are transmitted Provenance graph is generated at the end of job execution and displayed . | ||||||||
> > | Describe the ActivityDescription record with Param names, UCD , datatypes, Used , Generated section in the form. Includes defaults parameter values. When the job is run, values are transmitted. The Provenance graph is generated at the end of job execution and displayed . | ||||||||
DEPTH param in prov SAP: Option All agents - means track them all or show them all ?? To be discussed Identifying files locally and across various partners or a common store : Used name space to generate the unique ids. | |||||||||
Added: | |||||||||
> > | Anastasia - provenance for the Applause data base | ||||||||
Changed: | |||||||||
< < |
| ||||||||
> > | https://cloud.aip.de/index.php/s/aY6EccPAQFepH1I | ||||||||
Deleted: | |||||||||
< < | Log pages to be hooked to the scanning process : you could have a scanning activity . Not bound to a simple Prov object. Not clear . Provenance for light curves. The source for each point comes from a different observation. It makes the graph too crowded. | ||||||||
Added: | |||||||||
> > | Log pages to be hooked to the scanning process : you could have a scanning activity. Not bound to a simple Prov object. Not clear . Provenance for light curves. The source for each point comes from a different observation. It makes the graph too crowded. | ||||||||
How to distinguish data points provided, from observations provenance ? A source/ rather a detection/ as an entity . How can you adjust the granularity ? | |||||||||
Changed: | |||||||||
< < | ? Michael : Did you try to use summary functions from the ProvToolBox Summaries ? It helps to gather info in a coarse grain view. | ||||||||
> > | ? Michael : Did you try to use summary functions from the ProvToolBox Summaries ? It helps to gather info in a coarse grain view. | ||||||||
Added: | |||||||||
> > | Michael johnson - Astronomy WFlows | ||||||||
Deleted: | |||||||||
< < |
| ||||||||
2 use cases : | |||||||||
Changed: | |||||||||
< < |
| ||||||||
> > | a. Lsst differential photometry : is it a new detection or an error ? | ||||||||
Added: | |||||||||
> > | |||||||||
b. Was the image badly calibrated due to a bad choice in the standard star? Trace how the standard star is measured and chosen. What you pay for provenance : +45% info to store what you get : trust improve 99% for use case 1 , and % ( a bit less for use case 2 | |||||||||
Changed: | |||||||||
< < | UML2PROV Carlos … to generate Provenance helps to upgrade data quality and trust Highly use case dependent : you must track the info that may cause errors or improve trust | ||||||||
> > | UML2PROV Carlos … to generate Provenance templates helps to upgrade data quality and trust Highly use case dependent : you must design what to track by selecting the info that may cause errors or improve trust. | ||||||||
—> this feeds the Activity desc and parameters desc How to construct Prov templates ? What is the status ? | |||||||||
Added: | |||||||||
> > | Prov SAP for Pollux- Michèle | ||||||||
Changed: | |||||||||
< < |
| ||||||||
> > | Agents : which Agents ? All, only one activity? Mainly to track the history of a data set. | ||||||||
Deleted: | |||||||||
< < | Agents : which Agents ? All , only one activity? Mainly to track the history of a data set Prov tap is more general Parameters responseFormat | ||||||||
Changed: | |||||||||
< < | ML ? should RESPONSE Format be for graphics format ? How could we distinguish both outputs? Response : this can be chained to another application : keep only one output | ||||||||
> > | More focused than ProvTAP which will allow to ask a larger set of queries by design. | ||||||||
Changed: | |||||||||
< < | Probably we will have advanced viewers that develops groups of metadata from the viewer hierarchical browsing. | ||||||||
> > | ?Parameters responseFormat | ||||||||
Changed: | |||||||||
< < |
| ||||||||
> > | ML ? should RESPONSE Format be for graphics format ? How could we distinguish both outputs? Response : this can be chained to another application : keep only one output? need to be disussed further. | ||||||||
Deleted: | |||||||||
< < | Shows a ProvSap client interface Sankey representation for graphics Is a possible output format They are exclusive choices . Need to run it again if you want it again as metadata . Then it is not scriptable ? What to do with all the SVG files ? | ||||||||
Changed: | |||||||||
< < | What means DEPTH ? Relations counting or entity to entity generation layer? | ||||||||
> > | Probably we will have advanced viewers that develops groups of metadata from the viewer using a hierarchical browsing function. | ||||||||
Added: | |||||||||
> > | Kristin - Provenance for the Rave Survey | ||||||||
Changed: | |||||||||
< < |
| ||||||||
> > | Shows a ProvSap client interface. Sankey representation for graphics. | ||||||||
Deleted: | |||||||||
< < | Draft ready since April, please provide comments Prototype for TAP service : | ||||||||
Changed: | |||||||||
< < | This is an implementation on a simple database Tap service working Interface with aladin , topcat, etc . | ||||||||
> > | ?ML: Graphics is a possible output format but the options are exclusive choices . Need to run it again if you want it again as metadata . Then it is not scriptable ? What to do with all the SVG files ? | ||||||||
Changed: | |||||||||
< < | To do to enrich the CDS prototype? insert the 300 Hips files | ||||||||
> > | ?What means DEPTH ? Relations counting or entity to entity generation layer? | ||||||||
Changed: | |||||||||
< < | question Anastasia: How many Gbytes costs the Provenance metadata? | ||||||||
> > | Discussion: | ||||||||
Changed: | |||||||||
< < | Store data provenance for any source ? Is it possible? Discuss with the Gaia people ? | ||||||||
> > | Kristin : Propose to give up the IVOA JSON format. (and use only Prov-JSON) | ||||||||
Changed: | |||||||||
< < | François: If you can query the data base in SQL, and store all your sources, then TAP can do If you have a unique id for an entity and a dataset publisher id , | ||||||||
> > | Markus : we have several JSON flavours in the game. May be better to leave it open and see how people implement it. | ||||||||
Added: | |||||||||
> > | François Bonnarel ProvTAP | ||||||||
Changed: | |||||||||
< < | You can query both sides : the provenance data base , and the Obscore data base , by a data link , by a broadcast mechanism from one data base to the other ? | ||||||||
> > | the Prov TAP Draft is ready since April, please provide comments on the documents. | ||||||||
Changed: | |||||||||
< < | Give me the provenance of obs-publisherid =12345. —> query provtap or provsap with e-id =12345 And vice versa : find an e-id and build up the query on an Obstap service ….. Pbs with PosgresQL and server settings … May be try docker … a new package can include extra relations into a W3C provenance document and visualize them | ||||||||
> > | demonstrate a prototype for a Provenance TAP service : | ||||||||
Changed: | |||||||||
< < | Extensions of the visualisation | ||||||||
> > | This is an implementation on a simple database accessed through Tap service. Interface with aladin , topcat, etc . are working and allow to discover datasets served in Obscore , for instance from selection on the provenance metadata. | ||||||||
Changed: | |||||||||
< < | + Working Draft discussion : | ||||||||
> > | To do to enrich the CDS prototype? insert more Hips files ( planned 300 using an ingestion script) | ||||||||
Changed: | |||||||||
< < | Check the DM Requirements: if an entity exist, there is an Activity to create it. Update the expression of these constraints. Identifiers : must be unique . Replica of an entity ? is another entity. Must be specified as a copy Agreed on : Entities, Activities, Agent must be uniquely identifiable. | ||||||||
> > | question Anastasia: How many Gbytes costs the Provenance metadata? is the number of entities a pb in a relational database? | ||||||||
Changed: | |||||||||
< < | 12 . Contact info should be recorded for all Activities and entities … | ||||||||
> > | ichallenge: Store data provenance for any source ? Is it possible? Discuss with the Gaia people ? | ||||||||
Changed: | |||||||||
< < | i.e. there is always a known Agent linked with them ( even a general one) . | ||||||||
> > | François: If you can query the data base in SQL, and store all your sources, then TAP can do as long as you have a unique id for an entity and a dataset publisher id , | ||||||||
Changed: | |||||||||
< < | What is entityDescription? Do we need it ? Who did implement it? | ||||||||
> > | You can query both sides : the provenance data base and the Obscore data base, by a data link , by a broadcast mechanism from one data base to the other ? | ||||||||
Changed: | |||||||||
< < | How do we describe the role and the constraints to be used by an activity entities when they are to be used by an activity. Used and used description / What is needed for Input params ? An id A title For us we can have a role | ||||||||
> > | Give me the provenance of obs-publisherid =12345. —> query provtap or provsap with e-id =12345 And vice versa : find an e-id and build up the query on an Obstap service ….. Pbs with PosgresQL and server settings … May be try docker … a new package can include extra relations into a W3C provenance document and visualize them | ||||||||
Changed: | |||||||||
< < | What is needed for output params ? | ||||||||
> > | Extensions of the visualisation functions is needed if we extend the model. ? PROV Suite authors will be ok with this ( from a discussion Mathieu-Duong) | ||||||||
Added: | |||||||||
> > | Working Draft discussion / how to finalise it | ||||||||
Changed: | |||||||||
< < | MS: The description side is the degree of freedom for each project to feed its specific metadata. | ||||||||
> > | Check the DM Requirements: if an entity exist, there is an Activity to create it. Update the expression of these constraints. Identifiers : must be unique . Replica of an entity ? is another entity. Must be specified as a copy Agreed on : Entities, Activities, Agent must be uniquely identifiable. | ||||||||
Changed: | |||||||||
< < | EntityDescription could be one of this placeholder ??? We need more info about this… more practical usage examples. | ||||||||
> > | Requirement 12 . Contact info should be recorded for all Activities and entities …i.e. there is always a known Agent linked with them ( even a general one) . | ||||||||
Changed: | |||||||||
< < | Parameter : What is a parameter in our model? A parameter cannot be used or generated by ? Is an entity but restricted… | ||||||||
> > | What is entityDescription? Do we need it ? Who did implement it? | ||||||||
Changed: | |||||||||
< < | It can be derived from an Entity when it is Data. ?? Do we need to say it ?? We coud also setup this rule: If your parameter must be traced , define it as an entity . | ||||||||
> > | How do we describe the role and the constraints to be used by an activity entities when they are to be used by an activity. Used and used description / | ||||||||
Changed: | |||||||||
< < | EntityDescription | ||||||||
> > |
| ||||||||
Added: | |||||||||
> > |
Parameter : | ||||||||
Changed: | |||||||||
< < | What would you put here ? I would put the required format for my entities to be used with Activitydesc Axx The role | ||||||||
> > | What is a parameter in our model? A parameter cannot be used or generated by ? Is an entity but restricted… It can be derived from an Entity when it is Data. ?? Do we need to say it ?? | ||||||||
Changed: | |||||||||
< < | example : ActivityDescription : regridding Constraints input-format=Fits. ??? Mimetype ? output formats= fits errorMapPreviewformats = jpeg | ||||||||
> > | ML: We coud also setup this rule: If your parameter must be traced , define it as an entity . | ||||||||
Added: | |||||||||
> > | EntityDescription | ||||||||
Changed: | |||||||||
< < | Sextractor : with particular output styles : .dat --> check for an example to describe | ||||||||
> > | What would you put here ? I would put the required format for my entities to be used with Activitydescription 'Axyz'. | ||||||||
Changed: | |||||||||
< < | Do we specialize the table for each particular specialized entities. Yes for a full archive …. | ||||||||
> > |
| ||||||||
Added: | |||||||||
> > |
| ||||||||
What goes into the draft:
| |||||||||
Changed: | |||||||||
< < | this part is important to provide metadata to the Core Model. We explore the PROV-ONE proposition and have also checked the possibility of the W3C Plan . | ||||||||
> > | This part is important to provide metadata to the Core Model. We explore the PROV-ONE proposition and have also checked the possibility of the W3C Plan for ActivityDescription. | ||||||||
Changed: | |||||||||
< < | Mention that we consider we could reuse the input and output ports from PROV-ONE Process from PROV-ONE is similar to our Activity Description, but may be not with the same granularity. To be checked . | ||||||||
> > | Mention that we consider we could reuse the input and output ports from PROV-ONE Process from PROV-ONE DM is similar to our Activity Description, but may be not with the same granularity. To be checked . | ||||||||
Added: | |||||||||
> > | We notice that the EntityDescription even still fuzzy is interpreted as a way to convey metadata about entities usage and cannot just be removed straight. We need more practice to clarify the possible usage . It is really important to go for implementaions and sort out the refactoring and specialising possibilities from the feedback. 2 reference implementations are requested anyway for the IVOA review of the Proposed Recommendation. | ||||||||
— End of day 2
Useful links: | |||||||||
Changed: | |||||||||
< < | Add useful links here | ||||||||
> > | Most of the presentations are hoocked under the indico page mentionned above. | ||||||||
<--
|