(r26) ProvenanceRFC < IVOA

IVOA Web>IvoaDataModel>ProvenanceRFC (revision 26) (raw view)~~EditAttach~~
---+ Provenance Data Model RFC Page
---+++ Document

<span style="background-color: transparent;">The attached document presents the IVOA Provenance Data Model v1.0 which is proposed for review in this [[http://wiki.ivoa.net/internal/IVOA/ProvenanceRFC/PR-ProvenanceDM-1_0-20181015.pdf][PR document]].</span>

<span style="background-color: transparent;"> </span><span style="background-color: transparent;">The model has been examined thoroughly with respect to the original use cases. Recent contacts at the Provenance Week has led to a number of discussions in order to stabilise the model and compare to the W3C data model and its customisations in various application domains (ProvONE, UniProv, RDA Prov patterns, etc).</span>

<span style="background-color: transparent;"> </span><span style="background-color: transparent;">The provenance information is modeled with different focuses: the core model with the main classes bound to the tasks (activities) in which datasets (entities) are involved, and an extended model to attach information on the configuration of thoses tasks (input parameters, config file), their description, and the context (ambient conditions, execution environment, etc).</span>

<span style="background-color: transparent;"> </span><span style="background-color: transparent;">Not all projects need the full detail of the model, but the core model will answer the general needs to trace progenitors of a dataset, for instance.</span>
---+++ Reference Implementation
---+++++ %BLUE%Rave provenance data database access (K. Riebe)%ENDCOLOR%

%BLUE%<br />%ENDCOLOR%
---+++++ %BLUE%CTA implementation (M. Servillat)%ENDCOLOR%

<span style="background-color: transparent;">The Cherenkov Telescope Array (CTA) is the next generation ground-based very high energy gamma-ray instrument. Contrary to previous Cherenkov experiments, it will serve as an open observatory providing data to a wide astrophysics community, with the requirement to propose self-described data products to users that may be unaware of the Cherenkov astronomy specificities. Because of the complexity in the detection process and in the data processing chain, provenance information of data products are necessary to the user to perform a correct scientific analysis.</span>

Provenance concepts are relevant for different aspects of CTA:
   * Pipeline: the CTA Observatory must ensure that data processing is traceable and reproducible, making the capture of provenance information necessary.
   * Data diffusion: the diffused data products have to contain all the relevant context information as well as a description of the methods and algorithms used during the data processing.
   * Instrument Configuration: the characteristics of the instrument at a given time have to be available and traceable (hardware changes, measurements of e.g. a reflectivity curve of a mirror, ...)
In the context of CTA, two main implementations were developed:
   * a python Provenance class dedicated to the capture of provenance information of each CTA pipeline tool (including contextual information and connected to configuration information)<br /> https://cta-observatory.github.io/ctapipe/api/ctapipe.core.Provenance.html
   * OPUS: a job controller that can execute predefined jobs on a work cluster and expose the results. This job controller follows the IVOA UWS pattern and the definition of jobs proposed in the IVAO Provenance DM, so as to capture and expose the provenance information for each job and result via a ProvSAP interface.<br /> https://uws-server.readthedocs.io
---+++++ %BLUE% Pollux Provenance: a simple access protocol to provenance of theoretical spectra (M. Sanguillon)%ENDCOLOR%

<span style="background-color: transparent;">POLLUX is a stellar spectra database proposing access to high resolution synthetic spectra computed using the best available models of atmosphere (CMFGEN, ATLAS and MARCS), performant spectral synthesis codes (CMF_FLUX, SYNSPEC and TURBOSPECTRUM) and atomic line lists from VALD database and specific molecular line lists for cool stars. </span>

Currently the provenance information is given to the astronomer in the header of the spectra files (depending on the format: FITS, ASCII, XML, VOTable, ...) but in a non-normalized description format. The implementation of the provenance concepts in a standardized format allows users on one hand to benefit from tools to create, visualize and transform to another format the description of the provenance of these spectra and on a second hand to select data depending on provenance criteria.

In this context, the ProvSAP protocol has been implemented to retrieve provenance information in different formats of the serialized data: PROV-N, PROV-JSON, PROV-XML, VOTABLE and to build diagrams in the following graphic formats: PDF, PNG, SVG. These serializations and graphics are generated using the voprov python package derived from the prov Python library (MIT license) developed by Trung Dong Huynh (University of Southampton).
---+++++ %BLUE%SVOM Quick Analysis (L. Michel)%ENDCOLOR%

The [[http://www.svom.fr/en/][SVOM]] satellite is a Sino-French variable object monitor to be launched in 2021. When a transient object is detected, a fast alert is sent to the ground station through a worldwide VHF network. Burst advocates and instrument scientists are in charge of evaluating the scientific relevance of the event. To complete the assement, scientists have at their disposal high level data products such as light curves or spectra generated by an automatic data reduction pipeline. In some case, they might need to reprocess the raw data with refined parameters. To do so, scientific products (calib_leve &gt;= 2) embedd their JSON provenance serialization in a specific extension. This provenannce instance can be extracted, updated and then uploaded to a dedicated pipeline to reprocess the photon list with different parameters.
---+++++ <span style="color: blue;">PROV CDS Data base : Implementation of a TAP service for Prov metadata bound to ObsTAP metadata. (F . Bonnarel, M. Louys, G. Mantelet) TBC</span>

ProvTAP is a proposal for providing Provenance metadata via TAP services. The draft can be found here (http://volute.g-vo.org/svn/trunk/projects/dm/provenance/provtap/ProvTAP.pdf). It is basically providing a TAP-schema mapping the provenance model onto the relational schema.

The CDS ProvTAP service prototype is implementing this ProvTAP specification on top of a database providing metadata for HipS generation and also Schmidt plate digitization, cutouts opérations and RGB image construction. A full presentation of the prototype with a "slide demo" including a lot of ADQL queries can be found here : https://indico.obspm.fr/event/59/session/1/contribution/7/material/slides/

An oral presentation will be made at next ADASS
---+++++ %BLUE%MuseWise Provenance: Implementation of ProvSAP (O. Streicher)%ENDCOLOR%

MUSE is an <a href="https://en.wikipedia.org/wiki/Integral_field_spectrograph" title="Integral field spectrograph">integral field spectrograph</a> installed at the Very Large Telescope (VLT) of the European Southern Observatory (ESO). It consists of 24 spectrographs, providing a 1x1arcmin FOV (7.5" in Narrow Field Mode) with 300x300 pixel. For each pixel, a spectrum covering the range 465-930nm is provided. MuseWise is the data reduction framework that is used within the MUSE collaboration. It is built on the Astro-WISE system, which has been extended to support 3D spectroscopic data and integrates data reduction, provenance tracking, quality control and data analysis. The MUSE pipeline is very flexible and offers a variety of options and alternative data flows to fit to the different instrument modi and scientific requirements.

The implementation provides the information collected by the system in using the ProvSAP protocol. The problems that arised during this implementation are discussed below.
---+++ Implementations Validators

TBC
---+++ RFC Review Period: 2018 October 22 - 2018 November 19
---++++ Comments from WG members

The PR was made before the working group converged on a common draft, leaving some discussions for the RFC period. So, the draft represents only part of the working group.

_The situation came from the fact that the concerns you expressed where not firmly supported by difficulties in tackling real use cases,_ _despite your claims. See below why Parameters have been managed the way they are in the PR. I am pretty sure this can also work for your MUSE use-case, Ole_ -- IVOA.FrancoisBonnarel - 2018-10-23

The main concerns and alternative proposals are presented here.

For convenience, we compiled the changes from the first three topics (Parameters, special Entities, W3C serialization) into a <a href="http://wiki.ivoa.net/internal/IVOA/ProvenanceRFC/ProvenanceDM-alternate.pdf" title="PDF covering sections 2.2 and 3.2 of the draft">PDF covering sections 2.2 and 3.2 of the draft</a>, including updated class diagrams.

---+++++ Special handling of Parameters

While in the model =Parameter= is labelled as an =Entity=, they are used in a completely different way:

   * The relation between the =Entity= and the =EntityDescription= is a =hasDescription= relation, while the =Parameter= is linked to the =ParameterDescription= by an attribute.
   * The usage of an =Entity= by an =Activity= is specified by the =used= relation, while the usage of a =Parameter= by an =Activity= is specified by the =hadConfiguration= relation.
   * While the description of the usage for an =Entity= happens with a =UsedDescription=, the description of the usage for a =Parameter= is mangled into the =ParameterDescription=.
<em>This is not true. Used class contains a role attribute and HadConfiguration is a specialization of Used, which can give the role of the Parameter with respect to the Activity </em><span style="background-color: transparent;">-- </span><span style="background-color: transparent;">IVOA.FrancoisBonnarel</span><span style="background-color: transparent;"> - 2018-10-23</span>

%GRAY%The class hierarchy of =hadConfiguration= is not specified in your Proposal. And all properties of the =Parameter= in your proposal are only with respect to a certain activity (this is how you actually define the =Parameter=, see your comment below), so they are in the =ParameterDescription=. For example the =name=, =min=, =max=, =default=, =unit=. The =role= attribute for =hadConfiguration= is explicitly undefined, therefore =hadConfiguration= has intentionally no corresponding description. -- IVOA.OleStreicher - 2018-10-24%ENDCOLOR%

<span style="color: lightsteelblue;"> </span>

<span style="color: lightsteelblue;">The PR explains that hadConfiguration is to be seen as a simple association table, no role there, a simplification of used, not a specialization. Indeed, the ParameterDescription transports the attributes that desccribe the parameter, i.e. its name, ucd, utype, or finer definition such as min, max, options... So the ParameterDescription already transports the role of the parameter (its name in fact, completed by the ucd and utype) and it would be redundant to repeat the usage in a UsedDescription.</span> -- IVOA.MathieuServillat - 2018-10-24

%GRAY%So you agree that Francois comment was wrong. Maybe we remove the comments related to this to keep the discussion clean? Francois?-- IVOA.OleStreicher - 2018-10-24%ENDCOLOR%

Specifically the last item limits the usability of =Parameter=: =Parameter= is bound to a specific =ActivityDescription= (since the =Parameter= has a single link to the =ParameterDescription=, and this has a single link to the =ActivityDescription=):

   * They cannot be created independently from their usage
   * They cannot re-used by an =Activity= with a different =ActivityDescription= (even not just another version!).
   * =Activity= cannot accept =Parameters= with different =ParameterDescriptions= (f.e. different units) for the same place.
_This is intentional_ -- IVOA.FrancoisBonnarel - 2018-10-23

The specific handling of =Parameter= s is limited to values that are used in the configuration of an =Activity=. In contrast, values that carry non-configuration data are handled as a general =Entity=. This is also the case for values that are used for configuration, but don't fit into the listed restrictions.<br />A discussion within the working group pointed out three ways to handle configuration values:

   1 They may be handled as a normal =Parameter=,
   1 They may be defined as =Parameters=, but used with a general =Entity=. In this case it is unclear which =ParameterDescription= is relevant the one that is related to the =Entity= by the =hasDescription= relation or the one related to the =Activity= by the =hadConfiguration= relation.
   1 They may be handled like a general =Entity=. This contradicts however the class hierarchy, which only defines =Parameter=, =ConfigFile= and =ObsConfig= as subclasses of =Configuration=, but not =Entity= that carry a value. So it is unclear whether f.e. they shall related to the =Activity= via the =used= or the =hadConfiguration= relation.
The choice between these three ways may vary between different implementations, and even within one implementation. A client (or a query) therefore needs to check all possible ways to get a complete answer, and on the implementation side the selection of the correct way is not trivial either. So, the special handling of =Parameter= complicates the voprov data model significantly, without gaining an appropriate advantage.

Another sign of a bad model is that for UWS parameters it requires a hack of having =Parameter= that actually contain a reference instead of a value, which is explained in the Appendix B3 or the draft.

Therefore, we propose to homogenize the handling of =Parameter= with the handling of other =Entity= by re-using the =used=, =EntityDescription=, and =UsageDescription= classes also for =Parameter=. Then, =Parameter= is special only by the fact that they directly contain a value, while other =Entity= s would refer to their content by a link.

_No (Data) Entities have values too._ -- IVOA.FrancoisBonnarel - 2018-10-23

%GRAY%You did misunderstand this sentence: Our [[http://wiki.ivoa.net/internal/IVOA/ProvenanceRFC/ProvenanceDM-alternate.pdf][alternative proposal]] is to make =Parameter= be a special =Entity= that carries a value, while other (Data) =Entities= refer to the content via link.-- IVOA.OleStreicher - 2018-10-24%ENDCOLOR%

<span style="color: lightsteelblue;"> </span>

<span style="color: lightsteelblue;">The limit here is that we should not assume what a general entity will be (how can we?), so we do not describe the content/value of it, just the general category (specialized entity that are commonly manipulated in astronomy) or the type of container (in e.g. the EntityDescription with format, content_type... i.e. how to read it and not what we read). However, configuration information in the form of parameters as an input to an activity are relevevant provenance information (it helps assess the reliability and quality of the activity, see section 2.2.6 of the PR), so for this input parameter, we describe and restrict the expected value of it, so that it becomes queryable in the standard. The content/value of a Data entity is not expected to be queryable using a provenance system, nor to provide relevant provenance information.</span> -- IVOA.MathieuServillat - 2018-10-24

%GRAY%The value of a =Parameter= is also stored in the simpler [[http://wiki.ivoa.net/internal/IVOA/ProvenanceRFC/ProvenanceDM-alternate.pdf][alternative model]]. Our model does not restrict the queryability of that value. Also your proposal does not specify that BTW. -- IVOA.OleStreicher - 2018-10-24%ENDCOLOR%

This would also make the described hack unnecessary.

_General statement to refuse this evolution: It is true that Parameter has managed in a special way in the current PR. It is intentional. Why ?_ _The Parameter class is there to tackle what astronomy application users generally call "parameters" of the application._ _Think to SExtractor or HipsGen. They have a couple of parameters such as "ANALYSIS _THRESH", "MAG_ZEROPOINT" (SExtractor), fov, skyval, border, publisher (HipsGen)._ _They are definitly a different concept than DataEntity, which are the things we want to follow with our Provenance model, the things which are used or transformed by the activities._ _So parameters are so strongly bound to activities that they don't have existence outside their bounding to an activity._ _They have a value which is either a number or string, or a reference to external structures (files) or internal entities._ _In the latter case it is possible to use as a parameter value an id of something which has an history in the provenance._ _ActivityDescription and EntityDescription are there to gather all the properties common to activities sharing the same kind of processing and to Entities they use or generate._ _ParameterDescription gathers all the metadata common to Parameters of Activities sharing the same ActivityDescription._ _They have the same kind of binding to ActivityDescription than Parameters have to activities._ _The organisation you are proposing, Ole, simply miss the specificity of what is intended by the parameter concept._ _Parameter class has also a strong semantic value and so have ParameterDescription and hadConfiguration._ _Last point: Parameter is a derivation of Entity and ParameterDescription a derivation of EntityDescription._ _The benefit of this is that generic W3C-aware software can manage these classes. Parameter is an Entity with a special type= voprov:parameter._ _But this type changes all the "movie" (as we say in French) for behavior and interpretation._ -- IVOA.FrancoisBonnarel - 2018-10-23

%GRAY%For the discussion of (IVOA) parameters see point 7 in section "Several inconsistencies" below.%ENDCOLOR%

%GRAY%Independent of this, you did not explain why specifics of =Parameter= require a distinct structure in the model instead of completely integrating them into the normal =Entity= - =Activity= relations. All the requirements you specified for parameters are already fullfilled by the simpler [[http://wiki.ivoa.net/internal/IVOA/ProvenanceRFC/ProvenanceDM-alternate.pdf][alternative model]].-- IVOA.OleStreicher - 2018-10-24%ENDCOLOR%

<span style="color: lightsteelblue;"> </span>

<span style="color: lightsteelblue;">In addition to François' examples, I answer this above, and it is explained why the value of a parameter is relevant provenance information in the PR at the beginning of section 2.2.6.</span> -- IVOA.MathieuServillat - 2018-10-24
---+++++ Linking usage to =EntityDescription=

The voprov model defines a number of specialization of =Entity=, based on their usage:

   * =MainEntity= ( =Data=, =Visualization=, =Document=),
   * =Configuration= ( =Parameter=, =ConfigFile=, =ObsConfig=),
   * =Context= ( =AmbientConditions=, =InstrumentalContext=, =ExecutionEnvironment=).
These =Entitys= are linked to the =Activity= with different relations: =used=, =hadConfiguration=, and =hadContext=.<br />Binding the usage to the =Entity= specialization is wrong, since the same =Entity= may be used differently by different =Activities=: f.e. what is a configuration for a data processing =Activity= may be input data for the generation of some visualization. It is also redundant, since the usage type is already specified by the usage relation.

The description of the =Entity= itself (f.e. whether it is a "key=value" list in a given format) is already done in the =EntityDescription=. This makes further specializations unnecessary.

We therefore propose to remove the specialized =Entity= classes =Configuration= and =Context= and their subclasses from the standard and use the main class here, together with specialized usage relations. It should then also defined that the =hadConfiguration= and =hadContext= relations are specialized =used= relations.

The specialized =Entity= classes =Visualization=, =Document=, and =Device= can also be replaced by putting this information into their =EntityDescription= (this is what the =EntityDescription= was made for). In ProvONE, this could not be done, since ProvONE does not have an =EntityDescription= to describe the =Entity=.

---+++++ W3C serialization

The W3C serialization is integral part of the draft (Sec.3.2), but largely unspecified. It is f.e. undefined, how =hadConfiguration=, =hadContext=, and =hadDescription= relations for =Entity= are represented in the W3C serialization. For the representation of other attributes, only a suggestion is given. This makes it impossible to develop a client that consistently uses these attributes, which makes the format rather useless.

Since the model is already largely W3C consistent, we propose to include normative W3C representations for all attributes and classes directly in the model description.

The draft normatively defines the prefix for the namespaces. This is incomplete and unnecessarily unflexible. Usually the namespace is defined by (only) the URI, which should be followed here as well.

---+++++ VO-DML compatible model

The VO-DML compatible data model (Fig. 6) does not correspond to the original voprov model (f.e. Fig. 5) and is significantly restricted: While in the original model =Parameter= is an =Entity= and may have provenance information attached (like how the =Parameter= was created), in the VO-DML compatible model =Parameter= is not derived from =Entity= and therefore cannot have provenance information.

Since the VO-DML compatible model is the base for other standards like provTAP, full correspondence to the original model is important here. So, the VO-DML compatible model must still be updated to reflect the recent changes.

It is important to have serializations like provTAP fully compatible to the original model (and thus to W3C prov structure): for many dataprocessing frameworks that are used in astronomy (Kepler, Taverna etc.) there is ongoing work to produce W3C compatible provenance information, and it should be possible to use this to fill a database that can be f.e. queried with provTAP without major restructuration.

---+++++ Attributes designated to Workflow

While the draft explicitly excludes reproducibility in a workflow as a use case, there are a number of attributes that do not describe the actual provenance or how the =Activity= actually works, but document how an =Activity= *should* be used. This includes:

   * =min=, =max=, =options=, and (default) =value= in =ParameterDescription=
   * =multiplicity=, =entityDescription= in =UsedDescription=
   * =multiplicity=, =entityDescription= in =WasGeneratedByDescription=
These attributes describe the requirements of the input resp. the intended output, which is part of the workflow instead of the provenance, and should therefore removed from the draft.

---+++++ Several inconsistencies

There is a large number of inconsistencies in the document:

   * The class diagram (Fig. 5) limits =hadConfiguration= to =Parameter=, while the text implies that this relation should also used for =ConfigFile= and =ObsConfig=
   * The text allows a =Parameter= to be replaced by a general =Entity=, but this is not shown in the class diagram
   * It is unclear whether the usage of a =Context= or a (non- =Parameter=) =Configuration= ( =Configfile=, =ObsConfig=), expressed by a =hadContext= or a =hadConfig= shall/may be accompanied with a =UsedDescription=.
   * The class diagram specifies a m:n relation between =UsedDescription= resp. =WasGeneratedByDescription= and =EntityDescription=, but the according tables (Table 14 and 15) specify only a single link (1:n relation).
   * The =startTime= and =endTime= attributes in =Activity= (Table 3) are mandatory (cannot be empty). However, sometimes they may be unknown or not relevant. They should be optional (this does not affect the ability to sort or to query those attributes).
   * It was agreed in the WG that =ParameterDescription= is a subclass of =EntityDescription= to allow a non- =Parameter= =Entity= that carry a value to be linked to the same type of description. This is however not documented in the text or in the diagram, and also not in the definition table of =EntityDescription= (Table 12).
   * As motivation for the special handling of =Parameter=, the text lists several uses of the word "param(eter)" in the VO, claiming that they belong to a single concept. This is however not true: in UWS a parameter ( =uws:param=) corresponds to a generic =Entity= and a =used= relation in voprov, since it may contain either a value or a reference, and is unspecified whether it is data or configuration. A VOTABLE column with its =FIELD= element is somehow modelled similar to a list of =Parameter= s in the draft; however it is also not limited to configuration and may actually carry non-config data. It is also unrelated to a specific activity. So, this cannot serve as motivation for the definition of =voprov:Parameter= in its current form (carry a value, limited to configuration).
   * In the documentation of =Parameter=, =value= is documented as "[...] type depends on =ParameterDescription.datatype= and =xtype= ". However, the =ParameterDescription= is optional. It is unclear how the type is specified when the =ParameterDescription= is missing. Also is unclear whether the same is true for other =Entity= s that carry a =value=.
   * The attributes of the specialized relations ( =hasDescription=, =hadDescription=, =hadConfiguration=, =hadContext=) are not defined.
   * The tables often contain attributes that "must not be null" (bold), other attributes, and "Optional Attributes". It is unclear what the difference between other attributes and "Optional Attributes" is. Also, below the "Optional Attributes", links are listed (separated by a horizontal line). It is unclear whether they also belong to the Optional Attributes.
   * The =hadContext= and =hadConfiguration= relations have an empty =role= attribute. This makes it impossible to relate them to a certain description. If an =Activity= uses several =Entities= (files) for configuration with the =hadConfiguration= relation, there is no way to distinguish their role.
-- IVOA.OleStreicher - 2018-10-22

---+++ TCG Review Period: TCG_start_date - TCG_end_date

TBC
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pdf	PR-ProvenanceDM-1_0-20181015.pdf	r1	manage	1015.0 K	2018-10-16 - 20:06	MathieuServillat	Proposed Recommendation IVOA Provenance Data Model
pdf	PROVTAPAug2018.pdf	r1	manage	7802.8 K	2018-10-16 - 15:55	FrancoisBonnarel	Presentation of ProvTAP service - Provenance day meeting 2018-08-27
pdf	ProvQuerytest-3store.pdf	r2 r1	manage	47.8 K	2018-10-22 - 19:22	MireilleLouys	exampleQueries in SPARQL/ CDS ImageDB in Blazegraph
txt	ProvQuerytest-3store.txt	r1	manage	7.3 K	2018-10-23 - 18:13	MireilleLouys	exampleQueries in SPARQL/ CDS ImageDB in Blazegraph
pdf	ProvTAP.pdf	r1	manage	788.6 K	2018-10-16 - 15:55	FrancoisBonnarel
pdf	ProvenanceDM-alternate.pdf	r1	manage	308.6 K	2018-10-22 - 08:29	OleStreicher	Proposed changes to the data model
rdf	requetes_prov_3Store_adass2018.rdf	r1	manage	7.2 K	2018-10-23 - 18:03	MireilleLouys	exampleQueries in SPARQL/ CDS ImageDB in Blazegraph