Provenance Data Model RFC #2 Page

History

After the first RFC period (November 2018) during which no consensus was reached, the Provenance model has been revamped.

• Based on a similar skeleton, this new version has significant differences with the former one.
• The current document has been issued after some intensive sessions held in the Spring-19 meeting in Paris
• After this, the TCG sugested to go in a new WG stage, shorten to 4 weeks, which actually ended on 2019/07/19.

Document

The attached document presents the IVOA Provenance Data Model v1.0 which is proposed for review is accessible in ivoadoc.

This document describes how provenance information can be modeled, stored and exchanged within the astronomical community in a standardized way. We follow the definition of provenance as proposed by the W3C (https://www.w3.org/TR/prov-overview/), i.e. that "provenance is information about entities, activities, and people involved in producing a piece of data or thing, which can be used to form assessments about its quality, reliability or trustworthiness."

Reference Implementation

CTA implementations (M. Servillat)

The Cherenkov Telescope Array (CTA) is the next generation ground-based very high energy gamma-ray instrument. Contrary to previous Cherenkov experiments, it will serve as an open observatory providing data to a wide astrophysics community, with the requirement to propose self-described data products to users that may be unaware of the Cherenkov astronomy specificities. Because of the complexity in the detection process and in the data processing chain, provenance information of data products are necessary to the user to perform a correct scientific analysis.

Provenance concepts are relevant for different aspects of CTA:

• Pipeline: the CTA Observatory must ensure that data processing is traceable and reproducible, making the capture of provenance information necessary.
• Data diffusion: the diffused data products have to contain all the relevant context information as well as a description of the methods and algorithms used during the data processing.
• Instrument Configuration: the characteristics of the instrument at a given time have to be available and traceable (hardware changes, measurements of e.g. a reflectivity curve of a mirror, ...)

• a python Provenance class dedicated to the capture of provenance information of each CTA pipeline tool (including contextual information and connected to configuration information)
  https://cta-observatory.github.io/ctapipe/api/ctapipe.core.Provenance.html
• OPUS: a job controller that can execute predefined jobs on a work cluster and expose the results. This job controller follows the IVOA UWS pattern and the definition of jobs proposed in the IVAO Provenance DM (Description classes), so as to capture and expose the provenance information for each job and result via a ProvSAP interface.
  https://opus-job-manager.readthedocs.io

Pollux Provenance: a simple access protocol to provenance of theoretical spectra (M. Sanguillon)

POLLUX is a stellar spectra database proposing access to high resolution synthetic spectra computed using the best available models of atmosphere (CMFGEN, ATLAS and MARCS), performant spectral synthesis codes (CMF_FLUX, SYNSPEC and TURBOSPECTRUM) and atomic line lists from VALD database and specific molecular line lists for cool stars.

Currently the provenance information is given to the astronomer in the header of the spectra files (depending on the format: FITS, ASCII, XML, VOTable, ...) but in a non-normalized description format. The implementation of the provenance concepts in a standardized format allows users on one hand to benefit from tools to create, visualize and transform to another format the description of the provenance of these spectra and on a second hand to select data depending on provenance criteria.

In this context, the ProvSAP protocol has been implemented to retrieve provenance information in different formats of the serialized data: PROV-N, PROV-JSON, PROV-XML, VOTABLE and to build diagrams in the following graphic formats: PDF, PNG, SVG. These serializations and graphics are generated using the voprov python package derived from the prov Python library (MIT license) developed by Trung Dong Huynh (University of Southampton).

SVOM Quick Analysis (L. Michel)

The SVOM satellite is a Sino-French variable object monitor to be launched in 2021. When a transient object is detected, a fast alert is sent to the ground station through a worldwide VHF network. Burst advocates and instrument scientists are in charge of evaluating the scientific relevance of the event. To complete the assement, scientists have at their disposal high level data products such as light curves or spectra generated by an automatic data reduction pipeline. In some case, they might need to reprocess raw data with refined parameters. To do so, scientific products (calib_leve >= 2) embedd their JSON provenance serialization in a specific extension. This provenance instance can be extracted, updated and then uploaded to a dedicated pipeline to reprocess the photon list with different parameters.

ProvHIPS CDS prototype service providing provenance metadata for HiPS datasets stored at CDS. ( F. Bonnarel, A. Egner)

This prototype is both an implementation of Provenance Data Model and of the DAL ProvTAP protocol.

ProvTAP is a proposal for providing Provenance metadata via TAP services. The current draft for this DAL protocol definition (TBC) can be found here). It is basically providing a TAP-schema mapping the IVOA Provenance model onto a relational schema.

• A first implementation of a TAP service for Provenance metadata bound to ObsTAP metadata was done by F . Bonnarel, M. Louys and G. Mantelet in August 2018.

• A second version of ProvHiPS was presented at ADASS (http://adass2018.astro.umd.edu/abstracts/O11-3.pdf) and was gathering provenance metadata on the process of transformation of original data collections into HiPS datasets.

HiPS is an all sky organisation of astronomical data (images, catalogs) based on Healpix tesselation of the sky, with iso-area, iso-latitude property of the tiles building up the mesh structure. (see HiPS description) Each HiPS data collection is obtained by the reprojection on the celestial sphere of original FITS images belonging to the same image collection. The hipsgen code fis used at CDS to generate the HiPS format of data stored at CDS image data base. These image collections are themselves produced in various ways and stages , from raw data, mosaiced, calibrated, normalized formats before they can be ingested into the HiPS processing activity.

For example in the case of HST images it is possible to trace back from HST HiPS datasets to stacked images collection, single calibrated exposure collections or raw exposure image collections In the case of HiPS obtained from Schmidt plate datasets, it is possible to go back to the plates through the digitized images.

A triple Store implementation for an image database (M.Louys, F.X Pineau, L.Holzmann, F.Bonnarel).

We have implemented the IVOA provenance DM proposed using a BlazeGraph triplestore data base. It handles astronomical images as Entities with a succession of Activities that produce or consume them. Entities represent image files or photometric plates. Activities are described by an ActivityDescription instance which gives the template for execution of several different Activity instances based on the same template. / The Blazegraph prov-test prototype translates the classes via an Ontology of Object in OWL. The Prov_Owl diagram attached exposes the main objects of this ontology. This prototype implements:

• Agent, Activity, Entity ( mainly DatasetEntity)
• Used, wasGeneratedBy
• wasAssociatedTo, wasAttributedTo
• wasDerivedFrom
• describedBy
• ActivityDescription
• UsageDescription
• GenerationDescription
• ParameterDescription
• Parameter

Examples of queries available with this prototype implementation are given in the attached document ProvQuerytest-3store.pdf

The list shows how the data provider can take benefit of the encoded provenance to re-organise and optimise the image data base. It shows the various queries an astronomer can build in order to appreciate data quality from the recording of Activities, applied methods, parameter values, dataset dependency, typically the link to a progenitor Entity implemented using the wasDerivedFrom relation.

MuseWise Provenance: Implementation of ProvSAP (O. Streicher)

MUSE is an integral field spectrograph installed at the Very Large Telescope (VLT) of the European Southern Observatory (ESO). It consists of 24 spectrographs, providing a 1x1arcmin FOV (7.5" in Narrow Field Mode) with 300x300 pixel. For each pixel, a spectrum covering the range 465-930nm is provided. MuseWise is the data reduction framework that is used within the MUSE collaboration. It is built on the Astro-WISE system, which has been extended to support 3D spectroscopic data and integrates data reduction, provenance tracking, quality control and data analysis. The MUSE pipeline is very flexible and offers a variety of options and alternative data flows to fit to the different instrument modi and scientific requirements.

Rave provenance data database access (K. Riebe)

Implementations Validators

There is no validator for the model as such. Validation tools should be applied to specific implementations of the model.

RFC Review Period: 2019 July 23 - 2019 September 03

Comments from WG members

TCG Review Period: TCG_start_date - TCG_end_date

TBC

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->