TWiki
>
IVOA Web
>
IvoaDataModel
>
DataAnnotation
(revision 25) (raw view)
Edit
Attach
---+ Model Annotation in VOTables (MIVOT) %TOC{depth="2"}% ---++ MIVOT at a Glance Model Instances in VOTables (MIVOT) defines a syntax to map VOTable data to any model serizalized in VODML. The annotation operates as a bridge between the data and the model. It associates the column/param metadata from the VOTable to the data model elements (class, attributes, types, etc.) of a standardized IVOA data model, expressed in the Virtual Observatory Data Modeling Language (here after VO-DML). It also brings up VOTable data or metadata that were possibly missing in the table metadata. The data model elements are grouped in an independent annotation block complying with the MIVOT XML syntax. This annotation block is added as an extra resource element at the top of the result resource of the query response. The data annotation be operated in the context of any VO protocol. The MIVOT syntax allows to describe a data structure as a hierarchy of classes. It is also able to represent relations and composition between them. It can also build up data model objects by aggregating instances from different tables of the VOTable. Missing metadata can also be provided using MIVOT, for instance by completing coordinate system description, or by providing curation tracing. The annotation block is the VODML transcription of data model classes, with their attributes, types, and relations. It maps the VOTable data on the relevant model classes. It is made of re-usable bricks that facilitate the developement of tools on both client and server sides. The adopted design does not alter the original VOTable content. ---++ History The first proposal for a solution for mapping data on models was based on GROUPS and UTypes. This approach suffered some flaws and the IVOA decided to promote a mapping syntax closer to VO-DML, a new standard for model serializations in XML that became a REC in 2016. The baseline of this approach consists in inserting into the VOTable to be annotated, an XML block that is faithful to the model structure and that acts as a bridge between the model leaves and the actual data. * A first [[https://wiki.ivoa.net/twiki/bin/view/IVOA/VODML-Mapping][proposal]] has been published as a VO working draft in 2018. * Users were able to test it on their data during a hands-on [[https://wiki.ivoa.net/twiki/bin/view/IVOA/InterOpMay2018VODML][session]] in May 2018. The conclusions were that the approach was promising but we were missing at that time models applicable to real data. * This syntax proposal has also been tested on some [[https://olaurino.gitlab.io/ivoa-dm-examples/][model examples]] (STC2, Cube, TimeSeries and Tesselation...). The major objections were 2 folds 1) the verbosity should be reduced 2) the mapping of some ORM features had to be simplified. * After this, the working group ran a wide [[https://wiki.ivoa.net/twiki/bin/view/IVOA/SourceCatalogs][survey]] to gather requirements for annotating source data. A simplified mapping syntax (namely vodml-lite) that was driven by the need of improving both readability and compactness of the annotations, has been tested in this context as well as along with the work of the time domain interest group. * Finally, at the end of 2020, the DM-WG has been mandated by the TCG to run a virtual [[https://wiki.ivoa.net/twiki/bin/view/IVOA/DMWorkshop][workshop]] to clearly define the model usage policy in the VO and to issue a common syntax for the annotations. The present proposal, one of the major outcomes of that workshop, results from a lot of work put by many people. It is the product of an evolution which combines the best features of both proposals to create a highly effective solution. ---++ Standard and Tools The draft is managed following the !GitHub workflow. The proposal is hosted at [[https://github.com/ivoa-std/ModelInstanceInVot][https://github.com/ivoa-std/ModelInstanceInVot]] * The PDF, updated after each merge on the main branch, can be seen as a release [[https://github.com/ivoa-std/ModelInstanceInVot/releases/download/auto-pdf-preview/mivot-draft.pdf][asset]] . * This draft comes with another repository ([[https://github.com/ivoa/modelinstanceinvot-code"][modelinstanceinvot-code]] ) that gathers codes that are being developped to exercise the mapping syntax on real data. * This project contains a few Jupyter notebooks that can be launched [[https://mybinder.org/v2/gh/ivoa/modelinstanceinvot-code/package][online]] * This code is meant to be integrated in PyVO ---++ Repository The GitHub project [[https://github.com/ivoa-std/ModelInstanceInVot][repository]] contains the following section: * *Document sources:* one TEX file per section. * *XML schema*: XSD1.1 [[https://github.com/ivoa-std/ModelInstanceInVot/raw/master/schema/xsd/mivot-v1.0.xsd][schema file]] that is currently used for the validation. Using XSD1.1 has been made necessary to state the syntactic rules to be applied to elements depending on the context in which they are used. * *Test suite:* Huge Python test bench validating all indvidual MIVOT features. The tests check both valid features and forbidden patterns along with the reasons for which they are rejected. We encourage people interested in practicing the mapping to get snippet examples telling what to do and what not to do. ---++ Reference Interoperable Implementations Data annotation is one of the steps of a broader workflow that starts from raw data and ends-up with the science code. It is difficult to figure out what is a reference implemenation for the mapping without service able to provide annotated data or clients able to process them either. We overcame this difficulty by emulating the missing links. Our implementations work mainly with prototype services or with datasets annotated be hand. Client code shows that the data set content can be interpreted only by reading the annotations. * <strong>Code sample: </strong>the [[https://github.com/ivoa/modelinstanceinvot-code"][modelinstanceinvot-code]] package provide a lot of code able to process annotated VOtables. * This CODE is based on a model viewer object able to provide different serialisations of the mapped objects. It operates the followings steps: * Mapping block extraction * Reference resolution (model leaves are set with table data). At this point the user gets an XML serialization of the model instance that can be used in different ways: * Extracting model components with XPATH queries * Converting the XML serialization in JSON * Building Python instances of datamodel place-holder classes (see<em> client/class_wrapper</em>) * Building Astropy objects (see _client/class_wrapper/astropy_wrapper_) * WATCHOUT: * This code is being developed to exercice the annotation processing, it likely suffers some weakness though. * The mapped models are not necessarily VO standards. They can be prototype models (MANGO, SparseCube) or pre-PR version (MCT, PhotDM); but they are all VO-DML serializable. * JUPYTER notebooks ( _./jupyter_). These are notebooks based on a data sample annotated by hand and located in _./mivot_code/examples/data_.<br />The notebooks can be run with Binder (https://mybinder.org/v2/gh/ivoa/modelinstanceinvot-code/package) * _gaia_3D.ipynb_: 3D position plot of a star cluster (GAIA) based on MCT classes * _gaia_3D_astropy.ipynb_: 3D position plot of a star cluster (GAIA) based on _astropy.SkyCoord_ * _moving_source.ipynb_: Plot the positions of an XMM source along of 20 years of observations * _photdm_impl.ipynb_: Plot SEDs from a table of cross-matched XMM sources * UNIT TESTS: There are many unit tests checking that the parser is able to process all of the mapping features. * The MIVOT snippets are in _tests/data/input,_ the output references are in <em>tests/data/output <br /></em> * Pay attention to the test #14 which extracts a SparseCube instance from a VOTable including all mapping features for the record. <br />This VOTable has been written on purpose by Mark CD as a DM workshop usecase (2021). * EXAMPLES: ( _mivot_code/examples_) some standalone scripts processing each one specific VOTable. * The scripts named _example.1.xtapdb.*_ have the best documentaion in both python code and XML files * LAUNCHERS: ( _mivot_code/launchers_) * _xcatb_client.py_: unique active package launcher. Sends a user ADQL query on https://saada.unistra.fr/fov and tell you about the mapped quantities. * <strong>PhotDM </strong>Photometric calibrations serialized with MIVOT (M. Louys 2022) * Various photometric calibrations are available [[https://github.com/ivoa-std/PhotDM/tree/master/serializations][here]] * No processing code available, just look at the VOTables.<strong><br /></strong> * *Field of View*: * Instrument field of views serialized with MIVOT and consumed by Aladin (Lite and Desktop) (UTBM Intern Clément Nogueira, 2022). * Graphical editor for the FoV shapes. * Allows to download MIVOT serializations of the drawn !FoVs * A FITS image can be plotted in the background as a drawing template. * Can be tested [[https://saada.unistra.fr/fov][here]] * Follow the landing page instructions * Draw a FoV with the editor * Save it on disk * Look at the MIVOT file * Upload it in AladinLite * *XTapDB* <span style="background-color: transparent;"> TAP service mapping on the fly XMM data on MANGO (Unistra intern I. Errami, 2022)</span> * <span style="background-color: transparent;">This service extends a VOLLT instance. It contains 2 tables that can be annotated on the fly (catalogueentry and mergedentry). Both contain a subset of the XMM-Newton catalogue (4xmmdr11). <br /></span> * <span style="background-color: transparent;">The service can be explored with [[http://saada.unistra.fr/taphandle?url=https%3A//xcatdb.unistra.fr/xtapdb/][TapHandle]]<br /></span> * <span style="background-color: transparent;">This is a work in progress; the mapping is partially done and it does not match exactly the MANGO data model classes (still in design phase)</span> * <span style="background-color: transparent;">Getting annotated VOTables: <br /></span> * <span style="background-color: transparent;">Service base URL: https://xcatdb.unistra.fr/xtapdb</span> * <span style="background-color: transparent;">Use f<em>ormat=application/mango</em> to tell the server to do annotions<br /></span> * <span style="background-color: transparent;">e.g. Curl on this [[https://xcatdb.unistra.fr/xtapdb/sync?REQUEST=doQuery&FORMAT=application%2Fmango&PHASE=RUN&LANG=ADQL&QUERY=select+%2A+from+catalogueentry][https://xcatdb.unistra.fr/xtapdb/sync?REQUEST=doQuery&FORMAT=application%2Fmango&PHASE=RUN&LANG=ADQL&QUERY=select+%2A+from+catalogueentry]]</span> * <span style="background-color: transparent;"> </span> * Annotation processing * Install the Python package _modelinstanceinvot-code_ * run _xcatdb-client 'select * from catalogueentry'_ ---++ Implementations Validators [[https://github.com/ivoa/mivot-validator/][mivot-validator]] is a Python validator for VOTables annotated with MIVOT. The validation process is 2 steps; * VOTable validation (against 1.3) * MIVOT validation Both must succeed for the files to be considered as valid. The validator can process either individual files or directory contents (no recursivity) ---++ Annotations and Astropy The TCG mandate about the mapping syntax also included the commitment of providing tools that could help both data provider and client developers to assess the impact of working with annotated data. * Some work has been put on the [[https://github.com/ivoa/modelinstanceinvot-code/wiki/Data-Annotations-and-AstroPyVO][design]] for an integration of the mapping processing into the AstroPy/PyVO ecosystem. --- --- ---++ Comments from the IVOA Community during RFC/TCG review period: 2022-09-12 - 2022-10-24 The comments from the TCG members during the RFC/TCG review should be included in the next section. In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your Wiki Name so that authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment. Additional discussion about any of the comments or responses can be conducted on the WG mailing list. However, please be sure to enter your initial comments here for full consideration in any future revisions of this document Comments added as !GitHub isssues must also be reported here. * Sample comment by TWiki.WikiName * Response (by TWiki.WikiName) *%RED%Using !GitHub Issues:%ENDCOLOR%* You can use !GitHub issues to comment. In this case, just add here a short label with the issue URL. The disscussion can take place on !GitHub and the final editor answer will be summarized with a reference to the corresponding MR if any. --- --- ---++ Comments from TCG member during the RFC/TCG Review Period: 2022-09-16 - 2022-10-31 WG chairs or vice chairs must read the Document, provide comments if any (including on topics not directly linked to the Group matters) or indicate that they have no comment. IG chairs or vice chairs are also encouraged to do the same, althought their inputs are not compulsory. ---+++ TCG Chair & Vice Chair ---+++ [[IvoaApplications][Applications Working Group]] ---+++ [[IvoaDAL][Data Access Layer Working Group]] ---+++ [[IvoaDataModel][Data Model Working Group]] ---+++ [[IvoaGridAndWebServices][Grid & Web Services Working Group]] ---+++ [[IvoaResReg][Registry Working Group]] ---+++ [[IvoaSemantics][Semantics Working Group]] ---+++ [[IvoaDCP][Data Curation & Preservation Interest Group]] ---+++ [[IvoaEducation][Education Interest Group]] ---+++ [[IvoaKDD][Knowledge Discovery Interest Group]] ---+++ [[IvoaOps][Operations Interest Group]] Overall the text looks carefully written. I haven't yet tried using this proposal myself so I don't have a feel for how it works in practice, but implementations and validators seem to be in place. One suggestion: The examples in the text for each feature are welcome, but it would be nice as well to have a single complete example of a VOTable marked up using MIVOT, alongside an explanation of what the markup is doing or how it could be used. Because of the size of the standard it wouldn't make sense to show off all the features in such an example, I have in mind something fairly simple. I appreciate that there are multiple examples in the github repository, but it's not obvious to the reader where to start, and not all those examples correspond to the published version. I leave it to the authors to decide whether this suggestion is a good idea or practicable. A few minor corrections below (version 2022-09-16): * Section 4.6: listings 10 and 11 are identical. * Section 4.9: broken Appendix reference _"See more examples in Appendix ??"_ . Same thing in (at least) Listing 10 and Section 4.10 - missing =\label{appen_*}= definitions? * Listing 9: outermost closing tag is missing a "/"; example reads "<code><GLOBALS>...<GLOBALS></code>" rather than "<code><GLOBALS>...</GLOBALS></code>". Same thing in listings 10, (11), 12. * Table 22: _"he host"_ -> _"the host"_ * Section 4.12: there appear to be a couple of full stops missing. ---+++ [[IvoaRadio][Radio Astronomy Interest Group]] ---+++ [[IvoaSS][Solar System Interest Group]] ---+++ [[IvoaTheory][Theory Interest Group]] ---+++ [[IvoaVOEvent][Time Domain Interest Group]] ---+++ [[IvoaStdsDocsProc][Standards and Processes Committee]] --- ---++ TCG Vote : 2022-09-16 - 2022-10-31 If you have minor comments (typos) on the last version of the document please indicate it in the Comments column of the table and post them in the TCG comments section above with the date. | Group | Yes | No | Abstain | Comments | | TCG | | | | | | Apps | | | | | | DAL | | | | | | DM | | | | | | GWS | | | | | | Registry | | | | | | Semantics | | | | | | DCP | | | | | | Edu | | | | | | KDIG | | | | | | Ops | * | | | | | Radio | | | | | | SSIG | | | | | | Theory | | | | | | TD | | | | | | <nop>StdProc | | | | | <!-- * Set ALLOWTOPICRENAME = IVOA.TWikiAdminGroup -->
Edit
|
Attach
|
Watch
|
P
rint version
|
H
istory
:
r60
|
r27
<
r26
<
r25
<
r24
|
B
acklinks
|
V
iew topic
|
Raw edit
|
More topic actions...
Topic revision: r25 - 2022-09-26
-
MarkTaylor
IVOA
Log in
or
Register
IVOA.net
Wiki Home
WebChanges
WebTopicList
WebStatistics
Twiki Meta & Help
IVOA
Know
Main
Sandbox
TWiki
TWiki intro
TWiki tutorial
User registration
Notify me
Working Groups
Applications
Data Access Layer
Data Model
Grid & Web Services
Registry
Semantics
Interest Groups
Data Curation
Education
Knowledge Discovery
Operations
Radio Astronomy
Solar System
Theory
Time Domain
Committees
Stds&Procs
www.ivoa.net
Documents
Events
Members
XML Schema
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback