Difference: MeasRFC2 (9 vs. 10)

NOTICE : This RFC page replaces RFC#1

Rationale for a second RFC round:

• Many comments have been collected after RFC #1. Some of them just required text improvements but some others implied significant model changes, especially for Coords.
• A global RFC answer has been sent to WG mailing list in February 2020: answer
• These data models should support the description of datasets and DAL responses, by defining fundamental elements which are commonly used in many contexts. The intent is that they be imported as components in these more complex models so that they all build from the same basis, thereby enhancing interoperability.
• They should NOT be expected to fully support any use-cases outside of the described set. For example, they cannot currently support the complex error types found in various Catalogs. These use-cases are to be considered in future updates to the models.
• Measurements cannot be used without Coords because the latest is imported by Measurements but Coords can be used in some other contexts e.g. Transform.
• The nature of the changes in Coords, that impacts Measurements thus, led the TCG to decided on 2020-10-8 a second RFC round for both models.

Summary

Version 1 of STC was developed in 2007, prior to the development and adoption of vo-dml modeling practices. As we progress to the development of vo-dml compliant component models, it is necessary to revisit those models which define core content. Additionally, the scope of the STC-1.0 model is very broad, making a complete implementation and development of validators, very difficult. As such it may be prudent to break the content of STC-1.0 into component models itself, which as a group, cover the scope of the original.

This effort will start from first principles with respect to defining a specific project use-case, from which requirements will be drawn, satisfied by the model, and implemented in the use-case. We will make use of the original model to ensure that the coverage of concepts is complete and that the models will be compatible. However, the form and structure may be quite different. This model will use vo-dml modeling practices, and model elements may be structured differently to more efficiently represent the concepts.

This model covers the description of measured or determined astronomical data, and includes the following concepts:

• The association of the determined ’value’ with corresponding errors. In this model, the ’value’ is given by the various Coordinate types of the Coordinates data model (Rots and Cresitello-Dittmar et al., 2019).
• A description of the Error model.

• RFC2 Document version (IVOA repository)
  • PDF, schema, vodml/xml, html (hosted on volute)
• Post-RFC2 Document version (IVOA repository)
  • PDF, schema, vodml/xml, html
• Development Version: (Git repository)
  • Post RFC2 Tag

Implementation Requirements

(from DM Working group twiki):

The "IVOA Document Standards" standard has a broad outline of the implementation requirements for IVOA standards. These requirements fit best into the higher level standards for applications and protocols than for data models themselves. At the Oct 2017 interop in Trieste, the following implementation requirements for Data Model Standards was agreed upon, which allow the models to be vetted against their requirements and use cases, without needing full science use cases to be implemented.

• VO-DML models must validate against schema
• Serializations which touch each entity of the model. These serializations may be 'fake' (ie: not based on actual data files), and are to be produced by the modeler as unit tests/examples.
• Real world serializations covering use cases, produced by others following the model, in a mutually agreed upon format.
• Software which interprets these serializations and demonstrates proper interpretation of the content

Serializations:

• Example serializations:
  • Annotated VOTables:
    • all model elements as VOTable files annotated to the VODML Mapping Syntax ( WD:20170323), produced by Jovial software package.
      • coordinates model elements: here
        • includes xml, and jovial dsl files
      • measurement model elements: here
        • includes xml, and jovial dsl files
      • transform model elements: here
        • includes xml, and jovial dsl files
  • Various Formats:
    • independent python code, generated example serializations spanning all elements of the models in 4 formats:
      • *.vot: VOTable-1.3 standard syntax
        • Validates using votlint
      • *.avot: VOTable-1.3 annotated with VO-DML/Mapping syntax
        • Validates using xmllint to a VOTable-1.3 schema enhanced with an imported VO-DML mapping syntax schema
      • *.xml: XML format
        • Validates against the model schema
      • *.xxx: An internal DOC format
        • XML/DOM structure representing the instances generated when interpreting the templates.
    • measurement model elements: here
    • coordinates model elements: here
• Usage implementations include annotated serializations of the case data files
• Cube: Example files for Cube ( nD-Image and Sparce Cube ) incorporate Measurement and Coordinate model instances

Software:

A detailed study was performed to determine the compatibility of the Meas/Coords data models to the AstroPy package, a popular Python package with intensive support for Space and Time coordinates.

• The results of this comparison are described in the STC2 wiki page.

• Jovial: A Java toolset that helps build and generate serializations for VODML compliant data models.
  • Version used for DM workshop and example file generation: https://github.com/mcdittmar/jovial
  • Original implementation by Omar Laurino: https://github.com/olaurino/jovial
• Rama: Python package, parses annotation and instantiates instances of model classes. Includes adaptors to AstroPy classes.
  • Version used for DM workshop: https://github.com/mcdittmar/rama
  • Original implementation by Omar Laurino: https://github.com/olaurino/rama
  • Understands 'VODML Mapping syntax' annotations ( WD:20170323)
• ModelInstanceInVOT Code: Python package for processing annotated VOTables
  • Implementation by Laurent Michel: https://github.com/ivoa/modelinstanceinvot-code
  • Understands 'Merged syntax' annotations ( WD:in progress )
  • A notebook show how that code can be used.
• TDIG: Working project of Time Series as Cube.
  • An effort to enhance SPLAT to load/interpret/analyze TimeSeries data using data annotation
    • the tool was enhanced to use new annotations (eg: TIMESYS, UTypes) to identify and interpret the data automatically.
  • Delays in resolving on a standard annotation syntax has hindered progress on this project to fully realize the possibilities. This is a high-priority for upcoming work.
• pyVO: extract_skycoord_from_votable()
  • Demonstrated in Paris this product of the hack-a-thon generates AstroPy SkyCoord instances from VOTables using various elements embedded in the VOTable.
    • Interrogates a VOTable, identifies key information and uses that to automatically generate instances of SkyCoord.
      • UCD: 'pos.eq.ra', 'pos.eq.dec'
      • COOSYS.system: "ICRS", "FK4", "FK5"
      • COOSYS.equinox
    • The COOSYS maps directly to SpaceFrame, and the value of the system
    • The UCD 'pos.eq' maps directly to meas:EquatorialPosition; with 'pos.eq.ra|dec' identifying the corresponding attributes (EquatorialPosition.ra|dec) as coordinates coords:Longitude and coords:Latitude.
    • This illustrates that even with minimal annotation, this sort of automatic discovery/instantiation can take place. With a defined annotation syntax, this utility could be expanded to generate other AstroPyobjects very easily.

Validators

As noted above, the serializations may be validated to various degrees using the corresponding schema

• VOTable-1.3 using votlint: verifies the serialization complies with VOTable syntax
• VOTable-1.3 + VODML: verifies the serialization is properly annotated
• XML using xmllint with model schema: verifies the serialization is a valid instance of the model.
• NOTE: The modeler examples undergo all levels of validation, showing that the VOTable serializations are also valid instances of the model.

Usage

In the period since the close of the RFC2 review, a great deal of effort has been made to illustrate the usability of the Meas/Coords models in the context of real world scenarios. Each have confirmed the usability of the data models, and illustrate how annotating data to models can facilitate interoperability.

These include:

• Data Model Workshop - May 2021
  • This Git repository contains original implementations from all participants.
• DM Case Implementations
  • This Git repository contains a set of use case implementations maintained to the current model suite.
    • each workshop usecase implemented using Jovial and Rama tools.
    • Jupyter notebook illustrates the case thread.
  • Highlights
    • Proper Motion slider
    • Uniform TimeSeries extraction from multiple data providers
• AstroPy Wrapper
  • Using an AstroPy wrapper in the ModelInstanceInVOT code (see Software)
  • This Git repository holds case implementations
    • Meas/Coords model elements are mapped in VOTable
    • parser interprets annotation to generate model instances, and converts them to SkyCoord instances.
    • Threads:
      • Extract positions, parallax and proper motions from ESAC archive; generate 3D plot of source positions
        • using direct Measurements model instances
        • using converted AstroPy SkyCoord instances
      • Identify annotated source positions and reconcile the coordinate frames.
      • Extract observation history of a source from ESAC XMM TAP archive, track source movement over 20 year period.
  • Examples
    • https://github.com/ivoa/modelinstanceinvot-code/tree/merge-syntax/python/examples
  • Notebook
    • https://mybinder.org/v2/gh/ivoa/modelinstanceinvot-code/merge-syntax
• ADASS 2021 BoF - TAP and the Data Models
  • This BoF discussed the possibility and benefits for TAP services to apply on-the-fly annotation of the query responses to serve not only the data, but real model instances.
  • Annotated TAP responses can be consumed by software such as those described blow, to interpret the content in terms of IVOA data models, greatly enhancing the interoperability of manipulating query responses from various services in science threads.
  • Conclusions of the BoF include: "This session and the following discussions 4 highlighted that TAP services can already serve hierarchical data and that serving legacy data with annotations or even Provenance instances is within our reach."
  • Resources
    • ADASS programme overview
    • video recording of BoF
    • twiki page (presentations and notes)
    • proceedings on Astro-ph

Links with Coords

The Measurement model is heavily dependent on the Coordinates model (also in RFC) for its core elements. Information about its relation to the Coordinates model, and how the requirements are distributed can be found on the STC2 page

Comments from the IVOA Community during RFC/TCG review period: 2020-10-26 - 2020-12-07

Comments by Markus Demleitner

The introduction and points (1), (4), (7), (11) from my RFC 1 comments I'd still retain, and I essentially stand by the summary, which I'd update to:

I'd make the model a lot smaller, thus creating space we'll need once we tackle strict errors (i.e., explicit distributions) in earnest one day. We ought to have, perhaps NaiveMeasuremetnt (a pseudo-distribution saying "value is something like an expectation or perhaps a median, and error is something like a first moment or something like that), and perhaps AsymmetricNaiveMeasure. I'm not even convinced we have credible use cases for statError and sysError. And if we want correlations, these should be explicit as relations between, I guess, measurements (or their parameters?).

• The set of Measurement types included in this model are those needed to cover the basic properties of the data cube model ( Space - position and velocity, Time, Polarization, and Generic (for most other physical properties). The error model covers the basic error patterns. We will, no doubt see more complex error types in the future, so the model is designed to be extensible without necessarily effecting the other elements of the model (ie: its easy to do a minor version update to accommodate new content). -- MarkCresitelloDittmar - 2022-02-15

• The DM workshop cases include examples where Symmetrical, Bounds, and Ellipse are used. The TimeSeries case in particular, plots data with their associated errors. -- MarkCresitelloDittmar - 2022-02-15

• This text was specifically requested to be added to the model description in the PR1 comments. -- MarkCresitelloDittmar - 2022-02-15

• We have added a 'ucd' element to the Measure class to allow users to specify the physical nature of the GenericMeasure. However, this does not negate the need/benefits of having specific classes for the Time, Position, etc. types which have additional associated metadata which are not available to the Generic type. It is also worth noting that any UCD included in the VOTable element would, generally, not be appropriate for usage as the GenericMeasure type specification since these often have more complicated expressions which relate to the role they play, or the frame. -- MarkCresitelloDittmar - 2022-02-15

• Modeling the physics in this model is the feature that will most improve our ability to interoperate. Without this, a Cube is just a series of Measured data. With it, we can see it has Time, 2 Positions, some form of Energy, etc. As stated above, the UCD associated with the FIELD is typically not going to be appropriate for the Measure. For example, a magnitude field may have 'phot.mag;em.opt.V', which includes information about the associated band. For the Measure, we simply want 'phot.mag' to identify the nature of the measure as a magnitude. The band information is found in associated model elements. -- MarkCresitelloDittmar - 2022-02-15

Comments from TCG member during the RFC/TCG Review Period: 2022-02-24 - 2022-04-10

WG chairs or vice chairs must read the Document, provide comments if any (including on topics not directly linked to the Group matters) or indicate that they have no comment.

IG chairs or vice chairs are also encouraged to do the same, althought their inputs are not compulsory.

TCG Chair & Vice Chair

Applications Working Group

Data Access Layer Working Group

Data Model Working Group

Grid & Web Services Working Group

Registry Working Group

Semantics Working Group

Education Interest Group

Knowledge Discovery Interest Group

Radioastronomy Interest Group

Solar System Interest Group

Points of confusion for me:

• Section 2.1.1, "Pixelated Image Cube": I am not at all clear on what the term "pixel domain" means. In my mind, pixels are "image cells".

• Section 2.1.1 and subsequently: What does it mean to be "dimensionally compatible"? Are Angstroms "dimensionally compatible" with Astronomical Units, for example, because they both measure length? Are percentages "dimensionally compatible" with everything, or nothing?

• Section 1.1, first bullet: "instramental" should be "instrumental".
• Seciton 1.1, fourth bullet:"collections properties" should probably be "collections of properties"; "that source" should be "those sources" (grammatically - if the intention of the sentence is different it needs rewriting).
• Section 1.1, paragraph following bullets: "and different locations" should be "and from different locations" or "and at different locations".
• Section 5.1, second paragraph: "teh" should be "the"; "shpes" should be "shapes".

Theory Interest Group

Time Domain Interest Group

Operations

Standards and Processes Committee

TCG Vote : Vote_start_date - Vote_end_date

If you have minor comments (typos) on the last version of the document please indicate it in the Comments column of the table and post them in the TCG comments section above with the date.

<!--</p> <ul> <li> <ul> <li> Set ALLOWTOPICRENAME =<span class="WYSIWYG_PROTECTED"> TWikiAdminGroup</span> </li> </ul></li> </ul>-->

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->