IVOA Data Model Working Group

Introduction

The role of the Data Modeling group is to provide a framework for the description of metadata attached to observed or simulated data.
The activity of the Data Model WG activity focuses on logical relationships between these metadata, examines how an astronomer wants to retrieve, process and interpret astronomical data, and provides an architecture to handle them. What is defined in this WG can then be re-used in the protocols defined by the DAL WG or in VO aware applications .

Participation

Membership of this working group is informal as people and projects come and go. To join see our page on How to participate

Progress Meetings

The DM working group conducts, with varying levels of regularity, running meetings to discuss the progress of ongoing work and organize our efforts.
The DM Running Meeting Page is available to:

• Find day/time of the next scheduled meeting
• Add discussion items for addition to an upcoming meeting agenda
• Provide an access point to meeting notes.

The list is currently rather incomplete, but should improve going forward.

WG Roadmaps

Each group updates a roadmap of the goals and priorities for the upcoming semester. These are consolidated and organized at the TCG level:

• See the TCG roadmap

Mailing-list

dm@ivoa.net

• info - subscribe/unsubscribe/edit: listinfo form
• Working group discussions: archive

Standards and Documents

Here is a graphic illustrating the dependencies between data models used in the IVOA framework.

Standards repository

The IVOA Documents and Standards repository is the official resource for all current standards and release candidates that are ready for general review.

Development repository

The IVOA has been migrating the development work for standards to Git repositories. Prior to this, most development work was being managed by a Volute repository.
The following table lists all current data model documents along with references to their development repository and/or project twiki page(s).
These can be useful for submitting issues, participating in discussion, and directly engaging in project development.

Data Model	Version	State	Development Repository	Project Twiki Page
Characterisation	1.13	REC	None
Coordinates	1.0	REC	git	twiki
Cube	1.0	WD	Volute, git (TBD)
DatasetMetadata	1.0	WD	git
MANGO	1.0	WD	git	twiki
Measurements	1.0	REC	git	twiki
MIVOT	1.0	REC	git
ObsCore	1.1	REC	None	twiki, Errata
ObsCore - radio extension	1.0	PR	git
ObsCore - time extension	1.0	WD	git
ObsLocTap	1.0	REC	git
Photometry	1.1	REC	git	twiki
Provenance	1.0	REC	git	twiki
Simulation	1.0	REC	None
Spectral	1.2	REC	git
Spectral Lines	1.0	REC	None
	2.0	RFC (cancelled)	Volute
STC	1.33	REC	None
Transform	1.0	WD	git	twiki
CAOM	2.5	WD	cadc-git
VO-DML	1.0	REC	git	VO-DML tools

Implementation Requirements for Data Models

The "IVOA Document Standards" standard has a broad outline of the implementation requirements for IVOA standards. These requirements fit best into the higher level standards for applications and protocols than for data models themselves. At the Oct 2017 interop in Trieste, the following implementation requirements for Data Model Standards was agreed upon, which allow the models to be vetted against their requirements and use cases, without needing full science use cases to be implemented.

• VO-DML models must validate against schema
• Serializations which touch each entity of the model. These serializations may be 'fake' (ie: not based on actual data files), and are to be produced by the modeler as unit tests/examples.
• Real world serializations covering use cases, produced by others following the model, in a mutually agreed upon format.
• Software which interprets these serializations and demonstrates proper interpretation of the content.

General presentations on the DM modeling effort

Here are general presentations about the data modeling effort in the VO.

• DataModels in the Virtual Observatory Framework

Projects

Active Projects

• Source Data Model:
  • Source and Catalog Requirement survey
  • Source DM and Workshop 2021: see here
  • Project migrates to MANGO data model - providing a generic framework which applies to this project.
• Multi-Dimensional Data Model:
  • The project encompasses the description of Multi-Dimensional data products, from pixelated Images to n-dimensional hypercubes
  • The concept is that this single N-Dimensional Cube model should form the framework from which specialized data products can be defined.
  • As such: it forms the basis for supporting these products-of-interest in the community
    • Spectra (SpectralDM v2.0), including Eschelle spectra
    • TimeSeries
    • Event Lists
    • Pixellated images ( ImageDM v1.0)
  • The Cube model has been demonstrated to satisfy the requirements of these products, but lacks the implemented science oriented usage threads needed to gain general support.
    • We are working closely with the formation of a High Energy Interest Group, which we hope will bring resources and motivation to help see this project to fruition.
• VO-DML 1.1

Proposed Projects

• IVOA Note for FITS serializations
  • discussion started as question about lightcurve data model, but was concluding with a proposal to develop an IVOA Note with ‘preferred’ serializations for various products in FITS format.
  • see DM mail thread
• Characterisation v2.0
  • Project to create vo-dml compliant version of the model.
  • Currently idle.. pending resource availability
• Proposal Data Model
  • Data model for making Observation Proposals. It is being created as project within The Opticon RadioNet Pilot (https://www.orp-h2020.eu)
  • see git repository
• STC-2.0:
  • Project to create vo-dml compliant version of STC
  • The project has been broken up, with separate models covering portions of the domain space covered by STC-1
    • Coords, Measurements, Transforms - completed or in progress
    • Regions, Intervals, ?? - TBD
• VODML-Mapping:
  • Project to develop an annotation syntax mapping VOTable serializations to VO-DML compliant models.
  • The initial syntax proposal was rejected;
  • An alternative syntax was developed as MIVOT

Topic Pages

• UTYPEs discussion: Closed with decision to adopt VO-DML
• Mapping hands-on session page (May 2018)
• High Energy physics data modeled & serialised with VO tools page
• Running meeting AstrPyVO MIVOT page
• Obscore 1.1-Next wiki page

Archival Project Reference Information

RFC pages for our standards.

Project	Version	RFC comments
Characterization	1.13	RFC
Coordinates	1.0	RFC-1, RFC-2
Measurements	1.0	RFC-1, RFC-2
ObsCore	1.0	RFC
	1.1	RFC
ObsLocTAP	1.0	RFC
Photometry	1.0	RFC
	1.1	RFC
Provenance	1.0	RFC-1, RFC-2
Simulation	1.0	RFC
Spectrum	1.0	RFC
	2.0	RFC-1, RFC-2 canceled
	1.2	RFC
Spectral Lines	2.0	RFC
VO-DML	1.0	RFC

Older IVOA Drafts and Related Documents

• Quantity model current drafts (.ps, .pdf)
• Quantity model serialization examples (.html)
• Observation model current drafts (.ps, .pdf)
• 2004 Linear STC: linear (command line) implementation of STC (v1.2).
• 2002 IDHA Data Model for Astronomical Images: modeling of the acquisition process and data products
• 2005 Characterisation_Note_20051129.pdf: Characterization DM draft Note
• spec96d.pdf: Spectrum DM version 0.96 latest (Victoria)
• spec97d.pdf: Alternative Spectrum DM with STC
• SourceDM_Moscow_2006.pdf: Source Catalog Data Model Moscow Sept 2006
• CharacterisationWDversion1.0.pdf: Characterisation Data Model
• update of CharacterisationIvoadraft-v1.1.pdf: Characterisation Model Working draft version 1.1
• ivoa_char_2d_image_tutorial_1.0.pdf: User's guide for Characterisation DM by Alberto Micol
• CharacterisationImplementationv1.1.pdf: Implementations of the Characterisation DM v1.1

Projects using data models

Virtual Observatory Projects working on Data models and their implementations should list themselves here. * to fill and update***

• ObservatoireStrasbourg
  • SAADA CharDM and ObsCore

• CDS
  1. Aladin portal
  2. CAMEA, an interactive tool for Characterisation metadata publishing
  3. Vizier + Saada: interface for products associated with Vizier sources
• Lerma- Paris
  1. Aspid-SR CharDM

Meetings

Most of our face-to-face activity takes place during IVOA Interoperabiltiy Meetings. Follow the Data Model Page links to locate agendas, presentation slides, and action items.

Interop	Main page	DM pageRole
Shanghai, China (15-19 May 2017)	InterOpMay2017	InterOpMay2017-DM
Trieste, Italy (21-23 Oct 2016)	InterOpOct2016	InteropOct2016DM
Cape Town, Sth Africa (9-13 May 2016)	InterOpMay2016	InterOpMay2016
Sydney, Australia(30/10 - 01/11 2016)	InterOpOct2015	InteropOct2015DM
Sesto, Italy (14-19 Jun 2015)	InterOpJune2015	InterOpJune2015DM
Banff, Canada (10-12 Oct 2014)	InterOpOct2014	InterOpOct2014DM
Madrid, Spain (18-23 May 2014)	InterOpMay2014	InterOpMay2014DM
Heidelberg, Germany (12-17 May 2013)	InterOpMay2013	InterOpMay2013DM
Sao Paulo, Brazil (22-26 Oct 2012)	InterOpOct2012	InterOpOct2012DM
Urbana, USA (21-25 May 2012)	InterOpMay2012	InterOpMay2012DM
Pune, India (17-21 Oct 2011)	InterOpOct2011	InterOpOct2011DM
Naples, Italy (16-20 May 2011)	InterOpMay2011	InterOpMay2011DM
Nara, Japan (7-11 Dec 2010)	InterOpDec2010	InterOpDec2010DataModel
Victoria, Canada (17-21 May 2010)	InterOpMay2010	InterOpMay2010DM
Strasbourg, France(19-23 May 2009)	InterOpMay2009	DMInteropStrasbourg
Garching, Germany(19-23 Nov 2009)	InterOpNov2009	InterOpNov2009DM
Trieste, Italy (19-23 May 2008)	InterOpMay2008	InterOpMay2008DataModels
Cambridge, UK (27-28 Sep 2007)	InterOpSep2007	InterOpSep2007DataModel
Beijing, China (14-18 May 2007)	InterOpMay2007	InterOpMay2007DataModel
Moscow, Russia (18-21 Sep 2006)	InterOpSep2006	InterOpSep2006DM
Victoria, Canada (14-19 May 2006)	InterOpMay2006	InterOpMay2006DataModel
Madrid, Spain (6-7 Oct 2005)	InterOpOct2005	InterOpOct2005DataModel
Kyoto, Japan (16-20 May 2005)	InterOpMay2005	InterOpMay2005DataModel
Pune, India (27-29 Sept 2004)	InterOpSep2004	InterOpSep2004DataModel
Cambridge, Mass. USA (23-28 May 2004)	InterOpMay2004	InterOpMay2004DataModel
Strasbourg (16-17 Oct 2003)	InterOpOct2003	InterOpOct2003Registry
Cambridge, UK (12-16 May 2003)	InterOpMay2003	InterOpMay2003DataModel

Involved People

Name	Country	Organization	Role
Jonathan McDowell	USA	CfA, Boston
Francois Bonnarel	France	CDS Strasbourg
Igor Chilingarian	USA	CFA
Marie Lise Dubernet	France	Paris Observatory
Mireille Louys	France	CDS/ICube Laboratory, University of Strasbourg
Gerard Lemson	USA	JHU
Alberto Micol	Germany	ESO
Nicolas Moreau	France	Paris Observatory
Pedro Osuna	Spain	European Space Agency / ESAC
ArnoldRots	USA	CfA, Boston
Anita Richards	UK	Jodrell Bank
Jesus Salgado	Spain	ESA/ESAC
J.D. Santander-Vela	Germany	ESO / Archive dpt
Arancha Delgado	Germany	ESO / Archive dpt
Petr Skoda	Czech Rep.	Astron Inst. of Acad. Sciences
OmarLaurino	USA	Harvard-Smithsonian
KristinRiebe	Germany	GAVO, AIP, Potsdam (Germany)
MathieuServillat	France	LUTH - Observatoire de Paris	Vice-Chair
Mark Cresitello-Dittmar	USA	CfA, Boston	Chair
Laurent Michel	France	SSC XMM-Newton, Strasbourg
Michele Sanguillon	France	Montpellier
Ole Streicher	Germany	GAVO, AIP, Potsdam

Please put your name here if you wish ...

Other DM Meetings

• ADASS 2021: BoF TAP and the data models TapandTheDMs
• Check the above Interop pages for DM WG sessions
• ESO/Garching, Jan 2004: Data Model Technical Meeting 4 Draft 0.1 of Observation model
• Baltimore, Dec 2003: Data Model Technical Meeting 3 (no minutes yet; see draft 0.2 of Quantity)
• ESO/Garching on Jan. 23/24, 2003: Data Model Sessions minutes (WG creation and planning)
• Cambridge MA, Oct 2002: Data Model Technical Meeting 1 presentations & minutes