TWiki> IVOA Web>IvoaDataModel>SourceCatalogs>Dm2021 (revision 12)EditAttach
During Fall 2020 Interop we had a session, requested by Markus Demleitner, that aimed at opening a wide discussion on both Measure and Coordinate models and possibly at refining their usages.

This came after a long discussion on the mailing list, related to the necessity of keeping or not a tight dependancy between Measurements and Coordinates. It has also been suggested that product models (e.g. CubeDM ) could be replaced with domain related components.

Many people have participated to the discussion with many positive contributions.
Although the goal of this message is not to report on that meeting, we do need to note a few points:

  • There is a clear demand for using models
  • There is a clear demand for the models to be simple and modular.
  • The distinction between what is modelling and what is serialisation is unclear.
The need for a concrete solution for annotating catalog data, that resurfaced last week, has been tackled by the WG for a couple of years along with the development of Meas/Coordinates models.
  • 2018:Discussions have been engaged on this topic after a hands-on session in Victoria.
  • 2019: We asked data providers and client developers to present their needs in a dedicated session at Paris
After these 2 surveys the scope of the model usage has been exposed in Virtual May 2020 and we have been working on MANGO (a model for source data with some MC adaptations) that has been sketched in Groningen and in the virtual-May meetings and then presented in November 2020.

To continue this action we have been organising a virtual workshop (initially proposed by Tom to the TCG) with the hope of getting a widely accepted framework.

The idea is not to start afresh but to give to the participants the opportunity of matching their needs with the current proposals.

To get a clear picture of what has to be done, we proposed to work with a variety concrete use cases, mostly taken out of the SourceDM proposal ( 2016) and data samples provided by interested people.

Timeline

  • 2021 February 18: Public release of the use-case repository.
  • 2021 February 24: Running meeting to present the process
  • 2021 March: People can look and contribute at the repository
  • 2021 May: Virtual Workshop - Pre-Interop meeting & 3 sessions during Interop.

Running meeting(25/02/2021)

  • Etherpad [[https://yopad.eu/p/dm_virtual_meeeting_#2-365days][link]
SESSION #1


DM meeting - Workshop preparation - collective remarks and minutes
Participants 15

  • Mirreille Louys, Tom Donaldson Laurent Michel, Mark CD, Jesus Salgado, Gilles Landais, Francoise Genova, Kai Polsterer, Marco Molinaro, Francois Bonnarel , FX Pineau, Carlos Rodrigo, Sara Bertocco, Vincenzo Galluzzi, Markus Demleitner
The purpose of this meeting is to prepare the DM workshop forseen in April 2021:
Workshop goals
LM presents the goal of the meeting. Preparation of meeting on data models on VO and how
to improve consensus on data modeling generation.
Some DMs work well but disagreements on how to match models with scientific quantities.
Proposal to solve mapping problems is MANGO, developed by LM
Also, proposal from Tom D. to organize a workshop on how to solve this gap
Proposed process to prepare the workshop
Gather a list of concrete use cases and possible solutions
Gather contributions
  • propose new use cases
  • comment the use cases
  • propose a solution
  • comment proposed solutions
The dm-usecases repository
These use cases is created compiling some VO historical use cases that require data modeling
Two folders per use case, raw_data with an annotated xml and another folder with a mango proposal
fulfilling the use case
Time line for contribution to the use cases repository is during March to allow the pull requests merging
of all the contributions
poll created on use cases portal:
https://github.com/ivoa/dm-usecases/wiki/poll
FB: Procedure is to create a branch and ask for a pull request to be merged by LM? yes, either for commenting
existing or creating a new use case
TM: Lack of consensus are, in my view, due to the level of implementation of the solutions
LM: I contacted Mark T. about using data models in Topcat and the problem comes from heteregeneous data
modeling annotations in VOTables
FG: It is difficult to decide whether to implement before having an idea of the work required to implement
This means that the point has to be central to the workshop using the use cases
MD: Will implement. If it works for VizieR, it will work for him.
TM: Use case for proper motion representation?
LM: Yes, in astrometry
Mark CD: Orbital models not present, a new use case is needed
MD: Not clear if we need models for quantities as this could be solved just with UCDs
FB: UCDs are standalone, DM informs about quantities.
MD: physics: UCDs, use DM to annotate
GL: Mango enables to link two different quantities together, no need to have a specific model for each case.
Workshop conduct
virtual / in person ?
Proposal and decision for a Date
  • End of April ?
SESSION #2


DM meeting - Workshop preparation - collective remarks and minutes - Second session
Participants 8: Laurent Michel, Mireille Louys, Gregory Dubois-Felsmann, Jesus Salgado, Francois Bonnarel, Gerard Lemson, Tom Donaldson, Mark Allen, Harry Enke
LM presents objective of propossed workshop and historical background
poll created on use cases portal:
https://github.com/ivoa/dm-usecases/wiki/poll
Use cases page proposed:
https://github.com/ivoa/dm-usecases
GL: Is there a DM behind all the MANGO annotations?
LM: totally free
Changes to be done in another branch (to be merged with trunk after review)
If a specific model is proposed , a new directory should be inserted
Same if a new mapping strategy is proposed

Pre-interop meeting(17/05/2021)

  • Etherpad [[https://pad.unistra.fr/p/pre-interop][link]
Participants 17
Gerard Lemson, Janet Evans, Mark Cresitello-Ditmar, Tom Donaldson, Laurent Michel, Mark Allen, Francous Bonnarel, Ian Evans, Jean-Mark, Jesus Salgado, Gilles Landais, Jean-Michel Glorian
Markus Demleitner, Vicenzo Galluzzi, Stefania Amodeo, Hendrik Heinl, Eric Buchlin
This session will allow interested participants to evaluate our proposals with regard to the services they are operating or may need to build in the future.
The meeting is OPEN to everyone.
One thing we are trying to do though is target scientists/implementors who may consider volunteering to relate their experiences at the Interop Workshop.
We envision presenting feedback on the use-case work completed so far, or presenting a vision of where a more mature version of the use case could end up someday.
Agenda
Proposals
François Bonnarel (8+2’)
Minutes:
Introduction by LM:
- Objective on how to use data models into VO and technical ways to share and to implement interoperability using DMs
- Use cases:
Display
Plot
Filtering
X-Match
Data Sharing
...
  • - Description of elements of VOTable to identify if other elements are needed for data model serialization
  • - Quantities are well defined but tole and the way these quantitites are nested are poor. How quantities relate each other is very poor defined
  • - Proposal to add a mapping block without breaking existing VOTable blocks
  • - Most consensus on most of the data models issues but still not defined the mapping syntax
  • - Divided in three levels: Model fragments, Model Components and Integrated Models
  • - Main topic would to define which level should be mapped
  • - Presentation of the use cases https://github.com/ivoa/dm-usecases/wiki
VODML Overview (Gerard Lemson):
  • - Historical review of mapping DM problem and evolution from utypes to a complex VO/DML representation
Proposals
Mark Cresitello Dittmar
Packages: Jovial, Rama (both specific for VO DMs developed mainly by Omar Laurino) and Astropy, MatPlotLib
GL: Code to use annotation
MCD: Jovial
Markus Demleitner
  • Data model strategy
  • TD: About astropy implementation fixes. Which fixes?
  • MD: My astropy code
Laurent Michel
  • Presentation on MANGO
  • GL: MANGO and model instances mapping should go together?
  • LM: MANGO API relay on the syntax but mapping and syntax are indendent
  • TD: Merging on pyVO. What funtionality could be used?
  • LM: For example get quantity parsing MANGO syntax, or bind together different quantities
F. Bonnarel
  • Mapping Mango strucuture on top of Gaia and ZTF tables
Wrap-up
  • 26/05 15h UTC Mapping strategy
  • 27/05 15h UTC Model review
  • 28/05 15h UTC VOTable mapping syntax
Q/A
  • MD: Client writers are needed e.g. for measurements data model adoption. This simplified measurements would be just removing some boxes away
  • MCD: Use cases have started very simple that have been able to progress to complex use cases like Gaia. Another point is to show that different implementation are able to arise to the same result that is totally solved yet
  • MD: Posibility to go to DPAC to serialize time series in line with a proper VO format
  • FB: For new projects you can go to them to modify the serializations but the modification of legacy data is a lot more complex
  • GillesL: At least in ViZieR if photometry data is added, it is very important to also add the provenance
  • GL: Probably better to focus in the more popular datasets/missions. Why still using VOTable for annotating (VOTables are there only because it could be related to a service response)
  • Slides to be uploaded to https://wiki.ivoa.net/twiki/bin/view/IVOA/Dm2021
Chat dump
  • 18:59:54 From Jean-Marc : Thanks to the speakers for the very nice introduction to VODML and the problem at hand. Unfortunately I have to leave now. I hope to be able to attend the workshop itself. Best.
  • 19:01:46 From Tom Donaldson : Thanks, Mark. That was a very helpful review of that work. Very intuitive flow.
  • 19:42:01 From Markus : That´s why you disentangle things...
  • 20:05:54 From Markus : Client writers!
  • 20:06:04 From Markus : Who´ll hijack them?
  • 20:06:23 From Mark CD : +1
  • 20:07:38 From Markus : Right.
  • 20:07:54 From Tom Donaldson : More important is skeptical client writers. But that’s a harder sell because they don’t see the point in expending effort.
  • 20:08:12 From Markus : Sell them error bars first.
  • 20:08:19 From Markus : automatic error bars.
  • 20:08:26 From Mark CD : Is that interesting enough?
  • 20:12:03 From Tom Donaldson : Along with Bambi eyeing, we may need hand holding. Even the simplest examples are intimidating without some initial help.
  • 20:12:42 From Markus : very true.
  • 20:19:26 From Tom Donaldson : I need to sign off at 25 past the hour. Thanks to everyone for the presentations, discussion points, and all the work behind them.
  • 20:22:01 From Markus : Again, I´m not saying we can ignore the legacy, I´m just saying let´s not burden our first version with it.
  • 20:24:12 From Ian Evans : I have to sign off to go to another meeting. Thanks everyone.
  • 20:25:38 From Markus : Oh: Do we collect slides/notes somewhere?
  • 20:27:59 From Laurent Michel : Can you upload you slides on the Wiki https://wiki.ivoa.net/twiki/bin/view/IVOA/Dm2021
Links

Interop Sessions

Session #1 Wednesday May 26 15h UTC Model users have the floor floor
Participants 53
Laurent Michel Introduction
- Explanation of data model benefits for interoperability
- Presentation of data model use cases repository created by the team with examples of mapping techniques
- Presentation of goals of workshop
Jesus Salgado Data Models: Homogeneous view for heterogeneous data
High level, why are we creating data models in the IVOA?
- combining data from different missions is difficult and prone to error
- serialization o data is only partially standardized
IVOA Architecture:
- data models are a 'core' component, to be used by multiple aspects of IVOA standards
Data Models
- visual representation of data elements and relations between them
- assist data discover, database design, data documentation and interoperability
Photometry example:
- everyone uses magnitudes
- magnitudes are relative measurements; how do we compare between observatories?
- Photometry model associates necessary metadata to properly interpret/compare/combine magnitude data
"IVOA DM should be complex enough to allow scientific use cases but simple enough to..."
No questions
Jean Michel Glorian A developer's point of view on the use of models in the votable
- Presentation of CASSIS and use of data models on it
- Visualize and analyze electromagnetic spectra. Makes use of SAMP, SSAP, SLAP, TAP (with dataproduct=spectra) EPN-TAP
- From the data model point of view: Spectrum, Spectral Line, Spectral Data Cube, Image, Time Series
- utype is not enough. Some metadata is not present
- annotation could be used to fully characterize the spectrum (spectral axes, error spectral axes,...)
- Pointing to the client, link between different columns: flux value-> error flux value, flux value->normalized flux value
- In favour of mapping block preamble without breaking VOTable
- Also, a library for consuming this preamble would be very useful (e.g. java, python)
Questions:
LM: A point missing also is the library for sever side
MCD: UCDs versus Data models for physics of the meassurements?
JMG: based on a fixed table. Sometimes the UCD is not enough and it is difficult to identify when it is really a problem
Matthias Fuessling Science case requirement for CTA
- Description of use of data models on CTA
- CTA covers gamma-ray with two telescopes arrays (La Palma and Chile)
- from sub-TeV to multi-TeV
- Also different coordination centers across Europe
- Open observatory run by proposals
- Requirement to follow FAIR protocols and support VO
- PB/year, only reduced data to the user (PB/year)
- Different data levels: R0, R1, DL0-DL6 (DL6, skymaps, spectra, light curves, phaseograms)
- DL3 are event-list and technical tables, and it is one of the scientific products level
- DL3 in line with the Open Gamma-ray Astro Data Format (OGADF)
- OGADF will evolve in lines with science needs
- H.E.S.S. format already have been released via VO
- Also, CTA is part of the multi messengers initiatives, in ASTERICS, ESCAPE, connected to VO events, what would need also metadata to be characterized
Questions:
LM: Meaning of data VO complaint difficult to define
MCD: Comparison of Cube DM with gamma-ray data
MF: Not sure if this comparison has been done for CTA. All data is usually cubes so we will take a look into it
Catherin Boisson: Small test done with high level data (no exactly the same than X-Ray). Plans to do a deeper study
François Bonnarel Mango YAML serialisation
- Description of used cases
- VizieR use case: Photometry of 3 main belt asteroids
- Metadata from asteroids, fundamental metadata join with aspects shown
- Also, photometric measurements and positions relative to Earth and Sun
- Exercise to map all these table and characterize metadata using MANGO
- Generic MANGO elements used and Mango extension for TimeSeries created
- Serialization in YAML format with MANGO view
- FB showing and explaining the YAML file content
GDF: Origin of times (zero point) (Use of TIMESYS?)
FB: Taken from coordinates for the time (one single float value). Time system defined at the beginning
MCD: In the data models, a zero point for time is defined. JD and MJD are defined into the data model (class names)
Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf IVOA_DM_premeeting_20210517_v1.pdf r1 manage 1583.1 K 2021-05-17 - 16:43 GerardLemson VO-DML and Mapping
PDFpdf MangoViewONtwoTables.pdf r1 manage 1311.1 K 2021-05-22 - 13:47 FrancoisBonnarel Mango view on top of two tables with time variation
PDFpdf Workshop_pre-interop.pdf r1 manage 585.8 K 2021-05-17 - 14:00 LaurentMichel  
PDFpdf dm_workshop_preinterop.pdf r1 manage 2883.3 K 2021-05-18 - 13:18 MarkCresitelloDittmar Implementation review
PDFpdf markus.pdf r1 manage 2876.6 K 2021-05-17 - 18:30 MarkusDemleitner Lecture notes for Markus' talk
PDFpdf workshop_pre-interop_MANGO.pdf r1 manage 2858.9 K 2021-05-17 - 14:00 LaurentMichel  
Edit | Attach | Watch | Print version | History: r18 | r14 < r13 < r12 < r11 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r12 - 2021-05-26 - LaurentMichel
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback