(r10) Dm2021 < IVOA

IVOA Web>IvoaDataModel>SourceCatalogs>Dm2021 (revision 10) (raw view)~~EditAttach~~
During Fall 2020 Interop we had a [[InterOpNov2020DM][session]], requested by Markus Demleitner, that aimed at opening a wide discussion on both Measure and Coordinate models and possibly at refining their usages.

This came after a long discussion on the mailing list, related to the necessity of keeping or not a tight dependancy between Measurements and Coordinates. It has also been suggested that product models (e.g. CubeDM ) could be replaced with domain related components.

Many people have participated to the discussion with many positive contributions. <br />Although the goal of this message is not to report on that meeting, we do need to note a few points:
   * There is a clear demand for using models
   * There is a clear demand for the models to be simple and modular.
   * The distinction between what is modelling and what is serialisation is unclear.
The need for a concrete solution for annotating catalog data, that resurfaced last week, has been tackled by the WG for a couple of years along with the development of Meas/Coordinates models.
   * 2018:<a target="_blank" href="SourceCatalogs">Discussions</a> have been engaged on this topic after a hands-on session in Victoria.
   * 2019: We asked data providers and client developers to [[InterOpMay2019DM][present]] their needs in a dedicated session at Paris
After these 2 surveys the scope of the model usage has been exposed in Virtual May 2020 and we have been working on MANGO (a model for source data with some MC adaptations) that has been sketched in Groningen and in the virtual-May meetings and then [[%PUBURL%/IVOA/InterOpNov2020DM/Mango_interopFall2020.pdf][presented]] in November 2020.

To continue this action we have been organising a virtual workshop (initially proposed by Tom to the TCG) with the hope of getting a widely accepted framework.

The idea is not to start afresh but to give to the participants the opportunity of matching their needs with the current proposals.

To get a clear picture of what has to be done, we proposed to work with a variety concrete use cases, mostly taken out of the SourceDM proposal ( [[https://volute.g-vo.org/svn/trunk/projects/dm/SourceDM/][2016]]) and data samples provided by interested people.
---+++ Timeline
   * 2021 February 18: Public release of the use-case [[https://github.com/ivoa/dm-usecases][repository]].
   * 2021 February 24: Running meeting to present the process
   * 2021 March: People can look and contribute at the repository
   * 2021 May: Virtual Workshop - Pre-Interop meeting & 3 sessions during Interop.
---++ Running meeting(25/02/2021)
   * Etherpad [[https://yopad.eu/p/dm_virtual_meeeting_#2-365days][link]
*SESSION #1*

<strong><br /></strong>
<div id="magicdomid2"> *DM meeting - Workshop preparation - collective remarks and minutes* </div> <div id="magicdomid3">Participants 15</div> <div id="magicdomid4">

   * Mirreille Louys, Tom Donaldson Laurent Michel, Mark CD, Jesus Salgado, Gilles Landais, Francoise Genova, Kai Polsterer, Marco Molinaro, Francois Bonnarel , FX Pineau, Carlos Rodrigo, Sara Bertocco, Vincenzo Galluzzi, Markus Demleitner
</div> <div id="magicdomid20"><strong>The purpose of this meeting is to prepar</strong><strong>e</strong><strong> the DM workshop forseen in April 2021: </strong></div> <div id="magicdomid22"> *Workshop goals* </div> <div id="magicdomid23">LM presents the goal of the meeting. Preparation of meeting on data models on VO and how</div> <div id="magicdomid24">to improve consensus on data modeling generation.</div> <div id="magicdomid25">Some DMs work well but disagreements on how to match models with scientific quantities.</div> <div id="magicdomid26">Proposal to solve mapping problems is MANGO, developed by LM</div> <div id="magicdomid27">Also, proposal from Tom D. to organize a workshop on how to solve this gap</div> <div id="magicdomid29"> *Proposed process to prepare the workshop* </div> <div id="magicdomid30">Gather a list of concrete use cases and possible solutions</div> <div id="magicdomid31">Gather contributions</div> <div id="magicdomid32">
   * propose new use cases
</div> <div id="magicdomid33">
   * comment the use cases
</div> <div id="magicdomid34">
   * propose a solution
</div> <div id="magicdomid35">
   * comment proposed solutions
</div> <div id="magicdomid37"> *The dm-usecases repository* </div> <div id="magicdomid38">Link https://github.com/ivoa/dm-usecases</div> <div id="magicdomid39">These use cases is created compiling some VO historical use cases that require data modeling</div> <div id="magicdomid40">Two folders per use case, raw_data with an annotated xml and another folder with a mango proposal</div> <div id="magicdomid41">fulfilling the use case</div> <div id="magicdomid43">Time line for contribution to the use cases repository is during March to allow the pull requests merging</div> <div id="magicdomid44">of all the contributions</div> <div id="magicdomid46">poll created on use cases portal:</div> <div id="magicdomid47">https://github.com/ivoa/dm-usecases/wiki/poll</div> <div id="magicdomid49">FB: Procedure is to create a branch and ask for a pull request to be merged by LM? yes, either for commenting</div> <div id="magicdomid50">existing or creating a new use case</div> <div id="magicdomid52">TM: Lack of consensus are, in my view, due to the level of implementation of the solutions</div> <div id="magicdomid53">LM: I contacted Mark T. about using data models in Topcat and the problem comes from heteregeneous data</div> <div id="magicdomid54">modeling annotations in VOTables</div> <div id="magicdomid55">FG: It is difficult to decide whether to implement before having an idea of the work required to implement</div> <div id="magicdomid56">This means that the point has to be central to the workshop using the use cases</div> <div id="magicdomid58">MD: Will implement. If it works for VizieR, it will work for him.</div> <div id="magicdomid60">TM: Use case for proper motion representation?</div> <div id="magicdomid61">LM: Yes, in astrometry</div> <div id="magicdomid63">Mark CD: Orbital models not present, a new use case is needed</div> <div id="magicdomid65">MD: Not clear if we need models for quantities as this could be solved just with UCDs</div> <div id="magicdomid66">FB: UCDs are standalone, DM informs about quantities.</div> <div id="magicdomid67">MD: physics: UCDs, use DM to annotate</div> <div id="magicdomid69">GL: Mango enables to link two different quantities together, no need to have a specific model for each case.</div> <div id="magicdomid71"> *Workshop conduct* </div> <div id="magicdomid72">virtual / in person ?</div> <div id="magicdomid74"> *Proposal and decision for a* *Date* </div> <div id="magicdomid75">
   * End of April ?
<div id="magicdomid75"> *SESSION #2* </div>

<strong><br /></strong>
<div id="magicdomid109">DM meeting - Workshop preparation - collective remarks and minutes - Second session</div> <div id="magicdomid522">Participants 8: Laurent Michel, Mireille Louys, Gregory Dubois-Felsmann, Jesus Salgado, Francois Bonnarel, Gerard Lemson, Tom Donaldson, Mark Allen, Harry Enke</div> <div id="magicdomid261">LM presents objective of propossed workshop and historical background</div> <div id="magicdomid303">poll created on use cases portal:</div> <div id="magicdomid304">https://github.com/ivoa/dm-usecases/wiki/poll</div> <div id="magicdomid321">Use cases page proposed:</div> <div id="magicdomid324">https://github.com/ivoa/dm-usecases</div> <div id="magicdomid370">GL: Is there a DM behind all the MANGO annotations?</div> <div id="magicdomid371">LM: totally free</div> <div id="magicdomid421">Changes to be done in another branch (to be merged with trunk after review)</div> <div id="magicdomid489">If a specific model is proposed , a new directory should be inserted</div> <div id="magicdomid525">Same if a new mapping strategy is proposed</div> <div id="magicdomid527">
---++ Running meeting(17/05/2021)
</div>
   * Etherpad [[https://pad.unistra.fr/p/pre-interop][link]
<div id="magicdomid1147">Participants 17</div> <div id="magicdomid309">Gerard Lemson, Janet Evans, Mark Cresitello-Ditmar, Tom Donaldson, Laurent Michel, Mark Allen, Francous Bonnarel, Ian Evans, Jean-Mark, Jesus Salgado, Gilles Landais, Jean-Michel Glorian</div> <div id="magicdomid1145">Markus Demleitner, Vicenzo Galluzzi, Stefania Amodeo, Hendrik Heinl, Eric Buchlin</div> <div id="magicdomid4">This session will allow interested participants to evaluate our proposals with regard to the services they are operating or may need to build in the future.</div> <div id="magicdomid6">The meeting is OPEN to everyone.</div> <div id="magicdomid7">One thing we are trying to do though is target scientists/implementors who may consider volunteering to relate their experiences at the Interop Workshop.</div> <div id="magicdomid8">We envision presenting feedback on the use-case work completed so far, or presenting a vision of where a more mature version of the use case could end up someday.</div> <div id="magicdomid11"><span style="text-decoration: underline;">Agenda</span></div> <div id="magicdomid67">
   * Repo https://github.com/ivoa/dm-usecases
   * Wiki https://github.com/ivoa/dm-usecases/wiki
</div> <div id="magicdomid69">
   * VODML overview by G. Lemson (10&rsquo;)
   * VODML https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/
   * VODML mapping in ivoatex https://github.com/ivoa/mapping-vodml/tree/port-to-ivoatex
   * VODML mapping https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml-mapping/
</div> <div id="magicdomid72">Proposals</div> <div id="magicdomid73">Mark Cresitello Dittmar p (17+3&rsquo;) 
   * Coord DM https://volute.g-vo.org/svn/trunk/projects/dm/STC/Coords/doc/PR-Coords-1.0.pdf
   * Measure DM https://volute.g-vo.org/svn/trunk/projects/dm/STC/Meas/doc/PR-Meas-1.0.pdf
   * Cube DM https://volute.g-vo.org/svn/trunk/projects/dm/Cube/
</div> <div id="magicdomid77">
   * Markus Demleitner (17+3&rsquo;)
   * Code and use cases at https://github.com/msdemlei/astropy
</div> <div id="magicdomid79">Laurent Michel (17+3&rsquo;)
   * Mango DM https://github.com/ivoa-std/MANGO
   * Mapping syntax https://github.com/ivoa-std/ModelInstanceInVot
   * Mapping code: https://github.com/ivoa/modelinstanceinvot-code
   * API https://github.com/ivoa/dm-usecases/wiki/mango#lm-mango-parser
</div> <div id="magicdomid84">François Bonnarel (8+2&rsquo;) </div> <div id="magicdomid85"></div> <div id="magicdomid371">Minutes:</div> <div id="magicdomid814"> *Introduction by LM:* </div> <div id="magicdomid592">- Objective on how to use data models into VO and technical ways to share and to implement interoperability using DMs</div> <div id="magicdomid594">- Use cases:</div> <div id="magicdomid502"><span style="white-space: pre;"> </span>Display</div> <div id="magicdomid508"><span style="white-space: pre;"> </span>Plot</div> <div id="magicdomid514"><span style="white-space: pre;"> </span>Filtering</div> <div id="magicdomid520"><span style="white-space: pre;"> </span>X-Match</div> <div id="magicdomid529"><span style="white-space: pre;"> </span>Data Sharing</div> <div id="magicdomid537">...</div> <div id="magicdomid723">
   * - Description of elements of VOTable to identify if other elements are needed for data model serialization
   * - Quantities are well defined but tole and the way these quantitites are nested are poor. How quantities relate each other is very poor defined
   * - Proposal to add a mapping block without breaking existing VOTable blocks
   * - Most consensus on most of the data models issues but still not defined the mapping syntax
   * - Divided in three levels: Model fragments, Model Components and Integrated Models
   * - Main topic would to define which level should be mapped
   * - Presentation of the use cases https://github.com/ivoa/dm-usecases/wiki
</div> <div id="magicdomid964"> *VODML Overview (Gerard Lemson):* </div> <div id="magicdomid1021">
   * - Historical review of mapping DM problem and evolution from utypes to a complex VO/DML representation
</div> <div id="magicdomid1044"> *Proposals* </div> <div id="magicdomid1234"> *Mark Cresitello Dittmar* </div> <div id="magicdomid1132">Packages: Jovial, Rama (both specific for VO DMs developed mainly by Omar Laurino) and Astropy, MatPlotLib</div> <div id="magicdomid1150"></div> <div id="magicdomid1164">GL: Code to use annotation</div> <div id="magicdomid1177">MCD: Jovial</div> <div id="magicdomid1178"></div> <div id="magicdomid1232"> *Markus Demleitner* </div> <div id="magicdomid1235">
   * Data model strategy
   * TD: About astropy implementation fixes. Which fixes?
   * MD: My astropy code
</div> <div id="magicdomid1302"> *Laurent Michel* </div> <div id="magicdomid1318">
   * Presentation on MANGO
   * GL: MANGO and model instances mapping should go together?
   * LM: MANGO API relay on the syntax but mapping and syntax are indendent
   * TD: Merging on pyVO. What funtionality could be used?
   * LM: For example get quantity parsing MANGO syntax, or bind together different quantities
</div> <div id="magicdomid1566"></div> <div id="magicdomid1566"> *F. Bonnarel* </div>
   * Mapping Mango strucuture on top of Gaia and ZTF tables
<div id="magicdomid1604"></div> <div id="magicdomid1628"> *Wrap-up* </div> <div id="magicdomid1690">
   * 26/05 15h UTC Mapping strategy
   * 27/05 15h UTC Model review
   * 28/05 15h UTC VOTable mapping syntax
</div> <div id="magicdomid2293"> *Q/A* </div> <div id="magicdomid2295">
   * MD: Client writers are needed e.g. for measurements data model adoption. This simplified measurements would be just removing some boxes away
   * MCD: Use cases have started very simple that have been able to progress to complex use cases like Gaia. Another point is to show that different implementation are able to arise to the same result that is totally solved yet
   * MD: Posibility to go to DPAC to serialize time series in line with a proper VO format
   * FB: For new projects you can go to them to modify the serializations but the modification of legacy data is a lot more complex
   * GillesL: At least in ViZieR if photometry data is added, it is very important to also add the provenance
   * GL: Probably better to focus in the more popular datasets/missions. Why still using VOTable for annotating (VOTables are there only because it could be related to a service response)
   * Slides to be uploaded to [[Dm2021][https://wiki.ivoa.net/twiki/bin/view/IVOA/Dm2021]]
</div> <div id="magicdomid2358"> *Chat dump* </div> <div id="magicdomid2359">
   * 18:59:54 From Jean-Marc : Thanks to the speakers for the very nice introduction to VODML and the problem at hand. Unfortunately I have to leave now. I hope to be able to attend the workshop itself. Best.
   * 19:01:46 From Tom Donaldson : Thanks, Mark. That was a very helpful review of that work. Very intuitive flow.
   * 19:42:01 From Markus : That´s why you disentangle things...
   * 20:05:54 From Markus : Client writers!
   * 20:06:04 From Markus : Who´ll hijack them?
   * 20:06:23 From Mark CD : +1
   * 20:07:38 From Markus : Right.
   * 20:07:54 From Tom Donaldson : More important is skeptical client writers. But that&rsquo;s a harder sell because they don&rsquo;t see the point in expending effort.
   * 20:08:12 From Markus : Sell them error bars first.
   * 20:08:19 From Markus : automatic error bars.
   * 20:08:26 From Mark CD : Is that interesting enough?
   * 20:12:03 From Tom Donaldson : Along with Bambi eyeing, we may need hand holding. Even the simplest examples are intimidating without some initial help.
   * 20:12:42 From Markus : very true.
   * 20:19:26 From Tom Donaldson : I need to sign off at 25 past the hour. Thanks to everyone for the presentations, discussion points, and all the work behind them.
   * 20:22:01 From Markus : Again, I´m not saying we can ignore the legacy, I´m just saying let´s not burden our first version with it.
   * 20:24:12 From Ian Evans : I have to sign off to go to another meeting. Thanks everyone.
   * 20:25:38 From Markus : Oh: Do we collect slides/notes somewhere?
   * 20:27:59 From Laurent Michel : Can you upload you slides on the Wiki [[Dm2021][https://wiki.ivoa.net/twiki/bin/view/IVOA/Dm2021]]
</div> Links 
   * Usecase [[https://github.com/ivoa/dm-usecases][repository]]
   * Coords [[https://volute.g-vo.org/svn/trunk/projects/dm/STC/Coords/doc/PR-Coords-1.0.pdf][model]]
   * Measure [[https://volute.g-vo.org/svn/trunk/projects/dm/STC/Meas/doc/PR-Meas-1.0.pdf][model]]
   * Cube [[https://volute.g-vo.org/svn/trunk/projects/dm/Cube/][model]]
   * VODML [[https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/][repository]]
   * VODML [[https://github.com/ivoa/mapping-vodml/tree/port-to-ivoatex][mapping ivoatex]] with this [[https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml-org/xsd/ext/VOTable-1.4_vodml.xsd][xsd]] and derived from [[https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml-mapping/][mapping]]
   * MANGO [[https://github.com/ivoa-std/MANGO][model]]
   * ModelInstanceInVot [[https://github.com/ivoa-std/ModelInstanceInVot][mapping syntax]]
   * ModelInstanceInVot [[https://github.com/ivoa/modelinstanceinvot-code][test code]]
</div>
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pdf	IVOA_DM_premeeting_20210517_v1.pdf	r1	manage	1583.1 K	2021-05-17 - 16:43	GerardLemson	VO-DML and Mapping
pdf	Workshop_pre-interop.pdf	r1	manage	585.8 K	2021-05-17 - 14:00	LaurentMichel
pdf	dm_workshop_preinterop.pdf	r1	manage	2883.3 K	2021-05-18 - 13:18	MarkCresitelloDittmar	Implementation review
pdf	markus.pdf	r1	manage	2876.6 K	2021-05-17 - 18:30	MarkusDemleitner	Lecture notes for Markus' talk
pdf	workshop_pre-interop_MANGO.pdf	r1	manage	2858.9 K	2021-05-17 - 14:00	LaurentMichel
Topic revision: r10 - 2021-05-19 - JanetEvans
IVOA
Log in or Register
IVOA.net
Wiki Home
WebChanges
WebTopicList
WebStatistics
Twiki Meta & Help
IVOA
Know
Main
Sandbox
TWiki
TWiki intro
TWiki tutorial
User registration
Notify me
Working Groups
Interest Groups
Committees
Stds&Procs
www.ivoa.net
Documents
Events
Members
XML Schema