(r16) Dm2021 < IVOA

IVOA Web>IvoaDataModel>SourceCatalogs>Dm2021 (revision 16) (raw view)~~EditAttach~~
During Fall 2020 Interop we had a [[InterOpNov2020DM][session]], requested by Markus Demleitner, that aimed at opening a wide discussion on both Measure and Coordinate models and possibly at refining their usages.

This came after a long discussion on the mailing list, related to the necessity of keeping or not a tight dependancy between Measurements and Coordinates. It has also been suggested that product models (e.g. CubeDM ) could be replaced with domain related components.

Many people have participated to the discussion with many positive contributions. <br />Although the goal of this message is not to report on that meeting, we do need to note a few points:
   * There is a clear demand for using models
   * There is a clear demand for the models to be simple and modular.
   * The distinction between what is modelling and what is serialisation is unclear.
The need for a concrete solution for annotating catalog data, that resurfaced last week, has been tackled by the WG for a couple of years along with the development of Meas/Coordinates models.
   * 2018:<a target="_blank" href="SourceCatalogs">Discussions</a> have been engaged on this topic after a hands-on session in Victoria.
   * 2019: We asked data providers and client developers to [[InterOpMay2019DM][present]] their needs in a dedicated session at Paris
After these 2 surveys the scope of the model usage has been exposed in Virtual May 2020 and we have been working on MANGO (a model for source data with some MC adaptations) that has been sketched in Groningen and in the virtual-May meetings and then [[%PUBURL%/IVOA/InterOpNov2020DM/Mango_interopFall2020.pdf][presented]] in November 2020.

To continue this action we have been organising a virtual workshop (initially proposed by Tom to the TCG) with the hope of getting a widely accepted framework.

The idea is not to start afresh but to give to the participants the opportunity of matching their needs with the current proposals.

To get a clear picture of what has to be done, we proposed to work with a variety concrete use cases, mostly taken out of the SourceDM proposal ( [[https://volute.g-vo.org/svn/trunk/projects/dm/SourceDM/][2016]]) and data samples provided by interested people.
---+++ Timeline
   * 2021 February 18: Public release of the use-case [[https://github.com/ivoa/dm-usecases][repository]].
   * 2021 February 24: Running meeting to present the process
   * 2021 March: People can look and contribute at the repository
   * 2021 May: Virtual Workshop - Pre-Interop meeting & 3 sessions during Interop.
---++ Running meeting(25/02/2021)
   * Etherpad [[https://yopad.eu/p/dm_virtual_meeeting_#2-365days][link]
*SESSION #1*

<strong><br /></strong>
<div id="magicdomid2"> *DM meeting - Workshop preparation - collective remarks and minutes* </div> <div id="magicdomid3">Participants 15</div> <div id="magicdomid4">

   * Mirreille Louys, Tom Donaldson Laurent Michel, Mark CD, Jesus Salgado, Gilles Landais, Francoise Genova, Kai Polsterer, Marco Molinaro, Francois Bonnarel , FX Pineau, Carlos Rodrigo, Sara Bertocco, Vincenzo Galluzzi, Markus Demleitner
</div> <div id="magicdomid20"><strong>The purpose of this meeting is to prepar</strong><strong>e</strong><strong> the DM workshop forseen in April 2021: </strong></div> <div id="magicdomid22"> *Workshop goals* </div> <div id="magicdomid23">LM presents the goal of the meeting. Preparation of meeting on data models on VO and how</div> <div id="magicdomid24">to improve consensus on data modeling generation.</div> <div id="magicdomid25">Some DMs work well but disagreements on how to match models with scientific quantities.</div> <div id="magicdomid26">Proposal to solve mapping problems is MANGO, developed by LM</div> <div id="magicdomid27">Also, proposal from Tom D. to organize a workshop on how to solve this gap</div> <div id="magicdomid29"> *Proposed process to prepare the workshop* </div> <div id="magicdomid30">Gather a list of concrete use cases and possible solutions</div> <div id="magicdomid31">Gather contributions</div> <div id="magicdomid32">
   * propose new use cases
</div> <div id="magicdomid33">
   * comment the use cases
</div> <div id="magicdomid34">
   * propose a solution
</div> <div id="magicdomid35">
   * comment proposed solutions
</div> <div id="magicdomid37"> *The dm-usecases repository* </div> <div id="magicdomid38">Link https://github.com/ivoa/dm-usecases</div> <div id="magicdomid39">These use cases is created compiling some VO historical use cases that require data modeling</div> <div id="magicdomid40">Two folders per use case, raw_data with an annotated xml and another folder with a mango proposal</div> <div id="magicdomid41">fulfilling the use case</div> <div id="magicdomid43">Time line for contribution to the use cases repository is during March to allow the pull requests merging</div> <div id="magicdomid44">of all the contributions</div> <div id="magicdomid46">poll created on use cases portal:</div> <div id="magicdomid47">https://github.com/ivoa/dm-usecases/wiki/poll</div> <div id="magicdomid49">FB: Procedure is to create a branch and ask for a pull request to be merged by LM? yes, either for commenting</div> <div id="magicdomid50">existing or creating a new use case</div> <div id="magicdomid52">TM: Lack of consensus are, in my view, due to the level of implementation of the solutions</div> <div id="magicdomid53">LM: I contacted Mark T. about using data models in Topcat and the problem comes from heteregeneous data</div> <div id="magicdomid54">modeling annotations in VOTables</div> <div id="magicdomid55">FG: It is difficult to decide whether to implement before having an idea of the work required to implement</div> <div id="magicdomid56">This means that the point has to be central to the workshop using the use cases</div> <div id="magicdomid58">MD: Will implement. If it works for VizieR, it will work for him.</div> <div id="magicdomid60">TM: Use case for proper motion representation?</div> <div id="magicdomid61">LM: Yes, in astrometry</div> <div id="magicdomid63">Mark CD: Orbital models not present, a new use case is needed</div> <div id="magicdomid65">MD: Not clear if we need models for quantities as this could be solved just with UCDs</div> <div id="magicdomid66">FB: UCDs are standalone, DM informs about quantities.</div> <div id="magicdomid67">MD: physics: UCDs, use DM to annotate</div> <div id="magicdomid69">GL: Mango enables to link two different quantities together, no need to have a specific model for each case.</div> <div id="magicdomid71"> *Workshop conduct* </div> <div id="magicdomid72">virtual / in person ?</div> <div id="magicdomid74"> *Proposal and decision for a* *Date* </div> <div id="magicdomid75">
   * End of April ?
<div id="magicdomid75"> *SESSION #2* </div>

<strong><br /></strong>
<div id="magicdomid109">DM meeting - Workshop preparation - collective remarks and minutes - Second session</div> <div id="magicdomid522">Participants 8: Laurent Michel, Mireille Louys, Gregory Dubois-Felsmann, Jesus Salgado, Francois Bonnarel, Gerard Lemson, Tom Donaldson, Mark Allen, Harry Enke</div> <div id="magicdomid261">LM presents objective of propossed workshop and historical background</div> <div id="magicdomid303">poll created on use cases portal:</div> <div id="magicdomid304">https://github.com/ivoa/dm-usecases/wiki/poll</div> <div id="magicdomid321">Use cases page proposed:</div> <div id="magicdomid324">https://github.com/ivoa/dm-usecases</div> <div id="magicdomid370">GL: Is there a DM behind all the MANGO annotations?</div> <div id="magicdomid371">LM: totally free</div> <div id="magicdomid421">Changes to be done in another branch (to be merged with trunk after review)</div> <div id="magicdomid489">If a specific model is proposed , a new directory should be inserted</div> <div id="magicdomid525">Same if a new mapping strategy is proposed</div> <div id="magicdomid527">

---++ Pre-interop meeting(17/05/2021)
</div>
   * Etherpad [[https://pad.unistra.fr/p/pre-interop][link]
<div id="magicdomid1147">Participants 17</div> <div id="magicdomid309">Gerard Lemson, Janet Evans, Mark Cresitello-Ditmar, Tom Donaldson, Laurent Michel, Mark Allen, Francous Bonnarel, Ian Evans, Jean-Mark, Jesus Salgado, Gilles Landais, Jean-Michel Glorian</div> <div id="magicdomid1145">Markus Demleitner, Vicenzo Galluzzi, Stefania Amodeo, Hendrik Heinl, Eric Buchlin</div> <div id="magicdomid4">This session will allow interested participants to evaluate our proposals with regard to the services they are operating or may need to build in the future.</div> <div id="magicdomid6">The meeting is OPEN to everyone.</div> <div id="magicdomid7">One thing we are trying to do though is target scientists/implementors who may consider volunteering to relate their experiences at the Interop Workshop.</div> <div id="magicdomid8">We envision presenting feedback on the use-case work completed so far, or presenting a vision of where a more mature version of the use case could end up someday.</div> <div id="magicdomid11"><span style="text-decoration: underline;">Agenda</span></div> <div id="magicdomid67">
   * Repo https://github.com/ivoa/dm-usecases
   * Wiki https://github.com/ivoa/dm-usecases/wiki
</div> <div id="magicdomid69">
   * VODML overview by G. Lemson (10&rsquo;)
   * VODML https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/
   * VODML mapping in ivoatex https://github.com/ivoa/mapping-vodml/tree/port-to-ivoatex
   * VODML mapping https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml-mapping/
</div> <div id="magicdomid72">Proposals</div> <div id="magicdomid73">Mark Cresitello Dittmar p (17+3&rsquo;) 
   * Coord DM https://volute.g-vo.org/svn/trunk/projects/dm/STC/Coords/doc/PR-Coords-1.0.pdf
   * Measure DM https://volute.g-vo.org/svn/trunk/projects/dm/STC/Meas/doc/PR-Meas-1.0.pdf
   * Cube DM https://volute.g-vo.org/svn/trunk/projects/dm/Cube/
</div> <div id="magicdomid77">
   * Markus Demleitner (17+3&rsquo;)
   * Code and use cases at https://github.com/msdemlei/astropy
</div> <div id="magicdomid79">Laurent Michel (17+3&rsquo;) 
   * Mango DM https://github.com/ivoa-std/MANGO
   * Mapping syntax https://github.com/ivoa-std/ModelInstanceInVot
   * Mapping code: https://github.com/ivoa/modelinstanceinvot-code
   * API https://github.com/ivoa/dm-usecases/wiki/mango#lm-mango-parser
</div> <div id="magicdomid84">FranÃ§ois Bonnarel (8+2&rsquo;)</div> <div id="magicdomid371">Minutes:</div> <div id="magicdomid814"> *Introduction by LM:* </div> <div id="magicdomid592">- Objective on how to use data models into VO and technical ways to share and to implement interoperability using DMs</div> <div id="magicdomid594">- Use cases:</div> <div id="magicdomid502"><span style="white-space: pre;"> </span>Display</div> <div id="magicdomid508"><span style="white-space: pre;"> </span>Plot</div> <div id="magicdomid514"><span style="white-space: pre;"> </span>Filtering</div> <div id="magicdomid520"><span style="white-space: pre;"> </span>X-Match</div> <div id="magicdomid529"><span style="white-space: pre;"> </span>Data Sharing</div> <div id="magicdomid537">...</div> <div id="magicdomid723">
   * - Description of elements of VOTable to identify if other elements are needed for data model serialization
   * - Quantities are well defined but tole and the way these quantitites are nested are poor. How quantities relate each other is very poor defined
   * - Proposal to add a mapping block without breaking existing VOTable blocks
   * - Most consensus on most of the data models issues but still not defined the mapping syntax
   * - Divided in three levels: Model fragments, Model Components and Integrated Models
   * - Main topic would to define which level should be mapped
   * - Presentation of the use cases https://github.com/ivoa/dm-usecases/wiki
</div> <div id="magicdomid964"> *VODML Overview (Gerard Lemson):* </div> <div id="magicdomid1021">
   * - Historical review of mapping DM problem and evolution from utypes to a complex VO/DML representation
</div> <div id="magicdomid1044"> *Proposals* </div> <div id="magicdomid1234"> *Mark Cresitello Dittmar* </div> <div id="magicdomid1132">Packages: Jovial, Rama (both specific for VO DMs developed mainly by Omar Laurino) and Astropy, MatPlotLib</div> <div id="magicdomid1164">GL: Code to use annotation</div> <div id="magicdomid1177">MCD: Jovial</div> <div id="magicdomid1232"> *Markus Demleitner* </div> <div id="magicdomid1235">
   * Data model strategy
   * TD: About astropy implementation fixes. Which fixes?
   * MD: My astropy code
</div> <div id="magicdomid1302"> *Laurent Michel* </div> <div id="magicdomid1318">
   * Presentation on MANGO
   * GL: MANGO and model instances mapping should go together?
   * LM: MANGO API relay on the syntax but mapping and syntax are indendent
   * TD: Merging on pyVO. What funtionality could be used?
   * LM: For example get quantity parsing MANGO syntax, or bind together different quantities
</div> <div id="magicdomid1566"> *F. Bonnarel* </div>
   * Mapping Mango strucuture on top of Gaia and ZTF tables
<div id="magicdomid1628"> *Wrap-up* </div> <div id="magicdomid1690">
   * 26/05 15h UTC Mapping strategy
   * 27/05 15h UTC Model review
   * 28/05 15h UTC VOTable mapping syntax
</div> <div id="magicdomid2293"> *Q/A* </div> <div id="magicdomid2295">
   * MD: Client writers are needed e.g. for measurements data model adoption. This simplified measurements would be just removing some boxes away
   * MCD: Use cases have started very simple that have been able to progress to complex use cases like Gaia. Another point is to show that different implementation are able to arise to the same result that is totally solved yet
   * MD: Posibility to go to DPAC to serialize time series in line with a proper VO format
   * FB: For new projects you can go to them to modify the serializations but the modification of legacy data is a lot more complex
   * GillesL: At least in ViZieR if photometry data is added, it is very important to also add the provenance
   * GL: Probably better to focus in the more popular datasets/missions. Why still using VOTable for annotating (VOTables are there only because it could be related to a service response)
   * Slides to be uploaded to [[Dm2021][https://wiki.ivoa.net/twiki/bin/view/IVOA/Dm2021]]
</div> <div id="magicdomid2358"> *Chat dump* </div> <div id="magicdomid2359">
   * 18:59:54 From Jean-Marc : Thanks to the speakers for the very nice introduction to VODML and the problem at hand. Unfortunately I have to leave now. I hope to be able to attend the workshop itself. Best.
   * 19:01:46 From Tom Donaldson : Thanks, Mark. That was a very helpful review of that work. Very intuitive flow.
   * 19:42:01 From Markus : ThatÂ´s why you disentangle things...
   * 20:05:54 From Markus : Client writers!
   * 20:06:04 From Markus : WhoÂ´ll hijack them?
   * 20:06:23 From Mark CD : +1
   * 20:07:38 From Markus : Right.
   * 20:07:54 From Tom Donaldson : More important is skeptical client writers. But that&rsquo;s a harder sell because they don&rsquo;t see the point in expending effort.
   * 20:08:12 From Markus : Sell them error bars first.
   * 20:08:19 From Markus : automatic error bars.
   * 20:08:26 From Mark CD : Is that interesting enough?
   * 20:12:03 From Tom Donaldson : Along with Bambi eyeing, we may need hand holding. Even the simplest examples are intimidating without some initial help.
   * 20:12:42 From Markus : very true.
   * 20:19:26 From Tom Donaldson : I need to sign off at 25 past the hour. Thanks to everyone for the presentations, discussion points, and all the work behind them.
   * 20:22:01 From Markus : Again, IÂ´m not saying we can ignore the legacy, IÂ´m just saying letÂ´s not burden our first version with it.
   * 20:24:12 From Ian Evans : I have to sign off to go to another meeting. Thanks everyone.
   * 20:25:38 From Markus : Oh: Do we collect slides/notes somewhere?
   * 20:27:59 From Laurent Michel : Can you upload you slides on the Wiki [[Dm2021][https://wiki.ivoa.net/twiki/bin/view/IVOA/Dm2021]]
</div> Links 
   * Usecase [[https://github.com/ivoa/dm-usecases][repository]]
   * Coords [[https://volute.g-vo.org/svn/trunk/projects/dm/STC/Coords/doc/PR-Coords-1.0.pdf][model]]
   * Measure [[https://volute.g-vo.org/svn/trunk/projects/dm/STC/Meas/doc/PR-Meas-1.0.pdf][model]]
   * Cube [[https://volute.g-vo.org/svn/trunk/projects/dm/Cube/][model]]
   * VODML [[https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/][repository]]
   * VODML [[https://github.com/ivoa/mapping-vodml/tree/port-to-ivoatex][mapping ivoatex]] with this [[https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml-org/xsd/ext/VOTable-1.4_vodml.xsd][xsd]] and derived from [[https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml-mapping/][mapping]]
   * MANGO [[https://github.com/ivoa-std/MANGO][model]]
   * ModelInstanceInVot [[https://github.com/ivoa-std/ModelInstanceInVot][mapping syntax]]
   * ModelInstanceInVot [[https://github.com/ivoa/modelinstanceinvot-code][test code]]
---++ Interop Sessions

<div id="magicdomid25"> *Session #1 Wednesday May 26 15h UTC* Model users have the floor floor</div> <div id="magicdomid26">Participants 53</div> <div id="magicdomid28"><span style="text-decoration: underline;">Laurent Michel Introduction</span></div> <div id="magicdomid29">- Explanation of data model benefits for interoperability</div> <div id="magicdomid30">- Presentation of data model use cases repository created by the team with examples of mapping techniques</div> <div id="magicdomid31">- Presentation of goals of workshop</div> <div id="magicdomid33"><span style="text-decoration: underline;">Jesus Salgado Data Models: Homogeneous view for heterogeneous data</span></div> <div id="magicdomid34">High level, why are we creating data models in the IVOA?</div> <div id="magicdomid35">- combining data from different missions is difficult and prone to error</div> <div id="magicdomid36">- serialization o data is only partially standardized</div> <div id="magicdomid37">IVOA Architecture:</div> <div id="magicdomid38">- data models are a 'core' component, to be used by multiple aspects of IVOA standards</div> <div id="magicdomid39">Data Models</div> <div id="magicdomid40">- visual representation of data elements and relations between them</div> <div id="magicdomid41">- assist data discover, database design, data documentation and interoperability</div> <div id="magicdomid42">Photometry example:</div> <div id="magicdomid43">- everyone uses magnitudes</div> <div id="magicdomid44">- magnitudes are relative measurements; how do we compare between observatories?</div> <div id="magicdomid45">- Photometry model associates necessary metadata to properly interpret/compare/combine magnitude data</div> <div id="magicdomid46">"IVOA DM should be complex enough to allow scientific use cases but simple enough to..."</div> <div id="magicdomid48">No questions</div> <div id="magicdomid50"><span style="text-decoration: underline;">Jean Michel Glorian A developer's point of view on the use of models in the votable </span></div> <div id="magicdomid51">- Presentation of CASSIS and use of data models on it</div> <div id="magicdomid52">- Visualize and analyze electromagnetic spectra. Makes use of SAMP, SSAP, SLAP, TAP (with dataproduct=spectra) EPN-TAP</div> <div id="magicdomid53">- From the data model point of view: Spectrum, Spectral Line, Spectral Data Cube, Image, Time Series</div> <div id="magicdomid54">- utype is not enough. Some metadata is not present</div> <div id="magicdomid55">- annotation could be used to fully characterize the spectrum (spectral axes, error spectral axes,...)</div> <div id="magicdomid56">- Pointing to the client, link between different columns: flux value-&gt; error flux value, flux value-&gt;normalized flux value</div> <div id="magicdomid57">- In favour of mapping block preamble without breaking VOTable</div> <div id="magicdomid58">- Also, a library for consuming this preamble would be very useful (e.g. java, python)</div> <div id="magicdomid60">Questions:</div> <div id="magicdomid61">LM: A point missing also is the library for sever side</div> <div id="magicdomid62">MCD: UCDs versus Data models for physics of the meassurements?</div> <div id="magicdomid63">JMG: based on a fixed table. Sometimes the UCD is not enough and it is difficult to identify when it is really a problem</div> <div id="magicdomid65"><span style="text-decoration: underline;">Matthias Fuessling Science case requirement for CTA </span></div> <div id="magicdomid66">- Description of use of data models on CTA</div> <div id="magicdomid67">- CTA covers gamma-ray with two telescopes arrays (La Palma and Chile)</div> <div id="magicdomid68">- from sub-TeV to multi-TeV</div> <div id="magicdomid69">- Also different coordination centers across Europe</div> <div id="magicdomid70">- Open observatory run by proposals</div> <div id="magicdomid71">- Requirement to follow FAIR protocols and support VO</div> <div id="magicdomid72">- PB/year, only reduced data to the user (PB/year)</div> <div id="magicdomid73">- Different data levels: R0, R1, DL0-DL6 (DL6, skymaps, spectra, light curves, phaseograms)</div> <div id="magicdomid74">- DL3 are event-list and technical tables, and it is one of the scientific products level</div> <div id="magicdomid75">- DL3 in line with the Open Gamma-ray Astro Data Format (OGADF)</div> <div id="magicdomid76">- OGADF will evolve in lines with science needs</div> <div id="magicdomid77">- H.E.S.S. format already have been released via VO</div> <div id="magicdomid78">- Also, CTA is part of the multi messengers initiatives, in ASTERICS, ESCAPE, connected to VO events, what would need also metadata to be characterized</div> <div id="magicdomid80">Questions:</div> <div id="magicdomid81">LM: Meaning of data VO complaint difficult to define</div> <div id="magicdomid82">MCD: Comparison of Cube DM with gamma-ray data</div> <div id="magicdomid83">MF: Not sure if this comparison has been done for CTA. All data is usually cubes so we will take a look into it</div> <div id="magicdomid84">Catherin Boisson: Small test done with high level data (no exactly the same than X-Ray). Plans to do a deeper study</div> <div id="magicdomid86"><span style="text-decoration: underline;">FranÃ§ois Bonnarel Mango YAML serialisation </span></div> <div id="magicdomid87">- Description of used cases</div> <div id="magicdomid88">- VizieR use case: Photometry of 3 main belt asteroids</div> <div id="magicdomid89">- Metadata from asteroids, fundamental metadata join with aspects shown</div> <div id="magicdomid90">- Also, photometric measurements and positions relative to Earth and Sun</div> <div id="magicdomid91">- Exercise to map all these table and characterize metadata using MANGO</div> <div id="magicdomid92">- Generic MANGO elements used and Mango extension for TimeSeries created</div> <div id="magicdomid93">- Serialization in YAML format with MANGO view</div> <div id="magicdomid94">- FB showing and explaining the YAML file content</div> <div id="magicdomid96">GDF: Origin of times (zero point) (Use of TIMESYS?)</div> <div id="magicdomid97">FB: Taken from coordinates for the time (one single float value). Time system defined at the beginning</div> <div id="magicdomid98">MCD: In the data models, a zero point for time is defined. JD and MJD are defined into the data model (class names</div>

</div> <div id="magicdomid25"> *Session #2 Thursday May 27 15h UTC* Model users have the floor floor</div>

<div id="magicdomid101">Participants: 60</div> <div id="magicdomid103"><span style="text-decoration: underline;">Gilles Landais Annotated VOTable in VizieR</span></div> <div id="magicdomid104">- Presentation of VOTable responses from VizieR and annotations</div> <div id="magicdomid105">

   * - Provenance
</div> <div id="magicdomid106">
   * - Magnitude and system description
</div> <div id="magicdomid107">
   * - Groups for related meassure columns
</div> <div id="magicdomid108">
   * - Added values (e.g. coordinates into J2000)
</div> <div id="magicdomid110">- Example of mapping of radial velocities and associated errors</div> <div id="magicdomid111">- Photometry viewer implementation using photometry annotations</div> <div id="magicdomid112">- Analysis of possible MANGO annotated VizieR tables and presentation of prototype</div> <div id="magicdomid113">- DMs used, Source and Meassure annotations</div> <div id="magicdomid114">- In order to convert prototype to operational:</div> <div id="magicdomid115">
   * - Provenance information for filter origin
</div> <div id="magicdomid116">
   * - Provanence dublin core (DOI, authors, date)
</div> <div id="magicdomid117">
   * - Stable MANGO building block
</div> <div id="magicdomid118">
   * - VO validator
</div> <div id="magicdomid120">Questions:</div> <div id="magicdomid121">Ada Nebot: ask for link:</div> <div id="magicdomid122">[[http://viz-beta.u-strasbg.fr/viz-bin/Mango?-source=J/A+A/584/A5]]</div> <div id="magicdomid123">VizieR mango prototype:</div> <div id="magicdomid124">â€¨ [[http://viz-beta.u-strasbg.fr/viz-bin/Mango?-source=]][catalogue_name]</div> <div id="magicdomid125">Tom D.: in last point, about the need of new provenance.</div> <div id="magicdomid126">GillesL: For second part to prevent scientists looking for it but for photometry in order to consume it by clients</div> <div id="magicdomid127">Gilles: I like DataModel when a flat view exists ! (eg: Obscore, photDM)</div> <div id="magicdomid129"><span style="text-decoration: underline;">Ian Evans Science considerations for IVOA data models </span></div> <div id="magicdomid130">- Expose complex datasets managed by Chandra including sources, detections, ancilliary tables, and many types of FITS data products</div> <div id="magicdomid131">- Explanatin of complexity on X-ray astronomy data, composed from (x, y, t, PHA) in order to reconstruct photons</div> <div id="magicdomid132">- X-ray photometry depends on the source spectrum in order to calculate iteratively the real photon meassurement (forward fitting)</div> <div id="magicdomid133">- CSC makes use of Bayesian X-ray aperture photometry, so multiple detections are solved simultaneously</div> <div id="magicdomid134">- Optimization done by MCMC (Markov Chain MonteCarlo Chain)</div> <div id="magicdomid135">- Use of confidence intervals (independent lower and upper confidence limits)</div> <div id="magicdomid136">- 2nd lesson learned is temporal variability</div> <div id="magicdomid137">- Individual and multi-epoch data</div> <div id="magicdomid138">- Many to many relationship between detections and sources</div> <div id="magicdomid139">- Data models should contain shape and epoch information</div> <div id="magicdomid140">- IVOA data models can represent X-ray and X-ray source data?</div> <div id="magicdomid141">- Some changes needed</div> <div id="magicdomid142">- MANGO provides a good framework for mapping</div> <div id="magicdomid144">Questions:</div> <div id="magicdomid145">Kai P: Probability function is needed in general to make proper science</div> <div id="magicdomid146">Ian E: Yes, for a radial velocity sensor, e.g. PDF are also needed</div> <div id="magicdomid148"><span style="text-decoration: underline;">Jean Marc Petit Asteroids: astrometry, photometry, dynamics, survey modelling</span></div> <div id="magicdomid149">Different types of data:</div> <div id="magicdomid150">- Photometry:</div> <div id="magicdomid151">
   * Flux vs time lightcurves
   * Physical properties: lightcurve, phase function, colors
   * Heterogeneous time description (data, JD, MJD)
   * Showing example file and client to filter properties
   * Derived data: periods, lightcurve amplitude, shape
   * Provenance
</div> <div id="magicdomid157">- Astrometry:</div> <div id="magicdomid158">
   * MPC typical information shown
   * Survey internal information
</div> <div id="magicdomid160">- Dynamics</div> <div id="magicdomid161">
   * Orbit information
   * a, e_a, e, e_e, i, e_i, omega, e_omega, tperi, e_tperi and, in general, orbital elements information
   * Also, phometric information, distance, etc
   * Dynamics classification
</div> <div id="magicdomid165">- Comparison with models explanation</div> <div id="magicdomid166">- Question on how to represent metadata needed for orbital estimation</div> <div id="magicdomid168">Questions:</div> <div id="magicdomid169">Francois B: How is done the generation of orbital parameters from observational measurements (astrometry)?</div> <div id="magicdomid170">Jean-Mark: there is a data hierachy from meassurements to the orbital estimator calculation.</div> <div id="magicdomid172">JJK: what is the data provenance is the question, does it have any? Answer from JM is yes, but the provenance is not expressed in the model we use.</div> <div id="magicdomid173">Jean-Marc: Fully agree</div> <div id="magicdomid175">Mark C-D: Comparison of simulated and observed data is a very interesting case to me. Optimally, I think, the data itself could be described by the same model, but with VERY different associated Dataset Metadata, one describing the observation, the other describing the simulation. Does this approach seem feasible for your data?</div> <div id="magicdomid176">Jean-Marc: Models and real data behaves the same in the orbital simulator.</div> <div id="magicdomid177">GL: Link between real and simulated data is done through by defining the provenance of both sides (that is different) but trying to find similarities between both metadata</div> <div id="magicdomid179">AN : I also like this idea of the comparison of simulated and observed data. Same model for both helps for that. I guess that we have an example of something like this in the VOSA service (different context tho), where photometry (observed, collected from different surveys) is compared with synthetic photometry computed from stellar models., the aim is to determine stellar parameters from SED.</div> <div id="magicdomid180">In this specific science case it would be from positions, time, rvs, get the orbital parameters.</div> <div id="magicdomid182"><span style="text-decoration: underline;">Laurent Michel Binding PyVO with MANGO </span></div> <div id="magicdomid183">- Presentation of plans to include MANGO support on PyVO</div> <div id="magicdomid184">- Presentation of basic architectural concepts on VOTable handling in pyVO</div> <div id="magicdomid185">- MANGO is a container model</div> <div id="magicdomid186">- MANGO improves view of metadata</div> <div id="magicdomid187">- Example of elements to annotate VOTable with SpaceFrame, LonLatSkyPosition and ProperMotion (Global and Parameter Dock)</div> <div id="magicdomid188">- Showing example of MANGO annotation with previous elements</div> <div id="magicdomid189">- A MANGO parameter could contain different columns (e.g. value, error, frame)</div> <div id="magicdomid190">- Description of similarities between python dictionary and MANGO</div>

<div id="magicdomid25"> *Session #3 Thursday May 28 15h UTC* Mapping Syntax</div>

<div id="magicdomid205">Participants: 41</div> <div id="magicdomid207">Laurent Michel Introduction:</div> <div id="magicdomid208">

   * - This last session is more technical on mapping solutions
   * - utypes useful for complex protocols but limited for complex cases
   * - Complex mappings require architecture and techniques approaches
   * - Quantities and roles
   * - Annotation in VOTables without breaking exisiting VOTable structures
   * - mapping written in pure XML
   * - out of VOTable resources
   * - allow nested structures
   * - freedom to setup the schema
   * - XML schema mapping imported into VOTable schema
   * - Avoid name collisions between mapping and VOTable elements
   * - VODML mapping 2017
   * - 2019 MANGO mapping
   * - 2020 ModelInstanceInVot
<span style="text-decoration: underline;">Gerard Lemson VODML Mapping</span></div> <div id="magicdomid224">- Concepts:</div> <div id="magicdomid225">
   * - Mapping express how instances of data model are expressed on VOTable and TAP_SCHEMA
   * - Based on the VODML standard
   * - Example of a Source toy model to decribe the proposed mapping
   * - Explanation of mapping for real catalogues tables
   * - Complication of mapping is more than the syntax the impedance between data models and tabular data
</div> <div id="magicdomid230">- VO-DML Mapper</div> <div id="magicdomid231">
   * - tool that can read TAP_SCHEMA and VOTables
   * - able to read data models
   * - allow graphical connection between data elements and data model fields
   * - generates mapping block
</div> <div id="magicdomid235">- Mapping Meta-Models</div> <div id="magicdomid236">
   * - VO-DML is IVOA standard meta-model/language
   * - goals: mapping at the meta-model level
   * - mapping as much as possible 1-1
</div> <div id="magicdomid239">- Inspired by object-relational mapping</div> <div id="magicdomid240">- Explanation of VO-DML mapping, VODML-mapping.xsd</div> <div id="magicdomid241">- VODML/Model: Model1: xmatch -&gt; Model2: source &lt;- Model-2-sub: sdss-source</div> <div id="magicdomid242">- GLOBALS: container for "standalone" instances of VO/DML types</div> <div id="magicdomid243">- ATTRIBUTE</div> <div id="magicdomid244">- Explanation of the different elements into VO/DML mapping</div> <div id="magicdomid245">- CONTAINER/FOREIGNKEY</div> <div id="magicdomid247"><span style="text-decoration: underline;">Laurent Michel ModelInstanceInVot: alternative or evolution</span></div> <div id="magicdomid248">- More focused on the mapping generation than the object-relational mapping</div> <div id="magicdomid249">- Comparison of the approach of the path from mapping and models</div> <div id="magicdomid250">- XML elements faith to the model path</div> <div id="magicdomid251">- Use of dmrole and dmtype</div> <div id="magicdomid252">- Into the mapping block there are globals and the mapping of the tables included in the document</div> <div id="magicdomid253">- working draft written on ModelInstanceInVot</div> <div id="magicdomid254">- Similarities with VO/DML mapping but, also, significant differences</div> <div id="magicdomid256">Questions:</div> <div id="magicdomid257">GL: Elements can have structure so mapping is somehow more complex. Also connections from rows</div> <div id="magicdomid258">is too simplified. Container is very similar</div> <div id="magicdomid259">Some commonalities, some deviations but close enough to find elements missing</div> <div id="magicdomid260">LM: JSON can express complex data organizations, so the solution is based on this</div> <div id="magicdomid261">GL: Some elements in the VO/DML mapping so a different approach would be to evolve the old standard</div> <div id="magicdomid262">with these ideas</div> <div id="magicdomid264"><span style="text-decoration: underline;">Mark Cresitello Dittmar Summary </span></div> <div id="magicdomid265">- Impressions of the two approaches comparison</div> <div id="magicdomid266">- Both support the workshop use cases</div> <div id="magicdomid267">- Analysis of the Modelinstanceinvot mapping and VODML mapping on the use cases, including possible mapping limitations</div> <div id="magicdomid268">- Possibility of moving forward by using an hybrid solution</div> <div id="magicdomid270">Questions:</div> <div id="magicdomid271">GL: Same problem, elements is more general that attributes. Original idea was to have a very specific tecniques</div> <div id="magicdomid272">and it could be simplied</div> <div id="magicdomid273">FB: GROUPBY only have one reference but it can be stacked</div> <div id="magicdomid275">LM: to get the inputs and prepare summary to be sent to interop</div> <div id="magicdomid277">GL: question of SourceDM in MANGO</div> <div id="magicdomid278">LM: MANGO is not a SourceDM but annotations for source</div> <div id="magicdomid280">Mireille L: Question on the roadmap for a mapping strategy? 60% of simple datasets that can be already mapped. Do we need more complex use cases?</div> <div id="magicdomid282">TD: Probably mapping syntax is linked to VOTable 2.0 so first would be to ignore the block in the mapping and second the parser</div> <div id="magicdomid284">AN: If you have multiple tables we should wait to VOTable 2.0 and ensure we do not break backwards compatibility</div> <div id="magicdomid285">TD: multiple tables are allowed in VOtable but connections not really expressed</div> <div id="magicdomid287">LM: Internship at CDS to create a TAP that annotates on the fly</div>

<div id="magicdomid2"> *DM Workshop Session #4 Wrap up* </div>

<strong>Workshop overview [[%ATTACHURL%/MAY2021-ws41.pdf][MAY2021-ws41.pdf]]<br /></strong>
<div id="magicdomid4">Tuesday 2021 July 13 17h UTC</div>

<div id="magicdomid16">Participants :</div> <div id="magicdomid17">Laurent Michel, F. Bonnarel, Mark Taylor, Mark Cresitello-Dittmar, Tom Donaldson, JM Glorian, Mireille Louys, Jesus Salgado, Adrian Damian, Gerard Lemson</div>

<div id="magicdomid25">TD feel &gt;0 Can help for PyVO integration</div> <div id="magicdomid26">JMG we have to keep in touch</div> <div id="magicdomid29">TD: how do you validate wrt the xml schemata. votable element / instance ?</div> <div id="magicdomid30">how do the imports work?</div> <div id="magicdomid32">LM: shows unit tests on some instances in the mapping examples</div> <div id="magicdomid33">Separate the parts using name spaces</div> <div id="magicdomid34">at top of the votable there is a reference to the mapping syntax schema</div> <div id="magicdomid37">there must be a skip possibility to interpret the mapping , and the votable separately</div> <div id="magicdomid38">MT : can stiil ignore the mapping part and work with the VOtable part</div> <div id="magicdomid39">do we need an update of VOtable std?</div> <div id="magicdomid41">GL : Mapping part can be external to the VOTable, but it points into VOtable Fields for instance</div> <div id="magicdomid43">TD: how can we keep these parts separated</div> <div id="magicdomid44">do we want Votable and mapping syntax evolve separately for their future versions ?</div> <div id="magicdomid46">MT: what happens when the model is changed ?</div> <div id="magicdomid47">LM: if you rename or if you delete/ remove one attribute or instance</div> <div id="magicdomid49">JS : how difficult is it for a parser to cope with model changes ?</div> <div id="magicdomid50">TD: Parsing is relately straightforward</div> <div id="magicdomid51">depends on the model , more than on the syntax of the original VOTable.</div> <div id="magicdomid52">Parsing in a model agnostic way is easy but how to use properly this information could be tricky if the model is complex</div> <div id="magicdomid53">When the exercise for VO/DML was done to create a parser, it took not too long to create the parsing of the preamble</div> <div id="magicdomid54">propose reference interpretation to understand the problem pragmatically.</div> <div id="magicdomid55">GL: Also, the mapping tool could be adapted for this new syntax so the annotations are done in a more consistent way so parsing is simplified.</div> <div id="magicdomid57">MCD; About server side implmentations, obviusly ViZieR would be relevant implementation but we do not expect them to annotate all the catalogues</div> <div id="magicdomid58">JS: Maybe for luminosities is not so complex as they were already identified by Sebastien Derriere in the past and annotated in a temporary agreed syntax</div> <div id="magicdomid60">LM : model agnostic translation</div> <div id="magicdomid61">this was the experience of TD too in presenting syntax comparison in Santiago</div> <div id="magicdomid63">Topcat does not need the complex model structure to work</div> <div id="magicdomid64">Spectra tools are more suppposed to cope with Model complexity</div> <div id="magicdomid66">what if you would have a source model ? Could e.g. topcat/stilts be adapted to make use of the annotations for science use cases?</div> <div id="magicdomid68">Xmatch use-case : at best can interpret selection boxes, but much interpretation and choices come from the astrononer expertise</div> <div id="magicdomid69">Topcat does not do automatic conversions using metadata provided by the service. The user always select the information and use if in the way they consider it correct</div> <div id="magicdomid71">GL : try the Source model for annotation</div> <div id="magicdomid72">can be done by hand , crowd-source effort , or infered with rules from ucds / units</div> <div id="magicdomid74">reminder / sharing the merged syntax document</div> <div id="magicdomid75">Laurent to send the link around</div> <div id="magicdomid77">TD : how the Votable schema will change : we should clarify this</div> <div id="magicdomid78">can help to move the process further</div> <div id="magicdomid79">MT to chek how to interpret a VOtable with two imported schemata</div> <div id="magicdomid80">GL to test import of one schema?? please confirm , I may have not understood correctly (neither clear to me)</div>

FB On the mapping xml schema issue : 
<div id="magicdomid84">     in next version of VOTABLe we can introduce the optional VODML tag as a direct child of &lt;VOTABLE&gt;</div> <div id="magicdomid85">     the type of this element "VODML-type" would be defined in the external mapping schema will all its child elements</div> <div id="magicdomid86">     I think something like</div> <div id="magicdomid87">     &lt;VODML xsi-type="mapping:VODML-type" &gt; ....... &lt;/VODML&gt;</div> <div id="magicdomid88">     with mapping namespace imported and defined as whatever version of the mapping xml schema</div> <div id="magicdomid89"> </div>

*Chat Dump*
<div id="magicdomid85">chat 18:56:40 From Laurent Michel to Everyone:</div> <div id="magicdomid86">[[https://docs.google.com/presentation/d/1jvEeLPze750SaAYCUg_BH0Zv6SRW8ocgFI9vkMOEYtI/edit?usp=sharing]]</div> <div id="magicdomid87">18:56:51 From Laurent Michel to Everyone:</div> <div id="magicdomid88">etherpad [[https://us02web.zoom.us/j/89276419126?pwd=OHU0Y0dhd2Vmb3dFTkY5NzNmZGkxdz09]]</div> <div id="magicdomid89">18:57:11 From Laurent Michel to Everyone:</div> <div id="magicdomid90">etherpad https://pad.unistra.fr/p/DMWorkshop#4</div> <div id="magicdomid91">19:09:10 From Mark Taylor to Everyone:</div> <div id="magicdomid92">good summary of my concerns!</div> <div id="magicdomid93">19:30:13 From François Bonnarel to Everyone:</div> <div id="magicdomid94">concordia.xsd</div> <div id="magicdomid95">19:30:17 From François Bonnarel to Everyone:</div> <div id="magicdomid96">???*</div> <div id="magicdomid97">19:30:49 From Gerard Lemson to Everyone:</div> <div id="magicdomid98">hi all, sorry for being delayed.</div> <div id="magicdomid99">19:31:03 From Mark CD to Everyone:</div> <div id="magicdomid100">https://github.com/ivoa-std/ModelInstanceInVot/blob/master/schema/xsd/merged-syntax.xsd</div> <div id="magicdomid101">19:49:09 From François Bonnarel to Everyone:</div> <div id="magicdomid102">Where is the etherpad ?</div> <div id="magicdomid103">19:54:59 From Jean-Michel Glorian to Laurent Michel(Direct Message):</div> <div id="magicdomid104">Désolé, je dois partir du zoom. Comme je l'ai dit je n'ai pas eu le temps de regarder tout cela mais je vais essayer de voir ce que je peux faire avec le service de spectre polarbase que je maintiens et avec cassis coté client... Par conrte aucune idée au niveau du planning ...</div> <div id="magicdomid105">19:55:10 From Laurent Michel to Jean-Michel Glorian(Direct Message):</div> <div id="magicdomid106">etherpad [[https://us02web.zoom.us/j/89276419126?pwd=OHU0Y0dhd2Vmb3dFTkY5NzNmZGkxdz09]]</div> <div id="magicdomid107">19:55:56 From Jean-Michel Glorian to Laurent Michel(Direct Message):</div> <div id="magicdomid108">tu viens de m'nevoyer le lien à moi directement ...</div> <div id="magicdomid109">19:56:08 From Laurent Michel to Everyone:</div> <div id="magicdomid110">etherpad [[https://us02web.zoom.us/j/89276419126?pwd=OHU0Y0dhd2Vmb3dFTkY5NzNmZGkxdz09]]</div> <div id="magicdomid111">19:56:27 From Laurent Michel to Everyone:</div> <div id="magicdomid112">etherpad https://pad.unistra.fr/p/DMWorkshop#4</div>
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pdf	IVOA_DM_premeeting_20210517_v1.pdf	r1	manage	1583.1 K	2021-05-17 - 16:43	GerardLemson	VO-DML and Mapping
pdf	MAY2021-ws41.pdf	r1	manage	2638.1 K	2021-07-15 - 08:26	LaurentMichel	Wrap-up session
pdf	MangoViewONtwoTables.pdf	r1	manage	1311.1 K	2021-05-22 - 13:47	FrancoisBonnarel	Mango view on top of two tables with time variation
pdf	Workshop_pre-interop.pdf	r1	manage	585.8 K	2021-05-17 - 14:00	LaurentMichel
pdf	dm_workshop_preinterop.pdf	r1	manage	2883.3 K	2021-05-18 - 13:18	MarkCresitelloDittmar	Implementation review
pdf	markus.pdf	r1	manage	2876.6 K	2021-05-17 - 18:30	MarkusDemleitner	Lecture notes for Markus' talk
pdf	workshop_pre-interop_MANGO.pdf	r1	manage	2858.9 K	2021-05-17 - 14:00	LaurentMichel
pdf	workshop_pre-interop_MANGO2.pdf	r1	manage	2768.9 K	2021-07-15 - 08:25	LaurentMichel	MANGO + MIVOT