AstroPyVOMivot < IVOA

---+ # Follow-up of the discussions on the design of a AstroPyVO API processing annotated data.

At last Interop (April 2022), we have had a PyVO hands-on session where we have open a discussion on the requirements for a PyVO API processing annotated data.
<div id="magicdomid8">The minutes of that session are on GitHUB</div> <div id="magicdomid9">

   * https://github.com/ivoa/modelinstanceinvot-code/wiki/hack-a-thon-contributions
</div> <div id="magicdomid10">We all agreed that this new API must have no impact on the current one and that its design must be based on real use-cases.</div> <div id="magicdomid11">The expectation for using annotated data is not necessary to allow doing new things. It is rather to facilitate common operations.To move on, we have decided to start from some typical PyVOT codes and to figure out how to do the same things with a code based on annotated data.</div> <div id="magicdomid12">The rule is to write realistic pseudo-code.We are proposing now a brief discussion focused on a first sketch based on the proposal presented by Gilles in May 2021 and taking advantage of the experience we got while exercising MIVOT parsing.</div> <div id="magicdomid13">
   * https://github.com/ivoa/modelinstanceinvot-code/wiki/vizier-proposal
</div> <div id="magicdomid15">Meeting Wiki page:</div> <div id="magicdomid16">
   * https://github.com/ivoa/modelinstanceinvot-code/wiki/21_07_22/_edit
</div>
---++ Note of the meeting held on 2022 July 21

<span style="color: firebrick;"> *Go on the GitHub Wiki to get more details about the work progress* </span>

<strong><br /></strong>
<div id="magicdomid26"> *Attendees - 6* </div> <div id="magicdomid27">Laurent Michel LM, Renaud Savalle RS, Tom Donaldson TD, Omar Laurino OL, Mark CD MCD, Jesus Salgado JS</div>

<div id="magicdomid29"> *Meeting notes* </div>

<span style="color: cornflowerblue;"> </span>
<div id="magicdomid31"> _LM: Presentation of a brief introduction on the subject, PyVO-DM_ </div> <div id="magicdomid32"> _The proposed model facilitate the way to map DMs into VOTables and how to read these VOTables in a efficient way in python using astropy (and PyVO)_ </div> <div id="magicdomid33"> _In the client side, there are two things to do with this annotation: extract the mapping block adn and process this mapping block_ </div> <div id="magicdomid35"> _Also, write the python instances that map the data model instances (e.g position, photometry, etc)_ </div> <div id="magicdomid37"> _The focus of today is design and astropy API compliant with the native API and supports working with models_ </div> <div id="magicdomid39"> _As a result of the hackathon, pages under https://github.com/ivoa/modelinstanceinvot-code/wiki were created_ </div> <div id="magicdomid41">_Examples created by Vizier (Gilles Landais) described at https://github.com/ivoa/modelinstanceinvot-code/wiki/vizier-proposal__</div>

<div id="magicdomid42"> __as they have experience on the difficulties of mapping dm elements into vizier tables_ </div> <div id="magicdomid44"> _Gilles propose to iterate on measures on the resources and for each one obtain the values (there is another examople for spectral data that has not been explored further)_ </div> <div id="magicdomid46"> _There is an interpretation of this pseudocode into pyvo code including a query to the service, requesting dm_mapping. That could allow to created mangoinstances from the response and obtain a modelview using modelviewer_ </div> <div id="magicdomid48"> _That allows the navigation of the result in terms of mangoObjects._ </div> <div id="magicdomid49"> _It can be also iterated on the object parameters to get the values of the labels_ </div> <div id="magicdomid51"> _Example of the parsing of a PhotometryFilter using the model description_ </div> <div id="magicdomid53"> _MCD: the sentence containsMangoInstances could be false for, e.g. cubes or time series (the object is fully returned)_ </div> <div id="magicdomid55"> _OL: how is this compared with the library RAMA?_ </div> <div id="magicdomid57"> _TD: we need to have client code to navigate the vo/dml_ </div> <div id="magicdomid59"> _LM: the example showed is based on mango but the idea is to create astropy classes to do the parsing. The process has been divided in three phases 1 parse VO/Table 2 work with the parameters inside classes 3 create classes_ </div> <div id="magicdomid61"> _OL: Why you did not start with the library RAMA as most of the requirements are already implemented there?_ </div> <div id="magicdomid63"> _LM: it was started the exercise as RAMA was not compatible with the mapping syntax_ </div> <div id="magicdomid65"> _OL: yes, but this is a detail that could be implemented as other things are already there_ </div> <div id="magicdomid67"> _The main target for today is to discuss about the model components vs astropy components at_ </div> <div id="magicdomid68"> _https://github.com/ivoa/modelinstanceinvot-code/wiki/21_07_22_ </div> <div id="magicdomid70"> _MCD: if the idea is to create an skycoord instance from an annotation, the first thing is to study the constructor_ </div> <div id="magicdomid72"> _TD: it is an overloaded constructor and "humans" knowing the location of the columns create the instances using a simple constructor version_ </div> <div id="magicdomid73"> _There is a affiliated package called SpecUtils that could be explored_ </div> <div id="magicdomid74"> _We just draft some ideas on the aspect of the API during the last interop. We could check RAMA and I have some ideas on how to handle this in astropy_ </div> <div id="magicdomid76"> _LM: Table with the correspondance between the model instances and astropy types (in the case of polarization, photometry, etc it would be astropy quantity)_ </div> <div id="magicdomid78"> _For cases like PhotCal and Flag, there is not a clear correspondance with astropy_ </div> <div id="magicdomid79"> _Also there is not clear link with magnitudes and error, it does not support data grouping and there is not a way to iterate over the list of available measures_ </div> <div id="magicdomid81"> _MCD: it could be used an array for iteration_ </div> <div id="magicdomid82"> _LM: but associated values are difficult to link_ </div> <div id="magicdomid84"> _The path is to reuse API astropy classes, prevent the use of affiliated classes, extend astropy for quantities, extend to support, missing metadata, errors and data grouping_ </div> <div id="magicdomid86"> _TD: we should not be just limited to astropy classes as many metadata is not included into the astropy classes and it should be available for clients_ </div> <div id="magicdomid87"> _I think astropy core team will resist to include all these new classes but what we can do is upgrade the VOTable parser ensuring that we can load all the required metadata_ </div> <div id="magicdomid88"> _PyVO is the natural place to put these extended classes_ </div> <div id="magicdomid90"> _LM: so the idea would be to extend astropy adding these classes to pyVO_ </div>

---++ Community Notes

---+++ Previous Work

I note in passing that (for a simpler annotation) I've implemented an
API and showed how it could be used in an astropy branch: https://github.com/msdemlei/astropy

And I would agree that most of this needs to go into astropy rather than
pyVO, in particular basically everything dealing dealing with the
annotation of coordinate metadata and error annotation.

<!--
      * Set ALLOWTOPICRENAME = IVOA.TWikiAdminGroup
-->
This topic: IVOA > WebHome > IvoaDataModel > AstroPyVOMivot
Topic revision: r3 - 2022-07-25 - MarkusDemleitner