(r23) ObsLocTAP10RFC < IVOA

IVOA Web>WebPreferences>ObsLocTAP10RFC (revision 23) (raw view)~~EditAttach~~
---+ <span style="color: #0000ff;"><span style="text-decoration: underline;">ObsLocTAP</span></span> 1.0 Proposed Recommendation: Request for Comments

%TOC{depth="2"}%

---++ Summary

<span style="background-color: transparent;">Observation Locator Table Access Protocol (ObsLocTAP) document describes the necessary data model elements and the access protocol to discover metadata about observations for a given Astronomical Observatory through a uniform interface within the VO framework. The used data model makes reference to IVOA Observation Data Model elements (Louys, et al., 2017), removing the ones associated to datasets access, as these elements are not available yet for future observations that are planned, scheduled, performed but not archived. In this way, present standard is focused on the access to metadata related to the planning of a certain observatory, more than in the access to the scientific data products of a certain observation. Also, the data model described in the present standard will be focused on metadata discovery, useful for multiwavelength coordination observations.</span>

<span style="background-color: transparent;">The latest version of <span style="color: #0000ff;"><span style="text-decoration: underline;">ObsLocTAP</span></span></span><span style="background-color: transparent;"> can be found at:</span>
   * https://www.ivoa.net/documents/ObsLocTAP/20201007/index.html
Also, ObsLocTAP is inside the IVOA github repository at:

https://github.com/ivoa-std/ObsLocTAP
---+++ Reference Interoperable Implementations

Two separate reference implementations of server-side architecture exist at ESDC, through docker containers and at CFA:
---+++ <span style="background-color: transparent; color: #630000; font-size: 14.3px; font-weight: bold;">1- Integral - ESDC server</span>

<span style="background-color: transparent;">The ESAC Science Data Centre (ESDC) at ESAC is creating a new version of the Integral mission Science Archive (Monica Fernandez). Following the same phylosophy like the Gaia Archive, the server side is based on TAP. For this case, the Integral SOC is producing the table of the future planned Integral observations and this table is shared using the ObsLocTAP data model.</span>

The TAP URL location is at:

   * *https://ila.esac.esa.int/tap/tap/*
ObsLocTAP table is ivoa.obsplan. Some example queries are the following:
---++++++ <span style="background-color: transparent;">Discover columns for table ivoa.obsplan: </span>
<verbatim>SELECT * FROM tap_schema.columns WHERE table_name = 'ivoa.obsplan'  </verbatim>
---++++++ Discover observations for between 31/05/2020 and 08/09/2020:
<verbatim>SELECT * FROM ivoa.obsplan WHERE t_min > 59000 AND t_max < 59100  </verbatim>
---++++++ Discover observations that overlaps with a circle centered on the Crab:
<verbatim>SELECT * FROM ivoa.obsplan WHERE 1=INTERSECTS(s_region, CIRCLE('ICRS', 83.633080, 22.014500, 0.016666 ))</verbatim>

Service can be tested on any TAP client application (e.g. TopCat) or directly in your browser using TapHandle:

<a target="_blank" href="http://saada.unistra.fr/taphandle?url=https://ila.esac.esa.int/tap/tap/"> http://saada.unistra.fr/taphandle?url=https://ila.esac.esa.int/tap/tap/ </a>
---+++ <span style="background-color: transparent; color: #630000; font-size: 14.3px; font-weight: bold;"> 2- Docker containers:</span>

Two docker containers have been created (Jesus Salgado) in order to allow the creation of a ObsLocTAP server with very basic knowledge of the standard. The first one is a web server with a TAP Tuto instance. The second one is a PostgreSQL database, with a pgsphere module installed and a TAP_SCHEMA in line with ObsLocTAP datamodel.

These docker instances are explained here:
   * https://www.cosmos.esa.int/web/vovisobs_protocols/implementation-guides
For the execution, they can be invoked by:
<pre><verbatim>docker pull jsalgadodocker/pgsphere-obsplan
docker pull jsalgadodocker/tapserver
docker network create --driver=bridge db-network
docker image ls
docker run -p 8080:8080 --net=db-network --name tap <tapserver_image_id>
docker run -p 5432:5432 --net=db-network --name db  <pgsphere-obsplan_image_id></verbatim>
</pre>

---+++ 3- Chandra CFA server:

Chandra CFA TAP server has been adapted (Michael Tibbetts and Janet Evans) to create the ObsLocTAP ivoa.obsplan table to include the future Chandra observations. The service can be found here:

   * <span style="background-color: transparent; color: #630000; font-size: 14.3px; font-weight: bold;">https://cda.cfa.harvard.edu/cxctap/</span>
This is the first X-Ray observatories service implemented (similar services are already in on development for XMM-Newton and NuStar) what will allow the preparation of coordinated observations for these three observatories in a short-term.

Similar queries to Integral TAP server could be instantiated, e.g.:

---++++++ Discover observations for between 31/05/2020 and 08/09/2020:
<verbatim>SELECT * FROM ivoa.obsplan WHERE t_min > 59000 AND t_max < 59100  </verbatim>

This service has the caveat of not supporting geometrical operators yet (although this is not needed for most of the use cases)

Service can be tested on any TAP client application (e.g. TopCat) or directly in your browser using TapHandle:

<a target="_blank" href="http://saada.unistra.fr/taphandle?url=https%3A//cda.cfa.harvard.edu/cxctap/"> http://saada.unistra.fr/taphandle?url=https://cda.cfa.harvard.edu/cxctap/ </a>
---+++ 4- TOBY- Client Application

Also a client has been implemented by ESA (Emilio Salazar), located at:
   * http://integral.esa.int/toby/
<img width="400" alt="" src="%ATTACHURL%/Screenshot_2020-06-17_at_17.56.35.png" />

This graphical client application combines in a calendar view the output of ObsLocTAP servers for Integral, Chandra and the similar to ObsLocTAP servers (NuSTAR and XMM-Newton - in process of adaptation). Also, it shows the Visibility services output (ObjVisSAP) for some missions.

---++ Comments from the IVOA Community during RFC/TCG review period: 2020-10-19 - 2020-11-27

Minor comments on 14/05 PR1.0 have been covered on version 07/10.

The comments from the TCG members during the RFC/TCG review should be included in the next section.

In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your Wiki Name so that authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.

Additional discussion about any of the comments or responses can be conducted on the WG mailing list. However, please be sure to enter your initial comments here for full consideration in any future revisions of this document

   * Sample comment by TWiki.WikiName 
      * Response (by TWiki.WikiName)

---
---

---++ Comments from TCG member during the RFC/TCG Review Period: 2020-10-19 - 2020-11-27

WG chairs or vice chairs must read the Document, provide comments if any (including on topics not directly linked to the Group matters) or indicate that they have no comment.

IG chairs or vice chairs are also encouraged to do the same, althought their inputs are not compulsory.

---+++ TCG Chair & Vice Chair
---+++ [[IvoaApplications][Applications Working Group]]

---+++ [[IvoaDAL][Data Access Layer Working Group]]

The ObsLocTAP standard is a valuable one, as demonstrated by take up from the community and the resources the community itself invested in it. From the DAL perspective, it is another proof that the TAP specification can be leveraged to provide answers to different use cases.

DAL considers important the connection to and overlap with ObsCore elements.

DAL alerts, however, that the differences in registering these two *TAP solutions (tableset+utype in ObsLocTAP versus model-id in ObsTAP) lead to an extra join in RegTAP queries. A common solution (e.g., through TAPRegExt clarification and ObsCore change) should be seeked for in the future. We look forward to connecting to Registry WG to work on this.

DAL supports this standard to become a Recommendation.

Here we report to changes in the document to be fixed:
   * p10 - execution_status: UnScheduled -&gt; Unscheduled
   * p13 ucd for target_name should be meta.id;src - has a comma currently
Other proposed changes to the text and typo fixing will be sent as a Pull Request to the ObsLocTAP git repo.

-- IVOA.MarcoMolinaro - 2020-11-27

Thanks for your comments. Your pull request was accepted

-- IVOA.JesusSalgado - 2021-01-18
---+++ [[IvoaDataModel][Data Model Working Group]]

---+++ [[IvoaGridAndWebServices][Grid & Web Services Working Group]]

---+++ [[IvoaResReg][Registry Working Group]]

---+++ [[IvoaSemantics][Semantics Working Group]]

While Semantics, as far as I can tell, is largely unconcerned by this standard, from the perspective of validating implementations I have to say I'm a bit concerned that no obsloctap service exists in the Registry as of 2020-11-25. Having a few of them and having some code in the clients that uses that to discover them would make my approval a lot more heart-felt.

But then there's little that could go wrong there, I think, and so you have Semantics' vote.

If I may, here are a few remarks without my Semantics hat on; where I thought things are probably uncontentious, I've included fixes in https://github.com/ivoa-std/ObsLocTAP/pull/8

(1) In the PR, I'm re-formatting the proposed discovery query to look a bit less scary (at least to my taste); if you agree this style make things a bit more readable, I'd be happy to re-format the sample queries accordingly.

(2) Why do you explicitly scale the architecture diagram? I give you the text in it comes out a bit small, but changing the aspect ratio, I think, doesn't really improve things, and it'll look odd for people who know architecture diagrams from other standards. There's nothing wrong with changing the 0.9\textwidth from the template to \textwidth (or, with some precautions, even 1.1\textwidth) -- but you really shouldn't change the aspect ratio.

(3) Again on this, I'd advise to use the pattern from ivoatex/document.template to give an explicit caption to the architecture diagram and then reference it from the text; saying "the figure below" is sure to break in the next release, when the floating figure somehow ends up above that text. If you absolutely insist on "the figure below", un-float the figure (i.e., do away with the figure environment) -- but that will lead to bad page breaks.

(4) In general, I advise against too many rules in tables, and your column tables, I'd argue, are cases in point; but that may be a matter of taste. My PR, however, at least opens up the headline a bit, which I'd say helps the appearance a bit.

(5) I'd still advise to replace the data types in the TAP_SCHEMA.columns table with generic descriptions, i.e., (float), (string), (integer), and then explaining the the text something like: "The concrete types in the TAP_SCHEMA are up to the implementation; however, (float)-typed columns must contain floating point data (like DOUBLE PRECISION or REAL), (string)-typed columns must contain text (e.g, VARCHAR(n), CHAR(n)), and (integer)-typed columns must contain integers (e.g., INTEGER, BIGINT, SMALLINT)." [Note to self: we ought to have a cross-standard convention for this kind of thing]

(6) Since I suspect many actual use cases involve multiple regions of interest, I think giving an example involving uploads (perhaps using stilts syntax, as that can't be done quite as generically) would be a bonus.

(7) I'd not show the sample registry record in a section of its own. *If* you want it in the document, put it into an appendix. On the other hand, I'd say that's a classic candiate for auxiliaryurl (see https://ivoa.net/documents/Notes/IVOATex/20180814/NOTE-ivoatexDoc-1.2-20180814.html#tth_sEc3.6).

-- IVOA.MarkusDemleitner - 2020-11-25

Many thanks for your comments. All the editorial changes have been included into pull request:

https://github.com/ivoa-std/ObsLocTAP/pull/11

A registry record has been published for the Integral ObsLocTAP server (EuroVO Registry)

-- IVOA.JesusSalgado - 2021-01-18
---++++ Summary response

---+++ [[IvoaDCP][Data Curation & Preservation Interest Group]]

---+++ [[IvoaTheory][Education Interest Group]]

---+++ [[IvoaKDD][Knowledge Discovery Interest Group]]

---+++ [[IvoaSS][Solar System Interest Group]]

---+++ [[IvoaTheory][Theory Interest Group]]

---+++ [[IvoaVOEvent][Time Domain Interest Group]]

---+++ [[IvoaOps][Operations]]

This will be a useful standard, as confirmed by the level of community interest. But there are some issues that need to be addressed.

Reference implementations and validator tools:

   * *No validator is listed!* As the !DocStd requires, _"PRs being brought forward for promotion to REC should, when applicable, have at least two interoperable implementations and validation tools should be available."_ If you ask the author of =taplint= nicely he might be wiling to add an !ObsLocTAP validation stage, perhaps subject to some of the issues below being cleared up.
   * In the service at https://ila.esac.esa.int/tap/tap/, the =s_region= field when downloaded as a VOTable has a non-standard format (I think it's "<code>&lt;(ra-in-radians,dec-in-radians),radius&gt;</code>"). It would be nice to have this in some !DALI-compliant format, see e.g. !DALI 1.1 sec 3.3.6. (see also other =s_region= comments below).
   * The service at https://cda.cfa.harvard.edu/cxctap/ has some HTTP/HTTPS redirection issues that prevents it being used with TOPCAT (I have contacted Janet Evans about this - may be fixed 01/2021?). The =ivoa.obsplan= table from this service also has no =s_region= column. I haven't checked it for other deficiencies.
Document Content:

   * _"Until a new standard becomes authoritative for such fields, s_region will be serialised according to section 6 of TAP 1.0"._ !DALI section 3.3 does provide a standards-based way to serialize geometries ( =xtype="circle"=, =xtype="polygon"=); future !DALI versions are expected to support addtional xtypes "<code>multipolygon</code>", "<code>shape</code>", "<code>moc</code>". Since the !STC-S serialization from TAP 1.0 is deprecated/not-really-standardised I would suggest *either* to require !DALI-like serialization of =s_region= *or* allow either !DALI-like or !STC-S-like (with an appropriate xtype - xtype="ADQL:region"?) serialization. This point would probably benefit from input from DAL WG and/or !DALI authors (Pat).
   * I agree with Markus that more generic rather than specific data types for serialised results and TAP_SCHEMA.columns are a very good idea. I would suggest pseudo-types "integer", "floating point", "string", "region" and "timestamp" or similar, with references to !DALI for the serialization details. For instance the !ESAC reference implementation does not follow the requirement in sec 3.4 that target name has TAP_SCHEMA type "<code>CHAR(*)</code>", it uses "<code>VARCHAR</code>" instead, which is obviously quite reasonable.
   * I suppose that all the columns listed in the tables are supposed to be mandatory, to facilitate executing the same query against different services (though as written the two tables have different column lists, and the two reference implementations do not include all the listed columns, I assume both are mistakes). But this is not spelt out. I suggest to add explicit language saying that the obsplan table MUST include all the columns listed, even though null values may be permitted in some of the columns. There should also be some explicit policy on whether additional non-standard columns are permitted in the obsplan table (I suggest: yes).
   * Why are there two tables listing the column characteristics, one in section 3.1 and one in section 3.4, rather than doing it all in one place? This is an unnecessary opportunity for inconsistencies to arise (and I see at least one: =obs_release_date= exists in one table but not the other). Column name is duplicated; unit is duplicated but with slight differences ("unitless" vs. "") and data type is represented in different forms, which I argue above should be reduced in both cases to a generic direction (string, integer, floating point, region, timestamp). I agree that fitting all that information in one table would require some formatting ingenuity ... but unless there is a compelling reason to split this information into two parts, I would suggest to try to get them as close as possible together.
   * _"The planning exposure time (t_plan_exptime) has been added in order to show any possible inconsistency between the scheduled exposure time and the real performed exposure time (t_exptime). If an observation was performed without problems, both quantities must be exactly the same. Any discrepancy between these two values will reflect problems or deviations be tween scheduled observations and performed observations."_ I can imagine a case where the =t_plan_exptime= is an estimate and the actual =t_exptime= is within the planned error bounds but represents a more accurate result, in which case the test for discrepancy would appear to represent _"problems or deviations"_ that did not actually occur. But I'm not an observatory planning specialist, so if that's not a realistic scenario then fine.
   * _"Filtering of services using registration metadata will be described into the TAP Registration Extension definition for !ObsLocTAP services."_ I don't quite follow this sentence. Does it mean that an !ObsLocTAP registry extension standard will be submitted in the future as a separate document? (following discussion with Markus: he suggests removing this sentence or replacing by a pointer to section 5).
   * Section 4.4 examples use geometry function invocations like "<code>CIRCLE('ICRS', ra, dec)</code>". The initial string coordsys argument ("<code>'ICRS'</code>") will be deprecated in !ADQL 2.1 and in most cases doesn't do anything. To avoid confusion, especially with future !ADQL versions, write e.g. "<code>CIRCLE('', ra, dec)</code>" instead?
   * No bibliography is included in the formatted PDF version of the document, so all the =\cite= references in the text appear as "(?)".
   * Some examples in sec 4 could be made more readable using syntax "<code>&lt;target&gt; BETWEEN &lt;min&gt; AND &lt;max&gt;</code>" rather than "<code>&lt;target&gt; &lt; &lt;max&gt; AND &lt;target&gt; &gt; &lt;min&gt;</code>" (though semantics is not strictly identical)
   * I don't know if _"planification"_ is a word - replace by _"planning"_? But the meaning is clear, so if you want to keep it OK.
   * I suggested some hopefully uncontroversial typos at [[https://github.com/ivoa-std/ObsLocTAP/pull/9][PR#9]]
-- IVOA.MarkTaylor - 2020-11-27

Thanks Mark for your comments! I have included all the text changes on:

<a href="https://github.com/ivoa-std/ObsLocTAP/pull/14" title="PR#14">https://github.com/ivoa-std/ObsLocTAP/pull/14<br /></a>

for your review.

I have added here taplint as the main validator, as ObsLocTAP is a TAP server. I would be really appreciated if the taplint main author :) could add a specific ObsLocTAP check (perhaps on the obsplan table structure).

After discussing with various server implementors, I have decided to add as possible ways to serialize the regions STC-S and DALI. Integral will be migrated to on STC-S compliant server in a very short term.

I have been forced to maintain the two tables for the obsplan table description as it was really hard to put everything in one place but I have left one with only a description and another with all the technical descriptions so this would be the one people writing the server implementation will check more often.

About ranges, it I have maintain the &gt; min &lt; max as it looks to reflect better the overlap.

-- IVOA.JesusSalgado - 2021-01-21

   * Jesus, thanks for these changes, I think you've addressed all of my points apart from validation. Regarding the validator, I will add some code to taplint that checks !ObsLocTAP services are following the recommendations of the document, *subject to* incorporation of [[https://github.com/ivoa-std/ObsLocTAP/pull/15][my PR#15]] - I'm not insisting that you use the text I've provided, but I think this section needs to be modified a bit in order to state requirements that can be checked by a validator. -- IVOA.MarkTaylor - 2021-01-28
   * Mark, I have merged your proposed text with the current version. I find it quite clear. Thanks! -- IVOA.JesusSalgado - 2021-01-29
   * Thanks, action on me to come up with !ObsLocTAP validation in <code>taplint</code>.  I'll try to do that in the forseeable future (by end of 02/2021?). -- IVOA.MarkTaylor - 2021-01-29
      * Past experience indicates it is pretty likely that this activity will show up more issues with the document.

---+++ [[IvoaStdsDocsProc][Standards and Processes Committee]]

---++ TCG Vote : Vote_start_date - Vote_end_date

If you have minor comments (typos) on the last version of the document please indicate it in the Comments column of the table and post them in the TCG comments section above with the date.

| Group | Yes | No | Abstain | Comments |
| TCG | | | | |
| Apps | | | | |
| DAL | * | | | |
| DM | | | | |
| GWS | | | | |
| Registry | | | | |
| Semantics | * | | | Having registred services would be nice, though |
| DCP | | | | |
| KDIG | | | | |
| SSIG | | | | |
| Theory | | | | |
| TD | | | | |
| Ops | | | | |
| !StdProc | | | | |
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
png	Screenshot_2020-06-17_at_17.56.35.png	r1	manage	365.1 K	2020-06-17 - 16:00	JesusSalgado
Topic revision: r23 - 2021-02-04 - MarkTaylor
IVOA
Log in or Register
IVOA.net
Wiki Home
WebChanges
WebTopicList
WebStatistics
Twiki Meta & Help
IVOA
Know
Main
Sandbox
TWiki
TWiki intro
TWiki tutorial
User registration
Notify me
Working Groups
Interest Groups
Committees
Stds&Procs
www.ivoa.net
Documents
Events
Members
XML Schema