(r54) ObsLocTAP10RFC < IVOA

IVOA Web>WebPreferences>ObsLocTAP10RFC (revision 54) (raw view)~~EditAttach~~
---+ <span style="color: #0000ff;"><span style="text-decoration: underline;">ObsLocTAP</span></span> 1.0 Proposed Recommendation: Request for Comments

%TOC{depth="2"}%

---++ Summary

<span style="background-color: transparent;">Observation Locator Table Access Protocol (ObsLocTAP) document describes the necessary data model elements and the access protocol to discover metadata about observations for a given Astronomical Observatory through a uniform interface within the VO framework. The used data model makes reference to IVOA Observation Data Model elements (Louys, et al., 2017), removing the ones associated to datasets access, as these elements are not available yet for future observations that are planned, scheduled, performed but not archived. In this way, present standard is focused on the access to metadata related to the planning of a certain observatory, more than in the access to the scientific data products of a certain observation. Also, the data model described in the present standard will be focused on metadata discovery, useful for multiwavelength coordination observations.</span>

<span style="background-color: transparent;">The latest version of <span style="color: #0000ff;"><span style="text-decoration: underline;">ObsLocTAP</span></span></span><span style="background-color: transparent;"> can be found at:</span>
   * https://www.ivoa.net/documents/ObsLocTAP/20210609/index.html
Also, ObsLocTAP working version is inside the IVOA github repository at:

https://github.com/ivoa-std/ObsLocTAP

(this version could contain changes and corrections not included yet into the /documents version
---+++ Reference Interoperable Implementations

Two separate reference implementations of server-side architecture exist at ESDC, through docker containers and at CFA:
---+++ <span style="background-color: transparent; color: #630000; font-size: 14.3px; font-weight: bold;">1- Integral - ESDC server</span>

<span style="background-color: transparent;">The ESAC Science Data Centre (ESDC) at ESAC is creating a new version of the Integral mission Science Archive (Monica Fernandez). Following the same phylosophy like the Gaia Archive, the server side is based on TAP. For this case, the Integral SOC is producing the table of the future planned Integral observations and this table is shared using the ObsLocTAP data model.</span>

The TAP URL location is at:

   * *https://isla.esac.esa.int/tap/tap/*
ObsLocTAP table is ivoa.obsplan. Some example queries are the following:
---++++++ <span style="background-color: transparent;">Discover columns for table ivoa.obsplan: </span>
<verbatim>SELECT * FROM tap_schema.columns WHERE table_name = 'ivoa.obsplan'  </verbatim>
---++++++ Discover observations for between 31/05/2020 and 08/09/2020:
<verbatim>SELECT * FROM ivoa.obsplan WHERE t_min > 59000 AND t_max < 59100  </verbatim>
---++++++ Discover observations that overlaps with a circle centered on the Crab:
<verbatim>SELECT * FROM ivoa.obsplan WHERE 1=INTERSECTS(s_region, CIRCLE('ICRS', 83.633080, 22.014500, 0.016666 ))</verbatim>

Service can be tested on any TAP client application (e.g. TopCat) or directly in your browser using TapHandle:

<a target="_blank" href="http://saada.unistra.fr/taphandle?url=https://isla.esac.esa.int/tap/tap/"> http://saada.unistra.fr/taphandle?url=https://isla.esac.esa.int/tap/tap/ </a>
---+++ <span style="background-color: transparent; color: #630000; font-size: 14.3px; font-weight: bold;"> 2- Docker containers:</span>

Two docker containers have been created (Jesus Salgado) in order to allow the creation of a ObsLocTAP server with very basic knowledge of the standard. The first one is a web server with a TAP Tuto instance. The second one is a PostgreSQL database, with a pgsphere module installed and a TAP_SCHEMA in line with ObsLocTAP datamodel.

These docker instances are explained here:
   * https://www.cosmos.esa.int/web/vovisobs_protocols/implementation-guides
For the execution, they can be invoked by:
<pre><verbatim>docker pull jsalgadodocker/pgsphere-obsplan
docker pull jsalgadodocker/tapserver
docker network create --driver=bridge db-network
docker image ls
docker run -p 8080:8080 --net=db-network --name tap <tapserver_image_id>
docker run -p 5432:5432 --net=db-network --name db  <pgsphere-obsplan_image_id></verbatim>
</pre>

---+++ 3- Chandra CFA server:

Chandra CFA TAP server has been adapted (Michael Tibbetts and Janet Evans) to create the ObsLocTAP ivoa.obsplan table to include the future Chandra observations. The service can be found here:

   * <span style="background-color: transparent; color: #630000; font-size: 14.3px; font-weight: bold;">https://cda.cfa.harvard.edu/cxctap/</span>
This is the first X-Ray observatories service implemented (similar services are already in on development for XMM-Newton and NuStar) what will allow the preparation of coordinated observations for these three observatories in a short-term.

Similar queries to Integral TAP server could be instantiated, e.g.:

---++++++ Discover observations for between 31/05/2020 and 08/09/2020:
<verbatim>SELECT * FROM ivoa.obsplan WHERE t_min > 59000 AND t_max < 59100  </verbatim>

This service has the caveat of not supporting geometrical operators yet (although this is not needed for most of the use cases)

Service can be tested on any TAP client application (e.g. TopCat) or directly in your browser using TapHandle:

<a target="_blank" href="http://saada.unistra.fr/taphandle?url=https%3A//cda.cfa.harvard.edu/cxctap/"> http://saada.unistra.fr/taphandle?url=https://cda.cfa.harvard.edu/cxctap/ </a>
---+++ 4- TOBY- Client Application

Also a client has been implemented by ESA (Emilio Salazar), located at:
   * http://integral.esa.int/toby/
<img width="400" alt="" src="%ATTACHURL%/Screenshot_2020-06-17_at_17.56.35.png" />

This graphical client application combines in a calendar view the output of ObsLocTAP servers for Integral, Chandra and the similar to ObsLocTAP servers (NuSTAR and XMM-Newton - in process of adaptation). Also, it shows the Visibility services output (ObjVisSAP) for some missions.

---+++ Validator

The [[http://www.starlink.ac.uk/stilts/sun256/taplint.html][taplint]] TAP validator tool now includes !ObsLocTAP validation in its *LOC* stage; this will be present in STILTS versions &gt;3.4, and in the mean time is available in pre-release at [[ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/stilts.jar]]. It checks the required =ivoa.obsplan= table utype; the presence of required columns with their datatypes, units, utypes and UCDs; declaration of required geometry functions; and content and nullability of columns with restricted values.

-- IVOA.MarkTaylor - 2021-04-08

---+++ Registration example

A real ObsLocTAP registration could be found at the following entry:

<span style="background-color: transparent;">ivo://esavo/isla/obsloctap</span>

<span style="background-color: transparent;">-- IVOA.JesusSalgado - 2021-05-24</span>
---++ Comments from the IVOA Community during RFC/TCG review period: 2020-10-19 - 2020-11-27

Minor comments on 14/05 PR1.0 have been covered on version 07/10.

The comments from the TCG members during the RFC/TCG review should be included in the next section.

In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your Wiki Name so that authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.

Additional discussion about any of the comments or responses can be conducted on the WG mailing list. However, please be sure to enter your initial comments here for full consideration in any future revisions of this document

   * Sample comment by TWiki.WikiName 
      * Response (by TWiki.WikiName)

---
---

---++ Comments from TCG member during the RFC/TCG Review Period: 2020-10-19 - 2020-11-27

WG chairs or vice chairs must read the Document, provide comments if any (including on topics not directly linked to the Group matters) or indicate that they have no comment.

IG chairs or vice chairs are also encouraged to do the same, althought their inputs are not compulsory.

---+++ TCG Chair & Vice Chair

The relational mapping of the model is comparable to ObsCore, but I would like to echo some comments below about the s_region column. The DALI spec provides standard alternatives for the serialisation of values (circle and polygon are srtandard in DALI-1.1 and shape - polymorphism - is in WD-DALI-1.2). It would be much better if this standard didn't tie itself so tightly to past practices (adql REGION in TAP-1.0 section 6 in particular) and versions of standards so that as new mechanisms become available (in DALI or TAP) those could be used by implementers of ObsLocTAP. The same could be (was) said about ObsCore but at the time of the last revision the evolution of DALI and TAP was less clear. Specifically with respect to s_region, implementers should be allowed to return "adql REGION" aka ther non-standard STC-S from a TAP-1.0 service,or they could return DALI-1.1 polygon (or circle if appropriate), and in future they should be allowed to start using DALI-1.2 shape once that becomes a REC.

Technicality: The table on p13 (tap_schema section) doesn't have a caption or table number to reference.

p3 table: In general, the datatype values in this table are not very explicit. Since TAP-1.1, DALI and TAP (tap_schema) use the VOTable type system, so the effective type is specified by one or more of (votable) datatype, arraysize, and xtype. For example, a "polygon" is really datatype="double" arraysize="*" xtype="polygon" (or "float"). <br /><br />p13 table: I know what the doc means by "Floating Point" and "INTEGER", but that should be stated explicitly in reference to that table. I would suggest using the type names from the IVOA base model (in VO-DML) because those were carefull chosen to be ambiguous about size and leave size up to implementers. I think that is "real" and "integer" but that should be verified and it should be explained that implementers can chose the concrete type (eg float or double).

p13 table: A little care needs to be taken in spedcifying "CHAR(*)" columns to make it clear that an implementer can place a limit (eg datatype="char" arraysize="32*") and don't have to use arraysize="*". This is something where services may fail validation because they provide additional info to clients and the validator took that datatype description too literally, so it's good to make it clear that services are allowed to do that.

So mainly: relax the spec so it isn't so tighly tied to current RECs so that future enhancements (eg DALI) can be used. And clarify that table on p13 because the tap_schema descirption is the main thing.

-- IVOA.PatrickDowler - 2021-05-11

Adding some changes at:

https://github.com/ivoa-std/ObsLocTAP/pull/22

   * Relaxed link with certain DALI version
   * Tables with labels
   * After comments from other groups chairs and following their advise, now only an "abstract" type declaration is done (real, integer, string, region). Exact mapping would be in line with other VO protocols
-- IVOA.JesusSalgado - 2021-05-25

The changes in the PR are OK.

Approved. -- IVOA.PatrickDowler - 2021-05-27
---+++ [[IvoaApplications][Applications Working Group]]

This document seems generally well-written and understandable. It has clearly benefitted from the strong community involvement which also indicates that it's likely to be a useful standard.

I have done some very limited testing of the service implementations and found a few very fixable issues. I have not exercised the Docker containers or Toby (but hope to do so).

In addition to the items Mark identified, I saw that for the Chandra service:

<div id="_mcePaste">

   * Most utypes missing.
   * Column t_max expected ucd "time.end;obs.exposure", but TAP_SCHEMA had "time.stop;obs.exposure"
   * Column em_res_power expected ucd "spect.resolution", but TAP_SCHEMA had "spec.resolution"
   * Column pol_xel expected ucd "meta.number", but TAP_SCHEMA had "phot.count"
   * Column obs_release_date expected ucd "time.release;obs.exposure", but TAP_SCHEMA had "time.release"
</div>

Some minor editorial comments based on the 2021-02-04 preview on github:

   * For me the orientation is 90 degrees off for the tables on pages 10 and 14.

   * The reference to DALI in the reference section is to version 1.0 instead of 1.1 as mentioned in the s_region description.

   * The s_region section on page 11 includes, "In the WHERE clause, s_region constraints will generally make use of ADQL geometries (CIRCLE, POLYGON, POINT) and the geometry predicate functions (CONTAINS, INTERSECTS)." But section 3.3 says, "MUST provide, at least, the following ADQL (Osuna and Ortiz et al. 2008) geometrical features: CIRCLE, POLYGON, INTERSECTS and CONTAINS." Is "POINT" intentionally missing from section 3.3?

   * s_region can be returned using the DALI 1.1 specification. This is a new format compared to that mentioned in ObsCore, especially for TAP 1.0. I understand the move forward as recommended by DALI and TAP 1.1. But since the field name "s_region" is the same as in ObsCore, there could be confusion there for clients, who may not be expecting the DALI format. Perhaps it's worth an extra sentence in the s_region section explaining the evolution of that serialization.
-- IVOA.TomDonaldson - 2021-02-20

Thanks for your comments. About the Chandra implementation, the team is looking for resources to modify all the identified discrepancies from the standard.

About the comments on the text, I have created a new version and a new pull request:

https://github.com/ivoa-std/ObsLocTAP/pull/17

setting you as reviewers to identify the changes I have done folllowing your recommendations.

Tables orientation is rotated 90 degrees on purpose as the big table content was easier to be displayed landscape.

Thanks again!

-- IVOA.JesusSalgado - 2021-03-03

Pull request merged

-- IVOA.JesusSalgado - 2021-03-04

Thank you for these changes! I approve.

-- IVOA.TomDonaldson - 2021-05-27

---+++ [[IvoaDAL][Data Access Layer Working Group]]

The ObsLocTAP standard is a valuable one, as demonstrated by take up from the community and the resources the community itself invested in it. From the DAL perspective, it is another proof that the TAP specification can be leveraged to provide answers to different use cases.

DAL considers important the connection to and overlap with ObsCore elements.

DAL alerts, however, that the differences in registering these two *TAP solutions (tableset+utype in ObsLocTAP versus model-id in ObsTAP) lead to an extra join in RegTAP queries. A common solution (e.g., through TAPRegExt clarification and ObsCore change) should be seeked for in the future. We look forward to connecting to Registry WG to work on this.

DAL supports this standard to become a Recommendation.

Here we report to changes in the document to be fixed:
   * p10 - execution_status: UnScheduled -&gt; Unscheduled
   * p13 ucd for target_name should be meta.id;src - has a comma currently
Other proposed changes to the text and typo fixing will be sent as a Pull Request to the ObsLocTAP git repo.

-- IVOA.MarcoMolinaro - 2020-11-27

Thanks for your comments. Your pull request was accepted

-- IVOA.JesusSalgado - 2021-01-18
---+++ [[IvoaDataModel][Data Model Working Group]]

After checking comments from all the TCG colleagues, nothing really serious seems to affect the spec so we also provide the approaval

-- IVOA.JesusSalgado - 2021-05-25
---+++ [[IvoaGridAndWebServices][Grid & Web Services Working Group]]

The standard is well written, clear and useful. It is not strictly related to the GWS standards and activity and I have no

particular comment from that side. Just a warning about the actual possibility to access information about planned and scheduled

observations that maybe sometimes remain "private" according to the policies of the different observatories.

A few typos:

   * page 6 section 2, second line "fulfill" --&gt; o fulfil
   * page 4 last line making use --&gt; by making use
   * page 9 characterization bullet: associated to --&gt; associated with (also in other places in the PDF doc)
   * page 11 line 16: according DALI --&gt; according to DALI
Changes merged into master at https://github.com/ivoa-std/ObsLocTAP -- IVOA.JesusSalgado - 2021-05-24
---+++ [[IvoaResReg][Registry Working Group]]

From the Registry POV, this is well put together. Having looked through the ESAVO implementation registered resource (ivo://esavo/isla/obsloctap) as an example, I approve.

-- IVOA.TheresaDower - 2021-05-24
---+++ [[IvoaSemantics][Semantics Working Group]]

While Semantics, as far as I can tell, is largely unconcerned by this standard, from the perspective of validating implementations I have to say I'm a bit concerned that no obsloctap service exists in the Registry as of 2020-11-25. Having a few of them and having some code in the clients that uses that to discover them would make my approval a lot more heart-felt.

But then there's little that could go wrong there, I think, and so you have Semantics' vote.

If I may, here are a few remarks without my Semantics hat on; where I thought things are probably uncontentious, I've included fixes in https://github.com/ivoa-std/ObsLocTAP/pull/8

(1) In the PR, I'm re-formatting the proposed discovery query to look a bit less scary (at least to my taste); if you agree this style make things a bit more readable, I'd be happy to re-format the sample queries accordingly.

(2) Why do you explicitly scale the architecture diagram? I give you the text in it comes out a bit small, but changing the aspect ratio, I think, doesn't really improve things, and it'll look odd for people who know architecture diagrams from other standards. There's nothing wrong with changing the 0.9\textwidth from the template to \textwidth (or, with some precautions, even 1.1\textwidth) -- but you really shouldn't change the aspect ratio.

(3) Again on this, I'd advise to use the pattern from ivoatex/document.template to give an explicit caption to the architecture diagram and then reference it from the text; saying "the figure below" is sure to break in the next release, when the floating figure somehow ends up above that text. If you absolutely insist on "the figure below", un-float the figure (i.e., do away with the figure environment) -- but that will lead to bad page breaks.

(4) In general, I advise against too many rules in tables, and your column tables, I'd argue, are cases in point; but that may be a matter of taste. My PR, however, at least opens up the headline a bit, which I'd say helps the appearance a bit.

(5) I'd still advise to replace the data types in the TAP_SCHEMA.columns table with generic descriptions, i.e., (float), (string), (integer), and then explaining the the text something like: "The concrete types in the TAP_SCHEMA are up to the implementation; however, (float)-typed columns must contain floating point data (like DOUBLE PRECISION or REAL), (string)-typed columns must contain text (e.g, VARCHAR(n), CHAR(n)), and (integer)-typed columns must contain integers (e.g., INTEGER, BIGINT, SMALLINT)." [Note to self: we ought to have a cross-standard convention for this kind of thing]

(6) Since I suspect many actual use cases involve multiple regions of interest, I think giving an example involving uploads (perhaps using stilts syntax, as that can't be done quite as generically) would be a bonus.

(7) I'd not show the sample registry record in a section of its own. *If* you want it in the document, put it into an appendix. On the other hand, I'd say that's a classic candiate for auxiliaryurl (see https://ivoa.net/documents/Notes/IVOATex/20180814/NOTE-ivoatexDoc-1.2-20180814.html#tth_sEc3.6).

-- IVOA.MarkusDemleitner - 2020-11-25

Many thanks for your comments. All the editorial changes have been included into pull request:

https://github.com/ivoa-std/ObsLocTAP/pull/11

A registry record has been published for the Integral ObsLocTAP server (EuroVO Registry)

-- IVOA.JesusSalgado - 2021-01-18
---++++ Summary response

---+++ [[IvoaDCP][Data Curation & Preservation Interest Group]]

---+++ [[IvoaTheory][Education Interest Group]]

---+++ [[IvoaKDD][Knowledge Discovery Interest Group]]

---+++ [[IvoaRadio][RadioAstronomy Interest Group]]

Although our InterestGroup was created well after the specification was planned and drafted, we appreciate that severas radio astronomers participated to the writing and made this document useful for our field. We approve it. A few minor remarks may be taken into account before publication.

(1) in table 2, category, priority, execution_status and tracking_type miss both utypes and ucds. we wonder if some Provenance.ObsConfig* or Provenance.Proposal* utypes could not be invented, and category, at least, could have a "meta.code" ucd.

(2) in section 4.1 "Discovery of observations scheduled or planned for a certain observatory" the ADQL query doesn't show anything restricting explicitly the query to a certain facility.

(3) in section 4.2 "Identification of planned observations in a certain spectral range for a certain astronomical object" the ADQL query doesn't show anything restricting explicitly the query to a certain target

(4) in the registry record example (Appendix B) some columns miss existing utypes.

-- IVOA.FrancoisBonnarel - 2021-07-01 and IVOA.MarkLacy - 2021-07-01

---+++ [[IvoaSS][Solar System Interest Group]]

The document is well written and is very useful for generic sky coordinate observations, but this will only serve a small fraction of the Solar-System observations. For Earth based observations of solar system objects, RA/Dec coordinates are acceptable for the observation start time, but there is missing "tracking_type" parameter to advertise for "solar-system object tracking". This extra parameter would also require in turn to have a "target_name" parameter, to advertise what target we are tracking.

The next level would be to be able to advertise observations from space platforms aiming at solar-system bodies, with planetary frame coordinate pointing. But that'd be for a future version.

NB: This "tracking_type" parameter could also be useful for radio astronomy observation planning, by the way. On LOFAR or NenuFAR, there are 3 tracking modes for observation: a classical RA/Dec tracking mode, a solar-system object tracking mode, and a transit mode (fixed in Azimuth/Elevation).

-- IVOA.BaptisteCecconi - 2021-05-25

Many thanks for your comment. I am checking with some of the authors the solution proposed.

-- IVOA.JesusSalgado - 2021-05-26

Please take a look into the changes at:

https://github.com/ivoa-std/ObsLocTAP/pull/23

to see if this fulfills the science use case you mentioned

-- IVOA.JesusSalgado - 2021-05-27

---+++ [[IvoaTheory][Theory Interest Group]]

---+++ [[IvoaVOEvent][Time Domain Interest Group]]

---+++ [[IvoaOps][Operations]]

This will be a useful standard, as confirmed by the level of community interest. But there are some issues that need to be addressed.

Reference implementations and validator tools:

   * *No validator is listed!* As the !DocStd requires, _"PRs being brought forward for promotion to REC should, when applicable, have at least two interoperable implementations and validation tools should be available."_ If you ask the author of =taplint= nicely he might be wiling to add an !ObsLocTAP validation stage, perhaps subject to some of the issues below being cleared up.
   * In the service at https://ila.esac.esa.int/tap/tap/, the =s_region= field when downloaded as a VOTable has a non-standard format (I think it's "<code>&lt;(ra-in-radians,dec-in-radians),radius&gt;</code>"). It would be nice to have this in some !DALI-compliant format, see e.g. !DALI 1.1 sec 3.3.6. (see also other =s_region= comments below).
   * The service at https://cda.cfa.harvard.edu/cxctap/ has some HTTP/HTTPS redirection issues that prevents it being used with TOPCAT (I have contacted Janet Evans about this - may be fixed 01/2021?). The =ivoa.obsplan= table from this service also has no =s_region= column. I haven't checked it for other deficiencies.
Document Content:

   * _"Until a new standard becomes authoritative for such fields, s_region will be serialised according to section 6 of TAP 1.0"._ !DALI section 3.3 does provide a standards-based way to serialize geometries ( =xtype="circle"=, =xtype="polygon"=); future !DALI versions are expected to support addtional xtypes "<code>multipolygon</code>", "<code>shape</code>", "<code>moc</code>". Since the !STC-S serialization from TAP 1.0 is deprecated/not-really-standardised I would suggest *either* to require !DALI-like serialization of =s_region= *or* allow either !DALI-like or !STC-S-like (with an appropriate xtype - xtype="ADQL:region"?) serialization. This point would probably benefit from input from DAL WG and/or !DALI authors (Pat).
   * I agree with Markus that more generic rather than specific data types for serialised results and TAP_SCHEMA.columns are a very good idea. I would suggest pseudo-types "integer", "floating point", "string", "region" and "timestamp" or similar, with references to !DALI for the serialization details. For instance the !ESAC reference implementation does not follow the requirement in sec 3.4 that target name has TAP_SCHEMA type "<code>CHAR(*)</code>", it uses "<code>VARCHAR</code>" instead, which is obviously quite reasonable.
   * I suppose that all the columns listed in the tables are supposed to be mandatory, to facilitate executing the same query against different services (though as written the two tables have different column lists, and the two reference implementations do not include all the listed columns, I assume both are mistakes). But this is not spelt out. I suggest to add explicit language saying that the obsplan table MUST include all the columns listed, even though null values may be permitted in some of the columns. There should also be some explicit policy on whether additional non-standard columns are permitted in the obsplan table (I suggest: yes).
   * Why are there two tables listing the column characteristics, one in section 3.1 and one in section 3.4, rather than doing it all in one place? This is an unnecessary opportunity for inconsistencies to arise (and I see at least one: =obs_release_date= exists in one table but not the other). Column name is duplicated; unit is duplicated but with slight differences ("unitless" vs. "") and data type is represented in different forms, which I argue above should be reduced in both cases to a generic direction (string, integer, floating point, region, timestamp). I agree that fitting all that information in one table would require some formatting ingenuity ... but unless there is a compelling reason to split this information into two parts, I would suggest to try to get them as close as possible together.
   * _"The planning exposure time (t_plan_exptime) has been added in order to show any possible inconsistency between the scheduled exposure time and the real performed exposure time (t_exptime). If an observation was performed without problems, both quantities must be exactly the same. Any discrepancy between these two values will reflect problems or deviations be tween scheduled observations and performed observations."_ I can imagine a case where the =t_plan_exptime= is an estimate and the actual =t_exptime= is within the planned error bounds but represents a more accurate result, in which case the test for discrepancy would appear to represent _"problems or deviations"_ that did not actually occur. But I'm not an observatory planning specialist, so if that's not a realistic scenario then fine.
   * _"Filtering of services using registration metadata will be described into the TAP Registration Extension definition for !ObsLocTAP services."_ I don't quite follow this sentence. Does it mean that an !ObsLocTAP registry extension standard will be submitted in the future as a separate document? (following discussion with Markus: he suggests removing this sentence or replacing by a pointer to section 5).
   * Section 4.4 examples use geometry function invocations like "<code>CIRCLE('ICRS', ra, dec)</code>". The initial string coordsys argument ("<code>'ICRS'</code>") will be deprecated in !ADQL 2.1 and in most cases doesn't do anything. To avoid confusion, especially with future !ADQL versions, write e.g. "<code>CIRCLE('', ra, dec)</code>" instead?
   * No bibliography is included in the formatted PDF version of the document, so all the =\cite= references in the text appear as "(?)".
   * Some examples in sec 4 could be made more readable using syntax "<code>&lt;target&gt; BETWEEN &lt;min&gt; AND &lt;max&gt;</code>" rather than "<code>&lt;target&gt; &lt; &lt;max&gt; AND &lt;target&gt; &gt; &lt;min&gt;</code>" (though semantics is not strictly identical)
   * I don't know if _"planification"_ is a word - replace by _"planning"_? But the meaning is clear, so if you want to keep it OK.
   * I suggested some hopefully uncontroversial typos at [[https://github.com/ivoa-std/ObsLocTAP/pull/9][PR#9]]
-- IVOA.MarkTaylor - 2020-11-27

Thanks Mark for your comments! I have included all the text changes on:

<a href="https://github.com/ivoa-std/ObsLocTAP/pull/14" title="PR#14">https://github.com/ivoa-std/ObsLocTAP/pull/14<br /></a>

for your review.

I have added here taplint as the main validator, as ObsLocTAP is a TAP server. I would be really appreciated if the taplint main author :) could add a specific ObsLocTAP check (perhaps on the obsplan table structure).

After discussing with various server implementors, I have decided to add as possible ways to serialize the regions STC-S and DALI. Integral will be migrated to on STC-S compliant server in a very short term.

I have been forced to maintain the two tables for the obsplan table description as it was really hard to put everything in one place but I have left one with only a description and another with all the technical descriptions so this would be the one people writing the server implementation will check more often.

About ranges, it I have maintain the &gt; min &lt; max as it looks to reflect better the overlap.

-- IVOA.JesusSalgado - 2021-01-21

   * Jesus, thanks for these changes, I think you've addressed all of my points apart from validation. Regarding the validator, I will add some code to taplint that checks !ObsLocTAP services are following the recommendations of the document, *subject to* incorporation of [[https://github.com/ivoa-std/ObsLocTAP/pull/15][my PR#15]] - I'm not insisting that you use the text I've provided, but I think this section needs to be modified a bit in order to state requirements that can be checked by a validator. -- IVOA.MarkTaylor - 2021-01-28
   * Mark, I have merged your proposed text with the current version. I find it quite clear. Thanks! -- IVOA.JesusSalgado - 2021-01-29
   * Thanks, action on me to come up with !ObsLocTAP validation in =taplint=. I'll try to do that in the forseeable future (by end of 02/2021?). -- IVOA.MarkTaylor - 2021-01-29 
      * Past experience indicates it is pretty likely that this activity will show up more issues with the document.
I'm making progress with the validator. The more careful reading required to implement this leads to some more comments on the text:

   * The table name of "ivoa.obsplan" is first introduced only in passing in section 3.4. To people familiar with e.g. !ObsCore it's probably obvious what's going on, but I suggest making it explicit earlier in the document that an !ObsLocTAP service is one which contains a table named "ivoa.obsplan" (and which follows the requirements stated).
   * The standard doesn't say anything about what columns are allowed to have NULL values. Maybe they all are, which is OK, but I suggest that the nullability rules are made explicit.
   * The end of Section 3.1 says: _"Initially, observations are inserted as planned into the system. The final state will be one of executed or aborted."_ I suppose that the terms _"planned"_, _"executed"_ and _"aborted"_ here are supposed to refer to values of the controlled vocabulary listed in the description of the =execution_status= field in the table. But the capitalisation is different, and no state with a name like "executed" is present in the tabulated list. Please make it typographically clear what states are represented here (e.g. by quoting or using a different font, with exact matches to the controlled terms).
   * Section 3.2 says: _"For confidential observations that need to be hidden from the planning point of view, the period should be shown as blocked. That means that an entry should appear with the start time, end time and duration as relevant. The facility name should also be provided."_ I think I understand roughly what's meant here, but the text is confusing to me; I don't understand what "blocked" means. Can you rephrase to be more explicit about what you can/should put in what fields in this situation?
   * A couple of the utypes (for =facility_name= and =instrument_name=) in the table in section 3.4 have different capitalisation than the corresponding items in !ObsCore. Actually I can't remember if utypes are supposed to be case-sensitive (anyone?) but it would be better for them to match.
The validator is not finished yet, but testing so far reveals quite a few items of non-compliance for the *Integral ESDC server*. Since this is a reference implementation, these should be fixed before the standard can advance. Items spotted so far:

   * most (all?) of the columns have UCD values reported as Utypes and vice versa
   * the service metadata as declared in =TAP_SCHEMA.tables= and the =/tables= endpoint does not list the utype of the ivoa.obsplan table with the value mandated in section 5 of the standard
   * the mandatory =obs_release_date= column is missing
   * the =category= column contains some values not in the list of possibilities listed in the standard: "<code>TOO</code>", "<code>Normal</code>", "<code>FixedTime</code>"
   * the =execution_status= column contains the value "<code> Performed</code>", with a leading space, instead of "<code>Performed</code>"
   * the =s_region= column contains content which is neither a !DALI region nor !STC-S; it looks like some non-standard stringification of a !DALI CIRCLE.
   * the service does not _declare_ in its capabilities the =CIRCLE=, =POLYGON=, =INTERSECTS= and =CONTAINS= !ADQL geometry functions required by section 3.3. Should the validator flag this as a Warning (on the grounds that lack of declaration does not necessarily mean lack of implementation) or as an Error?
I can't run the validator against the CXC installation because of the redirection issues (scheduled for fixing, but AFAICT not done so far) on the host TAP server that I've already raised.

-- IVOA.MarkTaylor - 2021-02-05

I have added the comments on the text as a new pull request:

https://github.com/ivoa-std/ObsLocTAP/pull/18

The only point I have not added is the different capitalization of some utypes from ObsCore and ObsLocTAP. It looks to me that there are two different capitalizations in ObsCore itself (text and table) and the one inside the table looks to me more accurate (this is the one I have used in ObsLocTAP)

   * Thanks, that all seems fine. I have approved the pull request. -- IVOA.MarkTaylor - 2021-03-16
About the comments on the Integral implementation, I have notified the development team so I am waiting for a near future implementation response.

The only thing is that, after talking to them, we think that there is not need to flag as an error not to declare the geometrical functions in the capabilities but as a warning (as many toolkits are not adding this description inside the capabilities declaration)

-- IVOA.JesusSalgado - 2021-03-16

The Chandra CFA service now seems to have become fixed enough for me to run the validator against it. There are several validation errors that need to be fixed:
   * missing utype on =obsplan= table
   * missing utypes on all columns
   * missing required column =s_region=
   * incorrect/misspelt UCDs on columns =t_max=, =em_res_power=, =pol_xel=
   * undocumented UCDs on columns =category=, =execution_status=
   * incorrect units on columns =em_res_power=, =priority=
   * values outside of controlled vocabulary in columns =category=, =priority=, =execution_status= (including case mismatches and illegal NULLs)
-- IVOA.MarkTaylor - 2021-03-30

The ESA reference implementation now passes all the taplint LOC stage validation tests; as far as that goes, I'm happy with it. Still errors in the Chandra implementation - probably nothing serious or complicated but should get cleared up. -- IVOA.MarkTaylor - 2021-04-20

The Chandra team have addressed the issues and run the validator against their local pre-deployment copy, and it now gets a clean bill of health. Ops recommends acceptance! -- IVOA.MarkTaylor - 2021-05-25

---+++ [[IvoaStdsDocsProc][Standards and Processes Committee]]

---++ TCG Vote : Vote_start_date - Vote_end_date

If you have minor comments (typos) on the last version of the document please indicate it in the Comments column of the table and post them in the TCG comments section above with the date.

| Group | Yes | No | Abstain | Comments |
| TCG | * | | | |
| Apps | * | | | |
| DAL | * | | | |
| DM | * | | | |
| GWS | * | | | |
| Registry | * | | | |
| Semantics | * | | | Having registered services would be nice, though (has been addressed) |
| DCP | | | | |
| KDIG | | | | |
| RIG | * | | | |
| SSIG | * | | | |
| Theory | | | | |
| TD | | | | |
| Ops | * | | | |
| !StdProc | | | | |
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
png	Screenshot_2020-06-17_at_17.56.35.png	r1	manage	365.1 K	2020-06-17 - 16:00	JesusSalgado
Topic revision: r54 - 2021-07-15 - JanetEvans
IVOA
Log in or Register
IVOA.net
Wiki Home
WebChanges
WebTopicList
WebStatistics
Twiki Meta & Help
IVOA
Know
Main
Sandbox
TWiki
TWiki intro
TWiki tutorial
User registration
Notify me
Working Groups
Interest Groups
Committees
Stds&Procs
www.ivoa.net
Documents
Events
Members
XML Schema