ObsLocTAP 1.0 Proposed Recommendation: Request for Comments

Summary

Observation Locator Table Access Protocol (ObsLocTAP) document describes the necessary data model elements and the access protocol to discover metadata about observations for a given Astronomical Observatory through a uniform interface within the VO framework. The used data model makes reference to IVOA Observation Data Model elements (Louys, et al., 2017), removing the ones associated to datasets access, as these elements are not available yet for future observations that are planned, scheduled, performed but not archived. In this way, present standard is focused on the access to metadata related to the planning of a certain observatory, more than in the access to the scientific data products of a certain observation. Also, the data model described in the present standard will be focused on metadata discovery, useful for multiwavelength coordination observations.

The latest version of ObsLocTAP can be found at:

• https://www.ivoa.net/documents/ObsLocTAP/20201007/index.html

https://github.com/ivoa-std/ObsLocTAP

Reference Interoperable Implementations

Two separate reference implementations of server-side architecture exist at ESDC, through docker containers and at CFA:

1- Integral - ESDC server

The ESAC Science Data Centre (ESDC) at ESAC is creating a new version of the Integral mission Science Archive (Monica Fernandez). Following the same phylosophy like the Gaia Archive, the server side is based on TAP. For this case, the Integral SOC is producing the table of the future planned Integral observations and this table is shared using the ObsLocTAP data model.

The TAP URL location is at:

• https://ila.esac.esa.int/tap/tap/

Discover columns for table ivoa.obsplan:

SELECT * FROM tap_schema.columns WHERE table_name = 'ivoa.obsplan'  

Discover observations for between 31/05/2020 and 08/09/2020:

SELECT * FROM ivoa.obsplan WHERE t_min > 59000 AND t_max < 59100  

Discover observations that overlaps with a circle centered on the Crab:

SELECT * FROM ivoa.obsplan WHERE 1=INTERSECTS(s_region, CIRCLE('ICRS', 83.633080, 22.014500, 0.016666 ))

Service can be tested on any TAP client application (e.g. TopCat) or directly in your browser using TapHandle:

http://saada.unistra.fr/taphandle?url=https://ila.esac.esa.int/tap/tap/

2- Docker containers:

Two docker containers have been created (Jesus Salgado) in order to allow the creation of a ObsLocTAP server with very basic knowledge of the standard. The first one is a web server with a TAP Tuto instance. The second one is a PostgreSQL database, with a pgsphere module installed and a TAP_SCHEMA in line with ObsLocTAP datamodel.

These docker instances are explained here:

• https://www.cosmos.esa.int/web/vovisobs_protocols/implementation-guides

docker pull jsalgadodocker/pgsphere-obsplan
docker pull jsalgadodocker/tapserver
docker network create --driver=bridge db-network
docker image ls
docker run -p 8080:8080 --net=db-network --name tap <tapserver_image_id>
docker run -p 5432:5432 --net=db-network --name db  <pgsphere-obsplan_image_id>

3- Chandra CFA server:

Chandra CFA TAP server has been adapted (Michael Tibbetts and Janet Evans) to create the ObsLocTAP ivoa.obsplan table to include the future Chandra observations. The service can be found here:

• https://cda.cfa.harvard.edu/cxctap/

Similar queries to Integral TAP server could be instantiated, e.g.:

Discover observations for between 31/05/2020 and 08/09/2020:

SELECT * FROM ivoa.obsplan WHERE t_min > 59000 AND t_max < 59100  

This service has the caveat of not supporting geometrical operators yet (although this is not needed for most of the use cases)

Service can be tested on any TAP client application (e.g. TopCat) or directly in your browser using TapHandle:

http://saada.unistra.fr/taphandle?url=https://cda.cfa.harvard.edu/cxctap/

4- TOBY- Client Application

Also a client has been implemented by ESA (Emilio Salazar), located at:

• http://integral.esa.int/toby/

This graphical client application combines in a calendar view the output of ObsLocTAP servers for Integral, Chandra and the similar to ObsLocTAP servers (NuSTAR and XMM-Newton - in process of adaptation). Also, it shows the Visibility services output (ObjVisSAP) for some missions.

Validator

The taplint TAP validator tool now includes ObsLocTAP validation in its LOC stage; this will be present in STILTS versions >3.4, and in the mean time is available in pre-release at ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/stilts.jar. It checks the required ivoa.obsplan table utype; the presence of required columns with their datatypes, units, utypes and UCDs; declaration of required geometry functions; and content and nullability of columns with restricted values.

-- MarkTaylor - 2021-04-08

Comments from the IVOA Community during RFC/TCG review period: 2020-10-19 - 2020-11-27

Minor comments on 14/05 PR1.0 have been covered on version 07/10.

The comments from the TCG members during the RFC/TCG review should be included in the next section.

In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your Wiki Name so that authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.

Additional discussion about any of the comments or responses can be conducted on the WG mailing list. However, please be sure to enter your initial comments here for full consideration in any future revisions of this document

• Sample comment by WikiName
  • Response (by WikiName)

Comments from TCG member during the RFC/TCG Review Period: 2020-10-19 - 2020-11-27

WG chairs or vice chairs must read the Document, provide comments if any (including on topics not directly linked to the Group matters) or indicate that they have no comment.

IG chairs or vice chairs are also encouraged to do the same, althought their inputs are not compulsory.

TCG Chair & Vice Chair

Applications Working Group

This document seems generally well-written and understandable. It has clearly benefitted from the strong community involvement which also indicates that it's likely to be a useful standard.

I have done some very limited testing of the service implementations and found a few very fixable issues. I have not exercised the Docker containers or Toby (but hope to do so).

In addition to the items Mark identified, I saw that for the Chandra service:

Some minor editorial comments based on the 2021-02-04 preview on github:

• For me the orientation is 90 degrees off for the tables on pages 10 and 14.

• The reference to DALI in the reference section is to version 1.0 instead of 1.1 as mentioned in the s_region description.

• The s_region section on page 11 includes, "In the WHERE clause, s_region constraints will generally make use of ADQL geometries (CIRCLE, POLYGON, POINT) and the geometry predicate functions (CONTAINS, INTERSECTS)." But section 3.3 says, "MUST provide, at least, the following ADQL (Osuna and Ortiz et al. 2008) geometrical features: CIRCLE, POLYGON, INTERSECTS and CONTAINS." Is "POINT" intentionally missing from section 3.3?

• s_region can be returned using the DALI 1.1 specification. This is a new format compared to that mentioned in ObsCore, especially for TAP 1.0. I understand the move forward as recommended by DALI and TAP 1.1. But since the field name "s_region" is the same as in ObsCore, there could be confusion there for clients, who may not be expecting the DALI format. Perhaps it's worth an extra sentence in the s_region section explaining the evolution of that serialization.

Thanks for your comments. About the Chandra implementation, the team is looking for resources to modify all the identified discrepancies from the standard.

About the comments on the text, I have created a new version and a new pull request:

https://github.com/ivoa-std/ObsLocTAP/pull/17

setting you as reviewers to identify the changes I have done folllowing your recommendations.

Tables orientation is rotated 90 degrees on purpose as the big table content was easier to be displayed landscape.

Thanks again!

-- JesusSalgado - 2021-03-03

Pull request merged

-- JesusSalgado - 2021-03-04

Data Access Layer Working Group

The ObsLocTAP standard is a valuable one, as demonstrated by take up from the community and the resources the community itself invested in it. From the DAL perspective, it is another proof that the TAP specification can be leveraged to provide answers to different use cases.

DAL considers important the connection to and overlap with ObsCore elements.

DAL alerts, however, that the differences in registering these two *TAP solutions (tableset+utype in ObsLocTAP versus model-id in ObsTAP) lead to an extra join in RegTAP queries. A common solution (e.g., through TAPRegExt clarification and ObsCore change) should be seeked for in the future. We look forward to connecting to Registry WG to work on this.

DAL supports this standard to become a Recommendation.

Here we report to changes in the document to be fixed:

• p10 - execution_status: UnScheduled -> Unscheduled
• p13 ucd for target_name should be meta.id;src - has a comma currently

-- MarcoMolinaro - 2020-11-27

Thanks for your comments. Your pull request was accepted

-- JesusSalgado - 2021-01-18

Data Model Working Group

Grid & Web Services Working Group

Registry Working Group

Semantics Working Group

While Semantics, as far as I can tell, is largely unconcerned by this standard, from the perspective of validating implementations I have to say I'm a bit concerned that no obsloctap service exists in the Registry as of 2020-11-25. Having a few of them and having some code in the clients that uses that to discover them would make my approval a lot more heart-felt.

But then there's little that could go wrong there, I think, and so you have Semantics' vote.

If I may, here are a few remarks without my Semantics hat on; where I thought things are probably uncontentious, I've included fixes in https://github.com/ivoa-std/ObsLocTAP/pull/8

(1) In the PR, I'm re-formatting the proposed discovery query to look a bit less scary (at least to my taste); if you agree this style make things a bit more readable, I'd be happy to re-format the sample queries accordingly.

(2) Why do you explicitly scale the architecture diagram? I give you the text in it comes out a bit small, but changing the aspect ratio, I think, doesn't really improve things, and it'll look odd for people who know architecture diagrams from other standards. There's nothing wrong with changing the 0.9\textwidth from the template to \textwidth (or, with some precautions, even 1.1\textwidth) -- but you really shouldn't change the aspect ratio.

(3) Again on this, I'd advise to use the pattern from ivoatex/document.template to give an explicit caption to the architecture diagram and then reference it from the text; saying "the figure below" is sure to break in the next release, when the floating figure somehow ends up above that text. If you absolutely insist on "the figure below", un-float the figure (i.e., do away with the figure environment) -- but that will lead to bad page breaks.

(4) In general, I advise against too many rules in tables, and your column tables, I'd argue, are cases in point; but that may be a matter of taste. My PR, however, at least opens up the headline a bit, which I'd say helps the appearance a bit.

(5) I'd still advise to replace the data types in the TAP_SCHEMA.columns table with generic descriptions, i.e., (float), (string), (integer), and then explaining the the text something like: "The concrete types in the TAP_SCHEMA are up to the implementation; however, (float)-typed columns must contain floating point data (like DOUBLE PRECISION or REAL), (string)-typed columns must contain text (e.g, VARCHAR(n), CHAR(n)), and (integer)-typed columns must contain integers (e.g., INTEGER, BIGINT, SMALLINT)." [Note to self: we ought to have a cross-standard convention for this kind of thing]

(6) Since I suspect many actual use cases involve multiple regions of interest, I think giving an example involving uploads (perhaps using stilts syntax, as that can't be done quite as generically) would be a bonus.

(7) I'd not show the sample registry record in a section of its own. If you want it in the document, put it into an appendix. On the other hand, I'd say that's a classic candiate for auxiliaryurl (see https://ivoa.net/documents/Notes/IVOATex/20180814/NOTE-ivoatexDoc-1.2-20180814.html#tth_sEc3.6).

-- MarkusDemleitner - 2020-11-25

Many thanks for your comments. All the editorial changes have been included into pull request:

https://github.com/ivoa-std/ObsLocTAP/pull/11

A registry record has been published for the Integral ObsLocTAP server (EuroVO Registry)

-- JesusSalgado - 2021-01-18

Summary response

Data Curation & Preservation Interest Group

Education Interest Group

Knowledge Discovery Interest Group

Solar System Interest Group

Theory Interest Group

Time Domain Interest Group

Operations

This will be a useful standard, as confirmed by the level of community interest. But there are some issues that need to be addressed.

Reference implementations and validator tools:

• No validator is listed! As the DocStd requires, "PRs being brought forward for promotion to REC should, when applicable, have at least two interoperable implementations and validation tools should be available." If you ask the author of taplint nicely he might be wiling to add an ObsLocTAP validation stage, perhaps subject to some of the issues below being cleared up.
• In the service at https://ila.esac.esa.int/tap/tap/, the s_region field when downloaded as a VOTable has a non-standard format (I think it's "<(ra-in-radians,dec-in-radians),radius>"). It would be nice to have this in some DALI-compliant format, see e.g. DALI 1.1 sec 3.3.6. (see also other s_region comments below).
• The service at https://cda.cfa.harvard.edu/cxctap/ has some HTTP/HTTPS redirection issues that prevents it being used with TOPCAT (I have contacted Janet Evans about this - may be fixed 01/2021?). The ivoa.obsplan table from this service also has no s_region column. I haven't checked it for other deficiencies.

• "Until a new standard becomes authoritative for such fields, s_region will be serialised according to section 6 of TAP 1.0". DALI section 3.3 does provide a standards-based way to serialize geometries ( xtype="circle", xtype="polygon"); future DALI versions are expected to support addtional xtypes "multipolygon", "shape", "moc". Since the STC-S serialization from TAP 1.0 is deprecated/not-really-standardised I would suggest either to require DALI-like serialization of s_region or allow either DALI-like or STC-S-like (with an appropriate xtype - xtype="ADQL:region"?) serialization. This point would probably benefit from input from DAL WG and/or DALI authors (Pat).
• I agree with Markus that more generic rather than specific data types for serialised results and TAP_SCHEMA.columns are a very good idea. I would suggest pseudo-types "integer", "floating point", "string", "region" and "timestamp" or similar, with references to DALI for the serialization details. For instance the ESAC reference implementation does not follow the requirement in sec 3.4 that target name has TAP_SCHEMA type "CHAR(*)", it uses "VARCHAR" instead, which is obviously quite reasonable.
• I suppose that all the columns listed in the tables are supposed to be mandatory, to facilitate executing the same query against different services (though as written the two tables have different column lists, and the two reference implementations do not include all the listed columns, I assume both are mistakes). But this is not spelt out. I suggest to add explicit language saying that the obsplan table MUST include all the columns listed, even though null values may be permitted in some of the columns. There should also be some explicit policy on whether additional non-standard columns are permitted in the obsplan table (I suggest: yes).
• Why are there two tables listing the column characteristics, one in section 3.1 and one in section 3.4, rather than doing it all in one place? This is an unnecessary opportunity for inconsistencies to arise (and I see at least one: obs_release_date exists in one table but not the other). Column name is duplicated; unit is duplicated but with slight differences ("unitless" vs. "") and data type is represented in different forms, which I argue above should be reduced in both cases to a generic direction (string, integer, floating point, region, timestamp). I agree that fitting all that information in one table would require some formatting ingenuity ... but unless there is a compelling reason to split this information into two parts, I would suggest to try to get them as close as possible together.
• "The planning exposure time (t_plan_exptime) has been added in order to show any possible inconsistency between the scheduled exposure time and the real performed exposure time (t_exptime). If an observation was performed without problems, both quantities must be exactly the same. Any discrepancy between these two values will reflect problems or deviations be tween scheduled observations and performed observations." I can imagine a case where the t_plan_exptime is an estimate and the actual t_exptime is within the planned error bounds but represents a more accurate result, in which case the test for discrepancy would appear to represent "problems or deviations" that did not actually occur. But I'm not an observatory planning specialist, so if that's not a realistic scenario then fine.
• "Filtering of services using registration metadata will be described into the TAP Registration Extension definition for ObsLocTAP services." I don't quite follow this sentence. Does it mean that an ObsLocTAP registry extension standard will be submitted in the future as a separate document? (following discussion with Markus: he suggests removing this sentence or replacing by a pointer to section 5).
• Section 4.4 examples use geometry function invocations like "CIRCLE('ICRS', ra, dec)". The initial string coordsys argument ("'ICRS'") will be deprecated in ADQL 2.1 and in most cases doesn't do anything. To avoid confusion, especially with future ADQL versions, write e.g. "CIRCLE('', ra, dec)" instead?
• No bibliography is included in the formatted PDF version of the document, so all the \cite references in the text appear as "(?)".
• Some examples in sec 4 could be made more readable using syntax "<target> BETWEEN <min> AND <max>" rather than "<target> < <max> AND <target> > <min>" (though semantics is not strictly identical)
• I don't know if "planification" is a word - replace by "planning"? But the meaning is clear, so if you want to keep it OK.
• I suggested some hopefully uncontroversial typos at PR#9

Thanks Mark for your comments! I have included all the text changes on:

https://github.com/ivoa-std/ObsLocTAP/pull/14

for your review.

I have added here taplint as the main validator, as ObsLocTAP is a TAP server. I would be really appreciated if the taplint main author could add a specific ObsLocTAP check (perhaps on the obsplan table structure).

After discussing with various server implementors, I have decided to add as possible ways to serialize the regions STC-S and DALI. Integral will be migrated to on STC-S compliant server in a very short term.

I have been forced to maintain the two tables for the obsplan table description as it was really hard to put everything in one place but I have left one with only a description and another with all the technical descriptions so this would be the one people writing the server implementation will check more often.

About ranges, it I have maintain the > min < max as it looks to reflect better the overlap.

-- JesusSalgado - 2021-01-21

• Jesus, thanks for these changes, I think you've addressed all of my points apart from validation. Regarding the validator, I will add some code to taplint that checks ObsLocTAP services are following the recommendations of the document, subject to incorporation of my PR#15 - I'm not insisting that you use the text I've provided, but I think this section needs to be modified a bit in order to state requirements that can be checked by a validator. -- MarkTaylor - 2021-01-28
• Mark, I have merged your proposed text with the current version. I find it quite clear. Thanks! -- JesusSalgado - 2021-01-29
• Thanks, action on me to come up with ObsLocTAP validation in taplint. I'll try to do that in the forseeable future (by end of 02/2021?). -- MarkTaylor - 2021-01-29
  • Past experience indicates it is pretty likely that this activity will show up more issues with the document.

• The table name of "ivoa.obsplan" is first introduced only in passing in section 3.4. To people familiar with e.g. ObsCore it's probably obvious what's going on, but I suggest making it explicit earlier in the document that an ObsLocTAP service is one which contains a table named "ivoa.obsplan" (and which follows the requirements stated).
• The standard doesn't say anything about what columns are allowed to have NULL values. Maybe they all are, which is OK, but I suggest that the nullability rules are made explicit.
• The end of Section 3.1 says: "Initially, observations are inserted as planned into the system. The final state will be one of executed or aborted." I suppose that the terms "planned", "executed" and "aborted" here are supposed to refer to values of the controlled vocabulary listed in the description of the execution_status field in the table. But the capitalisation is different, and no state with a name like "executed" is present in the tabulated list. Please make it typographically clear what states are represented here (e.g. by quoting or using a different font, with exact matches to the controlled terms).
• Section 3.2 says: "For confidential observations that need to be hidden from the planning point of view, the period should be shown as blocked. That means that an entry should appear with the start time, end time and duration as relevant. The facility name should also be provided." I think I understand roughly what's meant here, but the text is confusing to me; I don't understand what "blocked" means. Can you rephrase to be more explicit about what you can/should put in what fields in this situation?
• A couple of the utypes (for facility_name and instrument_name) in the table in section 3.4 have different capitalisation than the corresponding items in ObsCore. Actually I can't remember if utypes are supposed to be case-sensitive (anyone?) but it would be better for them to match.

• most (all?) of the columns have UCD values reported as Utypes and vice versa
• the service metadata as declared in TAP_SCHEMA.tables and the /tables endpoint does not list the utype of the ivoa.obsplan table with the value mandated in section 5 of the standard
• the mandatory obs_release_date column is missing
• the category column contains some values not in the list of possibilities listed in the standard: "TOO", "Normal", "FixedTime"
• the execution_status column contains the value " Performed", with a leading space, instead of "Performed"
• the s_region column contains content which is neither a DALI region nor STC-S; it looks like some non-standard stringification of a DALI CIRCLE.
• the service does not declare in its capabilities the CIRCLE, POLYGON, INTERSECTS and CONTAINS ADQL geometry functions required by section 3.3. Should the validator flag this as a Warning (on the grounds that lack of declaration does not necessarily mean lack of implementation) or as an Error?

-- MarkTaylor - 2021-02-05

I have added the comments on the text as a new pull request:

https://github.com/ivoa-std/ObsLocTAP/pull/18

The only point I have not added is the different capitalization of some utypes from ObsCore and ObsLocTAP. It looks to me that there are two different capitalizations in ObsCore itself (text and table) and the one inside the table looks to me more accurate (this is the one I have used in ObsLocTAP)

• Thanks, that all seems fine. I have approved the pull request. -- MarkTaylor - 2021-03-16

About the comments on the Integral implementation, I have notified the development team so I am waiting for a near future implementation response.

The only thing is that, after talking to them, we think that there is not need to flag as an error not to declare the geometrical functions in the capabilities but as a warning (as many toolkits are not adding this description inside the capabilities declaration)

-- JesusSalgado - 2021-03-16

The Chandra CFA service now seems to have become fixed enough for me to run the validator against it. There are several validation errors that need to be fixed:

• missing utype on obsplan table
• missing utypes on all columns
• missing required column s_region
• incorrect/misspelt UCDs on columns t_max, em_res_power, pol_xel
• undocumented UCDs on columns category, execution_status
• incorrect units on columns em_res_power, priority
• values outside of controlled vocabulary in columns category, priority, execution_status (including case mismatches and illegal NULLs)

Standards and Processes Committee

TCG Vote : Vote_start_date - Vote_end_date

If you have minor comments (typos) on the last version of the document please indicate it in the Comments column of the table and post them in the TCG comments section above with the date.