| |
| META TOPICPARENT |
name="WebPreferences" |
ObsLocTAP 1.0 Proposed Recommendation: Request for Comments
Summary
Observation Locator Table Access Protocol (ObsLocTAP) document describes the necessary data model elements and the access protocol to discover metadata about observations for a given Astronomical Observatory through a uniform interface within the VO framework. The used data model makes reference to IVOA Observation Data Model elements (Louys, et al., 2017), removing the ones associated to datasets access, as these elements are not available yet for future observations that are planned, scheduled, performed but not archived. In this way, present standard is focused on the access to metadata related to the planning of a certain observatory, more than in the access to the scientific data products of a certain observation. Also, the data model described in the present standard will be focused on metadata discovery, useful for multiwavelength coordination observations.
The latest version of ObsLocTAP can be found at:
Also, ObsLocTAP is inside the IVOA github repository at:
https://github.com/ivoa-std/ObsLocTAP
Reference Interoperable Implementations
Two separate reference implementations of server-side architecture exist at ESDC, through docker containers and at CFA:
1- Integral - ESDC server
The ESAC Science Data Centre (ESDC) at ESAC is creating a new version of the Integral mission Science Archive (Monica Fernandez). Following the same phylosophy like the Gaia Archive, the server side is based on TAP. For this case, the Integral SOC is producing the table of the future planned Integral observations and this table is shared using the ObsLocTAP data model.
The TAP URL location is at:
ObsLocTAP table is ivoa.obsplan. Some example queries are the following:
Discover columns for table ivoa.obsplan:
SELECT * FROM tap_schema.columns WHERE table_name = 'ivoa.obsplan'
Discover observations for between 31/05/2020 and 08/09/2020:
SELECT * FROM ivoa.obsplan WHERE t_min > 59000 AND t_max < 59100
Discover observations that overlaps with a circle centered on the Crab:
SELECT * FROM ivoa.obsplan WHERE 1=INTERSECTS(s_region, CIRCLE('ICRS', 83.633080, 22.014500, 0.016666 ))
Service can be tested on any TAP client application (e.g. TopCat) or directly in your browser using TapHandle:
http://saada.unistra.fr/taphandle?url=https://ila.esac.esa.int/tap/tap/
2- Docker containers:
Two docker containers have been created (Jesus Salgado) in order to allow the creation of a ObsLocTAP server with very basic knowledge of the standard. The first one is a web server with a TAP Tuto instance. The second one is a PostgreSQL database, with a pgsphere module installed and a TAP_SCHEMA in line with ObsLocTAP datamodel.
These docker instances are explained here:
For the execution, they can be invoked by:
docker pull jsalgadodocker/pgsphere-obsplan
docker pull jsalgadodocker/tapserver
docker network create --driver=bridge db-network
docker image ls
docker run -p 8080:8080 --net=db-network --name tap <tapserver_image_id>
docker run -p 5432:5432 --net=db-network --name db <pgsphere-obsplan_image_id>
3- Chandra CFA server:
Chandra CFA TAP server has been adapted (Michael Tibbetts and Janet Evans) to create the ObsLocTAP ivoa.obsplan table to include the future Chandra observations. The service can be found here:
- https://cda.cfa.harvard.edu/cxctap/
This is the first X-Ray observatories service implemented (similar services are already in on development for XMM-Newton and NuStar) what will allow the preparation of coordinated observations for these three observatories in a short-term.
Similar queries to Integral TAP server could be instantiated, e.g.:
Discover observations for between 31/05/2020 and 08/09/2020:
SELECT * FROM ivoa.obsplan WHERE t_min > 59000 AND t_max < 59100
This service has the caveat of not supporting geometrical operators yet (although this is not needed for most of the use cases)
Service can be tested on any TAP client application (e.g. TopCat) or directly in your browser using TapHandle:
http://saada.unistra.fr/taphandle?url=https://cda.cfa.harvard.edu/cxctap/
4- TOBY- Client Application
Also a client has been implemented by ESA (Emilio Salazar), located at:
This graphical client application combines in a calendar view the output of ObsLocTAP servers for Integral, Chandra and the similar to ObsLocTAP servers (NuSTAR and XMM-Newton - in process of adaptation). Also, it shows the Visibility services output (ObjVisSAP) for some missions.
Comments from the IVOA Community during RFC/TCG review period: 2020-10-19 - 2020-11-27
Minor comments on 14/05 PR1.0 have been covered on version 07/10.
The comments from the TCG members during the RFC/TCG review should be included in the next section.
In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your Wiki Name so that authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.
Additional discussion about any of the comments or responses can be conducted on the WG mailing list. However, please be sure to enter your initial comments here for full consideration in any future revisions of this document
Comments from TCG member during the RFC/TCG Review Period: 2020-10-19 - 2020-11-27
WG chairs or vice chairs must read the Document, provide comments if any (including on topics not directly linked to the Group matters) or indicate that they have no comment.
IG chairs or vice chairs are also encouraged to do the same, althought their inputs are not compulsory.
TCG Chair & Vice Chair
The ObsLocTAP standard is a valuable one, as demonstrated by take up from the community and the resources the community itself invested in it. From the DAL perspective, it is another proof that the TAP specification can be leveraged to provide answers to different use cases.
DAL considers important the connection to and overlap with ObsCore elements.
DAL alerts, however, that the differences in registering these two *TAP solutions (tableset+utype in ObsLocTAP versus model-id in ObsTAP) lead to an extra join in RegTAP queries. A common solution (e.g., through TAPRegExt clarification and ObsCore change) should be seeked for in the future. We look forward to connecting to Registry WG to work on this.
DAL supports this standard to become a Recommendation.
Here we report to changes in the document to be fixed:
- p10 - execution_status: UnScheduled -> Unscheduled
- p13 ucd for target_name should be meta.id;src - has a comma currently
Other proposed changes to the text and typo fixing will be sent as a Pull Request to the ObsLocTAP git repo.
-- MarcoMolinaro - 2020-11-27
While Semantics, as far as I can tell, is largely unconcerned by this standard, from the perspective of validating implementations I have to say I'm a bit concerned that no obsloctap service exists in the Registry as of 2020-11-25. Having a few of them and having some code in the clients that uses that to discover them would make my approval a lot more heart-felt.
But then there's little that could go wrong there, I think, and so you have Semantics' vote.
If I may, here are a few remarks without my Semantics hat on; where I thought things are probably uncontentious, I've included fixes in https://github.com/ivoa-std/ObsLocTAP/pull/8
(1) In the PR, I'm re-formatting the proposed discovery query to look a bit less scary (at least to my taste); if you agree this style make things a bit more readable, I'd be happy to re-format the sample queries accordingly.
(2) Why do you explicitly scale the architecture diagram? I give you the text in it comes out a bit small, but changing the aspect ratio, I think, doesn't really improve things, and it'll look odd for people who know architecture diagrams from other standards. There's nothing wrong with changing the 0.9\textwidth from the template to \textwidth (or, with some precautions, even 1.1\textwidth) -- but you really shouldn't change the aspect ratio.
(3) Again on this, I'd advise to use the pattern from ivoatex/document.template to give an explicit caption to the architecture diagram and then reference it from the text; saying "the figure below" is sure to break in the next release, when the floating figure somehow ends up above that text. If you absolutely insist on "the figure below", un-float the figure (i.e., do away with the figure environment) -- but that will lead to bad page breaks.
(4) In general, I advise against too many rules in tables, and your column tables, I'd argue, are cases in point; but that may be a matter of taste. My PR, however, at least opens up the headline a bit, which I'd say helps the appearance a bit.
(5) I'd still advise to replace the data types in the TAP_SCHEMA.columns table with generic descriptions, i.e., (float), (string), (integer), and then explaining the the text something like: "The concrete types in the TAP_SCHEMA are up to the implementation; however, (float)-typed columns must contain floating point data (like DOUBLE PRECISION or REAL), (string)-typed columns must contain text (e.g, VARCHAR(n), CHAR(n)), and (integer)-typed columns must contain integers (e.g., INTEGER, BIGINT, SMALLINT)." [Note to self: we ought to have a cross-standard convention for this kind of thing]
(6) Since I suspect many actual use cases involve multiple regions of interest, I think giving an example involving uploads (perhaps using stilts syntax, as that can't be done quite as generically) would be a bonus.
(7) I'd not show the sample registry record in a section of its own. If you want it in the document, put it into an appendix. On the other hand, I'd say that's a classic candiate for auxiliaryurl (see https://ivoa.net/documents/Notes/IVOATex/20180814/NOTE-ivoatexDoc-1.2-20180814.html#tth_sEc3.6).
-- MarkusDemleitner - 2020-11-25
Summary response
|
>
> |
This will be a useful standard, as confirmed by the level of community interest. But there are some issues that need to be addressed.
Reference implementations and validator tools:
- No validator is listed! As the DocStd requires, "PRs being brought forward for promotion to REC should, when applicable, have at least two interoperable implementations and validation tools should be available." If you ask the author of
taplint nicely he might be wiling to add an ObsLocTAP validation stage, perhaps subject to some of the issues below being cleared up.
- In the service at https://ila.esac.esa.int/tap/tap/, the
s_region field when downloaded as a VOTable has a non-standard format (I think it's "<(ra-in-radians,dec-in-radians),radius>"). It would be nice to have this in some DALI-compliant format, see e.g. DALI 1.1 sec 3.3.6. (see also other s_region comments below).
- The service at https://cda.cfa.harvard.edu/cxctap/ has some HTTP/HTTPS redirection issues that prevents it being used with TOPCAT (I have contacted Janet Evans about this). The
ivoa.obsplan table from this service also has no s_region column. I haven't checked it for other deficiencies.
Document Content:
- "Until a new standard becomes authoritative for such fields, s_region will be serialised according to section 6 of TAP 1.0". DALI section 3.3 does provide a standards-based way to serialize geometries (
xtype="circle", xtype="polygon"); future DALI versions are expected to support addtional xtypes "multipolygon", "shape", "moc". Since the STC-S serialization from TAP 1.0 is deprecated/not-really-standardised I would suggest either to require DALI-like serialization of s_region or allow either DALI-like or STC-S-like (with an appropriate xtype - xtype="ADQL:region"?) serialization. This point would probably benefit from input from DAL WG and/or DALI authors (Pat).
- I agree with Markus that more generic rather than specific data types for serialised results and TAP_SCHEMA.columns are a very good idea. I would suggest pseudo-types "integer", "floating point", "string", "region" and "timestamp" or similar, with references to DALI for the serialization details. For instance the ESAC reference implementation does not follow the requirement in sec 3.4 that target name has TAP_SCHEMA type "
CHAR(*)", it uses "VARCHAR" instead, which is obviously quite reasonable.
- I suppose that all the columns listed in the tables are supposed to be mandatory, to facilitate executing the same query against different services (though as written the two tables have different column lists, and the two reference implementations do not include all the listed columns, I assume both are mistakes). But this is not spelt out. I suggest to add explicit language saying that the obsplan table MUST include all the columns listed, even though null values may be permitted in some of the columns. There should also be some explicit policy on whether additional non-standard columns are permitted in the obsplan table (I suggest: yes).
- Why are there two tables listing the column characteristics, one in section 3.1 and one in section 3.4, rather than doing it all in one place? This is an unnecessary opportunity for inconsistencies to arise (and I see at least one:
obs_release_date exists in one table but not the other). Column name is duplicated; unit is duplicated but with slight differences ("unitless" vs. "") and data type is represented in different forms, which I argue above should be reduced in both cases to a generic direction (string, integer, floating point, region, timestamp). I agree that fitting all that information in one table would require some formatting ingenuity ... but unless there is a compelling reason to split this information into two parts, I would suggest to try to get them as close as possible together.
- "The planning exposure time (t_plan_exptime) has been added in order to show any possible inconsistency between the scheduled exposure time and the real performed exposure time (t_exptime). If an observation was performed without problems, both quantities must be exactly the same. Any discrepancy between these two values will reflect problems or deviations be tween scheduled observations and performed observations." I can imagine a case where the
t_plan_exptime is an estimate and the actual t_exptime is within the planned error bounds but represents a more accurate result, in which case the test for discrepancy would appear to represent "problems or deviations" that did not actually occur. But I'm not an observatory planning specialist, so if that's not a realistic scenario then fine.
- "Filtering of services using registration metadata will be described into the TAP Registration Extension definition for ObsLocTAP services." I don't quite follow this sentence. Does it mean that an ObsLocTAP registry extension standard will be submitted in the future as a separate document? (following discussion with Markus: he suggests removing this sentence or replacing by a pointer to section 5).
- Section 4.4 examples use geometry function invocations like "
CIRCLE('ICRS', ra, dec)". The initial string coordsys argument ("'ICRS'") will be deprecated in ADQL 2.1 and in most cases doesn't do anything. To avoid confusion, especially with future ADQL versions, write e.g. "CIRCLE('', ra, dec)" instead?
- No bibliography is included in the formatted PDF version of the document, so all the
\cite references in the text appear as "(?)".
- Some examples in sec 4 could be made more readable using syntax "
<target> BETWEEN <min> AND <max>" rather than "<target> < <max> AND <target> > <min>" (though semantics is not strictly identical)
- I don't know if "planification" is a word - replace by "planning"? But the meaning is clear, so if you want to keep it OK.
- I suggested some hopefully uncontroversial typos at PR#9
-- MarkTaylor - 2020-11-27 |
