Table Access Protocol 1.1 Proposed Recommendation: Request for Comments


TOC


The table access protocol (TAP) defines a service protocol for accessing general table data, including astronomical catalogs as well as general database tables. Access is provided for both database and table metadata as well as for actual table data.

This is the proposed revision (version 1.1) of the standard, including changes from community feedback, clarifications and a few optional features.

Latest version of IVOASTANDARD can be found at:

(This RFC was

  • started with respect to PR-TAP-1.1-20180416 PR - with comments before 2018-08-30
  • continued on with PR-TAP-1.1-20180830 PR - with comments after 2018-08-30 and roughly before end of Sep. '18
)

A chronological view on the authenticated endpoints in TAP-1.1 (up to the 2018 Fall Interop in College Park) is available here. This chronology was prepared for and then linked into this mail thread summarizing the outcome from College Park Interop's discussion.

Reference Interoperable Implementations and Validators

From Pat Dowler

The primary CADC TAP service (http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/) is a complete TAP-1.1 implementation. We have implemented TAP-1.1 changes over the whole time that the new version was developed, but I would like to draw attention to these in particular:

  • the VOSI-capabilities endpoint makes use of the prototype UWSRegExt interface types to denite sync and async endpoints with a variety of security methods
  • the tap_schema uses the new datatype,arraysize,xtype column description and this is directly seen in the VOSI-tables output use of VOTableType everywhere
  • we do use ObsCore compatible TAP-1.0 style description of the s_region column (char,*,adql:REGION in tap_schema.columns) so the ObsCore output in VOTable complies with the type and STC-S serialisation in ObsCore-1.1
  • we use the DALI-1.1 datatypes otherwise (point, interval, and polygon in use)
-- PatrickDowler - 2018-04-20

From Markus Demleitner

In Release 1.1, I believe DaCHS (http://soft.g-vo.org/DaCHS) implements all aspects of TAP 1.1 (though of course I also expect that a validator will uncover a couple of issues, which we'll fix as fast as possible). An endpoint to try things out is http://dc.g-vo.org/tap, which also has interoperated nicely with TAP 1.0 clients so far.

Pat's notes on tap_schema datatypes, adql:Region, and DALI 1.1 geometries apply here, as well.

-- MarkusDemleitner - 2018-04-23

From Mark Taylor

I have done client implementation, and partial validator updates, for the features in TAP 1.1 that have changed since TAP 1.0. In particular the authenticated service declaration/discovery described in the example in section 2.4 has been tested and covered by validator updates. It works.

Some other TAP 1.1-specific validator updates have been made, including tests of required foreign key declarations, "size"/arraysize consistency, removed REQUEST=doQuery parameters. Some other features, including timestamp comparisons, table column name upload behaviour, use of xtype and of DALI-1.1 datatypes, are not tested in taplint yet. In due course I hope to get round to adding validator coverage for these featues. There are just a few changes to the non-validator clients apart from authentication: avoid REQUEST=doQuery, report on new TAP_SCHEMA columns in metadata browser. I don't currently forsee other client code changes elsewhere in TOPCAT/STILTS related to TAP 1.1-specific features.

The updated client code (TOPCAT, taplint, other TAP-sensitive STILTS commands) can currently be found at ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/topcat-full_tap11.jar. Note however this implementation is provisional, and may be changed or withdrawn depending on future TAP 1.1 discussions.

-- MarkTaylor - 2018-10-22


Comments from the IVOA Community during RFC/TCG review period: 2018-04-23 - 2018-11-23

The comments from the TCG members during the RFC/TCG review should be included in the next section.

*Please note a new revision of the PR was released on October 24.*

The RFC has been twice extended (see PR details at start of page).

In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your Wiki Name so that authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.

Additional discussion about any of the comments or responses can be conducted on the WG mailing list. However, please be sure to enter your initial comments here for full consideration in any future revisions of this document



Comments from TCG member during the RFC/TCG Review Period: 2018-04-23 - 2018-11-23

*Please note a new revision of the PR was released on October 24.*

WG chairs or vice chairs must read the Document, provide comments if any (including on topics not directly linked to the Group matters) or indicate that they have no comment.

IG chairs or vice chairs are also encouraged to do the same, althought their inputs are not compulsory.

TCG Chair & Vice Chair

Applications Working Group

Data Access Layer Working Group

Data Model Working Group

Some minimal comments from my read through.. Laurent may have some more technical comments as well.

Section 1.2.6: 'simple key-vale' -> 'simple key-value'

Section 2.4: 'the mandatory chilsd' -> 'the mandatory child'

Section 2.7: 'Thus, for the asynchronous TAP queries, the parameter requirements must be satisfied (and errors returned if not) only when the query is run in (in the sense of UWS job execution).' Something seems missing.. the query mode maybe?

Section 2.7: ' may be created with with no parameters' == Extra 'with'

Section 5.1.3: 'URL should lead to a an error' == Extra 'a'

Grid & Web Services Working Group

Registry Working Group

Semantics Working Group

Data Curation & Preservation Interest Group

Education Interest Group

Knowledge Discovery in Databases Interest Group

Solar System Interest Group

Theory Interest Group

Time Domain Interest Group

Operations

Comments from Mark Taylor (2018-06-12)

I have some concerns relating to the approach taken in Section 2.4 to declaring service endpoints for a variety of security methods. The UWSRegExt extension referenced here has not been previously published, but that is now being addressed by the draft UWSRegExt Note currently in circulation. Some edits to section 2.4 will be required when the content of that document has been agreed. But even with a well-defined UWSRegExt, from a client point of view I see some difficulties with using a capabilities document like the one shown in Section 2.4 of PR-TAP-20180416.

The proposal there is fine for clients that know they want to find, say, a TAP sync endpoint using tls-with-certificate authentication; I believe that is the usage scenario for which this arrangement has been tested at CADC. However, a client like TOPCAT has different service discovery requirements: it needs to find a bundle of related endpoints (sync, async, tables, capabilities, examples) which together constitute a TAP service. From that point of view the mixed bag of interfaces-with-securityMethods proposed here is difficult to interpret. I made a short presentation in Shanghai complaining about this. To me, it would make more sense to declare a separate base URL for each securityMethod, with the subordinate endpoints (/sync, /async, /tables etc) hanging off each one in the standard TAP-1.0 pattern. I know Pat is not keen on that, and I admit I'm not sure if it could be made to fit easily with the existing VOResource/VOSI-Capabilities framework.

Response: These items from my DAL mailing list post of 2018-06-13 (this thread -- note by MarcoMolinaro) is the primary obstacle to this approach:

2. DALI-1.1 specifies (Section 2) that all endpoints in a service must be siblings of the VOSI-capabilities resource (except VOSI-availability); further, DALI also specifies that the capabilities resource must be named /capabilities (relative to the base).

3. VOSI-1.1 specifies that VOSI-capabilities must be provided with anonymous access and such capabilities must not include a securityMethod

So, you have a base URL and a single anonymous child {baseURL}/capabilities and all the TAP endpoints (eg sync and async query execution) have to be siblings of that (i.e. direct children of {baseURL}). That is just a flat list of interfaces and can't be presented as a bundle with current VOSI. Somewhat ironically, before that DALI-1.1 restriction the CADC TAP service used to place the username/password auth endpoints under a "subdirectory", so it used to be /tap/auth/sync and /tap/auth/async and now it is /tap/auth-sync and /tap/auth-async so everything is a sibling of /tap/capabilities... that is quite natural from a service deployment point of view and more or less conforms to the bundle and defaults approach described above. But the community thought it was a good idea to be able to navigate(?) from any endpoint to the capabilities endpoint. Either (flat or bundle) works fine if you use the capabilities the way that our cadc-registry client lib does (it is just cosmetic). But: the only way to go the bundle way is for TAP-1.1 to violate DALI-1.1 and/or retract that bit from DALI (admit mistake; issue erratum).

-- PatrickDowler - 2018-08-23

The alternative that does not break any rules and does not require any UWSRegExt is

  • /tap/sync
  • /tap/async
and

  • /authtap/sync
  • /authap/async
i.e. a separate auth interface which uses a different root URL

-- PaulHarrison - 2018-09-05

  • I agree with Paul - it's true that you addionally need a /authtap/capabilities to avoid breaking the sibling rule, but there's no reason you can't have that as well as (and perhaps with the same content as) /tap/capabilities. I've argued this in more detail on-list. -- MarkTaylor - 2018-09-12
Response: As already stated, this arrangement is not compliant with DALI-1.1 which says that all endpoints must be siblings of the /capabiliies endpoint (in the REST resource sense). The fundamental issue is that authentication is orthogonal to service functionaility -- VOResource included is (securityMethod) at the leaves of the xml hierarchy as that is the least intrusive way to do it . The CADC implementation used to be more hierarchical but we changed it when people wanted the DALI restriction. Please re-read the "but: ..." in the previous response. -- PatrickDowler - 2018-09-12

Failing that, I think the current text of section 2.4 (as adjusted following production of the UWSRegExt document or something similar) could be fixed up to be usable by TOPCAT-like clients with some text along the following lines:

A TAP service consists of a related set of resource types, as listed in [the table in Section 2] : TAP-sync, TAP-async, VOSI-capabilities, VOSI-availability, and optionally VOSI-tables and DALI-examples. In some cases a client needs to identify a related "bundle" of endpoints providing one each of some or all of these resources in order to orchestrate its interaction with the TAP service. In TAP 1.0, this identification was straightforward, since only one instance of each resource type existed, and these were present at fixed locations relative to the TAP service base URL. In TAP 1.1, the capabilities document may declare multiple variants for some or all of these resource types, for instance to support different authentication contexts, and identifying the related bundle of endpoints that should be used given particular authentication requirements is therefore no longer trivial. Clients are free to combine the declared interfaces in any way they see fit, but the recommended way to interpret a TAP capabilities document as a list of TAP endpoint bundles is as follows:

  • There is one default bundle, corresponding to the TAP 1.0 endpoints. If the service allows unauthenticated access at all, it should usually be provided by this default bundle.
  • There may also be one or more authenticated bundles. Each distinct security method declared on an interface in the capabilities document (value of an interface/securityMethod/@standardId XPath) defines one authenticated bundle. Each endpoint sharing the same security method is in the same bundle; if no endpoint is declared with that security method, the corresponding default endpoint is used.
The example [in section 2.4] therefore defines the default bundle and two authenticated bundles, defined by the security methods "ivo://ivoa.net/sso#BasicAA" and "ivo://ivoa.net/sso#tls-with-certificate" respectively. Each of these authenticated bundles explicitly declares its TAP-sync, TAP-async and VOSI-tables endpoints, and implicitly uses the default endpoints for VOSI-capabilities and VOSI-availability.

That text is my attempt to make explicit what I think are the implicit assumptions about how the services described by the capabilities example in the PR are supposed to fit together. If I have misread those, it may need some adjustment.

Response: Document updated with approximately the above descriptive text.

-- PatrickDowler - 2018-08-29

Note however that this text does not address the issue of multiple interface elements within a capability element having the same xsi:type but different version attributes and (the same? different?) securityMethods, which complicates all this further - do we expect to see those?. I can think of ways to adjust the text above to accommodate those possibilities, but it starts to get pretty messy.

As far as I know(?), CADC has the only TAP services and the only TAP clients to make use of the scheme described in this PR to work with non-trivial authentication requirements, and I worry that the details of the scheme may be driven by their particular requirements or implementation details. I might be wrong about that, but I would very much like to see other TAP services that need authentication (LSST, IRSA, MAST, CSIRO?) look closely at this proposal, and preferably implement against it, to see whether it fits with their requirements. If it does, I will do my best to get TOPCAT working smoothly with it.

Other than that, this document mostly looks good to me. I have some minor comments:

  • Sec 1.1: This is a very useful addition to the document, providing good signposting of TAP's position in the IVOA landscape.
  • Sec 2.7.2: the Note in the last paragraph warns about using the coordinate system string parameter in ADQL geometry functions. It would be relevant to note here that these parameters are deprecated in ADQL 2.1 (readers only familiar with ADQL 2.1+ may otherwise find this note hard to understand).
  • Sec 2.7.4: for clarity, the last paragraph should say something like "The output truncation caused by non-zero values of the MAXREC parameter...".
  • Appendix A: Especially since the history has been quite convoluted, with some items being added and later retracted, it would be useful for users of TAP 1.1, rather than IVOA archaeologists, to consolidate the change log subsections into one item "changes since TAP 1.0".
  • I've made some, hopefully uncontroversial, typographical fixes in Volute (r5045).
Response: Document updated to clarify these points, except the changelog (TBD with authors).

-- PatrickDowler - 2018-08-29

Response: Document updated to provide a "Changes from TAP-1.0 to TAP-1.1" appendix and separate detailed changelog.

-- PatrickDowler - 2018-09-04

Further Comments from Mark Taylor, regarding PR-TAP-1.1-20180830 (2018-09-12)

Note: the following comments concern PR-TAP-1.1-20180830 as written. I'm still arguing that this approach may not be the best one; if the discussion in this thread succeeds in persuading the authors to change the structure of the capabilities document, some of the following comments may be invalidated.

Thank you for including my text concerning bundles of services. Given this I think I will be able to write client code to work with this style of capabilities document. However, I still find it more unwieldy than the old services-at- well-known-relative-locations arrangement, and as I hinted in my earlier comments, I can still see a number of unanswered questions that this prescription leaves. Some of these are particularly from the point of view of a validator: it's not clear how much of the content of the example /capabilities document in section 2.4 is intended to be normative (the only way you're allowed to do it) and how much is just personal taste or dictated by the details of the particular usage scenario being illustrated here.

  • The 'TAP Base URL' interface no longer serves the purpose it used to (defining where to find the sync and async endpoints). Is its inclusion in the capabilities file mandatory?
Response: The base url with type ParamHTTP services to let TAP-1.0 client find the base url and use the service. The explicit anonymous sync and async endpoints can be included and this allows a generic capabilities-reading client that doesn't know anyhting about TAP naming conventions to correctly find endpoint URLs (this is how the cadc-registry java lib works -- without TAP knowledge). So, those can be included, but I didn't think about whether they must or not. If we didn't need backwards compat it would be must and not
Topic attachments
I Attachment Action Size Date Who Comment
PDFpdf PR-TAP-1.1-20180416.pdf manage 645.9 K 2018-04-18 - 06:24 MarcoMolinaro  
Topic revision: r25 - 2018-11-14 - MarcoMolinaro
 
This site is powered by the TWiki collaboration platformCopyright © 2008-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback