TOC
The table access protocol (TAP) defines a service protocol for accessing general table data, including astronomical catalogs as well as general database tables. Access is provided for both database and table metadata as well as for actual table data.
This is the proposed revision (version 1.1) of the standard, including the following changes from TAP 1.0:
A chronological view on the authenticated endpoints in TAP-1.1 (up to the 2018 Fall Interop in College Park) is available here. This chronology was prepared for and then linked into this mail thread summarizing the outcome from College Park Interop's discussion.
Latest PR (20160626) reflects discussions held at Paris 2019 May Interop, as said, and its scope is to clarify §2.4 in terms of interface URL uniqueness and multiple security methods attached.
We kindly invite (especially the TCG members) to focus their review and vote on this specific paragraph.
The primary CADC TAP service (http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/) is a complete TAP-1.1 implementation. We have implemented TAP-1.1 changes over the whole time that the new version was developed, but I would like to draw attention to these in particular:
In Release 1.1, I believe DaCHS (http://soft.g-vo.org/DaCHS) implements all aspects of TAP 1.1 (though of course I also expect that a validator will uncover a couple of issues, which we'll fix as fast as possible). An endpoint to try things out is http://dc.g-vo.org/tap, which also has interoperated nicely with TAP 1.0 clients so far.
Pat's notes on tap_schema datatypes, adql:Region, and DALI 1.1 geometries apply here, as well.
-- MarkusDemleitner - 2018-04-23
I have done client implementation, and partial validator updates, for some of the features in PR-TAP 1.1 that have changed since TAP 1.0. The recommendations for how to do securityMethod-specific service discovery from the capabilities file have changed several times during the course of the RFC period, and I've implemented this in various versions. Much of the TAP 1.0 standard is already covered by the taplint validator.
The most recent client implementation (PR-TAP-20190626) can be found at ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/topcat-full_tap11c.jar . This includes the ability to use a TAP 1.1 service with various different security methods from the TOPCAT and STILTS UIs (see my Paris presentation). I have not so far attempted to implement user choice between declared mirrorURL
s alongside user choice between declared securityMethod
s, but I think that, unlike under previous versions of the PR, it would be fairly straightforward. There are just a few changes to the non-validator clients apart from authentication-related functionality: avoid REQUEST=doQuery, and use/report new TAP_SCHEMA columns in the metadata browser. I don't currently forsee other client code changes elsewhere in TOPCAT/STILTS related to TAP 1.1-specific features. This is prototype code still under development (e.g. there may be GUI refinements), but I'm not aware of problems relating to the content of the TAP 1.1 standard as far as it goes.
Some TAP-1.1-specific (and version-sensitive) updates to the taplint validator have been made; these can be found in pre-release at ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/stilts.jar . They include tests of the new capabilities document content (sec 2.4), checks on the new TAP_SCHEMA columns, tests of modified REQUEST parameter handling, and checks that some required foreign key relationships are in place. Some other features, including timestamp comparisons, table column name upload behaviour, use of xtype and of DALI-1.1 datatypes, are not tested in taplint yet. I may get round to adding validator coverage for these features in the future.
-- MarkTaylor - 2019-07-16
The comments from the TCG members during the RFC/TCG review should be included in the next section.
*Please note a new revision of the PR was released on 20 April, 2019.*
The RFC has been twice extended (see PR details at start of page).
In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your Wiki Name so that authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.
Additional discussion about any of the comments or responses can be conducted on the WG mailing list. However, please be sure to enter your initial comments here for full consideration in any future revisions of this document
*Please note a new revision of the PR was released on 20 April, 2019*
WG chairs or vice chairs must read the Document, provide comments if any (including on topics not directly linked to the Group matters) or indicate that they have no comment.
IG chairs or vice chairs are also encouraged to do the same, althought their inputs are not compulsory.
I approve of this document.
Most of the changes seem useful and reasonably understandable. I did find the authN discussions (especially in section 2.4) a bit hard to digest, but ultimately it seems adequate for this version of the document.
-- TomDonaldson - 2019-08-15
Some minimal comments from my read through.. Laurent may have some more technical comments as well.
Section 1.2.6: 'simple key-vale' -> 'simple key-value'
Section 2.4: 'the mandatory chilsd' -> 'the mandatory child'
Section 2.7: 'Thus, for the asynchronous TAP queries, the parameter requirements must be satisfied (and errors returned if not) only when the query is run in (in the sense of UWS job execution).' Something seems missing.. the query mode maybe?
Section 2.7: ' may be created with with no parameters' == Extra 'with'
Section 5.1.3: 'URL should lead to a an error' == Extra 'a'
Section 2: I'm not fond of the word 'consequences' when describing the use of a single URL. It comes across as admitting a compromise. A more positive word would be preferable.
Section 2.4: typo: 'version attribute is equivalent to version="1.0".' ('is' should be 'as')
-- BrianMajor - 2019-05-08
Response: these issues have been addressed in rev 5572 or earlier. -- PatrickDowler - 2019-08-13
Comments on PR-TAP-1.1-20190626.
I do not have technical comment on the protocol its-self. Some comments on the document.
Section 1.1, please check the references carefully. I suggest to order them alphabetically (the UWS now is the first one). There are some errors in the references, please check them carefully. For example: OBScore is the old document from 2011, now there is a new one; in the SSO ref the order of the authors is wrong, it is Taffoni et al.; CDP authors are Graham et al.
Response on references: Many of the references use ADS bibcodes and the resulting references reflect the record in ADS; when those records were created, it was decided to make the author list include the editors so I guess some mangling occurs that no longer matches the author list of the actual document. Since ObsCore-1.1 and SSO-2.0 was referenced locally (not in ADS yet) I have fixed these two, but others seems out of scope (see point #8 in https://wiki.ivoa.net/internal/IVOA/IvoaRepMin/ivoa-tm56-20141216.pdf) -- PatrickDowler - 2019-08-14
Please check the MUST, MAY, SHALL etc. in the documents. Sometimes you are not following properly the uppercase convection.
Consider to give a caption to the tables, this makes them more readable and easy to reference.
Section 2.2 probably can be summarized as some of the examples are the default behavior of UWS.
As on other documents maybe the appendix could be summarized.
Some typos:
Response: all typos above have been fixed in rev 5573. Thank you for this detailed read of the document! -- PatrickDowler - 2019-08-13
I appreciate the chronological view linked; given the back-and-forth nature and long timeline of some of these changes, I think this standard is another good argument (along with the recent RegTAP 1.1) for in the future providing a "squashed" view (to borrow some Git terminology) with complex version changes, in the RFC and document appendices.
Section 2.4: Despite understanding the technical concepts, I still find the following paragraph a bit confusing, and perhaps unnecessarily so. Do we need to spell out the reasoning in "While one can provide" and on? The information is useful but possibly either as a footnote or could use some more rewording. Also, typo, implement its challenge-response.
I am generally OK with the changes. -- TheresaDower - 2019-07-10
"As a consequence of using a single base URL with fixed name child resources, all supported authentication methods must be able to co-exist on the same URL. At this time, both anonymous and other authentication methods described in SSO can be made to work in this way, with the exception of HTTP Basic and Digest authentication. With a single base URL, one can provide an anonymous-only TAP service with http or https; once support for ivo://ivoa.net/sso#tls-with-certificate
is added, however, the entire service must use https only. While one can provide a service with securityMethod="ivo://ivoa.net/sso#BasicAA"
, this authentication method uses the HTTP protocol to implement it's challenge-response and thus cannot co-exist with other methods (including anonymous); such a service would only be able to describe that single method. However, the SSO standard recommends that username-password authentication (including BasicAA) be used to login and obtain another type of credential (e.g. a cookie) and that services should not directly use username-password for authentication; the single base URL design here is consistent with the recommendations of SSO."
Response: these issues have been addressed in rev 5572 (mainly by deleting text). -- PatrickDowler - 2019-08-13
in the 2.4 VOSI-capabilities, the example appear to me as non schema valid due to multiple securityMethod
Semantics is largely unconcerned by the proposed changes; hence, we are fine with the PR.
From a wider perspective, the long-winding discussion in 2.4 on authentication issues is perhaps a bit out of place in TAP, which, as such, does not seem to have special requirements on authentication (as compared to, say, SSAP). But this has been discussed elsewhere and shouldn't hold up the other changes.
-- MarkusDemleitner - 2019-08-12
Minor issue about wording - probably skip for this version and add it as a GitHub issue for the next version.
Section 1.3.1 ADQL Queries - It is not 100% clear what the last sentence in this section is trying to say:
"It is also possible that the service registration (in an IVOA Registry) may include sufficient tablemetadata to enable queries to be written directly."
These comments obsoleted by later changes - see instead below
I have some concerns relating to the approach taken in Section 2.4 to declaring service endpoints for a variety of security methods. The UWSRegExt extension referenced here has not been previously published, but that is now being addressed by the draft UWSRegExt Note currently in circulation. Some edits to section 2.4 will be required when the content of that document has been agreed. But even with a well-defined UWSRegExt, from a client point of view I see some difficulties with using a capabilities document like the one shown in Section 2.4 of PR-TAP-20180416.
The proposal there is fine for clients that know they want to find, say, a TAP sync endpoint using tls-with-certificate authentication; I believe that is the usage scenario for which this arrangement has been tested at CADC. However, a client like TOPCAT has different service discovery requirements: it needs to find a bundle of related endpoints (sync, async, tables, capabilities, examples) which together constitute a TAP service. From that point of view the mixed bag of interfaces-with-securityMethods proposed here is difficult to interpret. I made a short presentation in Shanghai complaining about this. To me, it would make more sense to declare a separate base URL for each securityMethod, with the subordinate endpoints (/sync, /async, /tables etc) hanging off each one in the standard TAP-1.0 pattern. I know Pat is not keen on that, and I admit I'm not sure if it could be made to fit easily with the existing VOResource/VOSI-Capabilities framework.
Response: These items from my DAL mailing list post of 2018-06-13 (this thread -- note by MarcoMolinaro) is the primary obstacle to this approach:
2. DALI-1.1 specifies (Section 2) that all endpoints in a service must be siblings of the VOSI-capabilities resource (except VOSI-availability); further, DALI also specifies that the capabilities resource must be named /capabilities (relative to the base).
3. VOSI-1.1 specifies that VOSI-capabilities must be provided with anonymous access and such capabilities must not include a securityMethod
So, you have a base URL and a single anonymous child {baseURL}/capabilities and all the TAP endpoints (eg sync and async query execution) have to be siblings of that (i.e. direct children of {baseURL}). That is just a flat list of interfaces and can't be presented as a bundle with current VOSI. Somewhat ironically, before that DALI-1.1 restriction the CADC TAP service used to place the username/password auth endpoints under a "subdirectory", so it used to be /tap/auth/sync and /tap/auth/async and now it is /tap/auth-sync and /tap/auth-async so everything is a sibling of /tap/capabilities... that is quite natural from a service deployment point of view and more or less conforms to the bundle and defaults approach described above. But the community thought it was a good idea to be able to navigate(?) from any endpoint to the capabilities endpoint. Either (flat or bundle) works fine if you use the capabilities the way that our cadc-registry client lib does (it is just cosmetic). But: the only way to go the bundle way is for TAP-1.1 to violate DALI-1.1 and/or retract that bit from DALI (admit mistake; issue erratum).
-- PatrickDowler - 2018-08-23
The alternative that does not break any rules and does not require any UWSRegExt is
-- PaulHarrison - 2018-09-05
Failing that, I think the current text of section 2.4 (as adjusted following production of the UWSRegExt document or something similar) could be fixed up to be usable by TOPCAT-like clients with some text along the following lines:
A TAP service consists of a related set of resource types, as listed in [the table in Section 2] : TAP-sync, TAP-async, VOSI-capabilities, VOSI-availability, and optionally VOSI-tables and DALI-examples. In some cases a client needs to identify a related "bundle" of endpoints providing one each of some or all of these resources in order to orchestrate its interaction with the TAP service. In TAP 1.0, this identification was straightforward, since only one instance of each resource type existed, and these were present at fixed locations relative to the TAP service base URL. In TAP 1.1, the capabilities document may declare multiple variants for some or all of these resource types, for instance to support different authentication contexts, and identifying the related bundle of endpoints that should be used given particular authentication requirements is therefore no longer trivial. Clients are free to combine the declared interfaces in any way they see fit, but the recommended way to interpret a TAP capabilities document as a list of TAP endpoint bundles is as follows:
The example [in section 2.4] therefore defines the default bundle and two authenticated bundles, defined by the security methods "ivo://ivoa.net/sso#BasicAA" and "ivo://ivoa.net/sso#tls-with-certificate" respectively. Each of these authenticated bundles explicitly declares its TAP-sync, TAP-async and VOSI-tables endpoints, and implicitly uses the default endpoints for VOSI-capabilities and VOSI-availability.
- There is one default bundle, corresponding to the TAP 1.0 endpoints. If the service allows unauthenticated access at all, it should usually be provided by this default bundle.
- There may also be one or more authenticated bundles. Each distinct security method declared on an interface in the capabilities document (value of an interface/securityMethod/@standardId XPath) defines one authenticated bundle. Each endpoint sharing the same security method is in the same bundle; if no endpoint is declared with that security method, the corresponding default endpoint is used.
That text is my attempt to make explicit what I think are the implicit assumptions about how the services described by the capabilities example in the PR are supposed to fit together. If I have misread those, it may need some adjustment.
Response: Document updated with approximately the above descriptive text.
-- PatrickDowler - 2018-08-29
Note however that this text does not address the issue of multiple interface elements within a capability element having the same xsi:type but different version attributes and (the same? different?) securityMethods, which complicates all this further - do we expect to see those?. I can think of ways to adjust the text above to accommodate those possibilities, but it starts to get pretty messy.
As far as I know(?), CADC has the only TAP services and the only TAP clients to make use of the scheme described in this PR to work with non-trivial authentication requirements, and I worry that the details of the scheme may be driven by their particular requirements or implementation details. I might be wrong about that, but I would very much like to see other TAP services that need authentication (LSST, IRSA, MAST, CSIRO?) look closely at this proposal, and preferably implement against it, to see whether it fits with their requirements. If it does, I will do my best to get TOPCAT working smoothly with it.
Other than that, this document mostly looks good to me. I have some minor comments:
-- PatrickDowler - 2018-08-29
Response: Document updated to provide a "Changes from TAP-1.0 to TAP-1.1" appendix and separate detailed changelog.
-- PatrickDowler - 2018-09-04
These comments obsoleted by later changes - see instead below
Note: the following comments concern PR-TAP-1.1-20180830 as written. I'm still arguing that this approach may not be the best one; if the discussion in this thread succeeds in persuading the authors to change the structure of the capabilities document, some of the following comments may be invalidated.
Thank you for including my text concerning bundles of services. Given this I think I will be able to write client code to work with this style of capabilities document. However, I still find it more unwieldy than the old services-at- well-known-relative-locations arrangement, and as I hinted in my earlier comments, I can still see a number of unanswered questions that this prescription leaves. Some of these are particularly from the point of view of a validator: it's not clear how much of the content of the example /capabilities document in section 2.4 is intended to be normative (the only way you're allowed to do it) and how much is just personal taste or dictated by the details of the particular usage scenario being illustrated here.
I am now reasonably happy with the prescriptions in the contentious section 2.4, and have successfully implemented against it (see the Implementations section above for details). I'm still slightly concerned that there has not been much variety of service implementations exploring the requirements and options for declaring variant authentication schemes but ... it's probably OK. I would therefore now basically recommend PR-TAP-1.1-20190626 for acceptance, but I still have some comments and corrections for the authors to consider.
INFO
element example here is invalid (for VOTable >=1.1): INFO payload goes in a mandatory value
attribute not as element text content. So it should read instead " <INFO name="QUERY" value="select * from stuff.items"/>
"
Response: these issues have been addressed in rev 5572. -- PatrickDowler - 2019-08-13
If you have minor comments (typos) on the last version of the document please indicate it in the Comments column of the table and post them in the TCG comments section above with the date.
Group | Yes | No | Abstain | Comments |
TCG | * | -- PatrickDowler - 2019-09-06 | ||
Apps | * | -- TomDonaldson - 2019-08-15 | ||
DAL | * | -- JamesDempsey -- MarcoMolinaro - 2019-07-04 | ||
DM | * | -- LaurentMichel - 2019-07-25 | ||
GWS | =* | -- GiulianoTaffoni check my comments on 20190626 version | ||
ReR | * | -- TheresaDower - 2019-07-10 | ||
Semantics | * | -- MarkusDemleitner - 2019-08-12 | ||
DataCP | ||||
KDD | ||||
SSIG | * | -- BaptisteCecconi - 2019-08-22 | ||
Theory | ||||
TD | * | |||
Ops | * | -- MarkTaylor - 2019-07-16 | ||
StdProc |
I | Attachment | History | Action | Size | Date | Who | Comment |
---|---|---|---|---|---|---|---|
![]() |
PR-TAP-1.1-20180416.pdf | r1 | manage | 645.9 K | 2018-04-18 - 06:24 | MarcoMolinaro | |
![]() |
PR-TAP-1.1-20190626.pdf | r1 | manage | 662.0 K | 2019-07-03 - 07:01 | MarcoMolinaro | PR-TAP-1.1-20190626 to update RFC before submitting to the Doc repo |
IVOA.net
Wiki Home
WebChanges
WebTopicList
WebStatistics
Twiki Meta & Help
IVOA
Know
Main
Sandbox
TWiki
Working Groups
Interest Groups
Committees