Difference: TAP11RFC (24 vs. 25)

Revision 252018-11-14 - MarcoMolinaro

 
META TOPICPARENT name="TableAccess"

Table Access Protocol 1.1 Proposed Recommendation: Request for Comments


TOC


The table access protocol (TAP) defines a service protocol for accessing general table data, including astronomical catalogs as well as general database tables. Access is provided for both database and table metadata as well as for actual table data.

This is the proposed revision (version 1.1) of the standard, including changes from community feedback, clarifications and a few optional features.

Latest version of IVOASTANDARD can be found at:

(This RFC was

  • started with respect to PR-TAP-1.1-20180416 PR - with comments before 2018-08-30
  • continued on with PR-TAP-1.1-20180830 PR - with comments after 2018-08-30 and roughly before end of Sep. '18
)
Added:
>
>
A chronological view on the authenticated endpoints in TAP-1.1 (up to the 2018 Fall Interop in College Park) is available here. This chronology was prepared for and then linked into this mail thread summarizing the outcome from College Park Interop's discussion.
 

Reference Interoperable Implementations and Validators

From Pat Dowler

The primary CADC TAP service (http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/) is a complete TAP-1.1 implementation. We have implemented TAP-1.1 changes over the whole time that the new version was developed, but I would like to draw attention to these in particular:

  • the VOSI-capabilities endpoint makes use of the prototype UWSRegExt interface types to denite sync and async endpoints with a variety of security methods
  • the tap_schema uses the new datatype,arraysize,xtype column description and this is directly seen in the VOSI-tables output use of VOTableType everywhere
  • we do use ObsCore compatible TAP-1.0 style description of the s_region column (char,*,adql:REGION in tap_schema.columns) so the ObsCore output in VOTable complies with the type and STC-S serialisation in ObsCore-1.1
  • we use the DALI-1.1 datatypes otherwise (point, interval, and polygon in use)
-- PatrickDowler - 2018-04-20

From Markus Demleitner

In Release 1.1, I believe DaCHS (http://soft.g-vo.org/DaCHS) implements all aspects of TAP 1.1 (though of course I also expect that a validator will uncover a couple of issues, which we'll fix as fast as possible). An endpoint to try things out is http://dc.g-vo.org/tap, which also has interoperated nicely with TAP 1.0 clients so far.

Pat's notes on tap_schema datatypes, adql:Region, and DALI 1.1 geometries apply here, as well.

-- MarkusDemleitner - 2018-04-23

From Mark Taylor

I have done client implementation, and partial validator updates, for the features in TAP 1.1 that have changed since TAP 1.0. In particular the authenticated service declaration/discovery described in the example in section 2.4 has been tested and covered by validator updates. It works.

Some other TAP 1.1-specific validator updates have been made, including tests of required foreign key declarations, "size"/arraysize consistency, removed REQUEST=doQuery parameters. Some other features, including timestamp comparisons, table column name upload behaviour, use of xtype and of DALI-1.1 datatypes, are not tested in taplint yet. In due course I hope to get round to adding validator coverage for these featues. There are just a few changes to the non-validator clients apart from authentication: avoid REQUEST=doQuery, report on new TAP_SCHEMA columns in metadata browser. I don't currently forsee other client code changes elsewhere in TOPCAT/STILTS related to TAP 1.1-specific features.

The updated client code (TOPCAT, taplint, other TAP-sensitive STILTS commands) can currently be found at ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/topcat-full_tap11.jar. Note however this implementation is provisional, and may be changed or withdrawn depending on future TAP 1.1 discussions.

-- MarkTaylor - 2018-10-22


Comments from the IVOA Community during RFC/TCG review period: 2018-04-23 - 2018-11-23

The comments from the TCG members during the RFC/TCG review should be included in the next section.

*Please note a new revision of the PR was released on October 24.*

The RFC has been twice extended (see PR details at start of page).

In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your Wiki Name so that authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.

Additional discussion about any of the comments or responses can be conducted on the WG mailing list. However, please be sure to enter your initial comments here for full consideration in any future revisions of this document



Comments from TCG member during the RFC/TCG Review Period: 2018-04-23 - 2018-11-23

*Please note a new revision of the PR was released on October 24.*

WG chairs or vice chairs must read the Document, provide comments if any (including on topics not directly linked to the Group matters) or indicate that they have no comment.

IG chairs or vice chairs are also encouraged to do the same, althought their inputs are not compulsory.

TCG Chair & Vice Chair

Applications Working Group

Data Access Layer Working Group

Data Model Working Group

Some minimal comments from my read through.. Laurent may have some more technical comments as well.

Section 1.2.6: 'simple key-vale' -> 'simple key-value'

Section 2.4: 'the mandatory chilsd' -> 'the mandatory child'

Section 2.7: 'Thus, for the asynchronous TAP queries, the parameter requirements must be satisfied (and errors returned if not) only when the query is run in (in the sense of UWS job execution).' Something seems missing.. the query mode maybe?

Section 2.7: ' may be created with with no parameters' == Extra 'with'

Section 5.1.3: 'URL should lead to a an error' == Extra 'a'

Grid & Web Services Working Group

Registry Working Group

Semantics Working Group

Data Curation & Preservation Interest Group

Education Interest Group

Knowledge Discovery in Databases Interest Group

Solar System Interest Group

Theory Interest Group

Time Domain Interest Group

Operations

Comments from Mark Taylor (2018-06-12)

I have some concerns relating to the approach taken in Section 2.4 to declaring service endpoints for a variety of security methods. The UWSRegExt extension referenced here has not been previously published, but that is now being addressed by the draft UWSRegExt Note currently in circulation. Some edits to section 2.4 will be required when the content of that document has been agreed. But even with a well-defined UWSRegExt, from a client point of view I see some difficulties with using a capabilities document like the one shown in Section 2.4 of PR-TAP-20180416.

The proposal there is fine for clients that know they want to find, say, a TAP sync endpoint using tls-with-certificate authentication; I believe that is the usage scenario for which this arrangement has been tested at CADC. However, a client like TOPCAT has different service discovery requirements: it needs to find a bundle of related endpoints (sync, async, tables, capabilities, examples) which together constitute a TAP service. From that point of view the mixed bag of interfaces-with-securityMethods proposed here is difficult to interpret. I made a short presentation in Shanghai complaining about this. To me, it would make more sense to declare a separate base URL for each securityMethod, with the subordinate endpoints (/sync, /async, /tables etc) hanging off each one in the standard TAP-1.0 pattern. I know Pat is not keen on that, and I admit I'm not sure if it could be made to fit easily with the existing VOResource/VOSI-Capabilities framework.

Response: These items from my DAL mailing list post of 2018-06-13 (this thread -- note by MarcoMolinaro) is the primary obstacle to this approach:

2. DALI-1.1 specifies (Section 2) that all endpoints in a service must be siblings of the VOSI-capabilities resource (except VOSI-availability); further, DALI also specifies that the capabilities resource must be named /capabilities (relative to the base).

3. VOSI-1.1 specifies that VOSI-capabilities must be provided with anonymous access and such capabilities must not include a securityMethod

So, you have a base URL and a single anonymous child {baseURL}/capabilities and all the TAP endpoints (eg sync and async query execution) have to be siblings of that (i.e. direct children of {baseURL}). That is just a flat list of interfaces and can't be presented as a bundle with current VOSI. Somewhat ironically, before that DALI-1.1 restriction the CADC TAP service used to place the username/password auth endpoints under a "subdirectory", so it used to be /tap/auth/sync and /tap/auth/async and now it is /tap/auth-sync and /tap/auth-async so everything is a sibling of /tap/capabilities... that is quite natural from a service deployment point of view and more or less conforms to the bundle and defaults approach described above. But the community thought it was a good idea to be able to navigate(?) from any endpoint to the capabilities endpoint. Either (flat or bundle) works fine if you use the capabilities the way that our cadc-registry client lib does (it is just cosmetic). But: the only way to go the bundle way is for TAP-1.1 to violate DALI-1.1 and/or retract that bit from DALI (admit mistake; issue erratum).

-- PatrickDowler - 2018-08-23

The alternative that does not break any rules and does not require any UWSRegExt is

  • /tap/sync
  • /tap/async
and

  • /authtap/sync
  • /authap/async
i.e. a separate auth interface which uses a different root URL

-- PaulHarrison - 2018-09-05

  • I agree with Paul - it's true that you addionally need a /authtap/capabilities to avoid breaking the sibling rule, but there's no reason you can't have that as well as (and perhaps with the same content as) /tap/capabilities. I've argued this in more detail on-list. -- MarkTaylor - 2018-09-12
Response: As already stated, this arrangement is not compliant with DALI-1.1 which says that all endpoints must be siblings of the /capabiliies endpoint (in the REST resource sense). The fundamental issue is that authentication is orthogonal to service functionaility -- VOResource included is (securityMethod) at the leaves of the xml hierarchy as that is the least intrusive way to do it . The CADC implementation used to be more hierarchical but we changed it when people wanted the DALI restriction. Please re-read the "but: ..." in the previous response. -- PatrickDowler - 2018-09-12

Failing that, I think the current text of section 2.4 (as adjusted following production of the UWSRegExt document or something similar) could be fixed up to be usable by TOPCAT-like clients with some text along the following lines:

A TAP service consists of a related set of resource types, as listed in [the table in Section 2] : TAP-sync, TAP-async, VOSI-capabilities, VOSI-availability, and optionally VOSI-tables and DALI-examples. In some cases a client needs to identify a related "bundle" of endpoints providing one each of some or all of these resources in order to orchestrate its interaction with the TAP service. In TAP 1.0, this identification was straightforward, since only one instance of each resource type existed, and these were present at fixed locations relative to the TAP service base URL. In TAP 1.1, the capabilities document may declare multiple variants for some or all of these resource types, for instance to support different authentication contexts, and identifying the related bundle of endpoints that should be used given particular authentication requirements is therefore no longer trivial. Clients are free to combine the declared interfaces in any way they see fit, but the recommended way to interpret a TAP capabilities document as a list of TAP endpoint bundles is as follows:

  • There is one default bundle, corresponding to the TAP 1.0 endpoints. If the service allows unauthenticated access at all, it should usually be provided by this default bundle.
  • There may also be one or more authenticated bundles. Each distinct security method declared on an interface in the capabilities document (value of an interface/securityMethod/@standardId XPath) defines one authenticated bundle. Each endpoint sharing the same security method is in the same bundle; if no endpoint is declared with that security method, the corresponding default endpoint is used.
The example [in section 2.4] therefore defines the default bundle and two authenticated bundles, defined by the security methods "ivo://ivoa.net/sso#BasicAA" and "ivo://ivoa.net/sso#tls-with-certificate" respectively. Each of these authenticated bundles explicitly declares its TAP-sync, TAP-async and VOSI-tables endpoints, and implicitly uses the default endpoints for VOSI-capabilities and VOSI-availability.

That text is my attempt to make explicit what I think are the implicit assumptions about how the services described by the capabilities example in the PR are supposed to fit together. If I have misread those, it may need some adjustment.

Response: Document updated with approximately the above descriptive text.

-- PatrickDowler - 2018-08-29

Note however that this text does not address the issue of multiple interface elements within a capability element having the same xsi:type but different version attributes and (the same? different?) securityMethods, which complicates all this further - do we expect to see those?. I can think of ways to adjust the text above to accommodate those possibilities, but it starts to get pretty messy.

As far as I know(?), CADC has the only TAP services and the only TAP clients to make use of the scheme described in this PR to work with non-trivial authentication requirements, and I worry that the details of the scheme may be driven by their particular requirements or implementation details. I might be wrong about that, but I would very much like to see other TAP services that need authentication (LSST, IRSA, MAST, CSIRO?) look closely at this proposal, and preferably implement against it, to see whether it fits with their requirements. If it does, I will do my best to get TOPCAT working smoothly with it.

Other than that, this document mostly looks good to me. I have some minor comments:

  • Sec 1.1: This is a very useful addition to the document, providing good signposting of TAP's position in the IVOA landscape.
  • Sec 2.7.2: the Note in the last paragraph warns about using the coordinate system string parameter in ADQL geometry functions. It would be relevant to note here that these parameters are deprecated in ADQL 2.1 (readers only familiar with ADQL 2.1+ may otherwise find this note hard to understand).
  • Sec 2.7.4: for clarity, the last paragraph should say something like "The output truncation caused by non-zero values of the MAXREC parameter...".
  • Appendix A: Especially since the history has been quite convoluted, with some items being added and later retracted, it would be useful for users of TAP 1.1, rather than IVOA archaeologists, to consolidate the change log subsections into one item "changes since TAP 1.0".
  • I've made some, hopefully uncontroversial, typographical fixes in Volute (r5045).
Response: Document updated to clarify these points, except the changelog (TBD with authors).

-- PatrickDowler - 2018-08-29

Response: Document updated to provide a "Changes from TAP-1.0 to TAP-1.1" appendix and separate detailed changelog.

-- PatrickDowler - 2018-09-04

Further Comments from Mark Taylor, regarding PR-TAP-1.1-20180830 (2018-09-12)

Note: the following comments concern PR-TAP-1.1-20180830 as written. I'm still arguing that this approach may not be the best one; if the discussion in this thread succeeds in persuading the authors to change the structure of the capabilities document, some of the following comments may be invalidated.

Thank you for including my text concerning bundles of services. Given this I think I will be able to write client code to work with this style of capabilities document. However, I still find it more unwieldy than the old services-at- well-known-relative-locations arrangement, and as I hinted in my earlier comments, I can still see a number of unanswered questions that this prescription leaves. Some of these are particularly from the point of view of a validator: it's not clear how much of the content of the example /capabilities document in section 2.4 is intended to be normative (the only way you're allowed to do it) and how much is just personal taste or dictated by the details of the particular usage scenario being illustrated here.

  • The 'TAP Base URL' interface no longer serves the purpose it used to (defining where to find the sync and async endpoints). Is its inclusion in the capabilities file mandatory?
Changed:
<
<
Response: The base url with type ParamHTTP services to let TAP-1.0 client find the base url and use the service. The explicit anonymous sync and async endpoints can be included and this allows a generic capabilities-reading client that doesn't know anyhting about TAP naming conventions to correctly find endpoint URLs (this is how the cadc-registry java lib works -- without TAP knowledge). So, those can be included, but I didn't think about whether they must or not. If we didn't need backwards compat it would be must and not redundant. Every other service we have explicitly defines anon interfaces and it is only TAP-1.0 compat (where we mistakenly put two endpoints under one interface because of the whole sync vs async battle and not wanting to scare anyone) so I think must is the right solution and should or must include the base ParamHTTP for backwards compat.
>
>
Response: The base url with type ParamHTTP services to let TAP-1.0 client find the base url and use the service. The explicit anonymous sync and async endpoints can be included and this allows a generic capabilities-reading client that doesn't know anyhting about TAP naming conventions to correctly find endpoint URLs (this is how the cadc-registry java lib works -- without TAP knowledge). So, those can be included, but I didn't think about whether they must or not. If we didn't need backwards compat it would be must and not
Deleted:
<
<
  • Interface versions. The example capabilities document uses the attribute @version='1.1' on all the interface elements within the TAP capability (one base URL, 3 sync and 3 async). Is it mandatory to do that? If not, how is a client to understand different version values here? I raised this question on the grid mailing list here, and got replies along the lines of, well, probably people won't want to do that; but in a standards document I would like to see something more concrete (e.g.: all interface children of a given capability MUST have the same @version attribute).
Response: After the fix below, the current spec says should. If someone wanted to keep their old 1.0 endpoint around they could have a mix of 1.1 and 1.0 interfaces; I can't see why someone would want to do that but I don't see why it would not be allowed. In our cadc-registry client, the convience method that works 99% of the time is getServiceURL(resourceID, standardID, securityMethod, interfaceType) so we have not found any need for this. We have deployed some services with 2 minor versions but in those cases we used versioned standardID values... so in general there are reasons for deploying multiple versions of s/w, but I didnt need to with TAP. -- PatrickDowler - 2018-09-12

  • At the start of section 2.4, it says 'For TAP-1.1 we use the same standardID but the version attribute of the capability should use the minor version (e.g. version="1.1").' I think it must mean instead "... the version attribute of the interface ..." - is that right? Maybe it's not... I asked a similar question about the UWSRegExt Note here, but I didn't follow Markus's answer much beyond "That is confusing." If it really does mean to say capability rather than interface here, there should at least be some more explanation in the text, since the example document has lots of interface/@version but no capability/@version attributes.
Response: Good catch - this should say interface as capability does not have a version attribute (fixed). -- PatrickDowler - 2018-09-12

  • According to the table in Section 2, it is permitted to list multiple per-security-method capabilities endpoints in the same way as for sync, async, tables, etc endpoints. How are such alternative capabilities documents to be written, and how are they to be interpreted? I suppose the purpose of this is to contain different auth-specific TAPRegExt-type metadata such as resource limits. But such an alternative capabilities document could contain different service endpoints too. If a client wants to use security method X, MAY/SHOULD/MUST it examine the X-specific capabilities endpoint to see whether that in turn reports X-specific sync/async/tables/etc endpoints, and if so should it use the endpoints from the original capabilities document or the X-specific capabilities document referenced therein? What about an alternative capabilities document that contains additional capabilities endpoints itself? That sounds a bit silly, I'm guessing it's not the intention, but that should be clarified in the text. Perhaps the most sensible thing would be to allow only xsi:type="tr:TableAccess" capabilities in such security-method-specific capabilities documents? But I might be missing some use cases (or missing the point).
Response: Yes, there is a table entry that permits that but I think it is something old (trying to support different limits exactly as you guessed) that should have been removed because VOSI-1.1 says that capabilities has to be anonymous to be of any use, so you can only really have the one at baseURL/capabilities. That table entry should be removed. Aside: with the current TAPRegExt there isnt even a way to have differnet limits for sync and async so we decided to punt auth-specific limits down the road. Alternate capabilities removed. -- PatrickDowler - 2018-09-12

If these points are cleared up in the text I will try to update taplint to work with TAP 1.1-style capabilities documents.

Some other points related to the capabilities document:

  • The subsections in section 2 are headed "{sync}", "{async}", "availability", "/capabilities", "/tables" and "/examples". I haven't found an explanation of the curly-bracket syntax anywhere, but given (per the table in section 2) that none of these have fixed locations any more I guess they should all have section headings, as well as references elsewhere in the text, that look the same.
Response: I am going to rename those sections in DALI-style (TAP-async and TAP-sync) and remove the curly braces. -- PatrickDowler - 2018-09-12

  • I only see one TAP implementation (CADC) that uses the multi-security-method capabilities file, since the quoted DaCHS instance (http://dc.g-vo.org/tap) has no authenticated interfaces. As I've said before, it would be nice to see some other implementations doing this; not so much to prove that you can write the capabilities document, but to demonstrate that this arrangement of authenticated endpoints serves the purposes of data providers that need to do authentication. Several data providers (apart from CADC) have said that they have a pressing need for authentication, so perhaps some of them could be persuaded to look seriously at this.

  • The example in section 2.4 still relies on the draft UWSRegExt Note. I presume the content of that document will be finalised by the time that TAP 1.1 is accepted.
Response: The shift to refering to a note makes this non-normative; I have clarified this in the doc. -- PatrickDowler - 2018-09-18

Minor points concerning the rest of the document:

  • Sec 2.7.6: ".. but restricted as described in section 4" . I think(?) this concerns the table_name restrictions described in section 4.2 - if so it would be better to reference section 4.2 specifically. If not, what is it referring to?

  • Sec 4: "Implementors may also use alternate character or numeric types (e.g. short or long instead of int or unicodeChar instead of char) as long as query execution is consistent with the recommended types." This comment is no longer really applicable since the generalisation of the types mandated in the following subsections; it should be removed or adjusted (e.g. to explain what "string" and "integer" mean).

  • Sec 5.3: the example includes the function invocation "circle('ICRS', 34, -4, 2)". This does not follow the advice given in sec. 2.7.2: "Clients should avoid using the coordinate system argument to geometric functions (use null value or use an alternate function without the coordinate system argument if available)."

  • Appendix A.4: "Clarified required and optional ADQL geometric functions. Changed language about mandatory ADQL geometry function support back to optional (should in the case where the tables contain spatial coordinates) so TAP now recommends a set of functions to support and notes others are simply optional (or not supported in the case of REGION)." : I can't find the text mentioned here in the document; it seems to have gone missing between PR-TAP-1.1-20170830 and PR-TAP-1.1-20180416. I think it should be reinstated (? though I don't know why it got removed) but if not, then this change log item should be adjusted accordingly.

  • Appendix A.5: "Added arraysize in TAP_SCHEMA..." . The "xtype" column was also added, that should be noted here.
I also made some hopefully uncontroversial typo fixes at r5137 and 5139.

Response: minor points fixed -- PatrickDowler - 2018-09-12

-- MarkTaylor - 2018-09-12

Note: All changes to here are in svn commit 5141. -- PatrickDowler - 2018-09-18

Standards and Processes Committee


TCG Vote : 2018-10-25 - 2018-12-07

If you have minor comments (typos) on the last version of the document please indicate it in the Comments column of the table and post them in the TCG comments section above with the date.

Voting period has been extended twice due to new PRs released on August 30 and October 24.

Group Yes No Abstain Comments
TCG        
Apps        
DAL        
DM        
G&WS        
RoR        
Semantics        
DataCP        
KDD        
SSIG        
Theory        
TD        
Ops        
StdProc        



<--

-->


<--  
-->
 
META FILEATTACHMENT attachment="PR-TAP-1.1-20180416.pdf" attr="" comment="" date="1524032689" name="PR-TAP-1.1-20180416.pdf" path="PR-TAP-1.1-20180416.pdf" size="661394" user="MarcoMolinaro" version="1"
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback