TAP V1.0: Request for Comments

This document will act as RFC centre for the latest TAP V1.0 Proposed Recommendation.

Following the comments below on the 20091006 version and discussions at the November Interop meeting, I have uploaded the Holiday Version of the TAP V1.0 Proposed Recommendation to resolve the outstanding issues. In addition to addressing the comments below, the latest document includes a new informative section (6) with a suggested syntax for STC-S; this has been done because STC-S is currently an IVOA note and we do not want to promote it to the status of a non-reviewed standard simply by mandating it's use. The included syntax follows the Note, but it is not part of the normative standard and STC-S support does not effect compliance to the specification. I realise this appears to be adding substantially to the TAP specification, but this was the least disruptive way to resolve the issue.

Note: this version has change bars for non-editorial changes; that will be cleaned up for the final version, along with the references to other pending recommendations.

According to the discussions in Garching 2009, TCG review should now be able to proceed and finish by mid-January; thank you for your patience.
--PatrickDowler, 2009-12-25

Note that comments applicable to the above document are in the next section with comments for earlier versions can be found here.

---

Review periods

Initial Review period: 2009 July 10 – 2009 August 21
Second Review period: 2009 October 09 – 2009 November 06
Third TCG Review Period: 2010 January 18 – 2010 February 03

In order to add a comment to the document, please edit this page and add your comment to the list below (include your WikiName so authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.

Discussions about any of the comments or responses should be conducted on the DAL mailing list, dal@ivoa.net.

---

Comments from the TCG (2010 January 18 -- 2010 February 03)

Applications (Tom McGlynn, Mark Taylor) _[Approval TBD]

Data Access Layer (Keith Noddle, Jesus Salgado) [Approved]

I'll repeat my earlier comments here : my sincere thanks to Pat for all his hard work getting the document to its current state and further thanks to all who contributed along the way.

Data Model (Mireille Louys, AnitaRichards) _[Approved] MireilleLouys

Thanks to Pat and reviewers for this very detailed and useful document. I approve it to be an IVOA recommendation.

Grid&Web Sevices (Matthew Graham, Paul Harrison) [Approval TBD]

Registry (Ray Plante, Aurelien Stebe) [Approval TBD]

Semantics (Sebastien Derriere, Norman Gray) [Approved] SebastienDerriere

Apart from a few typos listed below, I approve the "Christmas version" of TAP ! (page numbers from the PDF version)

• 2.3 p13 : returnedif -> returned if; and "with with" -> with
• 2.5 p19 in footnotes 3 and 4 : chose -> choose
• 2.6 p21 : "Second, a the client" -> remove 'a' or 'the'
• 2.7.2 p25, 2nd item : describing its support? for TAP

VOEvent (Rob Seaman, Alasdair Allan) [Approval TBD]

VO Query Language (Pedro Osuna, Yuji Shirasaki) [Approval TBD]

VOTable (Francois Ochsenbein) _[Approval TBD]

Just a few typos which could be fixed in the final document:

• sec.2.3.2 VERION=1.0 --> VERSION=1.0
• sec.2.6 ... The VOSI tables source provides the the same metata as... [double the]
• sec.2.6 (TAP_SCHEMA): I guess the expression 'must contain the following columns' does allow additional columns in local TAP_SCHEMAs -- could be specified for clarity.
• sec.6 is an excellent addition and will help a lot the implementors ! I have however two problems related to these: (1) the <coordsys> following the UNION and INTERSECTION keywords which seems to be in contradiction with the definition of the region which has its own definition of the <coordsys> with clearly specified UNKNOWNFRAME default; and (2) the choice of the GEOCENTER default and the very end of this section because GEOCENTER requires an exact knowledge of the epoch to derive the Earth's position, whereas BARYCENTER is the actual IRCS reference position and we should recommend its usage.

I'm ready to approve the document if these few questions can be fixed; I also wish to thank a lot the contributor(s) (merci Pat)!

Standard and Processes (Francoise Genova) [Approved 3 February 2010]

I also think that we have to move on ASAP with TAP. Could there be a link to the reference implementations from the RFC page?

As remarked by others, there are strong dependencies with yet-to-be-REC standards, in particular VOSI and UWS. Mitigations measures have been applied in the TAP text, but as explained in the 2009 Roadmap (p. 14) VOSI and UWS should have been consolidated with TAP agreements and therefore have to go on in the Recommendation process ASAP if we want to pursue at least some level of consistency.

This will also help to adjust TAP if necessary as recommended by Bob.

Astro RG (Masatoshi Ohishi) _[Approval TBD]

Data Curation & Preservation (Bob Hanisch) [Approved, 2 Feb 2010]

I sent a list of typos to Pat, so these should be fixed along with taking out the change bars when the final version is published. It is clearly time to move forward and get implementations going, then come back as necessary to update and improve.

Theory (Herve Wozniak, Claudio Gheller) [Approved]

Approved without reservation!

TCG (ChristopheArviset, Severin Gaudet) _[Approved]

Thanks to Pat for all the detailed editing in the 20091225 version, following comments from TCG and discussions at Garching Interop. I think we should now move forward and therefore I approve the document.

---

Comments from the TCG (2009 October 09 -- 2009 November 06)

Note: something odd happened during html export that put odd characters in various places; will fix so no need to report on that; the PDF should be ok.

Applications (Tom McGlynn, Mark Taylor) _[Approval TBD]

This document clearly defines a core functionality for generic queries of tables using either ADQL and PQL. This functionality is central to the development of VO applications and with this document as a guide both data providers and data clients should be able to build very powerful services using these core capabilities. Promoting this document to a recommendation should encourage the widespread adoption of the core TAP functionality leading to an immense increase in the fundamental capabilities of the VO.

The TAP interface is very complex with with strong interactions with a myriad of other VO protocols. A number of issues in ancillary areas have arisen since the release of the RFC, in TAP metadata, table uploads, VOSI capabilities, version negotiation, geometry support, and the interfaces of TAP to W3C and other IVOA protocols. A few of these have been addressed in changes made to the document between RFC and TCG review. It is unclear if substantive changes of this kind were appropriate at that point, since they were not themselves subject to general review. Overall these issues raise doubts that this document is an adequate guide to building a fully conformant TAP implementation.

Given the juxtaposition of these opportunities and concerns, Applications abstains in the approval process, with a recommendation that, regardless of the outcome of this process, work begin immediately on a new version of the document which focuses not on new functionality but on a clarification and resolution of issues in these ancillary areas.

response from PatrickDowler: after further consultation with the Applications WG chair, several important clarifications and improvements have been made in producing the 20091225 version.

Data Access Layer (Keith Noddle, Jesus Salgado) [Approved (conditional)]

The revisions planned to the current document will produce a usable standard that meets the crticial needs of the community. I therefore approve this document.

I would like to extend my thanks to the authors, espiecially Pat Dowler without whom we would not have achieved this standard. The document represents an excellent baseline from which we can expand and improve in the future.

Data Model (Mireille Louys, AnitaRichards) [Approval TBD]

Grid&Web Sevices (Matthew Graham, Paul Harrison) [Approval TBD]

Registry (Ray Plante, Aurelien Stebe) [Approved (conditional)]

I have two TCG-level comments:

• Section 2.9 indicates that TAP requires the use of VOTable 1.2 or greater. Section 2.9.1 specifically talks about use of the INFO element based on the pre-TCG-reviewed version of VOTable 1.2. Subsequently the definition of the INFO element changed, so Section 2.9.1, including all examples need to be updated for this change.

response from PatrickDowler: Examples have been fixed in the 20091225 version.

• As has been pointed out in the mailing list, the description of VOSI outputs (section 2.7.2) is still a bit too vague; it is not clear exactly what should be returned. Part of the problem is that we have not yet defined a TAP-specific capability extension; however, my sense is that we will not want to delay the TAP spec to figure this out. Taking this into account, I have posted to the DAL list (dal/0910/1620.htm) some additional suggested text that could be put into this section to clarify what needs to be returned. Probably of most practical importance, an example is included.

response from PatrickDowler: Suggested content included in the 20091225 version.

I'll note that overall I was impressed with the level of detail covered in this document. I can approve this document once these issues are addressed.

-- RayPlante - 28 Oct 2009

Semantics (Sebastien Derriere, Norman Gray) [Approval TBD]

Comments from NormanGray. There are minimal interactions with the Semantics WG, so the following are comments on exposition and the document itself. I haven't worked through the various comments above, so some of these might be duplicates.

Hrumph. I was looking at the 2009 June 7 version of the document. It might be useful to put a link to the current version of the document in the first line of this page, and not a four-month old version. It's not clear to me, therefore, how many of the comments below are still useful.

Document as a whole:

• The document TITLE element is still given as 'IVOA Document Template'
• The document character set is wrong. It's served as ISO-8859-1, but contains UTF-8 characters (most noticeably double-quote characters, but also, in sect 2.4, some characters in code examples). One option would be to serve the document as UTF-8, but Bruno has said that's problematic, so the document should either be transcoded to 8859-1 or the problematic characters converted to entities.
• The page numbers in the table of contents make no sense in an HTML document, and would be misleadingly wrong if the document were printed out.
• There are several errors in references -- I didn't check all the cross references.

1 Introduction: ADQL and PQL are mentioned. It would be useful to have references to the relevant Recommendations.

1.1.3 VOSI: "specifies base service interface" should be "specifies a base service interface"

1.2.2 Synchronous queries: "Metadata queries are always executed synchronously". Is this a MUST. If so, it might be better documented as such in s1.1.2, which should include a cross-reference to where the metadata queries are more fully documented.

1.3 Interface overview (informative): Footnote 2 is redundant, given the discussion of the 303 See Other response later in this section. Also, this section doesn't seem notably more 'informative' than the presumably normative subsections earlier in this section.

2.2.2 /async: There's a reference to UWS, and reference [5]. However, reference [5] is to RFC2396. I haven't bothered checking other references and cross-references -- they're surely automatically generated and this error was just a glitch.

2.3.3 LANG: "The service should return an 'unknown query language' as described in 2.9" -- perhaps "return an 'unknown query language' error...". This section refers to "the subsequent query parameter(s)". However as noted in s2.1.12, the parameters can come in any order. Perhaps "the other query parameter(s)" would be better.

2.3.4 Query: The quoted "yyyy-MM-dd[ HH:mm:ss[.SSS]]" is not a legal ISO-8601 datetime format. If the included space in that string is deliberate, it's not part of an ISO-8601 datetime spec, but indicates a separate date and time specification. In any case, the more restricted format of RFC3339 or <http://www.w3.org/TR/NOTE-datetime> would make better specs: both agree that yyyy-mm-dd[Thh:mm:ss[.ss]] is a suitable time. The same is true in s2.3.5 (the TIME=... example has a correctly-formatted time, though with a trailing slash). Note that since this syntax doesn't include a possible time zone or UTC offset, it presumably requires that everything be in UTC -- it would be good to make that explicit.

2.3.6 FORMAT: MIME type strings are case-insensitive, too, according to RFC2045 (ie, APPLICATION/x-VOTable+XmL is a VOTable MIME type). Well, RFC2045 specifies that the Content-Type header's values are case-insensitive, but I think this can be taken to imply that MIME types are always case-insensitive.

2.3.8 MTIME: There are multiple ISO-8601 range list formats -- do you really want to require TAP services to have to support "2007-03-01T13:00:00Z/P1Y2M10DT2H30M"? Neither RFC3339 nor the W3C datetime note include ranges -- perhaps the best thing here would be to declare that the MTIME parameter format must be a pair of ISO-8601 instants separated by a slash (that's one of the legal ISO-8601 formats). Also (picky), this section doesn't indicate whether the time interval is open or closed (I'd suggest inclusive at the start, and exclusive at the end, of the interval).

2.5 Table Upload: there's no discussion of authentication and authorisation here. I presume that's been taken care of elsewhere, in which case a reference to that would probably be useful here.

2.5.2 Inline Table Upload: A reference to RFC2046 and RFC2388 would be useful here

2.7.1 Data and metadata queries: refers to the TSV spec as [15], when it should be [16]. This mentions "Column values may not contain the TAB character" -- this should be "must not", since "may not" is ambiguous between "must not" and "might not".

2.7.3 Errors: This section is terribly vague. If I were implementing a TAP client, I wouldn't have a clue what to expect here -- all it seems to say is that I'll get an HTTP status returned within the HTTP request! Providing a 200 (OK) response with an error document seems ... eccentric, when there are plenty of sensible alternatives in the HTTP speec (I know SOAP does this, but that can hardly be said to be an argument in favour).

2.8.2 Version number changes: paragraph one says "no more than two integers", whereas paragraph two talks of "[a] version number change at the third level". Also the paragraph one text permits a single integer ("no more than two"). Which is it? One, two or three integers?

2.9.2 Version mismatch errors: these are signalled by an "ERROR" in an info element. That seems a bit non-specific. Wouldn't a more specific error code be useful.

4 Extended capabilities: there seems to be no way of registering extension names, even implicitly with, say, a reversed domain name. In a standard for networked operations, this seems to be asking for trouble.

7.1.1 Introduction: RFC2616 defines the 'http' scheme, but (as noted in s7.1.2) it's RFC2396 which defines URIs.

8 References: Not all of the linked URLs are clickable. For example, the VOSI URL doesn't work -- whatever it is that's converted this to HTML has simply coloured this text blue!

VOSI is included in the reference list twice, as [6] and [14]. Note that VOSI is still a WD, not a Rec (can I insert my standard moan about this?) Reference [15] refers to "EITF", should be "IETF"

response from PatrickDowler: Issues still present in last version have been fixed in 20091225 version, except the charset issue (discussing a solution with doc coordinator)

VOEvent (Rob Seaman, Alasdair Allan) [Approval TBD]

VO Query Language (Pedro Osuna, Yuji Shirasaki) [Approval TBD]

VOTable (Francois Ochsenbein) [Approved (conditional)]

(Comments by FrancoisOchsenbein on 2009-11-05 on TAP document dated 2009-10-26, improved a lot compared to the previous version, thank you!)

• Two comments related to VOTable:
  1. Sorry for the changes in VOTable <INFO> tag, which was changed following the comments during RFC to ensure compatibility with previous versions; as pointed out by Ray, the examples should be corrected (<DESCRIPTION> and </DESCRIPTION> should be removed)
  2. the FORMAT parameter (section 2.3.6): is there any relation with the possible serialisations in VOTable (<TABLEDATA> in 'pure' xml, <FITS> and <BINARY>) ? Would it mean that, if application/x-votable+xml is specified, that the <BINARY> serialisation is acceptable ? Or is there any means of specifying which serialisation the client would prefer ? It can make a huge difference for an application processing many huge TAP outputs

response from PatrickDowler: Examples have been fixed in 20091225 version.

• One comment related to the TAP_SCHEMA: the indexing description still seems to be incomplete, the order of columns in a multi-column index should be specified somewhere, and the unicity of an index is an important feature not described. Anyway I still feel that describing completely the indexes is not relevant for TAP -- just the flag specifying that a column is indexed (or is the first column used in a multi-column index) is enough. It would be more important to describe the keys (which combination(s) of columns define uniquely a row)

response from PatrickDowler: WG deemed it not worthwhile, to fully describe multi-column indices in this version.

• My last comment is related to the specification of errors/overflows (sections 2.7.3 and 2.7.4): I feel it's dangerous to have no method to specify an error or an overflow in non-VOTable output. Isn't it possible to imagine some way of giving this information for other output formats than VOTable ?

response from PatrickDowler: too costly to write a proper spec for all output formats for this version (to support more meta information with the table).

Provided these questions are sorted out, I would be happy to approve the document.

Standard and Processes (Francoise Genova) [Approval TBD]

Astro RG (Masatoshi Ohishi) _[Approved (conditional)]

I can go along with the document after the suggested changes have been fixed. Let us have the TAP v1.0 as soon as possible.

Data Curation & Preservation (Bob Hanisch) _[Approved (conditional)]

These comments are mostly presentation details which are easily fixed, however there are a couple of issues of substance. None of these are show-stoppers in my view, however, and I would be ok going ahead to REC if things are tidied up. Another option would be to hold TAP at PR for a period of time so that we could see a larger number of fully functional, interoperable implementations, and get further feedback from implementers regarding where the specification needs more detail.

The labels “normative” and “informative” do not seem to be applied consistently. Only Section 1.4 of the Introduction, for example, is labeled “informative”, but it seems appropriate that all of Section 1 be labeled “informative”. On the other hand, Section 7 has the “must” and “should” language of a standard and would seem to be “normative”. If the point is to recap HTTP rules then it might be better to move this to an Appendix (e.g., content that is explicitly informative) so it is clearer that these rules are about HTTP and not about TAP.

Section 1, para 1, both ADQL and PQL should have references here, on their first occurrence in the document.

Para 2, “or in some other format” would imply that virtually anything is ok. Better would be something like “or in one of several other formats specified herein.”

Section 1.1, sentence is missing an “and” after “metadata queries”.

Section 1.1.2, first sentence has awkward phrasing in “a subset and patterned after”. Perhaps “...to standardized tables that are a subset of, and patterned after, information schema in RDBMSs...”

Section 1.1.3, first sentence needs to be with “The” and another “the” is needed before “base service”.

In last sentence, “IVOA registry” needs a reference.

Section 1.2.1, “...will not be usable with other services unless they implement the exact same data model.” This does not seem correct to me. Isn’t the point that the data models need to be sufficiently similar that like elements are comparable? For example, one could certainly compare two TAP services with very diverse contents if both provided the same contents in the area of interest to the user. So I would have said “a sufficiently similar data model” or something along those lines.

Section 1.2.2, I am not sure that “PQL is more abstract than ADQL” (actually I might have said ADQL is more abstract than PQL), but it really doesn't matter. I would drop this phrase and just have “PQL can be used in some cases without first querying the metadata tables; PQL parameters carry sufficient meaning to enable...”.

In last sentence, it should be “Details...are...”

Section 1.3.2, last sentence, insert “are” before “likely to fail”.

Section 1.4, first sentence needs “The” at the beginning.

Footnote 2 ends with “(see other)”. See other what? Actually, I know that this means that HTTP code 303 stands for “see other”, so it is just the parentheses that make this a little confusing. Writing “HTTP code 303: See Other” would solve the problem.

Section 2.2, reference [5] at end of 1st para should be [3]

The list of UWS resources is not completely consistent with the UWS specification. In particular, UWS does not define a “termination” URI, and the URI is “parameters”, not “param”.

Section 2.3, last sentence, I would have thought that a response to a spurious parameter should at least include a warning rather than being completely ignored.

Section 2.3.1, 3rd para, “This is...” should be “These are...” as it us referring to “additional parameters”.

Section 2.3.3, “...is ADQL...” should be “...are ADQL...”. The parenthetic descriptions of ADQL and PQL are unnecessary here, as this has been said elsewhere.

Section 2.3.4, 4th para, “...and the services supports...” should be “...and the service supports...”

Section 2.3.5, 1st para, “...are specified in elsewhere...” ???

3rd para, “...and the services wants...” should be “...and the service wants...”

End of 4th para, suggest rewriting as “Details on how to use table references in PQL are given in the PQL specification document. [ref]”

Section 2.3.7, 2nd para, “This value” should not be in italics.

Section 2.3.9, “(e.g., scheme is vos or param)” I think needs to be clarified

Section 2.5, 3rd sentence, missing “the” before “uploaded table” and missing “a” before “legal ADQL table”.

Section 2.5.2, 1st sentence, missing “the” before “caller” In the example, this reader does not grasp the “boundary=AaB03” stuff. Is this obvious to others?

Section 2.6, the 1st sentences seems to go around in circles. In skeleton form: “The [] schema defines tables in the SCHEMA schema that contain [] metadata required to use the tables...”. At some level I think it makes sense, but I’m hoping that this can be reworked!

Section 2.6.1, “The schema_name values are unique.” Unique within what scope? How is this to be enforced? I have the same question regarding the definition/scope of “unique” in 2.6.2. 2.6.3, and 2.6.4.

Section 2.6.3, column_name unit, cites “VO standard format”. We don’t actually have an explicit document on standard units. VOTable refers to SI units and FITS conventions. I think this needs a footnote or something to explain the situation.

Section 2.6.4, there is a block of text starting with “Data types and how they map...” which seems out of place here, as this is not to do with Foreign Keys. Does this belong in the general text of 2.6 at the beginning?

In the middle of this paragraph “...where the services chose...” should be “...where the service selects..”, and “...which columns are returned by default” would be better as “...those columns that are returned by default.”

Section 2.7.3, after bullets, “...service containers an cannot...” should be “...service containers and cannot...”

Section 2.8.1, remove hyphen in “version-number” and add reference after last sentence (to IVOA document conventions).

Section 2.8.3, what is meant by “...level two”?

Section 3, 5th paragraph, “The resource document should include the table metadata...”. I think this is getting a bit far afield from TAP and straying into best practices for registries. I’d rather the “should” were a “may”. The footnote (6) is very wishy-washy and I think can be deleted.

Section 5.1 and 5.2, there are several places where there is extraneous space in the example HTTP GET and POST commands, particularly “/tap /sync/”

In 5.1, as noted before, the UWS resources are not totally consistent with the UWS specification.

In the middle of this section, the sentence that begins “Practically, ...” is a very odd structure. I don’t think one typically starts a sentence with an adverb like that. It would be better as “In practice it is very hard...”.

Section 6, rather than “prototyping” I’d suggest “further implementation experience”.

response from PatrickDowler: All these editorial issues have been corrected or text clarified in the 20091225 version.

Theory (Herve Wozniak, Claudio Gheller) [Approval TBD]

TCG (ChristopheArviset, Severin Gaudet) _[Approved]

Thanks to Pat for all the detailed editing in the 20091225 version, following comments from TCG and discussions at Garching Interop. I think we should now move forward and therefore I approve the document.