DALI-1.0 RFC

This document will act as RFC centre for the Data Access Layer Interface 1.0 specification. The current version of the document is available at:

http://www.ivoa.net/Documents/DALI/index.html

Review period: 2013-02-22 to 2013-03-22

In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your WikiName so authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.

Discussion about any of the comments or responses should be conducted on the DAL mailing list (dal@ivoa.net). However, please be sure to enter your initial comments here for full consideration in any future revisions of this document.

Implementation details

Links and description of existing interoperable implementations

Comments from the Community

Points by MarkusDemleitner, 2013-03-14

• (Throughout) I'd prefer "see Section 3" rather than "see 3" in cross references
• fixed -- PatrickDowler - 2013-04-24

• p. 7/8, In the example starting with "POST http://example.com/base/async-jobs", "<" characters are used to indicate (I guess) a response; In the following example, no such indication is provided. I'd suggest some other way to distinguish query from response (though I can't make a useful suggestion here), or even none, which would still be better than the current, confusing status.
• fixed -- PatrickDowler - 2013-04-24

• At the bottom of p. 8 you let people POST parameters after job creation. Paul Harrison (as one of UWS' parents) thinks that the libertinage with posting parameters has gone a bit too far. I tend to agree with him; even if we want people to post to parameters, I think we should really clarify the semantics (if only by referring to the later chapter on parameters). In the TAP implementation notes, there's some language: https://volute.googlecode.com/svn/trunk/projects/dal/TAPNotes/TAPNotes-fmt.html#uc-initpost Frankly, I'd say we should restrict what UWS allows, and probably to just POSTing to /parameters.
• email to discuss on DAL mailing list -- PatrickDowler - 2013-04-24

• p. 9, top "A concrete DAL service specification will specify one or more asynchronous ..." -- I'd say that should be "zero or more" after what's said above. The same applies to "A concrete DAL service specification will specify one or more synchronous..." a bit further down on that page.
• fixed to allow zero or more -- PatrickDowler - 2013-04-24

• p. 9, "The parameters used to specify the job are the same for synchronous and asynchronous DAL jobs." -- doesn't that contradict the statement above that services are free to do one thing on sync and something else on async?
• clarified to account for different semantics on different resources -- PatrickDowler - 2013-04-24

• p. 10, "no other vocab attributes are allowed in the document, for example:" -- I propose to move the "no other..." phrase behind the example; the example is for what's in front of the semicolon, not for the "no other" part.
• fixed -- PatrickDowler - 2013-04-24

• p. 10, "located inside the first element (the one with the vocab attribute) would indicate the presence of a single example." -- I'd rather just write "would contain an example referrable via the x fragment identifier." I'd leave out the language about "first element" and "single example" -- it confused me, for one.
• fixed -- PatrickDowler - 2013-04-24

• p. 11, I don't think the base-url specification is a good idea (if a machine has the examples URL, it has the access URL). And anyway, we really should not require it in each example -- that would make for extremely tedious reading (think of TAP examples: "On our http://dc.g-vo.org/tap endpoint, you can query...", and that 20 times?)
• email to discuss on DAL mailing list -- PatrickDowler - 2013-04-24

• p. 11, http-action property -- about the same thing. I don't even think we should have it, but I'm sure we must not require it.
• email to discuss on DAL mailing list -- PatrickDowler - 2013-04-24

• p. 11, generic-parameter -- for the record, I'm not a big fan of generic-parameter, either -- examples should IMHO be examples for a concrete protocol, known to the client. Hence, the semantics should be up to the concrete protocol (TAP talks about a query, SIAP about a region of interest -- and not about POS and SIZE parameters, that's up to the client to figure out --, SCS about center positions and cone sizes -- and not about RA, DEC, and SR). With http-action and generic-parameter, /examples moves in the direction of a machine readable standards markup language, and I don't think we want that.
• not in favour of changing this as it allows generic code/library to extract examples from a document -- PatrickDowler - 2013-04-24

• p. 12, "element with a vocab and examples as described above. If present, links back to a parent examples document must not be marked up with continuation attributes as this would create a loop." -- hm; we should ponder the implications of this. This means that the "primary" examples document is somehow special. That may be what we want, but we should be clear on this. I'd prefer things to be symmetric, but if we make such a rule, why not state clearly: "To ward against loops, documents referenced using continuation MUST NOT contain continuations themselves"?
• The more strict suggestion here could be used, but then a chain of pages with "more examples..." would not be allowed either because each would be the target of and contain a continuation. Maybe there is a simple-to-explain middle ground, but I agree that simple-to-explain is worth striving for here. -- PatrickDowler - 2013-04-24

• p. 14, "services may be registered and described without such an extension." -- Yes! In the early TAP takeup, the misconception about this kept many people from registering TAP services although that would have been possible even without TAPRegExt. Hence, I'd like to add here: "The use of standardID -- which should contain the IVORN of the standard a capability adheres to -- does not imply a particular xsi:type." Or something like that.
• clarified -- PatrickDowler - 2013-04-24

• p. 14, VOSI-tables -- should we explicitely state that when tables is not implemented, services must return 404 there?
• agreed. fixed -- PatrickDowler - 2013-04-24

• p. 19, "If a client requests a format by specifying the mimetype [...] the response that delivers that content must set that mimetype in the Content-Type header." I've always considered that requirement as fairly funky (and, by the way, an implementation hassle). So, I'd suggest we add some motivation here. Is this actually just about pulling thorugh text/xml in hopes that the browser does something sensible with this as opposed to application/x-votable+xml?
• purpose clarified -- PatrickDowler - 2013-04-24

• p. 22, "an HTTP status code (??)". Well... If we want to specify anything here, we should list the HTTP status codes we want to see, which is somewhat tricky given there could be proxies, etc. The reason I wanted this in is that I'd liked to have seen that errors actually cause HTTP error codes -- I still maintain it's unwise to return an error message of any kind with a 200 HTTP status code (excepting, of course, the /error document in UWS); it's an abuse of HTTP. If we cannot agree to change what the current DAL protocols do, we should just say "Working DAL services always return 200 OK, even if the execution ended with an error, as long as they can produce a VOTable (that might contain an error message), or a redirect (302 or 303) eventually resolving to such a resource. Clients must nevertheless be prepared for other status codes, in particular 401, 404, and 500." Still sounds weird to me.
• status codes for errors clariifed, 4xx or 5xx requierd for errors except in the case of getting an error document from the error resourcse of a UWS job -- PatrickDowler - 2013-04-24

• p. 22, "Successfully executed requests should result in a response with HTTP status code 200 (OK)..." Why only "should"? Also, we should add "eventually" here, since the immediate response could also be a redirect.
• changed to must eventually... -- PatrickDowler - 2013-04-24

• p. 23, "If the error document is being returned directly after a DALI-sync request, the service should use a suitable status codes (e.g. 4xx or 5xx for HTTP transport)" -- uh, I'm confused. The stuff I wrote above (about 200 codes for VOTable error messages) appeared to be what came out of the Sao Paulo DALI session. If that's negotiable after all, can we just say "Any error messages come with non-200 status codes" or similar in the section introduction?
• as above, hopefully clarified to reflect Sao paulo agreement -- PatrickDowler - 2013-04-24

• p. 23, "...RESOURCE element identified with the attribute type="results", containing a single TABLE element..." -- what do we gain by requiring this to be a single table? This is, after all, a fairly serious limitation, and I could well imagine some IVOA-sanctioned protocol for services like NED or VizieR, where the result of the query contains many "fragments" from different sources that have different columns. So, what would we break if we lifted this limitation?
• single table restruction removed, multiple resources already allowed, INFO element named QUERY_STATUS to convey OK - ERROR - OVERFLOW is probably ill-named at this point, but it is compatible with existing DAL specs -- PatrickDowler - 2013-04-24

Comment by Arnold Rots

In an attempt to be more specific, I suggest replacing the following text in Section 3.1.2:

Values never include a time zone indicator and are always interpreted as UTC [17].
In cases where values may be expressed using Julian Date (JD) or Modified Julian Date (MJD), these follow the rules for double precision numbers above and may have additional metadata as described in 4.4.

by:

Values must never include a time zone indicator and shall be interpreted as stated below.
In cases where values may be expressed using Julian Date (JD) or Modified Julian Date (MJD), these follow the rules for double precision numbers above
and may have additional metadata as described in 4.4.
All date-time values (ISO-8601, JD, and MJD) shall be interpreted as referring to time scale UTC and time reference position UNKNOWN, unless either or both of these are explicitly specified to be different [17].

-- ArnoldRots - 2013-03-15

• fixed -- PatrickDowler - 2013-04-24

The VAO has reviewed this document and, aside from Arnold's comment above, has no further comments to make.

-- MatthewGraham - 2013-03-19

Comments by Mark Taylor

• I made a number of comments on the 2012-12-10 version of this document on the DALI wiki page. Apart from a couple of typos, these don't seem to have been addressed. Please consider those comments repeated here. I haven't checked if Markus's comments from that page have been addressed.
• A number of the URLs in the examples (e.g. http://example.com/tap/ ) are marked up as http links in the HTML and PDF versions of the document. This isn't particularly helpful.
• Following on from Markus's comment above on MIME types (p.19) this requirement could be a bit counter-productive. For instance, what if a client requests application/x-votable+xml and the service wants to return application/x-votable+xml; serialization=BINARY2 ? This requirement would presumably prevent the service from adding that MIME type parameter. I'd echo Markus that if this is not well-motivated we could drop it.
• Sec 2.6: VOSI-tables. The effort to TAP-ise VizieR has thrown up scalability issues with the provision of tables metadata in a single XML document (see the presentation of Gilles Landais from the Implementation Feedback session in Sao Paulo here, about p.7). We will probably need to do something about this in the future. Now this is tied up with VOSI and VODataService rather than being defined here, so it's not a problem that needs to be solved in this document, but would it be better for DALI to either flag this up as something that might change in the future, or just point to those standards by name rather than including example XML so that future changes get picked up by default?

Comments by Pierre Fernique

• page 24 - 4.4.1 Overflow
  If the client specified a MAXREC value, but the server decides to reduce the number of records (in case of too large queries or server overload). Does the QUERY_STATUS should be ERROR, or OVERFLOW ? Logically, it should be OVERFLOW, but in this case, #rec < MAXREC in contradiction with the sentence "In the above examples, the TABLE should have exactly MAXRECrows"
• page 24 - 4.4 Use of VOTable
  "For columns containing coordinate values, the coordinate system metadata should be provided as described in [15]. For columns containing photometric values (including fluxes), the system should be described using Utypes as specified in the appropriate data model (most likely [16])."
  I'm not totally sure that a REC should refer to documents not yet endorsed by IVOA. This way to describe coordinates and photometry is not an IVOA recommendation ([15] is a note without IVOA decision and usage of phot utypes seems to be deeply modified since [16]). I suggest to remove this paragraph and references which are not really related to DALI purpose.

Comments from NormanGray (Semantics) 2013-04-04:

These refer to version http://www.ivoa.net/Documents/DALI/20121210/WD-DALI-1.0-20121210.html

Sect. 2: The table at the top of this section is neither referred to nor explained. Also what does 'required' = 'service specific' mean (does that mean it is or isn't required)?

"Special requests to modify the phase of the job cause the job to execute or abort." Does this mean "There exist requests intended to cause the job to execute or abort", or "Any phase-changing requests cause (as an error/side-effect) the job to abort"? I'm guessing the former.

Sect 2.1: in the examples, the HTTP transaction is illustrated with a line which looks like an HTTP Request-Line, such as "POST http://example.com/base/async-jobs" and "GET http://example.com/base/async-jobs/123" (that is, apparently illustrating an element of the wire HTTP transaction). However we also see "POST FOO=bar http://example.com/base/async-jobs/123/parameters" which isn't a valid HTTP Request-Line. This subsection should perhaps include a forward reference to section 3.

The reference to the UWS standard ways of associating parameters with a POST is rather elliptical. A more detailed reference, or perhaps an example of each of the mechanisms, would be useful. After having read the text here (noting that the UWS spec describes parameters only in abstract terms), I still don't know how to specify parameters in a DALI request.

Section 2.3: I think there are very few SGML parsers on the web. Perhaps a useful distinction would be against HTML5, which isn't (if I recall correctly) specified in terms of XML.

The http://purl.org/astronomy/vocab/examples namespace doesn't at present exist. I can help set this up.

Sect. 3.2.3 RESPONSEFORMAT: since this is talking about MIME^WMedia types, it should be explicit about this. RFC 2616 Sect 3.7 (for example) gives a grammar for these, which matches the grammar in RFC 2045 Sect 5.1. It would probably be most appropriate to define the Media types in this specification to be equivalent to RFC 2616 sect 3.7.

What's the rationale for including the 'short form' response types? It seems to add only complication.

In this section, the string 'RESPONSEFORMAT' is in a couple of places half-italicised.

This section remarks "Individual DAL services [...] are free to support custom formats by accepting non-standard values"; but earlier it only talks of "the general usage", so appears to neither mandate nor exclude any values, so it's not clear what 'non-standard' means in this context.

"A DAL service [...] should fail ( 4.2 ) where the RESPONSEFORMAT parameter specifies a format not supported by the service implementation." Presumably the HTTP code 406 Not Acceptable would be used here, if the transaction is happening over HTTP -- perhaps this should be spelled out.

What happens if a client makes a DALI request over HTTP, and includes an Accept request header with one Media Type, and a RESPONSEFORMAT parameter with a conflicting one? Is the answer different between synchronous and asynchronous requests?

Sect. 4.2 Errors: 'otehr' -> 'other'