DALI-1.0 RFC

This document will act as RFC centre for the Data Access Layer Interface 1.0 specification. The current version of the document is available at:

http://www.ivoa.net/Documents/DALI/index.html

RFC Review Period: 2013-02-22 to 2013-03-22

• The RFC period is closed and a new version of the document has been submitted to the document repository with the changes descrbed below.

• The TCG review period will begin once the new version is available. An alternate representation of the revised document is attache dbelow with change bars (some changes not show for clarity, eg large block of text that were removed).

In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your WikiName so authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.

Discussion about any of the comments or responses should be conducted on the DAL mailing list (dal@ivoa.net). However, please be sure to enter your initial comments here for full consideration in any future revisions of this document.

TCG Review Period: 2013-05-29 to 2013-06-26

Implementation details

Links and description of existing interoperable implementations

The specification describes common patterns and requirements for DAL services; it does not describe a single service that can be implemented. Still, existing TAP services implement most of the recommended practices with only small technical differences that can be addressed in a future version. Other DAL services like SIA and SSA are also more or less consistent with the recommended practices.

Comments from the Communityduring the RFC Period

Points by MarkusDemleitner, 2013-03-14

• (Throughout) I'd prefer "see Section 3" rather than "see 3" in cross references
• fixed -- PatrickDowler - 2013-04-24

• p. 7/8, In the example starting with "POST http://example.com/base/async-jobs", "<" characters are used to indicate (I guess) a response; In the following example, no such indication is provided. I'd suggest some other way to distinguish query from response (though I can't make a useful suggestion here), or even none, which would still be better than the current, confusing status.
• fixed -- PatrickDowler - 2013-04-24

• At the bottom of p. 8 you let people POST parameters after job creation. Paul Harrison (as one of UWS' parents) thinks that the libertinage with posting parameters has gone a bit too far. I tend to agree with him; even if we want people to post to parameters, I think we should really clarify the semantics (if only by referring to the later chapter on parameters). In the TAP implementation notes, there's some language: https://volute.googlecode.com/svn/trunk/projects/dal/TAPNotes/TAPNotes-fmt.html#uc-initpost Frankly, I'd say we should restrict what UWS allows, and probably to just POSTing to /parameters.
• email to discuss on DAL mailing list -- PatrickDowler - 2013-04-24

• p. 9, top "A concrete DAL service specification will specify one or more asynchronous ..." -- I'd say that should be "zero or more" after what's said above. The same applies to "A concrete DAL service specification will specify one or more synchronous..." a bit further down on that page.
• fixed to allow zero or more -- PatrickDowler - 2013-04-24

• p. 9, "The parameters used to specify the job are the same for synchronous and asynchronous DAL jobs." -- doesn't that contradict the statement above that services are free to do one thing on sync and something else on async?
• clarified to account for different semantics on different resources -- PatrickDowler - 2013-04-24

• p. 10, "no other vocab attributes are allowed in the document, for example:" -- I propose to move the "no other..." phrase behind the example; the example is for what's in front of the semicolon, not for the "no other" part.
• fixed -- PatrickDowler - 2013-04-24

• p. 10, "located inside the first element (the one with the vocab attribute) would indicate the presence of a single example." -- I'd rather just write "would contain an example referrable via the x fragment identifier." I'd leave out the language about "first element" and "single example" -- it confused me, for one.
• fixed -- PatrickDowler - 2013-04-24

• p. 11, I don't think the base-url specification is a good idea (if a machine has the examples URL, it has the access URL). And anyway, we really should not require it in each example -- that would make for extremely tedious reading (think of TAP examples: "On our http://dc.g-vo.org/tap endpoint, you can query...", and that 20 times?)
• email to discuss on DAL mailing list -- PatrickDowler - 2013-04-24

• p. 11, http-action property -- about the same thing. I don't even think we should have it, but I'm sure we must not require it.
• email to discuss on DAL mailing list -- PatrickDowler - 2013-04-24
• replaced base-url and htp-action with a single property named capability where the document specifies which service capability the example pertains to; this is optional so services that have only one functional capability (eg not counting VOSI resources) do not need to include it. -- PatrickDowler - 2013-05-24

• p. 11, generic-parameter -- for the record, I'm not a big fan of generic-parameter, either -- examples should IMHO be examples for a concrete protocol, known to the client. Hence, the semantics should be up to the concrete protocol (TAP talks about a query, SIAP about a region of interest -- and not about POS and SIZE parameters, that's up to the client to figure out --, SCS about center positions and cone sizes -- and not about RA, DEC, and SR). With http-action and generic-parameter, /examples moves in the direction of a machine readable standards markup language, and I don't think we want that.
• not in favour of changing this as it allows generic code/library to extract examples from a document -- PatrickDowler - 2013-04-24

• p. 12, "element with a vocab and examples as described above. If present, links back to a parent examples document must not be marked up with continuation attributes as this would create a loop." -- hm; we should ponder the implications of this. This means that the "primary" examples document is somehow special. That may be what we want, but we should be clear on this. I'd prefer things to be symmetric, but if we make such a rule, why not state clearly: "To ward against loops, documents referenced using continuation MUST NOT contain continuations themselves"?
• The more strict suggestion here could be used, but then a chain of pages with "more examples..." would not be allowed either because each would be the target of and contain a continuation. Maybe there is a simple-to-explain middle ground, but I agree that simple-to-explain is worth striving for here. -- PatrickDowler - 2013-04-24

• p. 14, "services may be registered and described without such an extension." -- Yes! In the early TAP takeup, the misconception about this kept many people from registering TAP services although that would have been possible even without TAPRegExt. Hence, I'd like to add here: "The use of standardID -- which should contain the IVORN of the standard a capability adheres to -- does not imply a particular xsi:type." Or something like that.
• clarified -- PatrickDowler - 2013-04-24

• p. 14, VOSI-tables -- should we explicitely state that when tables is not implemented, services must return 404 there?
• agreed. fixed -- PatrickDowler - 2013-04-24

• p. 19, "If a client requests a format by specifying the mimetype [...] the response that delivers that content must set that mimetype in the Content-Type header." I've always considered that requirement as fairly funky (and, by the way, an implementation hassle). So, I'd suggest we add some motivation here. Is this actually just about pulling thorugh text/xml in hopes that the browser does something sensible with this as opposed to application/x-votable+xml?
• purpose clarified -- PatrickDowler - 2013-04-24

• p. 22, "an HTTP status code (??)". Well... If we want to specify anything here, we should list the HTTP status codes we want to see, which is somewhat tricky given there could be proxies, etc. The reason I wanted this in is that I'd liked to have seen that errors actually cause HTTP error codes -- I still maintain it's unwise to return an error message of any kind with a 200 HTTP status code (excepting, of course, the /error document in UWS); it's an abuse of HTTP. If we cannot agree to change what the current DAL protocols do, we should just say "Working DAL services always return 200 OK, even if the execution ended with an error, as long as they can produce a VOTable (that might contain an error message), or a redirect (302 or 303) eventually resolving to such a resource. Clients must nevertheless be prepared for other status codes, in particular 401, 404, and 500." Still sounds weird to me.
• status codes for errors clariifed, 4xx or 5xx requierd for errors except in the case of getting an error document from the error resourcse of a UWS job -- PatrickDowler - 2013-04-24

• p. 22, "Successfully executed requests should result in a response with HTTP status code 200 (OK)..." Why only "should"? Also, we should add "eventually" here, since the immediate response could also be a redirect.
• changed to must eventually... -- PatrickDowler - 2013-04-24

• p. 23, "If the error document is being returned directly after a DALI-sync request, the service should use a suitable status codes (e.g. 4xx or 5xx for HTTP transport)" -- uh, I'm confused. The stuff I wrote above (about 200 codes for VOTable error messages) appeared to be what came out of the Sao Paulo DALI session. If that's negotiable after all, can we just say "Any error messages come with non-200 status codes" or similar in the section introduction?
• as above, hopefully clarified to reflect Sao paulo agreement -- PatrickDowler - 2013-04-24

• p. 23, "...RESOURCE element identified with the attribute type="results", containing a single TABLE element..." -- what do we gain by requiring this to be a single table? This is, after all, a fairly serious limitation, and I could well imagine some IVOA-sanctioned protocol for services like NED or VizieR, where the result of the query contains many "fragments" from different sources that have different columns. So, what would we break if we lifted this limitation?
• single table restruction removed, multiple resources already allowed, INFO element named QUERY_STATUS to convey OK - ERROR - OVERFLOW is probably ill-named at this point, but it is compatible with existing DAL specs -- PatrickDowler - 2013-04-24

---

Comment by Arnold Rots

In an attempt to be more specific, I suggest replacing the following text in Section 3.1.2:

Values never include a time zone indicator and are always interpreted as UTC [17].
In cases where values may be expressed using Julian Date (JD) or Modified Julian Date (MJD), these follow the rules for double precision numbers above and may have additional metadata as described in 4.4.

by:

Values must never include a time zone indicator and shall be interpreted as stated below.
In cases where values may be expressed using Julian Date (JD) or Modified Julian Date (MJD), these follow the rules for double precision numbers above
and may have additional metadata as described in 4.4.
All date-time values (ISO-8601, JD, and MJD) shall be interpreted as referring to time scale UTC and time reference position UNKNOWN, unless either or both of these are explicitly specified to be different [17].

-- ArnoldRots - 2013-03-15

• fixed -- PatrickDowler - 2013-04-24

---

The VAO has reviewed this document and, aside from Arnold's comment above, has no further comments to make.

-- MatthewGraham - 2013-03-19

---

Comments by Mark Taylor

• I made a number of comments on the 2012-12-10 version of this document on the DALI wiki page. Apart from a couple of typos, these don't seem to have been addressed. Please consider those comments repeated here. I haven't checked if Markus's comments from that page have been addressed.
• all comments accepted and included -- PatrickDowler - 2013-04-25

• A number of the URLs in the examples (e.g. http://example.com/tap/ ) are marked up as http links in the HTML and PDF versions of the document. This isn't particularly helpful.
• fixed -- PatrickDowler - 2013-04-25

• Following on from Markus's comment above on MIME types (p.19) this requirement could be a bit counter-productive. For instance, what if a client requests application/x-votable+xml and the service wants to return application/x-votable+xml; serialization=BINARY2 ? This requirement would presumably prevent the service from adding that MIME type parameter. I'd echo Markus that if this is not well-motivated we could drop it.
• clarified to allow services to respond with parameterised mimetypes if the client only requests the base mimetype

• Sec 2.6: VOSI-tables. The effort to TAP-ise VizieR has thrown up scalability issues with the provision of tables metadata in a single XML document (see the presentation of Gilles Landais from the Implementation Feedback session in Sao Paulo here, about p.7). We will probably need to do something about this in the future. Now this is tied up with VOSI and VODataService rather than being defined here, so it's not a problem that needs to be solved in this document, but would it be better for DALI to either flag this up as something that might change in the future, or just point to those standards by name rather than including example XML so that future changes get picked up by default?
• example removed and text added to say that there are scalability issues and services can use future VOSI standards since the version is in the capabilities doc -- PatrickDowler - 2013-04-25

-- MarkTaylor - 2013-03-21

Comments by Pierre Fernique

• page 24 - 4.4.1 Overflow
  If the client specified a MAXREC value, but the server decides to reduce the number of records (in case of too large queries or server overload). Does the QUERY_STATUS should be ERROR, or OVERFLOW ? Logically, it should be OVERFLOW, but in this case, #rec < MAXREC in contradiction with the sentence "In the above examples, the TABLE should have exactly MAXRECrows"
• clarified 3.2.4 so it is clear that a service can limit the value of MAXREC (lower than client request); overflow is relative to the effective MAXREC value, not just the client request -- PatrickDowler - 2013-04-25
• page 24 - 4.4 Use of VOTable
  "For columns containing coordinate values, the coordinate system metadata should be provided as described in [15]. For columns containing photometric values (including fluxes), the system should be described using Utypes as specified in the appropriate data model (most likely [16])."
  I'm not totally sure that a REC should refer to documents not yet endorsed by IVOA. This way to describe coordinates and photometry is not an IVOA recommendation ([15] is a note without IVOA decision and usage of phot utypes seems to be deeply modified since [16]). I suggest to remove this paragraph and references which are not really related to DALI purpose.
• agreed ; replaced this text with more generic advice to use standardised metadata mechanisms once available -- PatrickDowler - 2013-04-25

-- PierreFernique - 2013-03-28

---

Comments from NormanGray (Semantics) 2013-04-04:

These refer to version http://www.ivoa.net/Documents/DALI/20121210/WD-DALI-1.0-20121210.html

Sect. 2: The table at the top of this section is neither referred to nor explained. Also what does 'required' = 'service specific' mean (does that mean it is or isn't required)?

• added text explaing purpose of table -- PatrickDowler - 2013-04-25

"Special requests to modify the phase of the job cause the job to execute or abort." Does this mean "There exist requests intended to cause the job to execute or abort", or "Any phase-changing requests cause (as an error/side-effect) the job to abort"? I'm guessing the former.

• clarified phase-changign requests-- PatrickDowler - 2013-04-25

Sect 2.1: in the examples, the HTTP transaction is illustrated with a line which looks like an HTTP Request-Line, such as "POST http://example.com/base/async-jobs" and "GET http://example.com/base/async-jobs/123" (that is, apparently illustrating an element of the wire HTTP transaction). However we also see "POST FOO=bar http://example.com/base/async-jobs/123/parameters" which isn't a valid HTTP Request-Line. This subsection should perhaps include a forward reference to section 3.

• removed malformed and uninformative POST example -- PatrickDowler - 2013-04-25

The reference to the UWS standard ways of associating parameters with a POST is rather elliptical. A more detailed reference, or perhaps an example of each of the mechanisms, would be useful. After having read the text here (noting that the UWS spec describes parameters only in abstract terms), I still don't know how to specify parameters in a DALI request.

• tried to clarify this -- PatrickDowler - 2013-04-25

Section 2.3: I think there are very few SGML parsers on the web. Perhaps a useful distinction would be against HTML5, which isn't (if I recall correctly) specified in terms of XML.

• changed -- PatrickDowler - 2013-04-25

The http://purl.org/astronomy/vocab/examples namespace doesn't at present exist. I can help set this up.

• the URL itself resolves to a stub votable document saying there is no such vocabulary; maaybe we need to create it? -- PatrickDowler - 2013-04-25
• the document has been changed to use an ivorn to identfy the vocab in use (ivo://ivoa.net/std/DALI#examples) -- PatrickDowler - 2013-05-29

Sect. 3.2.3 RESPONSEFORMAT: since this is talking about MIME^WMedia types, it should be explicit about this. RFC 2616 Sect 3.7 (for example) gives a grammar for these, which matches the grammar in RFC 2045 Sect 5.1. It would probably be most appropriate to define the Media types in this specification to be equivalent to RFC 2616 sect 3.7.

• added reference to [5] aka RFC2616 -- PatrickDowler - 2013-04-25

What's the rationale for including the 'short form' response types? It seems to add only complication.

• rationale: prior usage DAL standards; I agree it doesn't really accomplish anything -- PatrickDowler - 2013-04-25

In this section, the string 'RESPONSEFORMAT' is in a couple of places half-italicised.

• fixed -- PatrickDowler - 2013-04-25

This section remarks "Individual DAL services [...] are free to support custom formats by accepting non-standard values"; but earlier it only talks of "the general usage", so appears to neither mandate nor exclude any values, so it's not clear what 'non-standard' means in this context.

• DALI itself does not mandate support for any oen format; concrete service specs do that. The last paragraph simply clarifies that service implementors can extend the formats supported to include any format they want to support, including their own custom formats. -- PatrickDowler - 2013-04-25

"A DAL service [...] should fail ( 4.2 ) where the RESPONSEFORMAT parameter specifies a format not supported by the service implementation." Presumably the HTTP code 406 Not Acceptable would be used here, if the transaction is happening over HTTP -- perhaps this should be spelled out.

• we tried to avoid specifying exact response codes in DALI; concrete service spec may specify them or may leave some flexibility to the implementors -- PatrickDowler - 2013-04-25

What happens if a client makes a DALI request over HTTP, and includes an Accept request header with one Media Type, and a RESPONSEFORMAT parameter with a conflicting one? Is the answer different between synchronous and asynchronous requests?

• We do not specify anything about Accept headers and although we are talking about HTTP we are only using the RESPONSEFORMAT parameter for content-type negotiation. This makes protocols usable from web browsers (where users have no control over Accepts headers) or other tools that might not let people mess with those; it also avoids implementation compexity when web servers or app containers mess with Accepsty headers outside the service implementors control. Short answer: RESPONSEFORMAT rules, Accepts ignored -- PatrickDowler - 2013-04-25

Sect. 4.2 Errors: 'otehr' -> 'other'

• fixed earlier -- PatrickDowler - 2013-04-25

Comments from TCG member during the TCG Review Period: 2013-05-29 to 2013-06-26

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or not the Standard.

IG chairs or vice chairs are also encouraged to do the same, althought their inputs are not compulsory.

TCG Chair & Vice Chair ( _Séverin Gaudet, Matthew Graham )

Approved -- SeverinGaudet - 2013-09-19

Applications Working Group ( _Mark Taylor, Pierre Fernique )

Basically approved, but some suggestions to consider for the final version:

• Sec 2.3: "DAL services should implement the /examples resource" - would a MAY be more appropriate than a SHOULD here?

changed to may -- PatrickDowler - 2013-09-18

• Sec 2.3.4: The continuation examples all have the typeof="more" attribute alongside property="continuation". I don't know enough RDFa to understand the significance of this, but should the typeof value have some discussion in the text? The only other place that typeof is used in DALI is in sec 2.3.3, where the usage of the typeof attribute is explicitly mandated.

removed typeof="more" from link examples -- PatrickDowler - 2013-09-18

• Sec 3.2.2: I'm still a bit unhappy about the VERSION resolution rules, as per my comment in the DALI wiki page. As stated it seems that implementations MUST change their behaviour at the instant when a standard version is promoted to REC. (An implementation A supports SXAP v1.2. On Monday it's a PR, so A's default behaviour must be SXAP 1.1. On Tuesday SXAP 1.2 is accepted as REC and A's default behaviour must be SXAP 1.2). Can this behaviour be SHOULD instead? Also, there's an unmatched close parenthesis in "... must be the same as VERSION=1.1)."

Two possible solutions: We could just say the default must be the latest version supported by the service, independent of whether it is a WD, PR, or REC? I'm OK with that... do we need to clarify that it is the latest declared version, where declared means the latest one described in the VOSI-capabilities and registry? That way someone could have a non-default prototype as long as they didn't put it in VOSI-capabilities. We could also make it more flexible by saying "SHOULD be the latest" instead of MUST. We could do both...

relaxed the requirement so default version should be the latest version (but services can default to older if they think this is better, eg in the short term) and removed the bit about treating a REC differently than a PR or WD so that a PR could be the default. -- PatrickDowler - 2013-09-19

• Sec 4: "URL in the HTTP header" -> write "URL in the HTTP Location header" instead for clarity?

fixed -- PatrickDowler - 2013-09-18

• Sec 4.2: "First, a DALI-async resource return a 404" - missing a verb, either "should" or "must"?

fixed -- PatrickDowler - 2013-09-18

• Sec 4.4: The existing text is an improvement on the previous version following Markus's comments about requiring a single TABLE. However, the existing text suggests use of multiple RESOURCE elements each labelled with type="results", and each containing a single TABLE (not explicitly stated, but suggested by "must contain, before the TABLE element..."). I'd be inclined to mandate a single type="results" RESOURCE element containing zero or more TABLEs (or even a hierarchy of RESOURCEs, if that's what the standard wants). But if not, be explicit about whether the number of tables per RESOURCE is limited to 1.

clarified so that there is exactly one resource with type="results" and other resources may be included (either as specified for a service or by an implementer for their own reasons) -- PatrickDowler - 2013-09-18

• Sec 4.4: I don't understand what's achived by making the QUERY_STATUS mandatory before the table, if its value might be overridden by an overflow value at the end. Should it just be required to appear at least once (before or after)? But maybe this is a backward compatibility issue.

This is partly backwards compat with client expectations and because it is a good practice in the implementation; in the case of an overflow or error during streaming output, having the OK status up front does tell the client that the request worked even if an overflow or error happened later. -- PatrickDowler - 2013-09-18

• Sec 5: Typos:
  • "reoved" -> "removed"
  • "smaller limit that a requested MAXREC": "that" -> "than"

fixed -- PatrickDowler - 2013-09-18

• Sec 6: Refs [15] and [16] no longer used in text, can be removed.

removed -- PatrickDowler - 2013-09-18

-- MarkTaylor - 2013-06-05

Data Access Layer Working Group ( Patrick Dowler, Francois Bonnarel )

Approved -- PatrickDowler - 2013-09-18

Data Model Working Group ( _Jesus Salgado, Omar Laurino )

Approved with comments.

• p. 5 Architecture: I don't think PhotDMv1-1 should be there.

PhotDMv1-1 removed -- PatrickDowler - 2013-09-19

• p. 5 Reference to UTYPEs. I would make less controversial statements: there is no standard for UTYPEs yet, but there are standards for DMs that describe standardized serializations. I would suggest replacing "Other data models may describe [...] the Utypes in the response will be those specified by the data model unless [...] service-specific utype attributes" to "The concrete DAL service specification defines whether the returned resources are serializations of a particular standard data model. For preserving backwards compatibility or to enable service-specific use cases, the concrete DAL service specification may explicitly specify the use of ad-hoc Utypes."

text changed -- PatrickDowler - 2013-09-18

• p. 6 Example Usage. This specification effectively defines a template for DAL service specifications, which will certainly be very useful. It would of great practical use to translate the specification in an actual document template. The Example Usage could be indeed a sample instance of such template.
• p. 7 Resources. It is stated that "DAL services are implemented as HTTP REST web services". However, later in the document (e.g. section 4.2) it is often addressed the case where the implementation is not based on HTTP. It would make sense to remove such "fallback" statements, unless the statement in section 2 is somewhat relaxed as one of the possible implementation strategies: this does not seem to be of any actual use, though.

clarified (footnote at start of Sec 2) that HTTP is currently the only specified protocol but a future version may include other protocols; hopefully that explains the sometimes more flexible wording

• p. 10 "MUST". There is some inconsistency in the way the specification uses the "MUST" keywords (and the like): for instance, on p. 10 must is used both lower-case and capitalized, although in most of the document these keywords are rendered in bold face (which makes sense). I spotted another inconsistency on p. 26 (section 4.4.1).

made them all lower case bold -- PatrickDowler - 2013-09-18

• p. 23 Responses. I believe it would be useful to clarify the meaning of "unspecified" in "an unspecified error document"

clarified -- PatrickDowler - 2013-09-19

-- OmarLaurino - 2013-06-18

Grid & Web Services Working Group ( Andre Schaaff, Andreas Wicenec )

Approved -- AndreSchaaff - 2013-09-22

Registry Working Group ( _Gretchen Greene, Pierre Le Sidaner )

Approved -- GretchenGreene - 2013-09-20

Semantics Working Group ( _Norman Gray, Mireille Louys )

The document relies on the ISO8601 specification to express TIME stamps, as mentionned in Section 3.1.2.

As other IVOA standards , like VOTable , f. i., rely on the FITS restricted usage of this specification, with UTC values possibly eluding the Z suffix , should we clarify that DALI supports the full ISO8601 specification?

This document has no clash nor inconsistencies with respect to Semantic standards .

Approved -- -- MireilleLouys - 2013-10-30

Response: Yes, there is a subtle inconsistency between ISO8601 and IVOA usage, which follows FITS usage and has been also noted in STC data model. I will change the language and references to make the format consistent with FITS and STC instead of ISO8601.

-- PatrickDowler - 2013-11-07

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Approved -- MikeFitzpatrick - 2013-09-26

Data Curation & Preservation Interest Group ( Alberto Accomazzi )

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Rick Wagner )

Standards and Processes Committee ( Francoise Genova)

• PR-DALI-1.0-20130919-changebars.pdf: PR-DALI-1.0-20130919 with changebars for TCG review comments dealt with so far