(1 vs. 41) VOSpace21RFC < IVOA

Revision 412018-05-25 - FrancoisBonnarel

META TOPICPARENT	name="WebPreferences"

VOSpace v2.1 Proposed Recommendation: Request for Comments

RFC page for the IVOA VOSpace 2.1 Proposed Recommendation.

The latest VOSpace 2.1 Proposed Recommendation (the second PR of this RFC) can be viewed here:

VOSpace-20180502.pdf

This PR has taken into account all comments up until April 16, 2018.

Change Overview

The change notes since version 2.0:

From version 2.1-20170405:

Replaced the dated Custom Protocols section with a new Public Share Protocol section
Updates from working group feedback

From version 2.1-20150601:

Added REQUEST=redirect for further optimized pullFromVoSpace
Removed "search" and "find nodes" until use cases and implementations are presented
Changed pullToVoSpace, pushFromVoSpace to be client orchestrated.
Changed Transfer in XSD to accept parameters.
Recreated and expanded all webservice operation examples
Changed XSD element authType to SecurityMethod to match IVOA standard.
Fixed text inconsistencies
Clarified role of the node busy flag
Ported source to ivoatex format

From version 2.1-20140805:

Addition of optimized HTTP GET method of data transfer for pushToVoSpace, pullFromVoSpace
Addition of authType to Protocol in XML schema for transfer negotiation.
Added preliminary list of standard authType URIs
Deprecated view=data convenience method of data transfer
Corrections to minor XML format errors in the examples throughout the document.

An earlier VOSpace discussion page can be found here:

http://wiki.ivoa.net/twiki/bin/view/IVOA/VOSpace21Spec

Comments from the IVOA Community during RFC period: 2017-04-06 - 2017-05-18

Implementation and Validators

CADC and OATS-INAF implementations of VOSpace 2.1, both are based on the VOSpace source code from OpenCADC. One item remains to be fully 2.1 compliant: only redirect on a synctrans parameterized request when the parameter "request=redirect" is provided. This is described here:

Issue: v2.1 Compliance - Obey request=redirect parameter

The github project cadc-test-vos is a validator that tests for 2.0 compliance.

The CADC has two VOSpace client implementations, one written in java (mostly used for testing) and one written in python. The python client is the CADC production client that is distributed to users and can be seen/installed here: CADC VOSpace client

When the CADC implemented version 2.1 of the VOSpace service, the python client that was built on the 2.0 specification was not changed. The fact that no errors or disruption was seen by users is evidence of the backwards compatibility between version 2.1 and version 2.0. -- BrianMajor - 2018-05-16

Comments from Pierre Fernique

Implementation references

Presently, it seems that there is one VOSpace 2.1 server working (CANFAR) declared in the VO registry. MSSL, CASU and WFAU seems to be out of order. Is there already VOSpace 2.1 clients ?

Abstract

p2 - It is a little bit surprising to have directly in the abstract a note about the (un)compatibility with the previous releases of the standard. May be a dedicated section, just after the Abstract, should be more adapted.

=> I agree. I have made the abstract more general and created a 'Previous Versions' subsection in the Introduction. -- BrianMajor - 2017-04-12
More largelly, the fact that the compatibility between clients and servers of different versions is rarelly insured questions me. This choice has always the same big impact. The consequence is to force the servers and the clients to implement VOSpace 1.0, 2.0 and 2.1 the time that all potential clients (resp. servers) have upgraded their codes. This remark is not limited to VOSpace, we have the same challenge for SIAv2 which is a totally new standard compared to SIAv1.

=> This is true. In the case of VOSpace, I think the version with the most implementations will gain support in the clients. We can try to share and make widely available client implementations for use by many. -- BrianMajor - 2017-04-12

Introduction

p5 - This section provides the analogy "node = file". Should we have to extend this analogy as "node = file or directory" ? This remark comes from our experience for implementing VOSpace client in Aladin. The role and functions on "directories" are not so clear in the previous release documents. And concretely, our experience shown us that the directory manipulations are generally not, or very lighted implemented. The situation could be improved, if we specify immediately that a node is potentially a directory (cf p11 - figure 2)

=> Good point. I have changed the text to say that a node can be thought of as a file or a directory. -- BrianMajor - 2017-04-12

Typical use of a VOSpace service

p5 - This subsection is important to understand the role of VOSpace, but I am wondering if it is a little bit too detailed as this level of the introduction (XML examples, ...)

VOSpace identifiers

p9 - The alternative '!' or '~' substitution characters should be justified. Why two substitutions characters ? Is it really required ? I suspect that it could be a source of potential bugs (cf. section 2.1 - identifier resolution). It seems that the response is provided p64 "Note that the # sign in PROTOCOL is URL encoded as %23 and that the alternate separator ‘~’ is used instead of ‘!’ in the TARGET".

=> You are correct in that the ~ was introduced so that it doesn't not have to be escaped in HTTP requests. I have added a 'Note' in the VOSpace Identifiers section describing this reasoning. As for whether it is required or not: I suspect that the original specification would have been better off using only ~ to begin with, but the ! character was preserved so as to not break existing implementations. Perhaps in version 3.0 we can deprecate the use of !. -- BrianMajor - 2017-04-12
P10 - The end of this section cites the standardID of this specification. May be, a short explanation of what is a "standardID" would be welcome to avoid the possible reader confusion with the node identifier (same section). This standardID is also described in page 36. Maybe this last definition should be sufficiente.

=> To make it more clear on the first reference to standardID on P10, I have added a short sentence explaining how it is used in the VO Registry. -- BrianMajor - 2017-04-12

VOSpace data model

p12 - "LinkNode" is described. But we do not know if we have to understand only dataLinkNode, or potentially ContainerLinkNode ?

=> There is this sentence: "The link target can be a URI that points to any type of resource, including other VOSpace Nodes (within the same VOSpace service or in another service), or external resources outside VOSpace altogether." I think this makes it clear that a link node can point to any type of node. -- BrianMajor - 2017-04-12
p13 - The support of all type of nodes is not required (end of the section). But does it means that a VOSPace could be not supporting hierarchy (flat view only) ?

=> That is correct. If container nodes are not supported then a heirarchy of nodes is not supported. -- BrianMajor - 2017-04-12

Property identifiers

p14 - URN is introduced here but not described. I suggest a end page note, or a RFC reference on URN.

=> I have added a reference to the RFC for URN Syntax in this section. -- BrianMajor - 2017-04-12

Property descriptions

p15 & 16 "Note that at the time of writing, the schema for registering PropertyDescriptions in the IVO registry has not been finalized" - This remark is repeated at various places in the document, but concretely, there is no progress on this point since VOSpace 1.0 (2007). Maybe these remarks should be simply removed.

=> Good idea. I've removed the three instances of that remark. -- BrianMajor - 2017-04-12

Standard properties

p16 - The standard properties identifiers are IVOID. But following the last Registry recommendation, any IVOID must be declared in the VO registry otherwise they are not valid. Actually, "ivo://ivoa.net/vospace/core" is not registered yet (or not accessible by the two active Full queryable resistries: euroVO and NAVO).
p17 - "ivo://ivoa.net/vospace/core#group...". This is the first (and the unique) mention of "group" notion. Surprisingly, there is no notion of "owner" (only creator, publisher and contributor which are probably not at the same level). It seems that this document does not described these notions (except briefly p36 "Nodes are considered to be owned by the user who created them"). May be, it could help the implementors to have a section describing these notions.

Cababilities

p19 - "Note that at the time of writing, the schema for registering Capability-Descriptions in the IVO registry has not been finalized" - same remark as p16

=> Done. -- BrianMajor - 2017-04-12

View descriptions

p24 - "Note that at the time of writing... " - same remark as p16

=> Done. -- BrianMajor - 2017-04-12
p25&26 - "pullToVoSpace" and "pushToVoSpace" are cited here, but never introduced before. May be a reference to the associated section would be welcome.

Public share protocol

p29 - A "standard ID" for "public share protocol" is provided in this section (new in VOSpace 2.1) => http://wiki.ivoa.net/twiki/bin/view/IVOA/VOSpacePublicShare. This ID is not compatible with regular IVOA standardID based on IVOID contrary to the other "standard protocols" defined in 3.5.3 (ex: ivo://ivoa.net/vospace/core#httpget). Is there a reason for this alternative nomenclature ?

=> Hmm... the idea behind the ID for the public share protocol was that the URI conveniently pointed to the page documenting the rules of the protocol. I personally like this concept as it encourages implementors to follow the pattern. However, I could be convinced to change it to the standard IVOID form. Additional thoughts from anyone? -- BrianMajor - 2017-04-13

Transfers

p30 - In the XML example, there are security method identifiers for which the vocabulary seems to take various origins ( ivo://ivoa.net/sso#tls-with-certificate, http://www.w3.org/Protocols/HTTP/1.0/spec/html#BasicAA). Reference to associated documents would help.

=> I've added a reference to the SSO document so these securityMethods can be more clear. -- BrianMajor - 2017-04-12

REST binding

p35 - "These names are part of the VOSpace 2.1 standard". It lets suppose that other elements in the document may be not relevant to VOSpace 2.1. I suggest to remove any reference to VOSpace 2.1 as this document is 2.1 specification (except for history list...)

=> Your comment made me realize that this section needed to be updated to match the recommendations of IVO Identifiers 2.0. There is now a new 2.1 synctrans endpoint that has the version number in the ID (not in path). I hope it is more clear now how the resource IDs work with the different versions of the specification. -- BrianMajor - 2017-04-14
p35 - ".auto", ".null" are cited in this section, but without explanation. The description of their respective role should be appreciate.

=> Removed from this section. -- BrianMajor - 2017-04-21
p36 - "For backwards compatibility support, services SHALL include the securityMethod element"... but as there is no real compatibility support (cf abstract), this sentence is a little bit surprising.

=> A 2.1 service is backwards compatible with a 2.0 client, so I think this sentence is still important to state. -- BrianMajor - 2017-04-21

Create Node

p41 - "If the Node xs:type...". This is the first citation of "xs:type" which seems to refer to node types.This notion should be defined somewhere before in the document, maybe page 11 in "Nodes and node types"

=> Good idea. I've added a short description of how to indentify node types in XML representations in "Nodes and node types" -- BrianMajor - 2017-04-21
p43 - In the example, (and the following examples), there are 2 paragraphes: Request:... Response:... but the Response paragraph also contains the HTTP request part of the dialog. Strickly speaking the "Response" starts at the ligne "HTTP/1.1 nnn...", the lines above ("PUT....") are the "Request". I would suggest to change the "Response" title by "HTTP dialog" which seems to be more appropriate.

=> I think I agree with you, but before I go and make all those changes, does anyone else have an opinion on this? -- BrianMajor - 2017-04-21

Transfering data

p55 - The difference between pushToVoSpace (sending data to a service) and pullFromVoSpace (reading data from a service) is not obvious. Same for pullToVoSpace (importing data into a service) and pushFromVoSpace (sending data from a service). I suppose that depends of the origin of the action, but it is just a speculation. As these 4 actions are a key point of VOSpace, small examples, or a small graphic would be appreciate.

=> Okay, I've added a bit more text for now on each of the four transfer protocols. I can add a graphics too if you think it needs more. -- BrianMajor - 2017-04-21

Change since last version

p82 - We have the list of the changes from the previous "version 2.1-20150601". The full history is provided in appendix D. It is probably preferable to have all the history in the same place.

=> Okay, I've consolidated these two sections. -- BrianMajor - 2017-04-12
p82 - a "(?TBD)" is still present

=> Thank you, removed. -- BrianMajor - 2017-04-12

Message schema

p82 - "VOSPace 2.0" must be replaced by "VOSpace 2.1" in the introduction sentence.

=> Done. -- BrianMajor - 2017-04-12

-- PierreFernique - 2017-04-11

Comments from Markus Demleitner

(1) The compliance matrix has links to findnodes and searches sections, which are commented out; hence the referencing fails. I think you should just remove the matrix lines.

=> I don't see the referencing failures, but I've taken out those lines anyhow. -- BrianMajor - 2017-09-19

Hm. I still see them in svn rev. 4428. Check VOSpace.log after a build and look for Reference.*undefined, and you'll see the warnings. Btw. I took the liberty to fix your SSOAUTH2 citations, too, but they will only resolve once the record is in ADS, which is pending... ah well, I'll tell you live if you're interested. -- MarkusDemleitner - 2017-09-28

(2) You're changing the target namespace of your schema; from what I can see in the spec, this should break 2.0 clients, as they will not find their elements any more. Are you sure yours isn't a classic case for the XML schema versioning note?

=> Well, yes, this was actually the original use case for the XML schema versioning note and I had forgotten to bring in the change. Thanks for noticing. The XSD namespace is now 2.0 and there is a version attribute in the <transfer> root element. This is the only element that has XSD changes (introduction of security method and transfer param). The search and find features (which I think you are referring) were never actually in the XSD. (That's why they were dropped from 2.1.) -- BrianMajor - 2017-09-19

Hm -- in volute rev. 4429 there's still several examples with the v2.1 namespace. -- MarkusDemleitner - 2017-09-28

(3) I get a bit nervous about "uri: the vos:// identifier for the node, URI-encoded according to RFC3986 (Berners-Lee et al., 2005)." (p. 12) For one, I'm not sure if this is the right place to say anything about encoding, but more importantly: Are these URIs supposed to be comparable? If so, you should somewhere define how they're intended to be compared (e.g., authorities and schemes in general are case-insensitive -- are they here? What about normalising percent-encoded values?). Similarly for nodes and target, p. 13.

=> I don't think there's a use case involving comparing VOS Identifiers. If you can live it, I'd rather not tackle these references to encoding in this minor version. -- BrianMajor - 2017-09-19

(4) "URI or URL" (e.g., sect. 3.2, p. 14): RFC3986 discourages a distinction between URI, URL, and URN, so I'd suggest to just write "URI" in these instances.

=> Done. -- BrianMajor - 2017-09-19

(5) The example in 3.3.5, "One use case for this would be a VOSpace 2.1 client talking to a service that implements both VOSpace 2.0 and VOSpace 2.1, where the client is acting on behalf of a third party agent that only understands VOSpace 2.0...", isn't quite convincing to me -- when you just change the minor version, shouldn't the 2.0 side be interoperable with any 2.1 side? What purpose does it then serve to discover a 2.0 service?

=> Good point. I've removed that paragraph as it was really only confusing matters. -- BrianMajor - 2017-09-19

(6) p. 22, "uri: an optional boolean flag to indicate that the View preserves..." -- are you sure this is "uri"? If so, I think a few words on why you chose this name (it is a bit odd for a boolean flag, no?) was chosen might be helpful.

=> Editor error. This text was meant for the 'optional' flag. Thanks. -- BrianMajor - 2017-09-19

(7) Perhaps for the next version, I think having a single section on identifier rules that's then referenced from sections like 3.2.2, 3.3.2, 3.4.2 (just highlighting differences as necessary) will reduce repetition and make the document more readable.

=> I think that would be a good improvement. -- BrianMajor - 2017-09-19

(8) p. 26 "to the application/binary MIME type." After consulting my /etc/mime.types I don't think application/binary is defined. I think you mean application/octet-stream. I mention in passing that these guys aren't actually called MIME types but "media types". But since nobody calls them that I'll shut up about that particular point.

=> I think you're right so I've changed it to application/octet-stream. -- BrianMajor - 2017-09-19

(9) p. 26 "the available views (the server) thinks is the most" -- why the parentheses?

=> Removed. -- BrianMajor - 2017-09-19

(10) I'm a bit split on 3.5.4 -- on the one hand, this offloads what in effect is part of the spec to a wiki page, which I'd usually like to avoid, and the separation of concerns between what's specified on the wiki and what's in this main spec is unclear to me (what if we find contradictions later?). On the other hand, I see that implementors should be made aware of this protocol. Perhaps you can just have a section "Common external protocols defined so far", clarify their standard status ("If you provide the protocols with the following URIs, you should regard the referenced specifications as if they were part of this specification. On the other hand, incompatible changes to protocols listed here are forbidden by this specification.") and then just enumerate ("self-documenting") URLs?

=> Okay, good idea. I've made changes to that section that are a blend of your suggestions and the original content. Hopefully it sits better with you now. -- BrianMajor - 2017-09-19

(11) sect. 3.6.2 "limited to the UWS synchronous transfer endpoint" -- is there such a thing? I'd say the "UWS" needs to go from that statement.

=> "UWS" has been removed. -- BrianMajor - 2017-09-19

(12) In several places, there still are hardcoded section references in the document (search for "section [0-9]"; example: "section 5.5" on p. 66, which I'm pretty sure doesn't point to where it should any more). They need be replaced by label-ref pairs (I'd have done that myself, but I don't understand the document well enough to confidently do that).

=> Thanks, now cleaned up. -- BrianMajor - 2017-09-19

(13) With my hat as ivoatex author on, I remark that my recommendation is to keep physical lines shorter than 80 characters if at all possible; it makes for more informative diffs without further tooling.

=> Okay, thanks. -- BrianMajor - 2017-09-19

-- MarkusDemleitner - 2017-06-12

Comments from TCG members during RFC period: 2017-04-06 - 2017-05-18

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

First of all, I have to thanks the editors. This new version of the document has been improved compared to the 2.1 first version: more examples, more definitions and vocabulary explanations - clearly easier to read and to understand.
VOSpace is a protocol quite heavy to code and presently we have very few working implementations even with the VOSPace predecessors (clients and servers). I hope that this new document/version will facilitate the life of the future implementors.

Suggestions, questions and typos:

page 7. the IVOA archictecture figure is place at a very bad position in the page (latex decision probably)... just after the sentence "The description will resemble this:". I suggest to force to move the IVOA figure a little bit further in the document.
page 8 "Appendex A" -> "Appendix A"
page 14: Where is described the possibility of "multiple small properties". What is the syntax for the alternative "Uri in properties" (indirection). And what is the format of the pointed additional metadata ?
page 14: "... unless the property description defined a specific delimiter" - How to describe this alternative character of the "comma" value delimitor ? I do not have found information on this point.
page 17: The group notion described through the properties "groupread" and "groupwrite" is not described in the document. How groups are defined and managed ?" etc... Is it related to the "owner" list (page 37).
page 19 "openStack Swift" => may be a reference would be welcome here
page 21 "If a service implementation supports more than one version of the VOSpace interface then these capability URIs can be used with a VOSpace service to identify different VOSpace capabilities for a node." - Does it imply that we will have to specify these VOSpace capabilities for each node, individually ? even if globally my Vospace is supporting both 2.0 and 2.1 ? Is there an alternative ?
page 25 : "pullToVoSpace" and "pushToVoSpace" are introduced in this note. A forward reference should be appreciated (6.4.3..).
page 26 - Container views (3.4.5). How to specify a recursive or not recursive container views ? are there predefined views URIs for this distinction ?
page 27 Protocols: "note: endpoint will only contain a value after the response from the service is received"... not clear for me.
page 27: the "protocol" notion is not present in the "node hierarchy graph" (page 12). It would be helping. Is it related to each node individually, or globally
page 29 External standard protocols - "The following section lists protocols that are identified by ..." ... "there is only external standard protocol: the public share protocol" : these two sentences in the same paragraph are a little bit contradictory. In fact, this "external protocol" for providing a persistent URL has been just addded in this last document version. It is really required for VOSPace standard ? Or is it just an example of what can be an external protocol ?
page 34 - bad line folding: "-" at the begining of the line "-, and the server..."
page 36: not clear for me if we have to replace # per ? only for new 2.1 bindings, or for all, or as we want. The example at the end of page 36 is confusing : ivo://ivoa.net/std/VOSpace#sync-2.1"
page 38 "Node authorization is similar to that of Linux file system permission handling with ownership and group membership"... but VOSpace is distributed by nature. So how to manage owners and groups in this context. How VOSpace can support X509 system ? or YellowPage mechanism or ... cf my remark for page 17.
page 43 "The capabilities list for the Node cannot be set using this method"... A reference to the section explaining how to create the capabilities list would be welcome here.
page 94. Thanks for the compliance matrix in annex B

Suggestions for a future release 2.2 :

There are many OPTIONAL elements in the this standard. It is not so good for the interoperability (and concretelly we have very few VOSPace clients). I would suggest in the next standard version to avoid (if possible) too many optional features (either remove it, or impose it)
The auto generation of a unique URI is an important function. Presently it is implemented (optional) as a kind of "hack" via a reserved suffix keywork ".auto" to the URI. I am thinking that this important feature could be implemented in a better way, as a creatorNode parameter or something like this.

-- PierreFernique - 2018-04-24

==> Thanks for your detailed review Pierre. I have addressed some of your points above, but not all. I've left the ones that would require significant enough changes as to warrant another review.

-- BrianMajor - 2018-05-02

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Added:

>
>

We validate VOSPace which is a sophisticated and well written standard.

We would like to make a (non blocking) suggestion. Maybe between adoption and REC publication or for next version.

- A model diagram like the one in fig 2 with all the classes, not only "node". Relationships should be shown but no need to do a full UML diagram.

- A couple of diagrams showing where Web service operations occur (VOSPACE to client. Client to VOSPACE. Inside a VOSPACE. From: VOspace to another One etc...)

Typos/minor suggestions

Page 9 - example.com!vospace is the authority -> clashes with the usage of authority in registry

Page 9 - /siap-out-1.vot is the URI path -> is missing the /myresults path part

Page 22 - 3.4.1 "stores data as a binary files" remove "s" to files.:

Page 25 - last paragraph - pushToVoSpace is reported twice, but I think the second would be a pushFromVoSpace

Page 29 - I'm a bit dubious about using the IVOA Twiki as a good place to resolve the VOSpacePublicShare, but I suppose it's difficult to find a better place (also SAMP Mtypes suggestions live there...)

Page 33 - REQUEST=redirect. Missing word in "If the supplied"

Page 48 - 6.2.3 copyNode. Copy a node "within" a VOSPACE.

Page 83 - Appendix A.1 starts as "XML Schema The requests..." -> Is think a new line or other separator should be there. Also, since there's no "A.2" maybe it is worth removing the subsection title (unless there's a way to split the long xsd into smaller chunks).

Page 95: there's an item "xx" between 21 and 22: is that intentional?

Page 96: there's a missing section reference for matrix item 46.

-- FrancoisBonnarel - 2018-05-25 -- MarcoMolinaro - 2018-05-25

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

VOSpace is a rather complex standard I just discoverd by reading the paper. I appreciate the section 1.4 which is an indispensable guide for the reader

Section 1.2:

Starting a document with a typical use case is a great idea, but this section is a bit too much detailed. Something more concise withe less XML would be still more valuable
P6: Similarly when a user....., they will (plurial mismatch?)

=> You may be right, there is a lot of information in this section, but I am hesitant to make such a significant change to the document at this point. I have reworded the sentence to make the pronoun match the plurality. -- BrianMajor - 2017-09-12

Section 3.1

It is said here the VOSpace may interpret data. This a very imprortant feature to me which should be introduced before the data model section (1.2 e.g.). This is not a data model feature by the way.

=> Section 1.2 hints at 'interpreting data' with the mention of views, but goes no further. Again, I don't wish to make changes to these sections if not necessary. -- BrianMajor - 2017-09-12

Thrown Exception

It is said in some places(3.4.4#Default Import View e.g.) that the service can throw exceptions. I do not see what does it mean in term of interoperatbility. Throwing or not exceptions is rather an implementation issue.

=> I disagree. The exceptions a service can throw are a part of its API so must be documented. Clients and services are likely to be implemented by different groups. -- BrianMajor - 2017-09-12

-- LaurentMichel - 2017-05-05

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

(0) I'd be a lot more relaxed if there were some sort of validator, or at least the beginnings of one. Is there really no plan for one? A spec this complex (there are 164 lines in this spec with MUST or SHALL alone, and the 65 items in Appendix B are already a steep programme in themselves...) probably has lots of ugly corner cases or subtle contradictions, and if someone said "I've went through the spec and produced first validator code, and while doing that I've not ran into major issues", that'd be reassuring.

=> Yes. I had missed adding the CADC VOSpace compliance tool to the top of this page (there now), but it is a validator for 2.0 services. It does pass on our 2.1 implementation which is assuring for backwards compliance, but it does not validate new 2.1 functionality. This is something that can be discussed more. -- BrianMajor - 2017-09-20

(1) For proper Registry semantics, you need to provide a StandardsRegExt record ivo://ivoa.net/vospace/core, which is a bit non-standard in that it doesn't really match a standard at all (see also (2)), and another StandardsRegExt record ivo://ivoa.net/vospace/v2.0 (which is also suboptimal in that it, for instance, contains a minor version number). Under the assumption that we cannot change any of the existing URIs pointing to these two records without breaking the protocol, I've drafted these two records in addition to the main standard record in volute. If these reflect your intentions, please fix the dates marked with UPDATEME on acceptance of the standard and then mail them (or their URLs) to the RofR maintainers. Of course, further review and changes would be highly appreciated. On the other hand, if they can be changed, it'd be great if the changable ones would point to the record mentioned in (2).

=> Thanks for the standards record in volute, I will indeed update it. See (2) for more comments.

I don't think there's a reference to ivo://ivoa.net/vospace/v2.0

-- BrianMajor - 2017-09-21

(2) I am really unhappy with the standardID (sect. 2, p. 10). ivo://ivoa.net/std/VOSpace/v2.1. For one, with such a versioning, you'd probably break your protocol (or at least the usual discovery patterns) on each minor revision. Since this URI is new anyway, could we still change it to something in the style of Identifiers 2.0, i.e., ivo://ivoa.net/std/VOSpace? I've put in a StandardsRegExt record for that IVOID into volute. This is also where new terms should be created if at all possible, and hence I've put the new synctrans-2.0 there.

=> Yes, I think we can make that change since there is no operational discovery in place. So, the standardIDs are now:

ivo://ivoa.net/std/VOSpace/v2.0 (from the 2.0 spec)
ivo://ivoa.net/std/VOSpace (corrected moving forward)
ivo://ivoa.net/vospace/core (for the vocabulary terms)

In 3.0 we can move everything to ivo://ivoa.net/std/VOSpace

-- BrianMajor - 2017-09-21

(3) Sect 2.1, "Resolve HTTP service endpoint of VOSpace service with registry" -- there's a "the" missing here, but even with it I think you'd do your readers a favour if you're a bit more explicit, e.g., "The HTTP endpoint is a vs:ParamHTTP-typed interface with @std set to True that is a child of a capability with a standardID as specified above." Or so.

=> I've fixed the English, but I'm reluctant to put in more details in this introductory section of the document.

-- BrianMajor - 2017-09-21

(4) Sect 3.1, "the type is specified by the xs:type attribute" -- it doesn't matter much in this context, but in Registry we urge people to use "canonical" prefixes. The really only does matter when you have prefixed values in attributes, which probably is not the case here. Still, if you don't mind much, I'd prefer to have xsi: as the implied prefix for schema-instance in your examples.

=> Done.

-- BrianMajor - 2017-09-21

(5) Sect. 3.2.2, p. 15 says "Ideally, these [properties] should be IVO registry URIs that point to a description registered in the IVO registry..." I think if you say that you'll also have to say how such a "registration" is supposed to work (StandardRegExt records?). Since I don't know enough about how this mechanism is supposed to work, I can't really suggest some text. I'm not even sure you shouldn't be using plain RDF vocabularies (as pioneered by Datalink) rather than the Registry here in the first place. How about retracting the entire statement about registered property names and waiting until people have use cases? This even becomes worse with 3.2.3; there currently is no way to provide such a property description in the Registry. If this stays in, we should draft a registry extension that allows these tricks now. I'm happy to help, perferably when someone actually has some property to register (this could be part of a VOSpaceRegExt that might live outside of VOSpace proper, but then you need to reference it in 3.2.3).

=> There will be the three StandardRegExt records. So, I guess we need to create an extension that will allow descriptions of the keys. I am happy to do that but hope it will not hold up this standard.

-- BrianMajor - 2017-09-21

(6) Sect. 3.3.5, "Other standard service interfaces will also be registered..." -- I don't think this piece helps a lot, but I can see many ways in which it can be confusing ("will? do I have to do it? when? how?"). I think I'd prefer if it and the enumeration were to go. Or did I simply not get what you say here?

=> That text seems outdated. I've remove that list.

-- BrianMajor - 2017-09-21

(7) p. 24, "However, at the time of writing, the schema for registering ViewDescriptions in the IVO registry has not been finalized." -- I'm somewhat embarrassed to see such language in a version 2.1 (rather than 1.0 or even 2.0). If you really don't want to include the registry extension for 2.1, can we just get together right after 2.1 to do the registry extension (or whatever else)?

=> Also outdated. Removed. (And yes to the registry extension work)

-- BrianMajor - 2017-09-21

(8) p. 36, the URI ivo://ivoa.net/std/VOSpace/v2.0#/transfers/(job-id)/results/transferDetails -- by the book, this should resolve to a StandardsRegExt StandardKey, which is of course impossible given that there's a variable (right?) jobID in there. How should clients even use that standardID? Anyway, there's no way this can be represented in StandardsRegExt, so it shouldn't be in the standard in this form, and hence it's also not in the records I've prepared. To help working out a solution, I'd first need to understand how this is intended to be used.

=> This was wrong. The idea is that "(job-id)/results/transferDetails" can be appended to the resolved transfer URL to obtain the results. Thanks for catching that, it's been fixed. -- BrianMajor - 2017-09-21

(9) p. 37 has ivo://ivoa.net/std/VOSpace?synctrans-2.1.-- I don't think that is actually what you want. As far as I can work out, this should refer to a StandardsRegExt key, and thus it should have a hash mark, i.e., ivo://ivoa.net/std/VOSpace#synctrans-2.1 (the question mark signifies a query string, i.e., something that the resource at the URI would somehow respond to). This latter term is already present in the draft StandardsRegExt record.

=> This has been done. -- BrianMajor - 2017-09-20

(10) I can't work out the relationship of the capability URIs in sect. 3.3.5 (in ivo://ivoa.net/std/VOSpace/core) and those in sect. 4 (which live in ivo://ivoa.net/std/VOSpace/v2.0). Which are the ones intended to go into capabilitiy/@standardID? What's supposed to happen with the other ones? For now, I've assumed the ones in core were intended for the SOAP bindings and can just go, so they're not in the proposed records.

=> I don't understand that either. I've asked the original authors in the mailing list for an explanation but have not received a response. email thread -- BrianMajor - 2017-09-20

-- MarkusDemleitner - 2017-06-12

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

I did not see much overlap for this specification with the work covered by the Semantics WG.

Document approved.

A minor comment :

"3.2.3 Property descriptions" subsubsection mentions the use of UCD Unified Content Descriptor very popular inside the VO framework. The reference to the last version of the specification (http://www.ivoa.net/documents/UCD1+/) could be added in the References section .

-- MireilleLouys - 2018-05-16

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _Ada Nebot, Dave Morris )

Probably too late for this version, but for future reference :

In response to questions from MarkusDemleitner about the standard capability URIs in section 3.3.5.

The standard capability URIs for different version of the VOSpace service were added to section 3.3.5 between the following versions :

Version 1.12 2008
Version 1.13 2008

I can't find anything on the mailing list discussing these changes, so I assume these were proposed and discussed at a meeting.

The earlier text in version-1.12 refers to standard URIs for other types of VO services :

Cone Search
SIAP
SSAP
TAP

There is an (edge) use case for allowing services to include Capability elements for the VOSpace service itself. This could potentially allow a VOSpace client to bootstrap itself given just the XML for the node.

However, all of these use cases work perfectly well using the standard VOSI URIs for these Capabilities.

As one of the original advocates for the Capabilities functionality, I can see no reason for declaring a different set of Capability URIs replicating the standard VOSI Capability URIs for each service type.

Non IVOA services or capabilities will need their own URIs, but standard IVOA services can use the standard VOSI Capability URIs for each service type.

So at some point, I would suggest we revert the text in section 3.3.5 to something like the original text in section 2.3.5 of version-1.12.

-- DaveMorris - 2018-05-01

==> I agree Dave. As discussed in Santiago (Oct 2017), there is not a use case for these extra VOSpace capabilities. Let's ensure that section is updated in the next version.

-- BrianMajor - 2018-05-02

page 8 (line 199 in tex) In ~~Appendex~~Appendix A

(line 518 in tex) \label{subsubsection:ui display ~~ame~~ name}

page 42 (line 1428 in tex) If the node path of the ~~uri~~URI

page 54 (line 1960 in text) A Node containing the Node ~~uri~~URI

page 58 and 66 (lines 2222 and 2642 in tex) create a new Node using the ~~uri~~URI

page 71 (line 2875 in tex) HTTP POST to http://rest-rendpointendpoint/sync

page 78 (line 3283 in tex) REQUEST=redirect and ~~SecurityMethod~~ securityMethod

page 82 (line 3519 in tex) This is thrown with the ~~uri~~URI of the View.

page 99 (line 3649) vospace/core#publisher .... ~~uri~~URI

-- DaveMorris - 2018-05-02

==> Thanks. All these edits have been made to the document linked on this page.

-- BrianMajor - 2018-05-02

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

Knowledge Discovery in Databases Interest Group ( _ Kai Polsterer_ )

Theory Interest Group ( _Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova)

TCG Vote : 2018-04-23 - 2018-05-04

If you have minor comments (typos) on the last version of the document please indicate it in the Comments column of the table and post them in the TCG comments section above with the date.

Group	Yes	No	Abstain	Comments
TCG	*
Apps	*
DAL	*			(with suggestion to add a full but simple model diagram in section 3 and small operation diagrams in section 6)
DM	*
G&WS	*
Reg	*			(with a request to the TCG chairs to verify the interoperable implementations)
Semantics	*			(note: vote registed by PatrickDowler from comment above)
DataCP
KDIG
SSIG
Theory
TDIG	*
Ops
StdProc

<-- 
 
 
 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->

META FILEATTACHMENT	attachment="VOSpace.pdf" attr="" comment="" date="1526573186" name="VOSpace.pdf" path="VOSpace.pdf" size="1018085" user="BrianMajor" version="11"

Revision 402018-05-24 - FrancoisBonnarel

META TOPICPARENT	name="WebPreferences"

VOSpace v2.1 Proposed Recommendation: Request for Comments

RFC page for the IVOA VOSpace 2.1 Proposed Recommendation.

The latest VOSpace 2.1 Proposed Recommendation (the second PR of this RFC) can be viewed here:

VOSpace-20180502.pdf

This PR has taken into account all comments up until April 16, 2018.

Change Overview

The change notes since version 2.0:

From version 2.1-20170405:

Replaced the dated Custom Protocols section with a new Public Share Protocol section
Updates from working group feedback

From version 2.1-20150601:

Added REQUEST=redirect for further optimized pullFromVoSpace
Removed "search" and "find nodes" until use cases and implementations are presented
Changed pullToVoSpace, pushFromVoSpace to be client orchestrated.
Changed Transfer in XSD to accept parameters.
Recreated and expanded all webservice operation examples
Changed XSD element authType to SecurityMethod to match IVOA standard.
Fixed text inconsistencies
Clarified role of the node busy flag
Ported source to ivoatex format

From version 2.1-20140805:

Addition of optimized HTTP GET method of data transfer for pushToVoSpace, pullFromVoSpace
Addition of authType to Protocol in XML schema for transfer negotiation.
Added preliminary list of standard authType URIs
Deprecated view=data convenience method of data transfer
Corrections to minor XML format errors in the examples throughout the document.

An earlier VOSpace discussion page can be found here:

http://wiki.ivoa.net/twiki/bin/view/IVOA/VOSpace21Spec

Comments from the IVOA Community during RFC period: 2017-04-06 - 2017-05-18

Implementation and Validators

CADC and OATS-INAF implementations of VOSpace 2.1, both are based on the VOSpace source code from OpenCADC. One item remains to be fully 2.1 compliant: only redirect on a synctrans parameterized request when the parameter "request=redirect" is provided. This is described here:

Issue: v2.1 Compliance - Obey request=redirect parameter

The github project cadc-test-vos is a validator that tests for 2.0 compliance.

The CADC has two VOSpace client implementations, one written in java (mostly used for testing) and one written in python. The python client is the CADC production client that is distributed to users and can be seen/installed here: CADC VOSpace client

When the CADC implemented version 2.1 of the VOSpace service, the python client that was built on the 2.0 specification was not changed. The fact that no errors or disruption was seen by users is evidence of the backwards compatibility between version 2.1 and version 2.0. -- BrianMajor - 2018-05-16

Comments from Pierre Fernique

Implementation references

Presently, it seems that there is one VOSpace 2.1 server working (CANFAR) declared in the VO registry. MSSL, CASU and WFAU seems to be out of order. Is there already VOSpace 2.1 clients ?

Abstract

p2 - It is a little bit surprising to have directly in the abstract a note about the (un)compatibility with the previous releases of the standard. May be a dedicated section, just after the Abstract, should be more adapted.

=> I agree. I have made the abstract more general and created a 'Previous Versions' subsection in the Introduction. -- BrianMajor - 2017-04-12
More largelly, the fact that the compatibility between clients and servers of different versions is rarelly insured questions me. This choice has always the same big impact. The consequence is to force the servers and the clients to implement VOSpace 1.0, 2.0 and 2.1 the time that all potential clients (resp. servers) have upgraded their codes. This remark is not limited to VOSpace, we have the same challenge for SIAv2 which is a totally new standard compared to SIAv1.

=> This is true. In the case of VOSpace, I think the version with the most implementations will gain support in the clients. We can try to share and make widely available client implementations for use by many. -- BrianMajor - 2017-04-12

Introduction

p5 - This section provides the analogy "node = file". Should we have to extend this analogy as "node = file or directory" ? This remark comes from our experience for implementing VOSpace client in Aladin. The role and functions on "directories" are not so clear in the previous release documents. And concretely, our experience shown us that the directory manipulations are generally not, or very lighted implemented. The situation could be improved, if we specify immediately that a node is potentially a directory (cf p11 - figure 2)

=> Good point. I have changed the text to say that a node can be thought of as a file or a directory. -- BrianMajor - 2017-04-12

Typical use of a VOSpace service

p5 - This subsection is important to understand the role of VOSpace, but I am wondering if it is a little bit too detailed as this level of the introduction (XML examples, ...)

VOSpace identifiers

p9 - The alternative '!' or '~' substitution characters should be justified. Why two substitutions characters ? Is it really required ? I suspect that it could be a source of potential bugs (cf. section 2.1 - identifier resolution). It seems that the response is provided p64 "Note that the # sign in PROTOCOL is URL encoded as %23 and that the alternate separator ‘~’ is used instead of ‘!’ in the TARGET".

=> You are correct in that the ~ was introduced so that it doesn't not have to be escaped in HTTP requests. I have added a 'Note' in the VOSpace Identifiers section describing this reasoning. As for whether it is required or not: I suspect that the original specification would have been better off using only ~ to begin with, but the ! character was preserved so as to not break existing implementations. Perhaps in version 3.0 we can deprecate the use of !. -- BrianMajor - 2017-04-12
P10 - The end of this section cites the standardID of this specification. May be, a short explanation of what is a "standardID" would be welcome to avoid the possible reader confusion with the node identifier (same section). This standardID is also described in page 36. Maybe this last definition should be sufficiente.

=> To make it more clear on the first reference to standardID on P10, I have added a short sentence explaining how it is used in the VO Registry. -- BrianMajor - 2017-04-12

VOSpace data model

p12 - "LinkNode" is described. But we do not know if we have to understand only dataLinkNode, or potentially ContainerLinkNode ?

=> There is this sentence: "The link target can be a URI that points to any type of resource, including other VOSpace Nodes (within the same VOSpace service or in another service), or external resources outside VOSpace altogether." I think this makes it clear that a link node can point to any type of node. -- BrianMajor - 2017-04-12
p13 - The support of all type of nodes is not required (end of the section). But does it means that a VOSPace could be not supporting hierarchy (flat view only) ?

=> That is correct. If container nodes are not supported then a heirarchy of nodes is not supported. -- BrianMajor - 2017-04-12

Property identifiers

p14 - URN is introduced here but not described. I suggest a end page note, or a RFC reference on URN.

=> I have added a reference to the RFC for URN Syntax in this section. -- BrianMajor - 2017-04-12

Property descriptions

p15 & 16 "Note that at the time of writing, the schema for registering PropertyDescriptions in the IVO registry has not been finalized" - This remark is repeated at various places in the document, but concretely, there is no progress on this point since VOSpace 1.0 (2007). Maybe these remarks should be simply removed.

=> Good idea. I've removed the three instances of that remark. -- BrianMajor - 2017-04-12

Standard properties

p16 - The standard properties identifiers are IVOID. But following the last Registry recommendation, any IVOID must be declared in the VO registry otherwise they are not valid. Actually, "ivo://ivoa.net/vospace/core" is not registered yet (or not accessible by the two active Full queryable resistries: euroVO and NAVO).
p17 - "ivo://ivoa.net/vospace/core#group...". This is the first (and the unique) mention of "group" notion. Surprisingly, there is no notion of "owner" (only creator, publisher and contributor which are probably not at the same level). It seems that this document does not described these notions (except briefly p36 "Nodes are considered to be owned by the user who created them"). May be, it could help the implementors to have a section describing these notions.

Cababilities

p19 - "Note that at the time of writing, the schema for registering Capability-Descriptions in the IVO registry has not been finalized" - same remark as p16

=> Done. -- BrianMajor - 2017-04-12

View descriptions

p24 - "Note that at the time of writing... " - same remark as p16

=> Done. -- BrianMajor - 2017-04-12
p25&26 - "pullToVoSpace" and "pushToVoSpace" are cited here, but never introduced before. May be a reference to the associated section would be welcome.

Public share protocol

p29 - A "standard ID" for "public share protocol" is provided in this section (new in VOSpace 2.1) => http://wiki.ivoa.net/twiki/bin/view/IVOA/VOSpacePublicShare. This ID is not compatible with regular IVOA standardID based on IVOID contrary to the other "standard protocols" defined in 3.5.3 (ex: ivo://ivoa.net/vospace/core#httpget). Is there a reason for this alternative nomenclature ?

=> Hmm... the idea behind the ID for the public share protocol was that the URI conveniently pointed to the page documenting the rules of the protocol. I personally like this concept as it encourages implementors to follow the pattern. However, I could be convinced to change it to the standard IVOID form. Additional thoughts from anyone? -- BrianMajor - 2017-04-13

Transfers

p30 - In the XML example, there are security method identifiers for which the vocabulary seems to take various origins ( ivo://ivoa.net/sso#tls-with-certificate, http://www.w3.org/Protocols/HTTP/1.0/spec/html#BasicAA). Reference to associated documents would help.

=> I've added a reference to the SSO document so these securityMethods can be more clear. -- BrianMajor - 2017-04-12

REST binding

p35 - "These names are part of the VOSpace 2.1 standard". It lets suppose that other elements in the document may be not relevant to VOSpace 2.1. I suggest to remove any reference to VOSpace 2.1 as this document is 2.1 specification (except for history list...)

=> Your comment made me realize that this section needed to be updated to match the recommendations of IVO Identifiers 2.0. There is now a new 2.1 synctrans endpoint that has the version number in the ID (not in path). I hope it is more clear now how the resource IDs work with the different versions of the specification. -- BrianMajor - 2017-04-14
p35 - ".auto", ".null" are cited in this section, but without explanation. The description of their respective role should be appreciate.

=> Removed from this section. -- BrianMajor - 2017-04-21
p36 - "For backwards compatibility support, services SHALL include the securityMethod element"... but as there is no real compatibility support (cf abstract), this sentence is a little bit surprising.

=> A 2.1 service is backwards compatible with a 2.0 client, so I think this sentence is still important to state. -- BrianMajor - 2017-04-21

Create Node

p41 - "If the Node xs:type...". This is the first citation of "xs:type" which seems to refer to node types.This notion should be defined somewhere before in the document, maybe page 11 in "Nodes and node types"

=> Good idea. I've added a short description of how to indentify node types in XML representations in "Nodes and node types" -- BrianMajor - 2017-04-21
p43 - In the example, (and the following examples), there are 2 paragraphes: Request:... Response:... but the Response paragraph also contains the HTTP request part of the dialog. Strickly speaking the "Response" starts at the ligne "HTTP/1.1 nnn...", the lines above ("PUT....") are the "Request". I would suggest to change the "Response" title by "HTTP dialog" which seems to be more appropriate.

=> I think I agree with you, but before I go and make all those changes, does anyone else have an opinion on this? -- BrianMajor - 2017-04-21

Transfering data

p55 - The difference between pushToVoSpace (sending data to a service) and pullFromVoSpace (reading data from a service) is not obvious. Same for pullToVoSpace (importing data into a service) and pushFromVoSpace (sending data from a service). I suppose that depends of the origin of the action, but it is just a speculation. As these 4 actions are a key point of VOSpace, small examples, or a small graphic would be appreciate.

=> Okay, I've added a bit more text for now on each of the four transfer protocols. I can add a graphics too if you think it needs more. -- BrianMajor - 2017-04-21

Change since last version

p82 - We have the list of the changes from the previous "version 2.1-20150601". The full history is provided in appendix D. It is probably preferable to have all the history in the same place.

=> Okay, I've consolidated these two sections. -- BrianMajor - 2017-04-12
p82 - a "(?TBD)" is still present

=> Thank you, removed. -- BrianMajor - 2017-04-12

Message schema

p82 - "VOSPace 2.0" must be replaced by "VOSpace 2.1" in the introduction sentence.

=> Done. -- BrianMajor - 2017-04-12

-- PierreFernique - 2017-04-11

Comments from Markus Demleitner

(1) The compliance matrix has links to findnodes and searches sections, which are commented out; hence the referencing fails. I think you should just remove the matrix lines.

=> I don't see the referencing failures, but I've taken out those lines anyhow. -- BrianMajor - 2017-09-19

Hm. I still see them in svn rev. 4428. Check VOSpace.log after a build and look for Reference.*undefined, and you'll see the warnings. Btw. I took the liberty to fix your SSOAUTH2 citations, too, but they will only resolve once the record is in ADS, which is pending... ah well, I'll tell you live if you're interested. -- MarkusDemleitner - 2017-09-28

(2) You're changing the target namespace of your schema; from what I can see in the spec, this should break 2.0 clients, as they will not find their elements any more. Are you sure yours isn't a classic case for the XML schema versioning note?

=> Well, yes, this was actually the original use case for the XML schema versioning note and I had forgotten to bring in the change. Thanks for noticing. The XSD namespace is now 2.0 and there is a version attribute in the <transfer> root element. This is the only element that has XSD changes (introduction of security method and transfer param). The search and find features (which I think you are referring) were never actually in the XSD. (That's why they were dropped from 2.1.) -- BrianMajor - 2017-09-19

Hm -- in volute rev. 4429 there's still several examples with the v2.1 namespace. -- MarkusDemleitner - 2017-09-28