Difference: VOSpace10Spec (1 vs. 35)

Revision 352012-06-26 - root

 
META TOPICPARENT name="IvoaGridAndWebServices"
Changed:
<
<
VOSpace home page
>
>
VOSpace home page
 

Discussion of the VOSpace 1.0 specification Document

The VOSpace-1.0 specification has been completed and submitted as an IVOA working draft.

The final versions are available here :

Changed:
<
<
>
>
 

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • 0 if you have no particular preference
  • -1 if you think the proposal is useful but needs more work
  • -2 if you disagree with the proposal

If you register a -1 or -2 vote, then please add a link to a page outlining your objections or comments.

Details and discussion of implementation plans are here.



Version 0.22

Changed:
<
<
>
>
  This document was produced as a result of the discussions that have occured since the Victoria Interop meeting.

Change Requests

Allow typing of properties

The current scheme is limited to key-value pairs where the value is interpreted as a string. A problem with this that some key-values pairs might be intended to represent other datatypes, e.g. a date or a float, and without this typing information, it is impossible to check the validity of the value. It is always possible for a client to add this information with an xsi:type attribute, e.g. <property uri="ivo://net.ivoa/properties/date" xsi:type="xs:dateTime">2006-11-22T18:50:03Z</property> but this might not be interpreted properly by the browser. However, if we actually add a type attribute then we can cover this: <property uri="ivo://net.ivoa/properties/date" type="xs:dateTime">2006-11-22T18:50:03Z</property>. The attribute is optional and non-inclusion implies that the datatype is string. The value of the attribute can either be an XML datatype or a reference to an XML schema that describes the data structure thus allowing for more complicated properties such as:
<property uri="ivo://net.ivoa/properties/color" type="myschema.xsd">
 <color>
   <red>123</red>
   <blue>234</blue>
   <green>89</green>
 </color>
</property> 

Votes

name vote comment
MatthewGraham +1 proposer
DaveMorris -1 Needs more detail, or defer to 1.1 ? (see email for details)

Rename views to formats

This needs to be renamed to what it actually is, i.e. format(s), since the current name is universally confusing.

Votes

name vote comment
MatthewGraham +1 proposer
DaveMorris -1 Needs more detail, or defer to 1.1 ? (see email for details)

Decoupled data servers

Under the current scheme, it is assumed that there is some communication channel between the VOSpace and a data server, e.g. a gridftp server, so that when a pushTo or pullFrom is completed, the data server can notify the VOSpace service that the transfer has completed. This sort of activity is particularly necessary when the endpoint is a logical one, e.g. a one-time-use URL. This design is fine for the cases where we have implemented the data servers ourselves or have access to the source code so that we can add the callback; however, what happens when you are dealing with an off-the-shelf data server where this is not the case or non-trivial, e.g. the Globus gridftp server. One solution is to have the client notify the VOSpace when the transaction is complete (since this really is only a problem for the asynchronous services) so pushToVoSpace would become:

  1. Client calls pushToVoSpace(<node>, <transfer>) returns <node> and <transfer> - the latter containing details for the data server
  2. Client transfers data to data server
  3. Client notifies VOSpace that transfer has been completed, e.g.
transferComplete(<node>).

There are a couple of problems with this, however: the client has to call the space twice and might forget to do the notification call and what happens if the transfer fails or is not done. An alternate approach is to do the data transfer first of all and then register the data object with the node including its physical location so pushToVoSpace becomes:

  1. Client transfer data to data server
  2. Client registers data with VOSpace: register(, URI of location) returns the registered <node>

This is actually the only transfer method which needs a modification: all the others work fine with decoupled servers. In fact, instead of adding an additional operation, we can modify pushToVoSpace either to have an additional URI argument: pushToVoSpace(<node>, <transfer>, location-uri) or we could just incorporate the location-uri into the transfer so that if the protocol contains an endpoint then that endpoint is interpreted to be the physical location of the data object.

One thing that would be useful is another operation to return the list of (decoupled) data servers (resources in SRB speak) that the VOSpace is using so I would suggest that we add a getDataServers operation.

Votes

name vote comment
MatthewGraham +1 proposer
DaveMorris -2 Current version is enough, full asynch with callbacks is later (see email for details)

copyNode and moveNode to operate across stores

See message for rationale of above itemized list:

  1. make (file)name a mandatory node property
  2. use pullDataToVOSpace with copyNode and moveNode to operate across stores
  3. {copy|move}Node to support exception StoreFull
  4. drop list paging in favor of filtering by property value

Apparently this democratic page accepts votes: Well, obviously I vote in favor of my comments, however, in case of disagreement I'd rather see how to use level 1 as is based on some real scenarios. Markus

Four points in one post, so I'll add comments as a similar list (DaveMorris).

  1. make (file)name a mandatory node property
    • The name is in the node URI, so it is already mandatory
  2. use pullDataToVOSpace with copyNode and moveNode to operate across stores
    • If services exchange SOAP messages, then this causes problems interoperability problems later on. All future services would have to support all the previous versions of the SOAP messages.
    • The current specification does support direct data transfer between services, but the control messages always come from the client. i.e. The client contacts service A and asks for a URL to access the data, and then calls service B and asks it to import the data from the URL.
    • See email and wiki for more details.
  3. {copy|move}Node to support exception StoreFull
    • This is a good point, and we don't have anything in the specification to handle this. Should this be a new fault type, or do we just wrap it as a PermissionDenied fault with an appropriate message ? Either way, we probably need to add something to the specification to say what should happen.
  4. drop list paging in favor of filtering by property value
    • Simple filtering on names (similar to Unix ls) is going to be added to version 1.0.
    • Complex filtering on property values (similar to Unix find) has been deferred version 1.1.
    • The paging is primarily intended to enable clients to handle large lists without exploding.
    • See email for more details.

Votes

name vote comment
MarkusDolensky +1 proposer (yeah)
DaveMorris -2 (items 1 and 2)
DaveMorris +1 (item 3 needs to be addressed in the specification)
DaveMorris -1 (parts of item 4 will be included in version 1.0 or 1.1)

Make the destination of CopyNode and MoveNode a simple uri

The advantage is it simplifies the standard doc (so many restrictions on node) and is much easier for client to use... as does not have to create a node object - simply create a uri for the destination.

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris 0 It would be useful to have a list of updated properties for a copy, but possibly not worth the extra complexity in the schema

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1  
DaveMorris +1 Can you outline the proposed changes in a wiki page ?

Version 0.21

Changed:
<
<
>
>
  This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Changes

VOspace10Spec21Archive

Changes for later versions of VOSpace

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
*Result*: consider for later version

Add a FindNodes operation

This operation would allow a simple search on the VOSpace - the level of functionality would be similar to the unix "find" command.

Revision 342007-05-17 - DaveMorris

 
META TOPICPARENT name="IvoaGridAndWebServices"
Added:
>
>
VOSpace home page
 

Discussion of the VOSpace 1.0 specification Document

The VOSpace-1.0 specification has been completed and submitted as an IVOA working draft.

The final versions are available here :


This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • 0 if you have no particular preference
  • -1 if you think the proposal is useful but needs more work
  • -2 if you disagree with the proposal

If you register a -1 or -2 vote, then please add a link to a page outlining your objections or comments.

Details and discussion of implementation plans are here.



Version 0.22

This document was produced as a result of the discussions that have occured since the Victoria Interop meeting.

Change Requests

Allow typing of properties

The current scheme is limited to key-value pairs where the value is interpreted as a string. A problem with this that some key-values pairs might be intended to represent other datatypes, e.g. a date or a float, and without this typing information, it is impossible to check the validity of the value. It is always possible for a client to add this information with an xsi:type attribute, e.g. <property uri="ivo://net.ivoa/properties/date" xsi:type="xs:dateTime">2006-11-22T18:50:03Z</property> but this might not be interpreted properly by the browser. However, if we actually add a type attribute then we can cover this: <property uri="ivo://net.ivoa/properties/date" type="xs:dateTime">2006-11-22T18:50:03Z</property>. The attribute is optional and non-inclusion implies that the datatype is string. The value of the attribute can either be an XML datatype or a reference to an XML schema that describes the data structure thus allowing for more complicated properties such as:
<property uri="ivo://net.ivoa/properties/color" type="myschema.xsd">
 <color>
   <red>123</red>
   <blue>234</blue>
   <green>89</green>
 </color>
</property> 

Votes

name vote comment
MatthewGraham +1 proposer
DaveMorris -1 Needs more detail, or defer to 1.1 ? (see email for details)

Rename views to formats

This needs to be renamed to what it actually is, i.e. format(s), since the current name is universally confusing.

Votes

name vote comment
MatthewGraham +1 proposer
DaveMorris -1 Needs more detail, or defer to 1.1 ? (see email for details)

Decoupled data servers

Under the current scheme, it is assumed that there is some communication channel between the VOSpace and a data server, e.g. a gridftp server, so that when a pushTo or pullFrom is completed, the data server can notify the VOSpace service that the transfer has completed. This sort of activity is particularly necessary when the endpoint is a logical one, e.g. a one-time-use URL. This design is fine for the cases where we have implemented the data servers ourselves or have access to the source code so that we can add the callback; however, what happens when you are dealing with an off-the-shelf data server where this is not the case or non-trivial, e.g. the Globus gridftp server. One solution is to have the client notify the VOSpace when the transaction is complete (since this really is only a problem for the asynchronous services) so pushToVoSpace would become:

  1. Client calls pushToVoSpace(<node>, <transfer>) returns <node> and <transfer> - the latter containing details for the data server
  2. Client transfers data to data server
  3. Client notifies VOSpace that transfer has been completed, e.g.
transferComplete(<node>).

There are a couple of problems with this, however: the client has to call the space twice and might forget to do the notification call and what happens if the transfer fails or is not done. An alternate approach is to do the data transfer first of all and then register the data object with the node including its physical location so pushToVoSpace becomes:

  1. Client transfer data to data server
  2. Client registers data with VOSpace: register(, URI of location) returns the registered <node>

This is actually the only transfer method which needs a modification: all the others work fine with decoupled servers. In fact, instead of adding an additional operation, we can modify pushToVoSpace either to have an additional URI argument: pushToVoSpace(<node>, <transfer>, location-uri) or we could just incorporate the location-uri into the transfer so that if the protocol contains an endpoint then that endpoint is interpreted to be the physical location of the data object.

One thing that would be useful is another operation to return the list of (decoupled) data servers (resources in SRB speak) that the VOSpace is using so I would suggest that we add a getDataServers operation.

Votes

name vote comment
MatthewGraham +1 proposer
DaveMorris -2 Current version is enough, full asynch with callbacks is later (see email for details)

copyNode and moveNode to operate across stores

See message for rationale of above itemized list:

  1. make (file)name a mandatory node property
  2. use pullDataToVOSpace with copyNode and moveNode to operate across stores
  3. {copy|move}Node to support exception StoreFull
  4. drop list paging in favor of filtering by property value

Apparently this democratic page accepts votes: Well, obviously I vote in favor of my comments, however, in case of disagreement I'd rather see how to use level 1 as is based on some real scenarios. Markus

Four points in one post, so I'll add comments as a similar list (DaveMorris).

  1. make (file)name a mandatory node property
    • The name is in the node URI, so it is already mandatory
  2. use pullDataToVOSpace with copyNode and moveNode to operate across stores
    • If services exchange SOAP messages, then this causes problems interoperability problems later on. All future services would have to support all the previous versions of the SOAP messages.
    • The current specification does support direct data transfer between services, but the control messages always come from the client. i.e. The client contacts service A and asks for a URL to access the data, and then calls service B and asks it to import the data from the URL.
    • See email and wiki for more details.
  3. {copy|move}Node to support exception StoreFull
    • This is a good point, and we don't have anything in the specification to handle this. Should this be a new fault type, or do we just wrap it as a PermissionDenied fault with an appropriate message ? Either way, we probably need to add something to the specification to say what should happen.
  4. drop list paging in favor of filtering by property value
    • Simple filtering on names (similar to Unix ls) is going to be added to version 1.0.
    • Complex filtering on property values (similar to Unix find) has been deferred version 1.1.
    • The paging is primarily intended to enable clients to handle large lists without exploding.
    • See email for more details.

Votes

name vote comment
MarkusDolensky +1 proposer (yeah)
DaveMorris -2 (items 1 and 2)
DaveMorris +1 (item 3 needs to be addressed in the specification)
DaveMorris -1 (parts of item 4 will be included in version 1.0 or 1.1)

Make the destination of CopyNode and MoveNode a simple uri

The advantage is it simplifies the standard doc (so many restrictions on node) and is much easier for client to use... as does not have to create a node object - simply create a uri for the destination.

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris 0 It would be useful to have a list of updated properties for a copy, but possibly not worth the extra complexity in the schema

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1  
DaveMorris +1 Can you outline the proposed changes in a wiki page ?

Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Changes

VOspace10Spec21Archive

Changes for later versions of VOSpace

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
*Result*: consider for later version

Add a FindNodes operation

This operation would allow a simple search on the VOSpace - the level of functionality would be similar to the unix "find" command.

<--  
-->

Revision 332007-05-01 - DaveMorris

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

Added:
>
>
The VOSpace-1.0 specification has been completed and submitted as an IVOA working draft.

The final versions are available here :


  This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • 0 if you have no particular preference
  • -1 if you think the proposal is useful but needs more work
  • -2 if you disagree with the proposal

If you register a -1 or -2 vote, then please add a link to a page outlining your objections or comments.

Details and discussion of implementation plans are here.



Version 0.22

This document was produced as a result of the discussions that have occured since the Victoria Interop meeting.

Change Requests

Allow typing of properties

The current scheme is limited to key-value pairs where the value is interpreted as a string. A problem with this that some key-values pairs might be intended to represent other datatypes, e.g. a date or a float, and without this typing information, it is impossible to check the validity of the value. It is always possible for a client to add this information with an xsi:type attribute, e.g. <property uri="ivo://net.ivoa/properties/date" xsi:type="xs:dateTime">2006-11-22T18:50:03Z</property> but this might not be interpreted properly by the browser. However, if we actually add a type attribute then we can cover this: <property uri="ivo://net.ivoa/properties/date" type="xs:dateTime">2006-11-22T18:50:03Z</property>. The attribute is optional and non-inclusion implies that the datatype is string. The value of the attribute can either be an XML datatype or a reference to an XML schema that describes the data structure thus allowing for more complicated properties such as:
<property uri="ivo://net.ivoa/properties/color" type="myschema.xsd">
 <color>
   <red>123</red>
   <blue>234</blue>
   <green>89</green>
 </color>
</property> 

Votes

name vote comment
MatthewGraham +1 proposer
DaveMorris -1 Needs more detail, or defer to 1.1 ? (see email for details)

Rename views to formats

This needs to be renamed to what it actually is, i.e. format(s), since the current name is universally confusing.

Votes

name vote comment
MatthewGraham +1 proposer
DaveMorris -1 Needs more detail, or defer to 1.1 ? (see email for details)

Decoupled data servers

Under the current scheme, it is assumed that there is some communication channel between the VOSpace and a data server, e.g. a gridftp server, so that when a pushTo or pullFrom is completed, the data server can notify the VOSpace service that the transfer has completed. This sort of activity is particularly necessary when the endpoint is a logical one, e.g. a one-time-use URL. This design is fine for the cases where we have implemented the data servers ourselves or have access to the source code so that we can add the callback; however, what happens when you are dealing with an off-the-shelf data server where this is not the case or non-trivial, e.g. the Globus gridftp server. One solution is to have the client notify the VOSpace when the transaction is complete (since this really is only a problem for the asynchronous services) so pushToVoSpace would become:

  1. Client calls pushToVoSpace(<node>, <transfer>) returns <node> and <transfer> - the latter containing details for the data server
  2. Client transfers data to data server
  3. Client notifies VOSpace that transfer has been completed, e.g.
transferComplete(<node>).

There are a couple of problems with this, however: the client has to call the space twice and might forget to do the notification call and what happens if the transfer fails or is not done. An alternate approach is to do the data transfer first of all and then register the data object with the node including its physical location so pushToVoSpace becomes:

  1. Client transfer data to data server
  2. Client registers data with VOSpace: register(, URI of location) returns the registered <node>

This is actually the only transfer method which needs a modification: all the others work fine with decoupled servers. In fact, instead of adding an additional operation, we can modify pushToVoSpace either to have an additional URI argument: pushToVoSpace(<node>, <transfer>, location-uri) or we could just incorporate the location-uri into the transfer so that if the protocol contains an endpoint then that endpoint is interpreted to be the physical location of the data object.

One thing that would be useful is another operation to return the list of (decoupled) data servers (resources in SRB speak) that the VOSpace is using so I would suggest that we add a getDataServers operation.

Votes

name vote comment
MatthewGraham +1 proposer
DaveMorris -2 Current version is enough, full asynch with callbacks is later (see email for details)

copyNode and moveNode to operate across stores

See message for rationale of above itemized list:

  1. make (file)name a mandatory node property
  2. use pullDataToVOSpace with copyNode and moveNode to operate across stores
  3. {copy|move}Node to support exception StoreFull
  4. drop list paging in favor of filtering by property value

Apparently this democratic page accepts votes: Well, obviously I vote in favor of my comments, however, in case of disagreement I'd rather see how to use level 1 as is based on some real scenarios. Markus

Four points in one post, so I'll add comments as a similar list (DaveMorris).

  1. make (file)name a mandatory node property
    • The name is in the node URI, so it is already mandatory
  2. use pullDataToVOSpace with copyNode and moveNode to operate across stores
    • If services exchange SOAP messages, then this causes problems interoperability problems later on. All future services would have to support all the previous versions of the SOAP messages.
    • The current specification does support direct data transfer between services, but the control messages always come from the client. i.e. The client contacts service A and asks for a URL to access the data, and then calls service B and asks it to import the data from the URL.
    • See email and wiki for more details.
  3. {copy|move}Node to support exception StoreFull
    • This is a good point, and we don't have anything in the specification to handle this. Should this be a new fault type, or do we just wrap it as a PermissionDenied fault with an appropriate message ? Either way, we probably need to add something to the specification to say what should happen.
  4. drop list paging in favor of filtering by property value
    • Simple filtering on names (similar to Unix ls) is going to be added to version 1.0.
    • Complex filtering on property values (similar to Unix find) has been deferred version 1.1.
    • The paging is primarily intended to enable clients to handle large lists without exploding.
    • See email for more details.

Votes

name vote comment
MarkusDolensky +1 proposer (yeah)
DaveMorris -2 (items 1 and 2)
DaveMorris +1 (item 3 needs to be addressed in the specification)
DaveMorris -1 (parts of item 4 will be included in version 1.0 or 1.1)

Make the destination of CopyNode and MoveNode a simple uri

The advantage is it simplifies the standard doc (so many restrictions on node) and is much easier for client to use... as does not have to create a node object - simply create a uri for the destination.

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris 0 It would be useful to have a list of updated properties for a copy, but possibly not worth the extra complexity in the schema

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1  
DaveMorris +1 Can you outline the proposed changes in a wiki page ?

Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Changes

VOspace10Spec21Archive

Changes for later versions of VOSpace

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
*Result*: consider for later version

Add a FindNodes operation

This operation would allow a simple search on the VOSpace - the level of functionality would be similar to the unix "find" command.

<--  
-->

Revision 322006-12-18 - DaveMorris

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • 0 if you have no particular preference
  • -1 if you think the proposal is useful but needs more work
  • -2 if you disagree with the proposal

If you register a -1 or -2 vote, then please add a link to a page outlining your objections or comments.

Details and discussion of implementation plans are here.



Version 0.22

This document was produced as a result of the discussions that have occured since the Victoria Interop meeting.

Change Requests

Allow typing of properties

The current scheme is limited to key-value pairs where the value is interpreted as a string. A problem with this that some key-values pairs might be intended to represent other datatypes, e.g. a date or a float, and without this typing information, it is impossible to check the validity of the value. It is always possible for a client to add this information with an xsi:type attribute, e.g. <property uri="ivo://net.ivoa/properties/date" xsi:type="xs:dateTime">2006-11-22T18:50:03Z</property> but this might not be interpreted properly by the browser. However, if we actually add a type attribute then we can cover this: <property uri="ivo://net.ivoa/properties/date" type="xs:dateTime">2006-11-22T18:50:03Z</property>. The attribute is optional and non-inclusion implies that the datatype is string. The value of the attribute can either be an XML datatype or a reference to an XML schema that describes the data structure thus allowing for more complicated properties such as:
<property uri="ivo://net.ivoa/properties/color" type="myschema.xsd">
 <color>
   <red>123</red>
   <blue>234</blue>
   <green>89</green>
 </color>
</property> 

Votes

name vote comment
MatthewGraham +1 proposer
DaveMorris -1 Needs more detail, or defer to 1.1 ? (see email for details)

Rename views to formats

This needs to be renamed to what it actually is, i.e. format(s), since the current name is universally confusing.

Votes

name vote comment
MatthewGraham +1 proposer
DaveMorris -1 Needs more detail, or defer to 1.1 ? (see email for details)

Decoupled data servers

Under the current scheme, it is assumed that there is some communication channel between the VOSpace and a data server, e.g. a gridftp server, so that when a pushTo or pullFrom is completed, the data server can notify the VOSpace service that the transfer has completed. This sort of activity is particularly necessary when the endpoint is a logical one, e.g. a one-time-use URL. This design is fine for the cases where we have implemented the data servers ourselves or have access to the source code so that we can add the callback; however, what happens when you are dealing with an off-the-shelf data server where this is not the case or non-trivial, e.g. the Globus gridftp server. One solution is to have the client notify the VOSpace when the transaction is complete (since this really is only a problem for the asynchronous services) so pushToVoSpace would become:

  1. Client calls pushToVoSpace(<node>, <transfer>) returns <node> and <transfer> - the latter containing details for the data server
  2. Client transfers data to data server
  3. Client notifies VOSpace that transfer has been completed, e.g.
transferComplete(<node>).

There are a couple of problems with this, however: the client has to call the space twice and might forget to do the notification call and what happens if the transfer fails or is not done. An alternate approach is to do the data transfer first of all and then register the data object with the node including its physical location so pushToVoSpace becomes:

  1. Client transfer data to data server
  2. Client registers data with VOSpace: register(, URI of location) returns the registered <node>

This is actually the only transfer method which needs a modification: all the others work fine with decoupled servers. In fact, instead of adding an additional operation, we can modify pushToVoSpace either to have an additional URI argument: pushToVoSpace(<node>, <transfer>, location-uri) or we could just incorporate the location-uri into the transfer so that if the protocol contains an endpoint then that endpoint is interpreted to be the physical location of the data object.

One thing that would be useful is another operation to return the list of (decoupled) data servers (resources in SRB speak) that the VOSpace is using so I would suggest that we add a getDataServers operation.

Votes

name vote comment
MatthewGraham +1 proposer
DaveMorris -2 Current version is enough, full asynch with callbacks is later (see email for details)
Changed:
<
<

>
>

copyNode and moveNode to operate across stores

  See message for rationale of above itemized list:

  1. make (file)name a mandatory node property
  2. use pullDataToVOSpace with copyNode and moveNode to operate across stores
  3. {copy|move}Node to support exception StoreFull
  4. drop list paging in favor of filtering by property value

Apparently this democratic page accepts votes: Well, obviously I vote in favor of my comments, however, in case of disagreement I'd rather see how to use level 1 as is based on some real scenarios. Markus

Four points in one post, so I'll add comments as a similar list (DaveMorris).

  1. make (file)name a mandatory node property
    • The name is in the node URI, so it is already mandatory
  2. use pullDataToVOSpace with copyNode and moveNode to operate across stores
    • If services exchange SOAP messages, then this causes problems interoperability problems later on. All future services would have to support all the previous versions of the SOAP messages.
    • The current specification does support direct data transfer between services, but the control messages always come from the client. i.e. The client contacts service A and asks for a URL to access the data, and then calls service B and asks it to import the data from the URL.
    • See email and wiki for more details.
  3. {copy|move}Node to support exception StoreFull
Changed:
<
<
    • This is a good point, and we don't have anything in the specification to handle this. Should this be a new fault type, or do we just wrap it as a PermissionDenied fault with an appropriate message ? Either way, we probably need to add something to the specification to say what should happen.
>
>
    • This is a good point, and we don't have anything in the specification to handle this. Should this be a new fault type, or do we just wrap it as a PermissionDenied fault with an appropriate message ? Either way, we probably need to add something to the specification to say what should happen.
 
  1. drop list paging in favor of filtering by property value
    • Simple filtering on names (similar to Unix ls) is going to be added to version 1.0.
    • Complex filtering on property values (similar to Unix find) has been deferred version 1.1.
    • The paging is primarily intended to enable clients to handle large lists without exploding.
    • See email for more details.

Votes

name vote comment
MarkusDolensky +1 proposer (yeah)
Changed:
<
<
DaveMorris -1 (items 1, 2 and 4)
>
>
DaveMorris -2 (items 1 and 2)
 
DaveMorris +1 (item 3 needs to be addressed in the specification)
Changed:
<
<
>
>
DaveMorris -1 (parts of item 4 will be included in version 1.0 or 1.1)
Deleted:
<
<

 

Make the destination of CopyNode and MoveNode a simple uri

The advantage is it simplifies the standard doc (so many restrictions on node) and is much easier for client to use... as does not have to create a node object - simply create a uri for the destination.

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris 0 It would be useful to have a list of updated properties for a copy, but possibly not worth the extra complexity in the schema

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1  
DaveMorris +1 Can you outline the proposed changes in a wiki page ?

Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Changes

VOspace10Spec21Archive

Changes for later versions of VOSpace

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
*Result*: consider for later version

Add a FindNodes operation

This operation would allow a simple search on the VOSpace - the level of functionality would be similar to the unix "find" command.

<--  
-->

Revision 312006-12-18 - DaveMorris

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • 0 if you have no particular preference
  • -1 if you think the proposal is useful but needs more work
  • -2 if you disagree with the proposal

If you register a -1 or -2 vote, then please add a link to a page outlining your objections or comments.

Details and discussion of implementation plans are here.



Version 0.22

This document was produced as a result of the discussions that have occured since the Victoria Interop meeting.

Change Requests

Allow typing of properties

The current scheme is limited to key-value pairs where the value is interpreted as a string. A problem with this that some key-values pairs might be intended to represent other datatypes, e.g. a date or a float, and without this typing information, it is impossible to check the validity of the value. It is always possible for a client to add this information with an xsi:type attribute, e.g. <property uri="ivo://net.ivoa/properties/date" xsi:type="xs:dateTime">2006-11-22T18:50:03Z</property> but this might not be interpreted properly by the browser. However, if we actually add a type attribute then we can cover this: <property uri="ivo://net.ivoa/properties/date" type="xs:dateTime">2006-11-22T18:50:03Z</property>. The attribute is optional and non-inclusion implies that the datatype is string. The value of the attribute can either be an XML datatype or a reference to an XML schema that describes the data structure thus allowing for more complicated properties such as:
<property uri="ivo://net.ivoa/properties/color" type="myschema.xsd">
 <color>
   <red>123</red>
   <blue>234</blue>
   <green>89</green>
 </color>
</property> 

Votes

name vote comment
MatthewGraham +1 proposer
DaveMorris -1 Needs more detail, or defer to 1.1 ? (see email for details)

Rename views to formats

This needs to be renamed to what it actually is, i.e. format(s), since the current name is universally confusing.

Votes

name vote comment
MatthewGraham +1 proposer
DaveMorris -1 Needs more detail, or defer to 1.1 ? (see email for details)

Decoupled data servers

Under the current scheme, it is assumed that there is some communication channel between the VOSpace and a data server, e.g. a gridftp server, so that when a pushTo or pullFrom is completed, the data server can notify the VOSpace service that the transfer has completed. This sort of activity is particularly necessary when the endpoint is a logical one, e.g. a one-time-use URL. This design is fine for the cases where we have implemented the data servers ourselves or have access to the source code so that we can add the callback; however, what happens when you are dealing with an off-the-shelf data server where this is not the case or non-trivial, e.g. the Globus gridftp server. One solution is to have the client notify the VOSpace when the transaction is complete (since this really is only a problem for the asynchronous services) so pushToVoSpace would become:

  1. Client calls pushToVoSpace(<node>, <transfer>) returns <node> and <transfer> - the latter containing details for the data server
  2. Client transfers data to data server
  3. Client notifies VOSpace that transfer has been completed, e.g.
transferComplete(<node>).

There are a couple of problems with this, however: the client has to call the space twice and might forget to do the notification call and what happens if the transfer fails or is not done. An alternate approach is to do the data transfer first of all and then register the data object with the node including its physical location so pushToVoSpace becomes:

  1. Client transfer data to data server
  2. Client registers data with VOSpace: register(, URI of location) returns the registered <node>

This is actually the only transfer method which needs a modification: all the others work fine with decoupled servers. In fact, instead of adding an additional operation, we can modify pushToVoSpace either to have an additional URI argument: pushToVoSpace(<node>, <transfer>, location-uri) or we could just incorporate the location-uri into the transfer so that if the protocol contains an endpoint then that endpoint is interpreted to be the physical location of the data object.

One thing that would be useful is another operation to return the list of (decoupled) data servers (resources in SRB speak) that the VOSpace is using so I would suggest that we add a getDataServers operation.

Votes

name vote comment
MatthewGraham +1 proposer
DaveMorris -2 Current version is enough, full asynch with callbacks is later (see email for details)


See message for rationale of above itemized list:

  1. make (file)name a mandatory node property
  2. use pullDataToVOSpace with copyNode and moveNode to operate across stores
  3. {copy|move}Node to support exception StoreFull
  4. drop list paging in favor of filtering by property value

Apparently this democratic page accepts votes: Well, obviously I vote in favor of my comments, however, in case of disagreement I'd rather see how to use level 1 as is based on some real scenarios. Markus

Added:
>
>
Four points in one post, so I'll add comments as a similar list (DaveMorris).
  1. make (file)name a mandatory node property
    • The name is in the node URI, so it is already mandatory
  2. use pullDataToVOSpace with copyNode and moveNode to operate across stores
    • If services exchange SOAP messages, then this causes problems interoperability problems later on. All future services would have to support all the previous versions of the SOAP messages.
    • The current specification does support direct data transfer between services, but the control messages always come from the client. i.e. The client contacts service A and asks for a URL to access the data, and then calls service B and asks it to import the data from the URL.
    • See email and wiki for more details.
  3. {copy|move}Node to support exception StoreFull
    • This is a good point, and we don't have anything in the specification to handle this. Should this be a new fault type, or do we just wrap it as a PermissionDenied fault with an appropriate message ? Either way, we probably need to add something to the specification to say what should happen.
  4. drop list paging in favor of filtering by property value
    • Simple filtering on names (similar to Unix ls) is going to be added to version 1.0.
    • Complex filtering on property values (similar to Unix find) has been deferred version 1.1.
    • The paging is primarily intended to enable clients to handle large lists without exploding.
    • See email for more details.
 

Votes

name vote comment
MarkusDolensky +1 proposer (yeah)
Added:
>
>
DaveMorris -1 (items 1, 2 and 4)
DaveMorris +1 (item 3 needs to be addressed in the specification)
 
Added:
>
>

 

Make the destination of CopyNode and MoveNode a simple uri

The advantage is it simplifies the standard doc (so many restrictions on node) and is much easier for client to use... as does not have to create a node object - simply create a uri for the destination.

Votes

name vote comment
PaulHarrison +1 proposer
Added:
>
>
DaveMorris 0 It would be useful to have a list of updated properties for a copy, but possibly not worth the extra complexity in the schema
 

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1  
Changed:
<
<
>
>
DaveMorris +1 Can you outline the proposed changes in a wiki page ?
 

Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Changes

VOspace10Spec21Archive

Changes for later versions of VOSpace

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
*Result*: consider for later version

Add a FindNodes operation

This operation would allow a simple search on the VOSpace - the level of functionality would be similar to the unix "find" command.

<--  
-->

Revision 302006-12-18 - DaveMorris

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
Deleted:
<
<
  • -1 if you disagree
 
  • 0 if you have no particular preference
Added:
>
>
  • -1 if you think the proposal is useful but needs more work
  • -2 if you disagree with the proposal
 
Added:
>
>
If you register a -1 or -2 vote, then please add a link to a page outlining your objections or comments.
 Details and discussion of implementation plans are here.



Version 0.22

This document was produced as a result of the discussions that have occured since the Victoria Interop meeting.

Change Requests

Allow typing of properties

The current scheme is limited to key-value pairs where the value is interpreted as a string. A problem with this that some key-values pairs might be intended to represent other datatypes, e.g. a date or a float, and without this typing information, it is impossible to check the validity of the value. It is always possible for a client to add this information with an xsi:type attribute, e.g. <property uri="ivo://net.ivoa/properties/date" xsi:type="xs:dateTime">2006-11-22T18:50:03Z</property> but this might not be interpreted properly by the browser. However, if we actually add a type attribute then we can cover this: <property uri="ivo://net.ivoa/properties/date" type="xs:dateTime">2006-11-22T18:50:03Z</property>. The attribute is optional and non-inclusion implies that the datatype is string. The value of the attribute can either be an XML datatype or a reference to an XML schema that describes the data structure thus allowing for more complicated properties such as:
<property uri="ivo://net.ivoa/properties/color" type="myschema.xsd">
 <color>
   <red>123</red>
   <blue>234</blue>
   <green>89</green>
 </color>
</property> 

Votes

name vote comment
MatthewGraham +1 proposer
Added:
>
>
DaveMorris -1 Needs more detail, or defer to 1.1 ? (see email for details)
 

Rename views to formats

This needs to be renamed to what it actually is, i.e. format(s), since the current name is universally confusing.

Votes

name vote comment
MatthewGraham +1 proposer
Added:
>
>
DaveMorris -1 Needs more detail, or defer to 1.1 ? (see email for details)
 

Decoupled data servers

Under the current scheme, it is assumed that there is some communication channel between the VOSpace and a data server, e.g. a gridftp server, so that when a pushTo or pullFrom is completed, the data server can notify the VOSpace service that the transfer has completed. This sort of activity is particularly necessary when the endpoint is a logical one, e.g. a one-time-use URL. This design is fine for the cases where we have implemented the data servers ourselves or have access to the source code so that we can add the callback; however, what happens when you are dealing with an off-the-shelf data server where this is not the case or non-trivial, e.g. the Globus gridftp server. One solution is to have the client notify the VOSpace when the transaction is complete (since this really is only a problem for the asynchronous services) so pushToVoSpace would become:

  1. Client calls pushToVoSpace(<node>, <transfer>) returns <node> and <transfer> - the latter containing details for the data server
  2. Client transfers data to data server
  3. Client notifies VOSpace that transfer has been completed, e.g.
transferComplete(<node>).

There are a couple of problems with this, however: the client has to call the space twice and might forget to do the notification call and what happens if the transfer fails or is not done. An alternate approach is to do the data transfer first of all and then register the data object with the node including its physical location so pushToVoSpace becomes:

  1. Client transfer data to data server
  2. Client registers data with VOSpace: register(, URI of location) returns the registered <node>

This is actually the only transfer method which needs a modification: all the others work fine with decoupled servers. In fact, instead of adding an additional operation, we can modify pushToVoSpace either to have an additional URI argument: pushToVoSpace(<node>, <transfer>, location-uri) or we could just incorporate the location-uri into the transfer so that if the protocol contains an endpoint then that endpoint is interpreted to be the physical location of the data object.

One thing that would be useful is another operation to return the list of (decoupled) data servers (resources in SRB speak) that the VOSpace is using so I would suggest that we add a getDataServers operation.

Votes

name vote comment
MatthewGraham +1 proposer
Added:
>
>
DaveMorris -2 Current version is enough, full asynch with callbacks is later (see email for details)
 

See message for rationale of above itemized list:

  1. make (file)name a mandatory node property
  2. use pullDataToVOSpace with copyNode and moveNode to operate across stores
  3. {copy|move}Node to support exception StoreFull
  4. drop list paging in favor of filtering by property value

Apparently this democratic page accepts votes: Well, obviously I vote in favor of my comments, however, in case of disagreement I'd rather see how to use level 1 as is based on some real scenarios. Markus

Votes

name vote comment
MarkusDolensky +1 proposer (yeah)

Make the destination of CopyNode and MoveNode a simple uri

The advantage is it simplifies the standard doc (so many restrictions on node) and is much easier for client to use... as does not have to create a node object - simply create a uri for the destination.

Votes

name vote comment
PaulHarrison +1 proposer

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1  

Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Changes

VOspace10Spec21Archive

Changes for later versions of VOSpace

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
*Result*: consider for later version

Add a FindNodes operation

This operation would allow a simple search on the VOSpace - the level of functionality would be similar to the unix "find" command.

<--  
-->

Revision 292006-11-23 - MatthewGraham

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • -1 if you disagree
  • 0 if you have no particular preference

Details and discussion of implementation plans are here.



Version 0.22

This document was produced as a result of the discussions that have occured since the Victoria Interop meeting.

Change Requests

Added:
>
>

Allow typing of properties

The current scheme is limited to key-value pairs where the value is interpreted as a string. A problem with this that some key-values pairs might be intended to represent other datatypes, e.g. a date or a float, and without this typing information, it is impossible to check the validity of the value. It is always possible for a client to add this information with an xsi:type attribute, e.g. <property uri="ivo://net.ivoa/properties/date" xsi:type="xs:dateTime">2006-11-22T18:50:03Z</property> but this might not be interpreted properly by the browser. However, if we actually add a type attribute then we can cover this: <property uri="ivo://net.ivoa/properties/date" type="xs:dateTime">2006-11-22T18:50:03Z</property>. The attribute is optional and non-inclusion implies that the datatype is string. The value of the attribute can either be an XML datatype or a reference to an XML schema that describes the data structure thus allowing for more complicated properties such as:
<property uri="ivo://net.ivoa/properties/color" type="myschema.xsd">
 <color>
   <red>123</red>
   <blue>234</blue>
   <green>89</green>
 </color>
</property> 

Votes

name vote comment
MatthewGraham +1 proposer

Rename views to formats

This needs to be renamed to what it actually is, i.e. format(s), since the current name is universally confusing.

Votes

name vote comment
MatthewGraham +1 proposer

Decoupled data servers

Under the current scheme, it is assumed that there is some communication channel between the VOSpace and a data server, e.g. a gridftp server, so that when a pushTo or pullFrom is completed, the data server can notify the VOSpace service that the transfer has completed. This sort of activity is particularly necessary when the endpoint is a logical one, e.g. a one-time-use URL. This design is fine for the cases where we have implemented the data servers ourselves or have access to the source code so that we can add the callback; however, what happens when you are dealing with an off-the-shelf data server where this is not the case or non-trivial, e.g. the Globus gridftp server. One solution is to have the client notify the VOSpace when the transaction is complete (since this really is only a problem for the asynchronous services) so pushToVoSpace would become:

  1. Client calls pushToVoSpace(<node>, <transfer>) returns <node> and <transfer> - the latter containing details for the data server
  2. Client transfers data to data server
  3. Client notifies VOSpace that transfer has been completed, e.g.
transferComplete(<node>).

There are a couple of problems with this, however: the client has to call the space twice and might forget to do the notification call and what happens if the transfer fails or is not done. An alternate approach is to do the data transfer first of all and then register the data object with the node including its physical location so pushToVoSpace becomes:

  1. Client transfer data to data server
  2. Client registers data with VOSpace: register(, URI of location) returns the registered <node>

This is actually the only transfer method which needs a modification: all the others work fine with decoupled servers. In fact, instead of adding an additional operation, we can modify pushToVoSpace either to have an additional URI argument: pushToVoSpace(<node>, <transfer>, location-uri) or we could just incorporate the location-uri into the transfer so that if the protocol contains an endpoint then that endpoint is interpreted to be the physical location of the data object.

One thing that would be useful is another operation to return the list of (decoupled) data servers (resources in SRB speak) that the VOSpace is using so I would suggest that we add a getDataServers operation.

Votes

name vote comment
MatthewGraham +1 proposer


  See message for rationale of above itemized list:

  1. make (file)name a mandatory node property
  2. use pullDataToVOSpace with copyNode and moveNode to operate across stores
  3. {copy|move}Node to support exception StoreFull
  4. drop list paging in favor of filtering by property value

Apparently this democratic page accepts votes: Well, obviously I vote in favor of my comments, however, in case of disagreement I'd rather see how to use level 1 as is based on some real scenarios. Markus

Votes

name vote comment
MarkusDolensky +1 proposer (yeah)

Make the destination of CopyNode and MoveNode a simple uri

The advantage is it simplifies the standard doc (so many restrictions on node) and is much easier for client to use... as does not have to create a node object - simply create a uri for the destination.

Votes

name vote comment
PaulHarrison +1 proposer

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1  

Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Changes

VOspace10Spec21Archive

Changes for later versions of VOSpace

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
*Result*: consider for later version

Add a FindNodes operation

This operation would allow a simple search on the VOSpace - the level of functionality would be similar to the unix "find" command.

<--  
-->

Revision 282006-08-14 - MarkusDolensky

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • -1 if you disagree
  • 0 if you have no particular preference

Details and discussion of implementation plans are here.



Version 0.22

This document was produced as a result of the discussions that have occured since the Victoria Interop meeting.

Change Requests

Added:
>
>
See message for rationale of above itemized list:

  1. make (file)name a mandatory node property
  2. use pullDataToVOSpace with copyNode and moveNode to operate across stores
  3. {copy|move}Node to support exception StoreFull
  4. drop list paging in favor of filtering by property value

Apparently this democratic page accepts votes: Well, obviously I vote in favor of my comments, however, in case of disagreement I'd rather see how to use level 1 as is based on some real scenarios. Markus

Votes

name vote comment
MarkusDolensky +1 proposer (yeah)
 

Make the destination of CopyNode and MoveNode a simple uri

The advantage is it simplifies the standard doc (so many restrictions on node) and is much easier for client to use... as does not have to create a node object - simply create a uri for the destination.

Votes

name vote comment
PaulHarrison +1 proposer

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1  

Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Changes

VOspace10Spec21Archive

Changes for later versions of VOSpace

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
*Result*: consider for later version

Add a FindNodes operation

This operation would allow a simple search on the VOSpace - the level of functionality would be similar to the unix "find" command.

<--  
-->

Revision 272006-08-09 - PaulHarrison

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • -1 if you disagree
  • 0 if you have no particular preference

Details and discussion of implementation plans are here.



Version 0.22

Changed:
<
<
>
>
  This document was produced as a result of the discussions that have occured since the Victoria Interop meeting.

Change Requests

Make the destination of CopyNode and MoveNode a simple uri

The advantage is it simplifies the standard doc (so many restrictions on node) and is much easier for client to use... as does not have to create a node object - simply create a uri for the destination.

Votes

name vote comment
PaulHarrison +1 proposer

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1  

Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Changes

VOspace10Spec21Archive

Changes for later versions of VOSpace

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
*Result*: consider for later version

Add a FindNodes operation

This operation would allow a simple search on the VOSpace - the level of functionality would be similar to the unix "find" command.

<--  
-->

Revision 262006-07-20 - PaulHarrison

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • -1 if you disagree
  • 0 if you have no particular preference

Details and discussion of implementation plans are here.



Version 0.22

This document was produced as a result of the discussions that have occured since the Victoria Interop meeting.

Change Requests

Make the destination of CopyNode and MoveNode a simple uri

The advantage is it simplifies the standard doc (so many restrictions on node) and is much easier for client to use... as does not have to create a node object - simply create a uri for the destination.

Votes

name vote comment
PaulHarrison +1 proposer

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1  

Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Changes

VOspace10Spec21Archive

Changes for later versions of VOSpace

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
*Result*: consider for later version
Added:
>
>

Add a FindNodes operation

 
Added:
>
>
This operation would allow a simple search on the VOSpace - the level of functionality would be similar to the unix "find" command.
 
<--  
-->

Revision 252006-07-19 - PaulHarrison

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • -1 if you disagree
  • 0 if you have no particular preference

Details and discussion of implementation plans are here.



Version 0.22

This document was produced as a result of the discussions that have occured since the Victoria Interop meeting.

Change Requests

Added:
>
>

Make the destination of CopyNode and MoveNode a simple uri

The advantage is it simplifies the standard doc (so many restrictions on node) and is much easier for client to use... as does not have to create a node object - simply create a uri for the destination.

Votes

name vote comment
PaulHarrison +1 proposer
 

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1  

Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Changes

VOspace10Spec21Archive
Deleted:
<
<

Mandate and define at least one transport protocol

without this a compliant VOSpace will not be able to transfer data to another compliant VOSpace.

I recommend http be mandatory.

see detailed discussion in email thread http://www.ivoa.net/forum/vospace/0606/0097.htm

-- PaulHarrison

  • Which version, http-1.0 or http-1.1 ?
  • Which methods http-get, http-put or both ?
    • For a space that stores public images, then http-1.1-get makes sense.
    • For a space that allows upload to sensitive database tables, then http-1.x-put does not have sufficient authentication.

-- DaveMorris - 16 Jun 2006

I think that where VOSpace is acting as a http server then it would be good to mandate http-1.1 compliance - it does fix issues with 1.0 and afterall it is a 7 year old specification now....

As far as a compliance statement goes - how about

(http-1.1-get or https-1.1-get) and (http-1.1-put or https-1.1-put)

-- PaulHarrison - 19 Jun 2006

Making some form of HTTP mandatory makes it easier to write clients and harder to write services. VOSpace fails unless we get useful services, and services are already harder to write than clients.

-- GuyRixon - 03 Jul 2006

we are already implicitly mandating http for the "control" interface - i.e. the web service - as far as implementors are concerned adding http-get would be not a great burden for the implementor - without a mandatory transport the success of the use case where a client sets up direct space to space transfers is not guaranteed, so the fallback would have to be copying the data to the client which understands the transport of each space and then transferring to the second space.

-- PaulHarrison - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris -1
GuyRixon -1

*result*: not done

Specify as optional a small list of well known transport protocols

so that at least implementations will do the same thing for common protocols - this list to include
  • ftp
  • gridftp
  • file - direct access to the file system

-- PaulHarrison

Agree with the list of common protocols.

Defined in a annex to the main specification, including a standard URI and details of the protocol specification (can be just short note and a reference to the external specification).

e.g.

HTTP 1.1 get
URI
    ivo://....vospace/protocols/http-1.1-get
Description :
    Get data using the HTTP-1.1 GET method as defined in RFC2616.
    http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.3

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

*Result*: specified as registry entries

Specify the key names and meanings for a small group of "essential" property keys

The minimum set which I would say we need mandatory names are
  • vos.Owner
  • vos.ModificationDate
  • vos.Size
  • vos.MimeType

-- PaulHarrison

Agree with the list of common properties.

Defined in a annex to the main specification, including a standard URI and details of what each property means and how it is represented.

e.g.

Data created date
KEY
    vos.data.created.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were originally created.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]
----
Data modified date
KEY
    vos.data.modified.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were last modified.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]

Note - As VOSpace-1.0 does not support append, the only way the created and modified dates will be different is if the server modifies the data underneath.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

*Result*: specified as registry entries

Clarify the use of the Format parameter in some calls

I believe that the original intention of Format was to specify a possible transformation of the data (mainly on export of table data from RDBMS based stores). In the current version of the document it reads as if this parameter is merely a description of the data - in which case mime-type suffices. In addition I am not sure if this parameter has any meaning for import operations.

Votes

name vote comment
PaulHarrison +1

*Result*: Format concept is now represented by the view concept

Change "exception" to "fault"

The document uses the term exception for the "faults" rather than "fault" -- PaulHarrison

Votes

name vote comment
PaulHarrison +1
DaveMorris -1 I can't see a significant benefit from changing this

*Result*: not done

Consider adding an optional identifier list to the parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a client is currently interacting with a V2.0 VOSpace that has links to a small subset (e.g. 100 data items) - the client needs to make 100 getNodeProperties SOAP calls to the V1.0 space to get the latest metadata about the data objects - with an optional identifier list it can make one call to ListNodes.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris -1 I can't see a significant benefit from changing this

*Result*: done

Reconsider getNodeProperties and setNodeProperties

the naming of these operations makes them appear as a pair, where in fact they are not - you get a Node out of the getNodeProperties and have to put a PropertyPairList into the setNodeProperties - If the change to ListNodes were made as above, then getNodeProperties would be redundant anyway. -- PaulHarrison

Votes

name vote comment
PaulHarrison +1 the benefit is improved clarity - in addition there is the slightly irritating issue (see below) that you cannot take the property list from the "get" and use it in the "set" because of read-only properties
DaveMorris -1 I can't see a significant benefit from changing this

*Result*: done

Semantics not clear for setNodeProperties

It is not clear
  • how to delete a property - does a null value of the property pair denote this?
  • does the whole set of node properties given as an argument replace the whole set for the node, or is a union operation performed.
  • interaction with the "read-only" properties in these scenarios...
-- PaulHarrison

Agree, need to make this clearer.

  • Null value deletes property.
  • Operation is union not replace.
  • Server throws PermissionDenined for read-only properties.
    • what if only one of the list of properties is read-only? - need to signal which are the bad properties in the exception - also PermissionDenied could be confusing as this normally refers to the permissions on the data object which could potentially be changed in future versions of VOSpace - certain properties are fundamentally readOnly and so would

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

*Result*: done

Change names of parameters for moveNode and copyNode

Target is a confusing name for the "source" part of a move operation it sounds more like "destination" - recommend use "source" and "destination" -- PaulHarrison

Yep, source and destination are probably better. As long as we make it clear that these are internal locations, not references to external locations.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1
GuyRixon +1

*Result*: done

Rename bulk data transfer operations

  • I stilll find the data transfer operations confusing- I think that the basic problem is that the push and pull verbs are opposite in meaning and viewed from opposite perspectives. I have a new set of proposals

old name new name"
pushDataToVoSpace importDataClientPush
pullDataToVoSpace importDataServerPull
pushDataFromVoSpace exportDataServerPush
pullDataFromVoSpace exportDataClientPull

which I think are better because

  1. the overall objective of the operation is the first verb
  2. the active party in the transfer is identified
  3. the direction of the transfer is then related to the active party.

-- PaulHarrison - 13 Jun 2006

This is a change to the specification not the WSDL, move this to the specification change page.

-- DaveMorris - 16 Jun 2006

I find the newly-proposed names more confusing. It's not obvious to me whether "import" moves data into VOSpace or into the client of VOSpace; similarly with "export".

-- GuyRixon - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1
MatthewGraham 0
DaveMorris -1 Change the specification document first
GuyRixon -1  

*Result*: not done

New GetPropertyKeys operation

This proposal is basically because I am still a little worried about interoperability problems with the completely untyped nature of the property-key pairs - particularly as they are expected to carry some fundamental metadata about the data objects in the current implementation. This call would return the complete list of key names that have been used in the VOSpace, which would then allow clients to attempt to be consistent in the use of key names - it is not much but at least it does provide a mechanism to voluntarily avoid complete anarchy.

Should return a list of the keys with an indication of which ones relate to read-only properties.

-- PaulHarrison - 13 Jun 2006

I agree with adding the method. However, this should be changed in the specification first, not the WSDL. Move this to the specification change page.

-- DaveMorris - 16 Jun 2006

Suggestion 1: this operation only returns the set of properties understood by the service. Collating the other properties doesn't seem useful and might be expensive to implement.

Suggestion 2: for the properties of interest, return metadata roughly equivalent to those for a CEA parameter:

  • Key (i.e. formal name)
  • Name to show in UI
  • Description
  • Whether writeable by client (probably as boolean attribute "readOnly")
  • UCD 1+
  • Unit

-- GuyRixon - 04 Jul 2006

Suggestion 2 does have some appeal. Suggestion 1, however, I am not so keen on, the original intention of this proposal was to be able to find out about the "custom properties" that clients might be using, rather than simply the ones that the service "understands" - for the extra metadata to be supplied though it would imply that an extra "registerPropertyMetadata" operation would be needed before a client could set the property.

And I do not see this being at all expensive to implement - it is much easier to implement than storing the property values of each of the property instances for each of the stored data objects, and we already require that.

-- PaulHarrison - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris +1
GuyRixon +1 With details above
MatthewGraham +1

*Result*: done

Re-assess the semantics of creating a server named data object.

see discussion on mailing list for more background http://www.ivoa.net/forum/vospace/0606/0095.htm

Votes

name vote comment

*Result*: done

Rename Status (of Node object) to TransferStatus

The Status member of the Node object is really referring to the transfer status.

Votes

name vote comment
PaulHarrison +1 proposer - benefit is increased clarity
GuyRixon +1

*Result*: new status that just indicates "busy" or not.

Consider interactions when two clients attempt operations on same data object

use case

client 2 attempts to delete a file that client 1 is currently downloading via a "pull" transfer.

As specified at the moment, Client 1 could have a data transfer cut off with out any real knowledge of why, or client 2 could receive a PermissionDenied fault

Possible solutions

  • add a DataInUse fault to the destructive manipulation methods
  • add a dataBeingRead to the list of Status values

This use case is actually quite an implementation challenge....involves close interaction with the states of the actual data streams for the transports.

Votes

name vote comment
PaulHarrison +1 proposer
*Result*: new status that just indicates "busy" or not.

Expand "expired" Node status to include "failed"

perhaps failed is a better name can then encompass the case where a partial data transfer has occured and failed half way, for some reason other than expiry

Votes

name vote comment
PaulHarrison +1 proposer

Different type of list operation that returns only the uris...

Efficiency...

Proposed (and provisionally agreed by DaveMorris, MatthewGraham and GuyRixon) that we include a "level" parameter to select what is returned. The returned elements are always node elements but:

  • level="minimal" returns nodes with only the URI child;
  • level="standard" returns nodes with the properties as well;
  • level="extended" alls the return of derived types of nodes.

(Naming of the levels can be tweaked if necessary.) "Derived types of nodes" allows implementors to design special nodes (e.g. LdapDataNode) that have all the parts of a normal node and also extra children. The level parameter protects the clients from parser grief when derived node-types are in play.

The URI child might become an attribute of the node element.

-- GuyRixon - 04 Jul 2006

This design does not seem very OO to me - too may meanings are being attached to level - we are saying that the ListNodes operation returns a type Node, but for level="minimal" actually something is returned that is an "incomplete" node though it is still being called a node. It would seem cleaner to have a different operation ListIdentifiers that returned an identifier array for this case.

the level=extended seems not to be within scope of 1.0 - It also seems to be confusing the metadata about a node with the data in a node.

-- PaulHarrison - 05 Jul 2006

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1 With changes listed above
*Result*: done

Confusing that pullDataToVoSpace, pushDataFromVoSpace return transfer object.

The transfer object contains a location element which is intended to be the uri that pullDataFromVoSpace and pushDataToVoSpace supply as an output parameter, so it is confusing that pullDataToVoSpace, pushDataFromVoSpace return a uri for a transfer when for those methods the uri for transfer is an input parameter.

Recommend that the pullDataToVoSpace, pushDataFromVoSpace return only what is necessary for the client to know - this might mean removing the transfer object from the return list, or refactoring the transfer object - global analysis needed.

Votes

name vote comment
PaulHarrison +1 proposer

Need to have describe how VOSpace should be registered

The registration of VOSpace servers will be crucial their operation - we need some discussion of this and preferably a VOResouce extension schema to accompany it. PaulHarrison - 27 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1

*Result*: registry schema done

Transports operation needs to return the direction of operation

The operation that returns the list of transports supported needs to return also the direction that the transport is supported in. e.g. a VOSpace server might support ftp as a client (in pullDataToVoSpace), but not as a server (pushDataToVoSpace).

Votes

name vote comment
PaulHarrison +1 proposer

*Result*: protocols are listed in two categories, "accepts" and "provides"

 

Changes for later versions of VOSpace

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
*Result*: consider for later version


<--  
-->

Revision 242006-07-19 - PaulHarrison

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • -1 if you disagree
  • 0 if you have no particular preference

Details and discussion of implementation plans are here.



Changed:
<
<

Version 0.21

>
>

Version 0.22

 
Changed:
<
<
This document was produced as a result of the discussions that occured at the Victoria Interop meeting.
>
>
This document was produced as a result of the discussions that have occured since the Victoria Interop meeting.
 

Change Requests

Added:
>
>

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1  

Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Changes

VOspace10Spec21Archive
 

Mandate and define at least one transport protocol

without this a compliant VOSpace will not be able to transfer data to another compliant VOSpace.

I recommend http be mandatory.

see detailed discussion in email thread http://www.ivoa.net/forum/vospace/0606/0097.htm

-- PaulHarrison

  • Which version, http-1.0 or http-1.1 ?
  • Which methods http-get, http-put or both ?
    • For a space that stores public images, then http-1.1-get makes sense.
    • For a space that allows upload to sensitive database tables, then http-1.x-put does not have sufficient authentication.

-- DaveMorris - 16 Jun 2006

I think that where VOSpace is acting as a http server then it would be good to mandate http-1.1 compliance - it does fix issues with 1.0 and afterall it is a 7 year old specification now....

As far as a compliance statement goes - how about

(http-1.1-get or https-1.1-get) and (http-1.1-put or https-1.1-put)

-- PaulHarrison - 19 Jun 2006

Making some form of HTTP mandatory makes it easier to write clients and harder to write services. VOSpace fails unless we get useful services, and services are already harder to write than clients.

-- GuyRixon - 03 Jul 2006

we are already implicitly mandating http for the "control" interface - i.e. the web service - as far as implementors are concerned adding http-get would be not a great burden for the implementor - without a mandatory transport the success of the use case where a client sets up direct space to space transfers is not guaranteed, so the fallback would have to be copying the data to the client which understands the transport of each space and then transferring to the second space.

-- PaulHarrison - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris -1
GuyRixon -1
Added:
>
>
*result*: not done
 

Specify as optional a small list of well known transport protocols

so that at least implementations will do the same thing for common protocols - this list to include
  • ftp
  • gridftp
  • file - direct access to the file system

-- PaulHarrison

Agree with the list of common protocols.

Defined in a annex to the main specification, including a standard URI and details of the protocol specification (can be just short note and a reference to the external specification).

e.g.

HTTP 1.1 get
URI
    ivo://....vospace/protocols/http-1.1-get
Description :
    Get data using the HTTP-1.1 GET method as defined in RFC2616.
    http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.3
Added:
>
>
 -- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1
Added:
>
>
*Result*: specified as registry entries
 

Specify the key names and meanings for a small group of "essential" property keys

The minimum set which I would say we need mandatory names are
  • vos.Owner
  • vos.ModificationDate
  • vos.Size
  • vos.MimeType

-- PaulHarrison

Agree with the list of common properties.

Defined in a annex to the main specification, including a standard URI and details of what each property means and how it is represented.

e.g.

Data created date
KEY
    vos.data.created.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were originally created.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]
----
Data modified date
KEY
    vos.data.modified.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were last modified.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]

Note - As VOSpace-1.0 does not support append, the only way the created and modified dates will be different is if the server modifies the data underneath.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1
Added:
>
>
*Result*: specified as registry entries
 

Clarify the use of the Format parameter in some calls

I believe that the original intention of Format was to specify a possible transformation of the data (mainly on export of table data from RDBMS based stores). In the current version of the document it reads as if this parameter is merely a description of the data - in which case mime-type suffices. In addition I am not sure if this parameter has any meaning for import operations.

Votes

name vote comment
PaulHarrison +1
Added:
>
>
*Result*: Format concept is now represented by the view concept
 

Change "exception" to "fault"

The document uses the term exception for the "faults" rather than "fault" -- PaulHarrison

Votes

name vote comment
PaulHarrison +1
DaveMorris -1 I can't see a significant benefit from changing this
Added:
>
>
*Result*: not done
 
Deleted:
<
<

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
 

Consider adding an optional identifier list to the parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a client is currently interacting with a V2.0 VOSpace that has links to a small subset (e.g. 100 data items) - the client needs to make 100 getNodeProperties SOAP calls to the V1.0 space to get the latest metadata about the data objects - with an optional identifier list it can make one call to ListNodes.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris -1 I can't see a significant benefit from changing this
Added:
>
>
*Result*: done
 

Reconsider getNodeProperties and setNodeProperties

the naming of these operations makes them appear as a pair, where in fact they are not - you get a Node out of the getNodeProperties and have to put a PropertyPairList into the setNodeProperties - If the change to ListNodes were made as above, then getNodeProperties would be redundant anyway. -- PaulHarrison

Votes

name vote comment
PaulHarrison +1 the benefit is improved clarity - in addition there is the slightly irritating issue (see below) that you cannot take the property list from the "get" and use it in the "set" because of read-only properties
DaveMorris -1 I can't see a significant benefit from changing this
Added:
>
>
*Result*: done
 

Semantics not clear for setNodeProperties

It is not clear
  • how to delete a property - does a null value of the property pair denote this?
  • does the whole set of node properties given as an argument replace the whole set for the node, or is a union operation performed.
  • interaction with the "read-only" properties in these scenarios...
-- PaulHarrison

Agree, need to make this clearer.

  • Null value deletes property.
  • Operation is union not replace.
  • Server throws PermissionDenined for read-only properties.
    • what if only one of the list of properties is read-only? - need to signal which are the bad properties in the exception - also PermissionDenied could be confusing as this normally refers to the permissions on the data object which could potentially be changed in future versions of VOSpace - certain properties are fundamentally readOnly and so would

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1
Added:
>
>
*Result*: done
 

Change names of parameters for moveNode and copyNode

Target is a confusing name for the "source" part of a move operation it sounds more like "destination" - recommend use "source" and "destination" -- PaulHarrison

Yep, source and destination are probably better. As long as we make it clear that these are internal locations, not references to external locations.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1
GuyRixon +1
Added:
>
>
*Result*: done
 

Rename bulk data transfer operations

  • I stilll find the data transfer operations confusing- I think that the basic problem is that the push and pull verbs are opposite in meaning and viewed from opposite perspectives. I have a new set of proposals

old name new name"
pushDataToVoSpace importDataClientPush
pullDataToVoSpace importDataServerPull
pushDataFromVoSpace exportDataServerPush
pullDataFromVoSpace exportDataClientPull

which I think are better because

  1. the overall objective of the operation is the first verb
  2. the active party in the transfer is identified
  3. the direction of the transfer is then related to the active party.

-- PaulHarrison - 13 Jun 2006

This is a change to the specification not the WSDL, move this to the specification change page.

-- DaveMorris - 16 Jun 2006

I find the newly-proposed names more confusing. It's not obvious to me whether "import" moves data into VOSpace or into the client of VOSpace; similarly with "export".

-- GuyRixon - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1
MatthewGraham 0
DaveMorris -1 Change the specification document first
GuyRixon -1  
Added:
>
>
*Result*: not done
 

New GetPropertyKeys operation

This proposal is basically because I am still a little worried about interoperability problems with the completely untyped nature of the property-key pairs - particularly as they are expected to carry some fundamental metadata about the data objects in the current implementation. This call would return the complete list of key names that have been used in the VOSpace, which would then allow clients to attempt to be consistent in the use of key names - it is not much but at least it does provide a mechanism to voluntarily avoid complete anarchy.

Should return a list of the keys with an indication of which ones relate to read-only properties.

-- PaulHarrison - 13 Jun 2006

I agree with adding the method. However, this should be changed in the specification first, not the WSDL. Move this to the specification change page.

-- DaveMorris - 16 Jun 2006

Suggestion 1: this operation only returns the set of properties understood by the service. Collating the other properties doesn't seem useful and might be expensive to implement.

Suggestion 2: for the properties of interest, return metadata roughly equivalent to those for a CEA parameter:

  • Key (i.e. formal name)
  • Name to show in UI
  • Description
  • Whether writeable by client (probably as boolean attribute "readOnly")
  • UCD 1+
  • Unit

-- GuyRixon - 04 Jul 2006

Suggestion 2 does have some appeal. Suggestion 1, however, I am not so keen on, the original intention of this proposal was to be able to find out about the "custom properties" that clients might be using, rather than simply the ones that the service "understands" - for the extra metadata to be supplied though it would imply that an extra "registerPropertyMetadata" operation would be needed before a client could set the property.

And I do not see this being at all expensive to implement - it is much easier to implement than storing the property values of each of the property instances for each of the stored data objects, and we already require that.

-- PaulHarrison - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris +1
GuyRixon +1 With details above
MatthewGraham +1
Changed:
<
<

Re-assess what metadata should be returned with various faults

>
>
*Result*: done
 
Deleted:
<
<
Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1  
 

Re-assess the semantics of creating a server named data object.

see discussion on mailing list for more background http://www.ivoa.net/forum/vospace/0606/0095.htm

Votes

name vote comment
Added:
>
>
*Result*: done
 

Rename Status (of Node object) to TransferStatus

The Status member of the Node object is really referring to the transfer status.

Votes

name vote comment
PaulHarrison +1 proposer - benefit is increased clarity
GuyRixon +1
Added:
>
>
*Result*: new status that just indicates "busy" or not.
 

Consider interactions when two clients attempt operations on same data object

use case

client 2 attempts to delete a file that client 1 is currently downloading via a "pull" transfer.

As specified at the moment, Client 1 could have a data transfer cut off with out any real knowledge of why, or client 2 could receive a PermissionDenied fault

Possible solutions

  • add a DataInUse fault to the destructive manipulation methods
  • add a dataBeingRead to the list of Status values

This use case is actually quite an implementation challenge....involves close interaction with the states of the actual data streams for the transports.

Votes

name vote comment
PaulHarrison +1 proposer
Added:
>
>
*Result*: new status that just indicates "busy" or not.
 

Expand "expired" Node status to include "failed"

perhaps failed is a better name can then encompass the case where a partial data transfer has occured and failed half way, for some reason other than expiry

Votes

name vote comment
PaulHarrison +1 proposer

Different type of list operation that returns only the uris...

Efficiency...

Proposed (and provisionally agreed by DaveMorris, MatthewGraham and GuyRixon) that we include a "level" parameter to select what is returned. The returned elements are always node elements but:

  • level="minimal" returns nodes with only the URI child;
  • level="standard" returns nodes with the properties as well;
  • level="extended" alls the return of derived types of nodes.

(Naming of the levels can be tweaked if necessary.) "Derived types of nodes" allows implementors to design special nodes (e.g. LdapDataNode) that have all the parts of a normal node and also extra children. The level parameter protects the clients from parser grief when derived node-types are in play.

The URI child might become an attribute of the node element.

-- GuyRixon - 04 Jul 2006

This design does not seem very OO to me - too may meanings are being attached to level - we are saying that the ListNodes operation returns a type Node, but for level="minimal" actually something is returned that is an "incomplete" node though it is still being called a node. It would seem cleaner to have a different operation ListIdentifiers that returned an identifier array for this case.

the level=extended seems not to be within scope of 1.0 - It also seems to be confusing the metadata about a node with the data in a node.

-- PaulHarrison - 05 Jul 2006

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1 With changes listed above
Added:
>
>
*Result*: done
 
Added:
>
>
 

Confusing that pullDataToVoSpace, pushDataFromVoSpace return transfer object.

The transfer object contains a location element which is intended to be the uri that pullDataFromVoSpace and pushDataToVoSpace supply as an output parameter, so it is confusing that pullDataToVoSpace, pushDataFromVoSpace return a uri for a transfer when for those methods the uri for transfer is an input parameter.

Recommend that the pullDataToVoSpace, pushDataFromVoSpace return only what is necessary for the client to know - this might mean removing the transfer object from the return list, or refactoring the transfer object - global analysis needed.

Votes

name vote comment
PaulHarrison +1 proposer

Need to have describe how VOSpace should be registered

The registration of VOSpace servers will be crucial their operation - we need some discussion of this and preferably a VOResouce extension schema to accompany it. PaulHarrison - 27 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1
Added:
>
>
*Result*: registry schema done
 
Added:
>
>
 

Transports operation needs to return the direction of operation

The operation that returns the list of transports supported needs to return also the direction that the transport is supported in. e.g. a VOSpace server might support ftp as a client (in pullDataToVoSpace), but not as a server (pushDataToVoSpace).

Votes

name vote comment
PaulHarrison +1 proposer
Added:
>
>
*Result*: protocols are listed in two categories, "accepts" and "provides"

Changes for later versions of VOSpace

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
*Result*: consider for later version
 
<--  
-->

Revision 232006-07-05 - PaulHarrison

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • -1 if you disagree
  • 0 if you have no particular preference

Details and discussion of implementation plans are here.



Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Change Requests

Mandate and define at least one transport protocol

without this a compliant VOSpace will not be able to transfer data to another compliant VOSpace.

I recommend http be mandatory.

see detailed discussion in email thread http://www.ivoa.net/forum/vospace/0606/0097.htm

-- PaulHarrison

  • Which version, http-1.0 or http-1.1 ?
  • Which methods http-get, http-put or both ?
    • For a space that stores public images, then http-1.1-get makes sense.
    • For a space that allows upload to sensitive database tables, then http-1.x-put does not have sufficient authentication.

-- DaveMorris - 16 Jun 2006

I think that where VOSpace is acting as a http server then it would be good to mandate http-1.1 compliance - it does fix issues with 1.0 and afterall it is a 7 year old specification now....

As far as a compliance statement goes - how about

(http-1.1-get or https-1.1-get) and (http-1.1-put or https-1.1-put)

-- PaulHarrison - 19 Jun 2006

Making some form of HTTP mandatory makes it easier to write clients and harder to write services. VOSpace fails unless we get useful services, and services are already harder to write than clients.

-- GuyRixon - 03 Jul 2006

we are already implicitly mandating http for the "control" interface - i.e. the web service - as far as implementors are concerned adding http-get would be not a great burden for the implementor - without a mandatory transport the success of the use case where a client sets up direct space to space transfers is not guaranteed, so the fallback would have to be copying the data to the client which understands the transport of each space and then transferring to the second space.

-- PaulHarrison - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris -1
GuyRixon -1

Specify as optional a small list of well known transport protocols

so that at least implementations will do the same thing for common protocols - this list to include
  • ftp
  • gridftp
  • file - direct access to the file system

-- PaulHarrison

Agree with the list of common protocols.

Defined in a annex to the main specification, including a standard URI and details of the protocol specification (can be just short note and a reference to the external specification).

e.g.

HTTP 1.1 get
URI
    ivo://....vospace/protocols/http-1.1-get
Description :
    Get data using the HTTP-1.1 GET method as defined in RFC2616.
    http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.3

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Specify the key names and meanings for a small group of "essential" property keys

The minimum set which I would say we need mandatory names are
  • vos.Owner
  • vos.ModificationDate
  • vos.Size
  • vos.MimeType

-- PaulHarrison

Agree with the list of common properties.

Defined in a annex to the main specification, including a standard URI and details of what each property means and how it is represented.

e.g.

Data created date
KEY
    vos.data.created.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were originally created.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]
----
Data modified date
KEY
    vos.data.modified.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were last modified.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]

Note - As VOSpace-1.0 does not support append, the only way the created and modified dates will be different is if the server modifies the data underneath.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Clarify the use of the Format parameter in some calls

I believe that the original intention of Format was to specify a possible transformation of the data (mainly on export of table data from RDBMS based stores). In the current version of the document it reads as if this parameter is merely a description of the data - in which case mime-type suffices. In addition I am not sure if this parameter has any meaning for import operations.

Votes

name vote comment
PaulHarrison +1

Change "exception" to "fault"

The document uses the term exception for the "faults" rather than "fault" -- PaulHarrison

Votes

name vote comment
PaulHarrison +1
DaveMorris -1 I can't see a significant benefit from changing this

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer

Consider adding an optional identifier list to the parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a client is currently interacting with a V2.0 VOSpace that has links to a small subset (e.g. 100 data items) - the client needs to make 100 getNodeProperties SOAP calls to the V1.0 space to get the latest metadata about the data objects - with an optional identifier list it can make one call to ListNodes.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris -1 I can't see a significant benefit from changing this

Reconsider getNodeProperties and setNodeProperties

the naming of these operations makes them appear as a pair, where in fact they are not - you get a Node out of the getNodeProperties and have to put a PropertyPairList into the setNodeProperties - If the change to ListNodes were made as above, then getNodeProperties would be redundant anyway. -- PaulHarrison

Votes

name vote comment
PaulHarrison +1 the benefit is improved clarity - in addition there is the slightly irritating issue (see below) that you cannot take the property list from the "get" and use it in the "set" because of read-only properties
DaveMorris -1 I can't see a significant benefit from changing this

Semantics not clear for setNodeProperties

It is not clear
  • how to delete a property - does a null value of the property pair denote this?
  • does the whole set of node properties given as an argument replace the whole set for the node, or is a union operation performed.
  • interaction with the "read-only" properties in these scenarios...
-- PaulHarrison

Agree, need to make this clearer.

  • Null value deletes property.
  • Operation is union not replace.
  • Server throws PermissionDenined for read-only properties.
    • what if only one of the list of properties is read-only? - need to signal which are the bad properties in the exception - also PermissionDenied could be confusing as this normally refers to the permissions on the data object which could potentially be changed in future versions of VOSpace - certain properties are fundamentally readOnly and so would

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Change names of parameters for moveNode and copyNode

Target is a confusing name for the "source" part of a move operation it sounds more like "destination" - recommend use "source" and "destination" -- PaulHarrison

Yep, source and destination are probably better. As long as we make it clear that these are internal locations, not references to external locations.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1
GuyRixon +1

Rename bulk data transfer operations

  • I stilll find the data transfer operations confusing- I think that the basic problem is that the push and pull verbs are opposite in meaning and viewed from opposite perspectives. I have a new set of proposals

old name new name"
pushDataToVoSpace importDataClientPush
pullDataToVoSpace importDataServerPull
pushDataFromVoSpace exportDataServerPush
pullDataFromVoSpace exportDataClientPull

which I think are better because

  1. the overall objective of the operation is the first verb
  2. the active party in the transfer is identified
  3. the direction of the transfer is then related to the active party.

-- PaulHarrison - 13 Jun 2006

This is a change to the specification not the WSDL, move this to the specification change page.

-- DaveMorris - 16 Jun 2006

I find the newly-proposed names more confusing. It's not obvious to me whether "import" moves data into VOSpace or into the client of VOSpace; similarly with "export".

-- GuyRixon - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1
MatthewGraham 0
DaveMorris -1 Change the specification document first
GuyRixon -1  

New GetPropertyKeys operation

This proposal is basically because I am still a little worried about interoperability problems with the completely untyped nature of the property-key pairs - particularly as they are expected to carry some fundamental metadata about the data objects in the current implementation. This call would return the complete list of key names that have been used in the VOSpace, which would then allow clients to attempt to be consistent in the use of key names - it is not much but at least it does provide a mechanism to voluntarily avoid complete anarchy.

Should return a list of the keys with an indication of which ones relate to read-only properties.

-- PaulHarrison - 13 Jun 2006

I agree with adding the method. However, this should be changed in the specification first, not the WSDL. Move this to the specification change page.

-- DaveMorris - 16 Jun 2006

Suggestion 1: this operation only returns the set of properties understood by the service. Collating the other properties doesn't seem useful and might be expensive to implement.

Suggestion 2: for the properties of interest, return metadata roughly equivalent to those for a CEA parameter:

  • Key (i.e. formal name)
  • Name to show in UI
  • Description
  • Whether writeable by client (probably as boolean attribute "readOnly")
  • UCD 1+
  • Unit

-- GuyRixon - 04 Jul 2006

Suggestion 2 does have some appeal. Suggestion 1, however, I am not so keen on, the original intention of this proposal was to be able to find out about the "custom properties" that clients might be using, rather than simply the ones that the service "understands" - for the extra metadata to be supplied though it would imply that an extra "registerPropertyMetadata" operation would be needed before a client could set the property.

And I do not see this being at all expensive to implement - it is much easier to implement than storing the property values of each of the property instances for each of the stored data objects, and we already require that.

-- PaulHarrison - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris +1
GuyRixon +1 With details above
MatthewGraham +1

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1  

Re-assess the semantics of creating a server named data object.

see discussion on mailing list for more background http://www.ivoa.net/forum/vospace/0606/0095.htm

Votes

name vote comment

Rename Status (of Node object) to TransferStatus

The Status member of the Node object is really referring to the transfer status.

Votes

name vote comment
PaulHarrison +1 proposer - benefit is increased clarity
GuyRixon +1

Consider interactions when two clients attempt operations on same data object

use case

client 2 attempts to delete a file that client 1 is currently downloading via a "pull" transfer.

As specified at the moment, Client 1 could have a data transfer cut off with out any real knowledge of why, or client 2 could receive a PermissionDenied fault

Possible solutions

  • add a DataInUse fault to the destructive manipulation methods
  • add a dataBeingRead to the list of Status values

This use case is actually quite an implementation challenge....involves close interaction with the states of the actual data streams for the transports.

Votes

name vote comment
PaulHarrison +1 proposer

Expand "expired" Node status to include "failed"

perhaps failed is a better name can then encompass the case where a partial data transfer has occured and failed half way, for some reason other than expiry

Votes

name vote comment
PaulHarrison +1 proposer

Different type of list operation that returns only the uris...

Efficiency...

Proposed (and provisionally agreed by DaveMorris, MatthewGraham and GuyRixon) that we include a "level" parameter to select what is returned. The returned elements are always node elements but:

  • level="minimal" returns nodes with only the URI child;
  • level="standard" returns nodes with the properties as well;
  • level="extended" alls the return of derived types of nodes.

(Naming of the levels can be tweaked if necessary.) "Derived types of nodes" allows implementors to design special nodes (e.g. LdapDataNode) that have all the parts of a normal node and also extra children. The level parameter protects the clients from parser grief when derived node-types are in play.

The URI child might become an attribute of the node element.

-- GuyRixon - 04 Jul 2006

This design does not seem very OO to me - too may meanings are being attached to level - we are saying that the ListNodes operation returns a type Node, but for level="minimal" actually something is returned that is an "incomplete" node though it is still being called a node. It would seem cleaner to have a different operation ListIdentifiers that returned an identifier array for this case.

the level=extended seems not to be within scope of 1.0 - It also seems to be confusing the metadata about a node with the data in a node.

-- PaulHarrison - 05 Jul 2006

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1 With changes listed above

Confusing that pullDataToVoSpace, pushDataFromVoSpace return transfer object.

The transfer object contains a location element which is intended to be the uri that pullDataFromVoSpace and pushDataToVoSpace supply as an output parameter, so it is confusing that pullDataToVoSpace, pushDataFromVoSpace return a uri for a transfer when for those methods the uri for transfer is an input parameter.

Recommend that the pullDataToVoSpace, pushDataFromVoSpace return only what is necessary for the client to know - this might mean removing the transfer object from the return list, or refactoring the transfer object - global analysis needed.

Votes

name vote comment
PaulHarrison +1 proposer

Need to have describe how VOSpace should be registered

The registration of VOSpace servers will be crucial their operation - we need some discussion of this and preferably a VOResouce extension schema to accompany it. PaulHarrison - 27 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1
Added:
>
>

Transports operation needs to return the direction of operation

The operation that returns the list of transports supported needs to return also the direction that the transport is supported in. e.g. a VOSpace server might support ftp as a client (in pullDataToVoSpace), but not as a server (pushDataToVoSpace).

Votes

name vote comment
PaulHarrison +1 proposer
 
<--  
-->

Revision 222006-07-05 - PaulHarrison

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • -1 if you disagree
  • 0 if you have no particular preference

Details and discussion of implementation plans are here.



Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Change Requests

Mandate and define at least one transport protocol

without this a compliant VOSpace will not be able to transfer data to another compliant VOSpace.

I recommend http be mandatory.

see detailed discussion in email thread http://www.ivoa.net/forum/vospace/0606/0097.htm

-- PaulHarrison

  • Which version, http-1.0 or http-1.1 ?
  • Which methods http-get, http-put or both ?
    • For a space that stores public images, then http-1.1-get makes sense.
    • For a space that allows upload to sensitive database tables, then http-1.x-put does not have sufficient authentication.

-- DaveMorris - 16 Jun 2006

I think that where VOSpace is acting as a http server then it would be good to mandate http-1.1 compliance - it does fix issues with 1.0 and afterall it is a 7 year old specification now....

As far as a compliance statement goes - how about

(http-1.1-get or https-1.1-get) and (http-1.1-put or https-1.1-put)

-- PaulHarrison - 19 Jun 2006

Making some form of HTTP mandatory makes it easier to write clients and harder to write services. VOSpace fails unless we get useful services, and services are already harder to write than clients.

-- GuyRixon - 03 Jul 2006

we are already implicitly mandating http for the "control" interface - i.e. the web service - as far as implementors are concerned adding http-get would be not a great burden for the implementor - without a mandatory transport the success of the use case where a client sets up direct space to space transfers is not guaranteed, so the fallback would have to be copying the data to the client which understands the transport of each space and then transferring to the second space.

-- PaulHarrison - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris -1
GuyRixon -1

Specify as optional a small list of well known transport protocols

so that at least implementations will do the same thing for common protocols - this list to include
  • ftp
  • gridftp
  • file - direct access to the file system

-- PaulHarrison

Agree with the list of common protocols.

Defined in a annex to the main specification, including a standard URI and details of the protocol specification (can be just short note and a reference to the external specification).

e.g.

HTTP 1.1 get
URI
    ivo://....vospace/protocols/http-1.1-get
Description :
    Get data using the HTTP-1.1 GET method as defined in RFC2616.
    http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.3

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Specify the key names and meanings for a small group of "essential" property keys

The minimum set which I would say we need mandatory names are
  • vos.Owner
  • vos.ModificationDate
  • vos.Size
  • vos.MimeType

-- PaulHarrison

Agree with the list of common properties.

Defined in a annex to the main specification, including a standard URI and details of what each property means and how it is represented.

e.g.

Data created date
KEY
    vos.data.created.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were originally created.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]
----
Data modified date
KEY
    vos.data.modified.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were last modified.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]

Note - As VOSpace-1.0 does not support append, the only way the created and modified dates will be different is if the server modifies the data underneath.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Clarify the use of the Format parameter in some calls

I believe that the original intention of Format was to specify a possible transformation of the data (mainly on export of table data from RDBMS based stores). In the current version of the document it reads as if this parameter is merely a description of the data - in which case mime-type suffices. In addition I am not sure if this parameter has any meaning for import operations.

Votes

name vote comment
PaulHarrison +1

Change "exception" to "fault"

The document uses the term exception for the "faults" rather than "fault" -- PaulHarrison

Votes

name vote comment
PaulHarrison +1
DaveMorris -1 I can't see a significant benefit from changing this

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer

Consider adding an optional identifier list to the parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a client is currently interacting with a V2.0 VOSpace that has links to a small subset (e.g. 100 data items) - the client needs to make 100 getNodeProperties SOAP calls to the V1.0 space to get the latest metadata about the data objects - with an optional identifier list it can make one call to ListNodes.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris -1 I can't see a significant benefit from changing this

Reconsider getNodeProperties and setNodeProperties

the naming of these operations makes them appear as a pair, where in fact they are not - you get a Node out of the getNodeProperties and have to put a PropertyPairList into the setNodeProperties - If the change to ListNodes were made as above, then getNodeProperties would be redundant anyway. -- PaulHarrison

Votes

name vote comment
PaulHarrison +1 the benefit is improved clarity - in addition there is the slightly irritating issue (see below) that you cannot take the property list from the "get" and use it in the "set" because of read-only properties
DaveMorris -1 I can't see a significant benefit from changing this

Semantics not clear for setNodeProperties

It is not clear
  • how to delete a property - does a null value of the property pair denote this?
  • does the whole set of node properties given as an argument replace the whole set for the node, or is a union operation performed.
  • interaction with the "read-only" properties in these scenarios...
-- PaulHarrison

Agree, need to make this clearer.

  • Null value deletes property.
  • Operation is union not replace.
  • Server throws PermissionDenined for read-only properties.
    • what if only one of the list of properties is read-only? - need to signal which are the bad properties in the exception - also PermissionDenied could be confusing as this normally refers to the permissions on the data object which could potentially be changed in future versions of VOSpace - certain properties are fundamentally readOnly and so would

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Change names of parameters for moveNode and copyNode

Target is a confusing name for the "source" part of a move operation it sounds more like "destination" - recommend use "source" and "destination" -- PaulHarrison

Yep, source and destination are probably better. As long as we make it clear that these are internal locations, not references to external locations.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1
GuyRixon +1

Rename bulk data transfer operations

  • I stilll find the data transfer operations confusing- I think that the basic problem is that the push and pull verbs are opposite in meaning and viewed from opposite perspectives. I have a new set of proposals

old name new name"
pushDataToVoSpace importDataClientPush
pullDataToVoSpace importDataServerPull
pushDataFromVoSpace exportDataServerPush
pullDataFromVoSpace exportDataClientPull

which I think are better because

  1. the overall objective of the operation is the first verb
  2. the active party in the transfer is identified
  3. the direction of the transfer is then related to the active party.

-- PaulHarrison - 13 Jun 2006

This is a change to the specification not the WSDL, move this to the specification change page.

-- DaveMorris - 16 Jun 2006

I find the newly-proposed names more confusing. It's not obvious to me whether "import" moves data into VOSpace or into the client of VOSpace; similarly with "export".

-- GuyRixon - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1
MatthewGraham 0
DaveMorris -1 Change the specification document first
GuyRixon -1  

New GetPropertyKeys operation

This proposal is basically because I am still a little worried about interoperability problems with the completely untyped nature of the property-key pairs - particularly as they are expected to carry some fundamental metadata about the data objects in the current implementation. This call would return the complete list of key names that have been used in the VOSpace, which would then allow clients to attempt to be consistent in the use of key names - it is not much but at least it does provide a mechanism to voluntarily avoid complete anarchy.

Should return a list of the keys with an indication of which ones relate to read-only properties.

-- PaulHarrison - 13 Jun 2006

I agree with adding the method. However, this should be changed in the specification first, not the WSDL. Move this to the specification change page.

-- DaveMorris - 16 Jun 2006

Suggestion 1: this operation only returns the set of properties understood by the service. Collating the other properties doesn't seem useful and might be expensive to implement.

Suggestion 2: for the properties of interest, return metadata roughly equivalent to those for a CEA parameter:

  • Key (i.e. formal name)
  • Name to show in UI
  • Description
  • Whether writeable by client (probably as boolean attribute "readOnly")
  • UCD 1+
  • Unit

-- GuyRixon - 04 Jul 2006

Added:
>
>
 Suggestion 2 does have some appeal. Suggestion 1, however, I am not so keen on, the original intention of this proposal was to be able to find out about the "custom properties" that clients might be using, rather than simply the ones that the service "understands" - for the extra metadata to be supplied though it would imply that an extra "registerPropertyMetadata" operation would be needed before a client could set the property.

And I do not see this being at all expensive to implement - it is much easier to implement than storing the property values of each of the property instances for each of the stored data objects, and we already require that.

-- PaulHarrison - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris +1
GuyRixon +1 With details above
MatthewGraham +1

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1  

Re-assess the semantics of creating a server named data object.

see discussion on mailing list for more background http://www.ivoa.net/forum/vospace/0606/0095.htm

Votes

name vote comment

Rename Status (of Node object) to TransferStatus

The Status member of the Node object is really referring to the transfer status.

Votes

name vote comment
PaulHarrison +1 proposer - benefit is increased clarity
GuyRixon +1

Consider interactions when two clients attempt operations on same data object

use case

client 2 attempts to delete a file that client 1 is currently downloading via a "pull" transfer.

As specified at the moment, Client 1 could have a data transfer cut off with out any real knowledge of why, or client 2 could receive a PermissionDenied fault

Possible solutions

  • add a DataInUse fault to the destructive manipulation methods
  • add a dataBeingRead to the list of Status values

This use case is actually quite an implementation challenge....involves close interaction with the states of the actual data streams for the transports.

Votes

name vote comment
PaulHarrison +1 proposer

Expand "expired" Node status to include "failed"

perhaps failed is a better name can then encompass the case where a partial data transfer has occured and failed half way, for some reason other than expiry

Votes

name vote comment
PaulHarrison +1 proposer

Different type of list operation that returns only the uris...

Efficiency...

Proposed (and provisionally agreed by DaveMorris, MatthewGraham and GuyRixon) that we include a "level" parameter to select what is returned. The returned elements are always node elements but:

  • level="minimal" returns nodes with only the URI child;
  • level="standard" returns nodes with the properties as well;
  • level="extended" alls the return of derived types of nodes.

(Naming of the levels can be tweaked if necessary.) "Derived types of nodes" allows implementors to design special nodes (e.g. LdapDataNode) that have all the parts of a normal node and also extra children. The level parameter protects the clients from parser grief when derived node-types are in play.

The URI child might become an attribute of the node element.

-- GuyRixon - 04 Jul 2006

Added:
>
>
This design does not seem very OO to me - too may meanings are being attached to level - we are saying that the ListNodes operation returns a type Node, but for level="minimal" actually something is returned that is an "incomplete" node though it is still being called a node. It would seem cleaner to have a different operation ListIdentifiers that returned an identifier array for this case.

the level=extended seems not to be within scope of 1.0 - It also seems to be confusing the metadata about a node with the data in a node.

-- PaulHarrison - 05 Jul 2006

 

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1 With changes listed above

Confusing that pullDataToVoSpace, pushDataFromVoSpace return transfer object.

The transfer object contains a location element which is intended to be the uri that pullDataFromVoSpace and pushDataToVoSpace supply as an output parameter, so it is confusing that pullDataToVoSpace, pushDataFromVoSpace return a uri for a transfer when for those methods the uri for transfer is an input parameter.

Recommend that the pullDataToVoSpace, pushDataFromVoSpace return only what is necessary for the client to know - this might mean removing the transfer object from the return list, or refactoring the transfer object - global analysis needed.

Votes

name vote comment
PaulHarrison +1 proposer

Need to have describe how VOSpace should be registered

The registration of VOSpace servers will be crucial their operation - we need some discussion of this and preferably a VOResouce extension schema to accompany it. PaulHarrison - 27 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1


<--  
-->

Revision 212006-07-04 - PaulHarrison

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • -1 if you disagree
  • 0 if you have no particular preference

Details and discussion of implementation plans are here.



Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Change Requests

Mandate and define at least one transport protocol

without this a compliant VOSpace will not be able to transfer data to another compliant VOSpace.

I recommend http be mandatory.

see detailed discussion in email thread http://www.ivoa.net/forum/vospace/0606/0097.htm

-- PaulHarrison

  • Which version, http-1.0 or http-1.1 ?
  • Which methods http-get, http-put or both ?
    • For a space that stores public images, then http-1.1-get makes sense.
    • For a space that allows upload to sensitive database tables, then http-1.x-put does not have sufficient authentication.

-- DaveMorris - 16 Jun 2006

I think that where VOSpace is acting as a http server then it would be good to mandate http-1.1 compliance - it does fix issues with 1.0 and afterall it is a 7 year old specification now....

As far as a compliance statement goes - how about

(http-1.1-get or https-1.1-get) and (http-1.1-put or https-1.1-put)

-- PaulHarrison - 19 Jun 2006

Making some form of HTTP mandatory makes it easier to write clients and harder to write services. VOSpace fails unless we get useful services, and services are already harder to write than clients.

-- GuyRixon - 03 Jul 2006

we are already implicitly mandating http for the "control" interface - i.e. the web service - as far as implementors are concerned adding http-get would be not a great burden for the implementor - without a mandatory transport the success of the use case where a client sets up direct space to space transfers is not guaranteed, so the fallback would have to be copying the data to the client which understands the transport of each space and then transferring to the second space.

-- PaulHarrison - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris -1
GuyRixon -1

Specify as optional a small list of well known transport protocols

so that at least implementations will do the same thing for common protocols - this list to include
  • ftp
  • gridftp
  • file - direct access to the file system

-- PaulHarrison

Agree with the list of common protocols.

Defined in a annex to the main specification, including a standard URI and details of the protocol specification (can be just short note and a reference to the external specification).

e.g.

HTTP 1.1 get
URI
    ivo://....vospace/protocols/http-1.1-get
Description :
    Get data using the HTTP-1.1 GET method as defined in RFC2616.
    http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.3

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Specify the key names and meanings for a small group of "essential" property keys

The minimum set which I would say we need mandatory names are
  • vos.Owner
  • vos.ModificationDate
  • vos.Size
  • vos.MimeType

-- PaulHarrison

Agree with the list of common properties.

Defined in a annex to the main specification, including a standard URI and details of what each property means and how it is represented.

e.g.

Data created date
KEY
    vos.data.created.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were originally created.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]
----
Data modified date
KEY
    vos.data.modified.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were last modified.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]

Note - As VOSpace-1.0 does not support append, the only way the created and modified dates will be different is if the server modifies the data underneath.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Clarify the use of the Format parameter in some calls

I believe that the original intention of Format was to specify a possible transformation of the data (mainly on export of table data from RDBMS based stores). In the current version of the document it reads as if this parameter is merely a description of the data - in which case mime-type suffices. In addition I am not sure if this parameter has any meaning for import operations.

Votes

name vote comment
PaulHarrison +1

Change "exception" to "fault"

The document uses the term exception for the "faults" rather than "fault" -- PaulHarrison

Votes

name vote comment
PaulHarrison +1
DaveMorris -1 I can't see a significant benefit from changing this

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer

Consider adding an optional identifier list to the parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a client is currently interacting with a V2.0 VOSpace that has links to a small subset (e.g. 100 data items) - the client needs to make 100 getNodeProperties SOAP calls to the V1.0 space to get the latest metadata about the data objects - with an optional identifier list it can make one call to ListNodes.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris -1 I can't see a significant benefit from changing this

Reconsider getNodeProperties and setNodeProperties

the naming of these operations makes them appear as a pair, where in fact they are not - you get a Node out of the getNodeProperties and have to put a PropertyPairList into the setNodeProperties - If the change to ListNodes were made as above, then getNodeProperties would be redundant anyway. -- PaulHarrison

Votes

name vote comment
PaulHarrison +1 the benefit is improved clarity - in addition there is the slightly irritating issue (see below) that you cannot take the property list from the "get" and use it in the "set" because of read-only properties
DaveMorris -1 I can't see a significant benefit from changing this

Semantics not clear for setNodeProperties

It is not clear
  • how to delete a property - does a null value of the property pair denote this?
  • does the whole set of node properties given as an argument replace the whole set for the node, or is a union operation performed.
  • interaction with the "read-only" properties in these scenarios...
-- PaulHarrison

Agree, need to make this clearer.

  • Null value deletes property.
  • Operation is union not replace.
  • Server throws PermissionDenined for read-only properties.
    • what if only one of the list of properties is read-only? - need to signal which are the bad properties in the exception - also PermissionDenied could be confusing as this normally refers to the permissions on the data object which could potentially be changed in future versions of VOSpace - certain properties are fundamentally readOnly and so would

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Change names of parameters for moveNode and copyNode

Target is a confusing name for the "source" part of a move operation it sounds more like "destination" - recommend use "source" and "destination" -- PaulHarrison

Yep, source and destination are probably better. As long as we make it clear that these are internal locations, not references to external locations.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1
GuyRixon +1

Rename bulk data transfer operations

  • I stilll find the data transfer operations confusing- I think that the basic problem is that the push and pull verbs are opposite in meaning and viewed from opposite perspectives. I have a new set of proposals

old name new name"
pushDataToVoSpace importDataClientPush
pullDataToVoSpace importDataServerPull
pushDataFromVoSpace exportDataServerPush
pullDataFromVoSpace exportDataClientPull

which I think are better because

  1. the overall objective of the operation is the first verb
  2. the active party in the transfer is identified
  3. the direction of the transfer is then related to the active party.

-- PaulHarrison - 13 Jun 2006

This is a change to the specification not the WSDL, move this to the specification change page.

-- DaveMorris - 16 Jun 2006

I find the newly-proposed names more confusing. It's not obvious to me whether "import" moves data into VOSpace or into the client of VOSpace; similarly with "export".

-- GuyRixon - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1
MatthewGraham 0
DaveMorris -1 Change the specification document first
GuyRixon -1  

New GetPropertyKeys operation

This proposal is basically because I am still a little worried about interoperability problems with the completely untyped nature of the property-key pairs - particularly as they are expected to carry some fundamental metadata about the data objects in the current implementation. This call would return the complete list of key names that have been used in the VOSpace, which would then allow clients to attempt to be consistent in the use of key names - it is not much but at least it does provide a mechanism to voluntarily avoid complete anarchy.

Should return a list of the keys with an indication of which ones relate to read-only properties.

-- PaulHarrison - 13 Jun 2006

I agree with adding the method. However, this should be changed in the specification first, not the WSDL. Move this to the specification change page.

-- DaveMorris - 16 Jun 2006

Suggestion 1: this operation only returns the set of properties understood by the service. Collating the other properties doesn't seem useful and might be expensive to implement.

Suggestion 2: for the properties of interest, return metadata roughly equivalent to those for a CEA parameter:

  • Key (i.e. formal name)
  • Name to show in UI
  • Description
  • Whether writeable by client (probably as boolean attribute "readOnly")
  • UCD 1+
  • Unit

-- GuyRixon - 04 Jul 2006

Added:
>
>
Suggestion 2 does have some appeal. Suggestion 1, however, I am not so keen on, the original intention of this proposal was to be able to find out about the "custom properties" that clients might be using, rather than simply the ones that the service "understands" - for the extra metadata to be supplied though it would imply that an extra "registerPropertyMetadata" operation would be needed before a client could set the property.

And I do not see this being at all expensive to implement - it is much easier to implement than storing the property values of each of the property instances for each of the stored data objects, and we already require that.

-- PaulHarrison - 04 Jul 2006

 

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris +1
GuyRixon +1 With details above
MatthewGraham +1

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1  

Re-assess the semantics of creating a server named data object.

see discussion on mailing list for more background http://www.ivoa.net/forum/vospace/0606/0095.htm

Votes

name vote comment

Rename Status (of Node object) to TransferStatus

The Status member of the Node object is really referring to the transfer status.

Votes

name vote comment
PaulHarrison +1 proposer - benefit is increased clarity
GuyRixon +1

Consider interactions when two clients attempt operations on same data object

use case

client 2 attempts to delete a file that client 1 is currently downloading via a "pull" transfer.

As specified at the moment, Client 1 could have a data transfer cut off with out any real knowledge of why, or client 2 could receive a PermissionDenied fault

Possible solutions

  • add a DataInUse fault to the destructive manipulation methods
  • add a dataBeingRead to the list of Status values

This use case is actually quite an implementation challenge....involves close interaction with the states of the actual data streams for the transports.

Votes

name vote comment
PaulHarrison +1 proposer

Expand "expired" Node status to include "failed"

perhaps failed is a better name can then encompass the case where a partial data transfer has occured and failed half way, for some reason other than expiry

Votes

name vote comment
PaulHarrison +1 proposer

Different type of list operation that returns only the uris...

Efficiency...

Proposed (and provisionally agreed by DaveMorris, MatthewGraham and GuyRixon) that we include a "level" parameter to select what is returned. The returned elements are always node elements but:

  • level="minimal" returns nodes with only the URI child;
  • level="standard" returns nodes with the properties as well;
  • level="extended" alls the return of derived types of nodes.

(Naming of the levels can be tweaked if necessary.) "Derived types of nodes" allows implementors to design special nodes (e.g. LdapDataNode) that have all the parts of a normal node and also extra children. The level parameter protects the clients from parser grief when derived node-types are in play.

The URI child might become an attribute of the node element.

-- GuyRixon - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1 With changes listed above

Confusing that pullDataToVoSpace, pushDataFromVoSpace return transfer object.

The transfer object contains a location element which is intended to be the uri that pullDataFromVoSpace and pushDataToVoSpace supply as an output parameter, so it is confusing that pullDataToVoSpace, pushDataFromVoSpace return a uri for a transfer when for those methods the uri for transfer is an input parameter.

Recommend that the pullDataToVoSpace, pushDataFromVoSpace return only what is necessary for the client to know - this might mean removing the transfer object from the return list, or refactoring the transfer object - global analysis needed.

Votes

name vote comment
PaulHarrison +1 proposer

Need to have describe how VOSpace should be registered

The registration of VOSpace servers will be crucial their operation - we need some discussion of this and preferably a VOResouce extension schema to accompany it. PaulHarrison - 27 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1


<--  
-->

Revision 202006-07-04 - PaulHarrison

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • -1 if you disagree
  • 0 if you have no particular preference

Details and discussion of implementation plans are here.



Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Change Requests

Mandate and define at least one transport protocol

without this a compliant VOSpace will not be able to transfer data to another compliant VOSpace.

I recommend http be mandatory.

see detailed discussion in email thread http://www.ivoa.net/forum/vospace/0606/0097.htm

-- PaulHarrison

  • Which version, http-1.0 or http-1.1 ?
  • Which methods http-get, http-put or both ?
    • For a space that stores public images, then http-1.1-get makes sense.
    • For a space that allows upload to sensitive database tables, then http-1.x-put does not have sufficient authentication.

-- DaveMorris - 16 Jun 2006

I think that where VOSpace is acting as a http server then it would be good to mandate http-1.1 compliance - it does fix issues with 1.0 and afterall it is a 7 year old specification now....

As far as a compliance statement goes - how about

(http-1.1-get or https-1.1-get) and (http-1.1-put or https-1.1-put)

-- PaulHarrison - 19 Jun 2006

Making some form of HTTP mandatory makes it easier to write clients and harder to write services. VOSpace fails unless we get useful services, and services are already harder to write than clients.

-- GuyRixon - 03 Jul 2006

Added:
>
>
we are already implicitly mandating http for the "control" interface - i.e. the web service - as far as implementors are concerned adding http-get would be not a great burden for the implementor - without a mandatory transport the success of the use case where a client sets up direct space to space transfers is not guaranteed, so the fallback would have to be copying the data to the client which understands the transport of each space and then transferring to the second space.

-- PaulHarrison - 04 Jul 2006

 

Votes

name vote comment
PaulHarrison +1
DaveMorris -1
GuyRixon -1

Specify as optional a small list of well known transport protocols

so that at least implementations will do the same thing for common protocols - this list to include
  • ftp
  • gridftp
  • file - direct access to the file system

-- PaulHarrison

Agree with the list of common protocols.

Defined in a annex to the main specification, including a standard URI and details of the protocol specification (can be just short note and a reference to the external specification).

e.g.

HTTP 1.1 get
URI
    ivo://....vospace/protocols/http-1.1-get
Description :
    Get data using the HTTP-1.1 GET method as defined in RFC2616.
    http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.3

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Specify the key names and meanings for a small group of "essential" property keys

The minimum set which I would say we need mandatory names are
  • vos.Owner
  • vos.ModificationDate
  • vos.Size
  • vos.MimeType

-- PaulHarrison

Agree with the list of common properties.

Defined in a annex to the main specification, including a standard URI and details of what each property means and how it is represented.

e.g.

Data created date
KEY
    vos.data.created.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were originally created.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]
----
Data modified date
KEY
    vos.data.modified.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were last modified.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]

Note - As VOSpace-1.0 does not support append, the only way the created and modified dates will be different is if the server modifies the data underneath.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Clarify the use of the Format parameter in some calls

I believe that the original intention of Format was to specify a possible transformation of the data (mainly on export of table data from RDBMS based stores). In the current version of the document it reads as if this parameter is merely a description of the data - in which case mime-type suffices. In addition I am not sure if this parameter has any meaning for import operations.

Votes

name vote comment
PaulHarrison +1

Change "exception" to "fault"

The document uses the term exception for the "faults" rather than "fault" -- PaulHarrison

Votes

name vote comment
PaulHarrison +1
DaveMorris -1 I can't see a significant benefit from changing this

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer

Consider adding an optional identifier list to the parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a client is currently interacting with a V2.0 VOSpace that has links to a small subset (e.g. 100 data items) - the client needs to make 100 getNodeProperties SOAP calls to the V1.0 space to get the latest metadata about the data objects - with an optional identifier list it can make one call to ListNodes.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris -1 I can't see a significant benefit from changing this

Reconsider getNodeProperties and setNodeProperties

the naming of these operations makes them appear as a pair, where in fact they are not - you get a Node out of the getNodeProperties and have to put a PropertyPairList into the setNodeProperties - If the change to ListNodes were made as above, then getNodeProperties would be redundant anyway. -- PaulHarrison

Votes

name vote comment
PaulHarrison +1 the benefit is improved clarity - in addition there is the slightly irritating issue (see below) that you cannot take the property list from the "get" and use it in the "set" because of read-only properties
DaveMorris -1 I can't see a significant benefit from changing this

Semantics not clear for setNodeProperties

It is not clear
  • how to delete a property - does a null value of the property pair denote this?
  • does the whole set of node properties given as an argument replace the whole set for the node, or is a union operation performed.
  • interaction with the "read-only" properties in these scenarios...
-- PaulHarrison

Agree, need to make this clearer.

  • Null value deletes property.
  • Operation is union not replace.
  • Server throws PermissionDenined for read-only properties.
    • what if only one of the list of properties is read-only? - need to signal which are the bad properties in the exception - also PermissionDenied could be confusing as this normally refers to the permissions on the data object which could potentially be changed in future versions of VOSpace - certain properties are fundamentally readOnly and so would

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Change names of parameters for moveNode and copyNode

Target is a confusing name for the "source" part of a move operation it sounds more like "destination" - recommend use "source" and "destination" -- PaulHarrison

Yep, source and destination are probably better. As long as we make it clear that these are internal locations, not references to external locations.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1
GuyRixon +1

Rename bulk data transfer operations

  • I stilll find the data transfer operations confusing- I think that the basic problem is that the push and pull verbs are opposite in meaning and viewed from opposite perspectives. I have a new set of proposals

old name new name"
pushDataToVoSpace importDataClientPush
pullDataToVoSpace importDataServerPull
pushDataFromVoSpace exportDataServerPush
pullDataFromVoSpace exportDataClientPull

which I think are better because

  1. the overall objective of the operation is the first verb
  2. the active party in the transfer is identified
  3. the direction of the transfer is then related to the active party.

-- PaulHarrison - 13 Jun 2006

This is a change to the specification not the WSDL, move this to the specification change page.

-- DaveMorris - 16 Jun 2006

I find the newly-proposed names more confusing. It's not obvious to me whether "import" moves data into VOSpace or into the client of VOSpace; similarly with "export".

-- GuyRixon - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1
MatthewGraham 0
DaveMorris -1 Change the specification document first
GuyRixon -1  

New GetPropertyKeys operation

This proposal is basically because I am still a little worried about interoperability problems with the completely untyped nature of the property-key pairs - particularly as they are expected to carry some fundamental metadata about the data objects in the current implementation. This call would return the complete list of key names that have been used in the VOSpace, which would then allow clients to attempt to be consistent in the use of key names - it is not much but at least it does provide a mechanism to voluntarily avoid complete anarchy.

Should return a list of the keys with an indication of which ones relate to read-only properties.

-- PaulHarrison - 13 Jun 2006

I agree with adding the method. However, this should be changed in the specification first, not the WSDL. Move this to the specification change page.

-- DaveMorris - 16 Jun 2006

Suggestion 1: this operation only returns the set of properties understood by the service. Collating the other properties doesn't seem useful and might be expensive to implement.

Suggestion 2: for the properties of interest, return metadata roughly equivalent to those for a CEA parameter:

  • Key (i.e. formal name)
  • Name to show in UI
  • Description
  • Whether writeable by client (probably as boolean attribute "readOnly")
  • UCD 1+
  • Unit

-- GuyRixon - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris +1
GuyRixon +1 With details above
MatthewGraham +1

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1  

Re-assess the semantics of creating a server named data object.

see discussion on mailing list for more background http://www.ivoa.net/forum/vospace/0606/0095.htm

Votes

name vote comment

Rename Status (of Node object) to TransferStatus

The Status member of the Node object is really referring to the transfer status.

Votes

name vote comment
PaulHarrison +1 proposer - benefit is increased clarity
GuyRixon +1

Consider interactions when two clients attempt operations on same data object

use case

client 2 attempts to delete a file that client 1 is currently downloading via a "pull" transfer.

As specified at the moment, Client 1 could have a data transfer cut off with out any real knowledge of why, or client 2 could receive a PermissionDenied fault

Possible solutions

  • add a DataInUse fault to the destructive manipulation methods
  • add a dataBeingRead to the list of Status values

This use case is actually quite an implementation challenge....involves close interaction with the states of the actual data streams for the transports.

Votes

name vote comment
PaulHarrison +1 proposer

Expand "expired" Node status to include "failed"

perhaps failed is a better name can then encompass the case where a partial data transfer has occured and failed half way, for some reason other than expiry

Votes

name vote comment
PaulHarrison +1 proposer

Different type of list operation that returns only the uris...

Efficiency...

Proposed (and provisionally agreed by DaveMorris, MatthewGraham and GuyRixon) that we include a "level" parameter to select what is returned. The returned elements are always node elements but:

  • level="minimal" returns nodes with only the URI child;
  • level="standard" returns nodes with the properties as well;
  • level="extended" alls the return of derived types of nodes.

(Naming of the levels can be tweaked if necessary.) "Derived types of nodes" allows implementors to design special nodes (e.g. LdapDataNode) that have all the parts of a normal node and also extra children. The level parameter protects the clients from parser grief when derived node-types are in play.

The URI child might become an attribute of the node element.

-- GuyRixon - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1 With changes listed above

Confusing that pullDataToVoSpace, pushDataFromVoSpace return transfer object.

The transfer object contains a location element which is intended to be the uri that pullDataFromVoSpace and pushDataToVoSpace supply as an output parameter, so it is confusing that pullDataToVoSpace, pushDataFromVoSpace return a uri for a transfer when for those methods the uri for transfer is an input parameter.

Recommend that the pullDataToVoSpace, pushDataFromVoSpace return only what is necessary for the client to know - this might mean removing the transfer object from the return list, or refactoring the transfer object - global analysis needed.

Votes

name vote comment
PaulHarrison +1 proposer

Need to have describe how VOSpace should be registered

The registration of VOSpace servers will be crucial their operation - we need some discussion of this and preferably a VOResouce extension schema to accompany it. PaulHarrison - 27 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1


<--  
-->

Revision 192006-07-04 - GuyRixon

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • -1 if you disagree
  • 0 if you have no particular preference

Details and discussion of implementation plans are here.



Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Change Requests

Mandate and define at least one transport protocol

without this a compliant VOSpace will not be able to transfer data to another compliant VOSpace.

I recommend http be mandatory.

see detailed discussion in email thread http://www.ivoa.net/forum/vospace/0606/0097.htm

-- PaulHarrison

  • Which version, http-1.0 or http-1.1 ?
  • Which methods http-get, http-put or both ?
    • For a space that stores public images, then http-1.1-get makes sense.
    • For a space that allows upload to sensitive database tables, then http-1.x-put does not have sufficient authentication.

-- DaveMorris - 16 Jun 2006

I think that where VOSpace is acting as a http server then it would be good to mandate http-1.1 compliance - it does fix issues with 1.0 and afterall it is a 7 year old specification now....

As far as a compliance statement goes - how about

(http-1.1-get or https-1.1-get) and (http-1.1-put or https-1.1-put)

-- PaulHarrison - 19 Jun 2006

Making some form of HTTP mandatory makes it easier to write clients and harder to write services. VOSpace fails unless we get useful services, and services are already harder to write than clients.

-- GuyRixon - 03 Jul 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris -1
GuyRixon -1

Specify as optional a small list of well known transport protocols

so that at least implementations will do the same thing for common protocols - this list to include
  • ftp
  • gridftp
  • file - direct access to the file system

-- PaulHarrison

Agree with the list of common protocols.

Defined in a annex to the main specification, including a standard URI and details of the protocol specification (can be just short note and a reference to the external specification).

e.g.

HTTP 1.1 get
URI
    ivo://....vospace/protocols/http-1.1-get
Description :
    Get data using the HTTP-1.1 GET method as defined in RFC2616.
    http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.3

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Specify the key names and meanings for a small group of "essential" property keys

The minimum set which I would say we need mandatory names are
  • vos.Owner
  • vos.ModificationDate
  • vos.Size
  • vos.MimeType

-- PaulHarrison

Agree with the list of common properties.

Defined in a annex to the main specification, including a standard URI and details of what each property means and how it is represented.

e.g.

Data created date
KEY
    vos.data.created.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were originally created.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]
----
Data modified date
KEY
    vos.data.modified.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were last modified.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]

Note - As VOSpace-1.0 does not support append, the only way the created and modified dates will be different is if the server modifies the data underneath.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Clarify the use of the Format parameter in some calls

I believe that the original intention of Format was to specify a possible transformation of the data (mainly on export of table data from RDBMS based stores). In the current version of the document it reads as if this parameter is merely a description of the data - in which case mime-type suffices. In addition I am not sure if this parameter has any meaning for import operations.

Votes

name vote comment
PaulHarrison +1

Change "exception" to "fault"

The document uses the term exception for the "faults" rather than "fault" -- PaulHarrison

Votes

name vote comment
PaulHarrison +1
DaveMorris -1 I can't see a significant benefit from changing this

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer

Consider adding an optional identifier list to the parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a client is currently interacting with a V2.0 VOSpace that has links to a small subset (e.g. 100 data items) - the client needs to make 100 getNodeProperties SOAP calls to the V1.0 space to get the latest metadata about the data objects - with an optional identifier list it can make one call to ListNodes.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris -1 I can't see a significant benefit from changing this

Reconsider getNodeProperties and setNodeProperties

the naming of these operations makes them appear as a pair, where in fact they are not - you get a Node out of the getNodeProperties and have to put a PropertyPairList into the setNodeProperties - If the change to ListNodes were made as above, then getNodeProperties would be redundant anyway. -- PaulHarrison

Votes

name vote comment
PaulHarrison +1 the benefit is improved clarity - in addition there is the slightly irritating issue (see below) that you cannot take the property list from the "get" and use it in the "set" because of read-only properties
DaveMorris -1 I can't see a significant benefit from changing this

Semantics not clear for setNodeProperties

It is not clear
  • how to delete a property - does a null value of the property pair denote this?
  • does the whole set of node properties given as an argument replace the whole set for the node, or is a union operation performed.
  • interaction with the "read-only" properties in these scenarios...
-- PaulHarrison

Agree, need to make this clearer.

  • Null value deletes property.
  • Operation is union not replace.
  • Server throws PermissionDenined for read-only properties.
    • what if only one of the list of properties is read-only? - need to signal which are the bad properties in the exception - also PermissionDenied could be confusing as this normally refers to the permissions on the data object which could potentially be changed in future versions of VOSpace - certain properties are fundamentally readOnly and so would

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Change names of parameters for moveNode and copyNode

Target is a confusing name for the "source" part of a move operation it sounds more like "destination" - recommend use "source" and "destination" -- PaulHarrison

Yep, source and destination are probably better. As long as we make it clear that these are internal locations, not references to external locations.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1
GuyRixon +1

Rename bulk data transfer operations

  • I stilll find the data transfer operations confusing- I think that the basic problem is that the push and pull verbs are opposite in meaning and viewed from opposite perspectives. I have a new set of proposals

old name new name"
pushDataToVoSpace importDataClientPush
pullDataToVoSpace importDataServerPull
pushDataFromVoSpace exportDataServerPush
pullDataFromVoSpace exportDataClientPull

which I think are better because

  1. the overall objective of the operation is the first verb
  2. the active party in the transfer is identified
  3. the direction of the transfer is then related to the active party.

-- PaulHarrison - 13 Jun 2006

This is a change to the specification not the WSDL, move this to the specification change page.

-- DaveMorris - 16 Jun 2006

I find the newly-proposed names more confusing. It's not obvious to me whether "import" moves data into VOSpace or into the client of VOSpace; similarly with "export".

-- GuyRixon - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1
MatthewGraham 0
DaveMorris -1 Change the specification document first
GuyRixon -1  

New GetPropertyKeys operation

This proposal is basically because I am still a little worried about interoperability problems with the completely untyped nature of the property-key pairs - particularly as they are expected to carry some fundamental metadata about the data objects in the current implementation. This call would return the complete list of key names that have been used in the VOSpace, which would then allow clients to attempt to be consistent in the use of key names - it is not much but at least it does provide a mechanism to voluntarily avoid complete anarchy.

Should return a list of the keys with an indication of which ones relate to read-only properties.

-- PaulHarrison - 13 Jun 2006

I agree with adding the method. However, this should be changed in the specification first, not the WSDL. Move this to the specification change page.

-- DaveMorris - 16 Jun 2006

Added:
>
>
Suggestion 1: this operation only returns the set of properties understood by the service. Collating the other properties doesn't seem useful and might be expensive to implement.

Suggestion 2: for the properties of interest, return metadata roughly equivalent to those for a CEA parameter:

  • Key (i.e. formal name)
  • Name to show in UI
  • Description
  • Whether writeable by client (probably as boolean attribute "readOnly")
  • UCD 1+
  • Unit

-- GuyRixon - 04 Jul 2006

 

Votes

name vote comment
PaulHarrison +1 proposer
Changed:
<
<
DaveMorris -1 Change the specification document first
GuyRixon -1 Add to spec document first
>
>
DaveMorris +1
GuyRixon +1 With details above
 
MatthewGraham +1

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer
Added:
>
>
GuyRixon +1  
 

Re-assess the semantics of creating a server named data object.

see discussion on mailing list for more background http://www.ivoa.net/forum/vospace/0606/0095.htm

Votes

name vote comment

Rename Status (of Node object) to TransferStatus

The Status member of the Node object is really referring to the transfer status.

Votes

name vote comment
PaulHarrison +1 proposer - benefit is increased clarity
Added:
>
>
GuyRixon +1
 

Consider interactions when two clients attempt operations on same data object

use case

client 2 attempts to delete a file that client 1 is currently downloading via a "pull" transfer.

As specified at the moment, Client 1 could have a data transfer cut off with out any real knowledge of why, or client 2 could receive a PermissionDenied fault

Possible solutions

  • add a DataInUse fault to the destructive manipulation methods
  • add a dataBeingRead to the list of Status values

This use case is actually quite an implementation challenge....involves close interaction with the states of the actual data streams for the transports.

Votes

name vote comment
PaulHarrison +1 proposer

Expand "expired" Node status to include "failed"

perhaps failed is a better name can then encompass the case where a partial data transfer has occured and failed half way, for some reason other than expiry

Votes

name vote comment
PaulHarrison +1 proposer

Different type of list operation that returns only the uris...

Efficiency...

Proposed (and provisionally agreed by DaveMorris, MatthewGraham and GuyRixon) that we include a "level" parameter to select what is returned. The returned elements are always node elements but:

  • level="minimal" returns nodes with only the URI child;
  • level="standard" returns nodes with the properties as well;
  • level="extended" alls the return of derived types of nodes.

(Naming of the levels can be tweaked if necessary.) "Derived types of nodes" allows implementors to design special nodes (e.g. LdapDataNode) that have all the parts of a normal node and also extra children. The level parameter protects the clients from parser grief when derived node-types are in play.

The URI child might become an attribute of the node element.

-- GuyRixon - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1 proposer
GuyRixon +1 With changes listed above

Confusing that pullDataToVoSpace, pushDataFromVoSpace return transfer object.

The transfer object contains a location element which is intended to be the uri that pullDataFromVoSpace and pushDataToVoSpace supply as an output parameter, so it is confusing that pullDataToVoSpace, pushDataFromVoSpace return a uri for a transfer when for those methods the uri for transfer is an input parameter.

Recommend that the pullDataToVoSpace, pushDataFromVoSpace return only what is necessary for the client to know - this might mean removing the transfer object from the return list, or refactoring the transfer object - global analysis needed.

Votes

name vote comment
PaulHarrison +1 proposer

Need to have describe how VOSpace should be registered

The registration of VOSpace servers will be crucial their operation - we need some discussion of this and preferably a VOResouce extension schema to accompany it. PaulHarrison - 27 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
Added:
>
>
GuyRixon +1
 


<--  
-->

Revision 182006-07-04 - GuyRixon

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • -1 if you disagree
  • 0 if you have no particular preference

Details and discussion of implementation plans are here.



Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Change Requests

Mandate and define at least one transport protocol

without this a compliant VOSpace will not be able to transfer data to another compliant VOSpace.

I recommend http be mandatory.

see detailed discussion in email thread http://www.ivoa.net/forum/vospace/0606/0097.htm

-- PaulHarrison

  • Which version, http-1.0 or http-1.1 ?
  • Which methods http-get, http-put or both ?
    • For a space that stores public images, then http-1.1-get makes sense.
    • For a space that allows upload to sensitive database tables, then http-1.x-put does not have sufficient authentication.

-- DaveMorris - 16 Jun 2006

I think that where VOSpace is acting as a http server then it would be good to mandate http-1.1 compliance - it does fix issues with 1.0 and afterall it is a 7 year old specification now....

As far as a compliance statement goes - how about

(http-1.1-get or https-1.1-get) and (http-1.1-put or https-1.1-put)

-- PaulHarrison - 19 Jun 2006

Making some form of HTTP mandatory makes it easier to write clients and harder to write services. VOSpace fails unless we get useful services, and services are already harder to write than clients.

-- GuyRixon - 03 Jul 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris -1
GuyRixon -1

Specify as optional a small list of well known transport protocols

so that at least implementations will do the same thing for common protocols - this list to include
  • ftp
  • gridftp
  • file - direct access to the file system

-- PaulHarrison

Agree with the list of common protocols.

Defined in a annex to the main specification, including a standard URI and details of the protocol specification (can be just short note and a reference to the external specification).

e.g.

HTTP 1.1 get
URI
    ivo://....vospace/protocols/http-1.1-get
Description :
    Get data using the HTTP-1.1 GET method as defined in RFC2616.
    http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.3

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Specify the key names and meanings for a small group of "essential" property keys

The minimum set which I would say we need mandatory names are
  • vos.Owner
  • vos.ModificationDate
  • vos.Size
  • vos.MimeType

-- PaulHarrison

Agree with the list of common properties.

Defined in a annex to the main specification, including a standard URI and details of what each property means and how it is represented.

e.g.

Data created date
KEY
    vos.data.created.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were originally created.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]
----
Data modified date
KEY
    vos.data.modified.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were last modified.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]

Note - As VOSpace-1.0 does not support append, the only way the created and modified dates will be different is if the server modifies the data underneath.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Clarify the use of the Format parameter in some calls

I believe that the original intention of Format was to specify a possible transformation of the data (mainly on export of table data from RDBMS based stores). In the current version of the document it reads as if this parameter is merely a description of the data - in which case mime-type suffices. In addition I am not sure if this parameter has any meaning for import operations.

Votes

name vote comment
PaulHarrison +1

Change "exception" to "fault"

The document uses the term exception for the "faults" rather than "fault" -- PaulHarrison

Votes

name vote comment
PaulHarrison +1
DaveMorris -1 I can't see a significant benefit from changing this

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer

Consider adding an optional identifier list to the parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a client is currently interacting with a V2.0 VOSpace that has links to a small subset (e.g. 100 data items) - the client needs to make 100 getNodeProperties SOAP calls to the V1.0 space to get the latest metadata about the data objects - with an optional identifier list it can make one call to ListNodes.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris -1 I can't see a significant benefit from changing this

Reconsider getNodeProperties and setNodeProperties

the naming of these operations makes them appear as a pair, where in fact they are not - you get a Node out of the getNodeProperties and have to put a PropertyPairList into the setNodeProperties - If the change to ListNodes were made as above, then getNodeProperties would be redundant anyway. -- PaulHarrison

Votes

name vote comment
PaulHarrison +1 the benefit is improved clarity - in addition there is the slightly irritating issue (see below) that you cannot take the property list from the "get" and use it in the "set" because of read-only properties
DaveMorris -1 I can't see a significant benefit from changing this

Semantics not clear for setNodeProperties

It is not clear
  • how to delete a property - does a null value of the property pair denote this?
  • does the whole set of node properties given as an argument replace the whole set for the node, or is a union operation performed.
  • interaction with the "read-only" properties in these scenarios...
-- PaulHarrison

Agree, need to make this clearer.

  • Null value deletes property.
  • Operation is union not replace.
  • Server throws PermissionDenined for read-only properties.
    • what if only one of the list of properties is read-only? - need to signal which are the bad properties in the exception - also PermissionDenied could be confusing as this normally refers to the permissions on the data object which could potentially be changed in future versions of VOSpace - certain properties are fundamentally readOnly and so would

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Change names of parameters for moveNode and copyNode

Target is a confusing name for the "source" part of a move operation it sounds more like "destination" - recommend use "source" and "destination" -- PaulHarrison

Yep, source and destination are probably better. As long as we make it clear that these are internal locations, not references to external locations.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1
GuyRixon +1

Rename bulk data transfer operations

  • I stilll find the data transfer operations confusing- I think that the basic problem is that the push and pull verbs are opposite in meaning and viewed from opposite perspectives. I have a new set of proposals

old name new name"
pushDataToVoSpace importDataClientPush
pullDataToVoSpace importDataServerPull
pushDataFromVoSpace exportDataServerPush
pullDataFromVoSpace exportDataClientPull

which I think are better because

  1. the overall objective of the operation is the first verb
  2. the active party in the transfer is identified
  3. the direction of the transfer is then related to the active party.

-- PaulHarrison - 13 Jun 2006

This is a change to the specification not the WSDL, move this to the specification change page.

-- DaveMorris - 16 Jun 2006

I find the newly-proposed names more confusing. It's not obvious to me whether "import" moves data into VOSpace or into the client of VOSpace; similarly with "export".

-- GuyRixon - 04 Jul 2006

Votes

name vote comment
PaulHarrison +1
MatthewGraham 0
DaveMorris -1 Change the specification document first
GuyRixon -1  

New GetPropertyKeys operation

This proposal is basically because I am still a little worried about interoperability problems with the completely untyped nature of the property-key pairs - particularly as they are expected to carry some fundamental metadata about the data objects in the current implementation. This call would return the complete list of key names that have been used in the VOSpace, which would then allow clients to attempt to be consistent in the use of key names - it is not much but at least it does provide a mechanism to voluntarily avoid complete anarchy.

Should return a list of the keys with an indication of which ones relate to read-only properties.

-- PaulHarrison - 13 Jun 2006

I agree with adding the method. However, this should be changed in the specification first, not the WSDL. Move this to the specification change page.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris -1 Change the specification document first
GuyRixon -1 Add to spec document first
MatthewGraham +1

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer

Re-assess the semantics of creating a server named data object.

see discussion on mailing list for more background http://www.ivoa.net/forum/vospace/0606/0095.htm

Votes

name vote comment

Rename Status (of Node object) to TransferStatus

The Status member of the Node object is really referring to the transfer status.

Votes

name vote comment
PaulHarrison +1 proposer - benefit is increased clarity

Consider interactions when two clients attempt operations on same data object

use case

client 2 attempts to delete a file that client 1 is currently downloading via a "pull" transfer.

As specified at the moment, Client 1 could have a data transfer cut off with out any real knowledge of why, or client 2 could receive a PermissionDenied fault

Possible solutions

  • add a DataInUse fault to the destructive manipulation methods
  • add a dataBeingRead to the list of Status values

This use case is actually quite an implementation challenge....involves close interaction with the states of the actual data streams for the transports.

Votes

name vote comment
PaulHarrison +1 proposer

Expand "expired" Node status to include "failed"

perhaps failed is a better name can then encompass the case where a partial data transfer has occured and failed half way, for some reason other than expiry

Votes

name vote comment
PaulHarrison +1 proposer

Different type of list operation that returns only the uris...

Efficiency...
Added:
>
>
Proposed (and provisionally agreed by DaveMorris, MatthewGraham and GuyRixon) that we include a "level" parameter to select what is returned. The returned elements are always node elements but:

  • level="minimal" returns nodes with only the URI child;
  • level="standard" returns nodes with the properties as well;
  • level="extended" alls the return of derived types of nodes.

(Naming of the levels can be tweaked if necessary.) "Derived types of nodes" allows implementors to design special nodes (e.g. LdapDataNode) that have all the parts of a normal node and also extra children. The level parameter protects the clients from parser grief when derived node-types are in play.

The URI child might become an attribute of the node element.

-- GuyRixon - 04 Jul 2006

 

Votes

name vote comment
PaulHarrison +1 proposer
Changed:
<
<
>
>
GuyRixon +1 With changes listed above
 

Confusing that pullDataToVoSpace, pushDataFromVoSpace return transfer object.

The transfer object contains a location element which is intended to be the uri that pullDataFromVoSpace and pushDataToVoSpace supply as an output parameter, so it is confusing that pullDataToVoSpace, pushDataFromVoSpace return a uri for a transfer when for those methods the uri for transfer is an input parameter.

Recommend that the pullDataToVoSpace, pushDataFromVoSpace return only what is necessary for the client to know - this might mean removing the transfer object from the return list, or refactoring the transfer object - global analysis needed.

Votes

name vote comment
PaulHarrison +1 proposer

Need to have describe how VOSpace should be registered

The registration of VOSpace servers will be crucial their operation - we need some discussion of this and preferably a VOResouce extension schema to accompany it. PaulHarrison - 27 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer


<--  
-->

Revision 172006-07-04 - GuyRixon

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • -1 if you disagree
  • 0 if you have no particular preference

Details and discussion of implementation plans are here.



Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Change Requests

Mandate and define at least one transport protocol

without this a compliant VOSpace will not be able to transfer data to another compliant VOSpace.

I recommend http be mandatory.

see detailed discussion in email thread http://www.ivoa.net/forum/vospace/0606/0097.htm

-- PaulHarrison

  • Which version, http-1.0 or http-1.1 ?
  • Which methods http-get, http-put or both ?
    • For a space that stores public images, then http-1.1-get makes sense.
    • For a space that allows upload to sensitive database tables, then http-1.x-put does not have sufficient authentication.

-- DaveMorris - 16 Jun 2006

I think that where VOSpace is acting as a http server then it would be good to mandate http-1.1 compliance - it does fix issues with 1.0 and afterall it is a 7 year old specification now....

As far as a compliance statement goes - how about

(http-1.1-get or https-1.1-get) and (http-1.1-put or https-1.1-put)

-- PaulHarrison - 19 Jun 2006

Making some form of HTTP mandatory makes it easier to write clients and harder to write services. VOSpace fails unless we get useful services, and services are already harder to write than clients.

-- GuyRixon - 03 Jul 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris -1
GuyRixon -1

Specify as optional a small list of well known transport protocols

so that at least implementations will do the same thing for common protocols - this list to include
  • ftp
  • gridftp
  • file - direct access to the file system

-- PaulHarrison

Agree with the list of common protocols.

Defined in a annex to the main specification, including a standard URI and details of the protocol specification (can be just short note and a reference to the external specification).

e.g.

HTTP 1.1 get
URI
    ivo://....vospace/protocols/http-1.1-get
Description :
    Get data using the HTTP-1.1 GET method as defined in RFC2616.
    http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.3

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Specify the key names and meanings for a small group of "essential" property keys

The minimum set which I would say we need mandatory names are
  • vos.Owner
  • vos.ModificationDate
  • vos.Size
  • vos.MimeType

-- PaulHarrison

Agree with the list of common properties.

Defined in a annex to the main specification, including a standard URI and details of what each property means and how it is represented.

e.g.

Data created date
KEY
    vos.data.created.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were originally created.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]
----
Data modified date
KEY
    vos.data.modified.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were last modified.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]

Note - As VOSpace-1.0 does not support append, the only way the created and modified dates will be different is if the server modifies the data underneath.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Clarify the use of the Format parameter in some calls

I believe that the original intention of Format was to specify a possible transformation of the data (mainly on export of table data from RDBMS based stores). In the current version of the document it reads as if this parameter is merely a description of the data - in which case mime-type suffices. In addition I am not sure if this parameter has any meaning for import operations.

Votes

name vote comment
PaulHarrison +1

Change "exception" to "fault"

The document uses the term exception for the "faults" rather than "fault" -- PaulHarrison

Votes

name vote comment
PaulHarrison +1
DaveMorris -1 I can't see a significant benefit from changing this

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer

Consider adding an optional identifier list to the parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a client is currently interacting with a V2.0 VOSpace that has links to a small subset (e.g. 100 data items) - the client needs to make 100 getNodeProperties SOAP calls to the V1.0 space to get the latest metadata about the data objects - with an optional identifier list it can make one call to ListNodes.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris -1 I can't see a significant benefit from changing this

Reconsider getNodeProperties and setNodeProperties

the naming of these operations makes them appear as a pair, where in fact they are not - you get a Node out of the getNodeProperties and have to put a PropertyPairList into the setNodeProperties - If the change to ListNodes were made as above, then getNodeProperties would be redundant anyway. -- PaulHarrison

Votes

name vote comment
PaulHarrison +1 the benefit is improved clarity - in addition there is the slightly irritating issue (see below) that you cannot take the property list from the "get" and use it in the "set" because of read-only properties
DaveMorris -1 I can't see a significant benefit from changing this

Semantics not clear for setNodeProperties

It is not clear
  • how to delete a property - does a null value of the property pair denote this?
  • does the whole set of node properties given as an argument replace the whole set for the node, or is a union operation performed.
  • interaction with the "read-only" properties in these scenarios...
-- PaulHarrison

Agree, need to make this clearer.

  • Null value deletes property.
  • Operation is union not replace.
  • Server throws PermissionDenined for read-only properties.
    • what if only one of the list of properties is read-only? - need to signal which are the bad properties in the exception - also PermissionDenied could be confusing as this normally refers to the permissions on the data object which could potentially be changed in future versions of VOSpace - certain properties are fundamentally readOnly and so would

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Change names of parameters for moveNode and copyNode

Target is a confusing name for the "source" part of a move operation it sounds more like "destination" - recommend use "source" and "destination" -- PaulHarrison

Yep, source and destination are probably better. As long as we make it clear that these are internal locations, not references to external locations.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1
Added:
>
>
GuyRixon +1
 

Rename bulk data transfer operations

  • I stilll find the data transfer operations confusing- I think that the basic problem is that the push and pull verbs are opposite in meaning and viewed from opposite perspectives. I have a new set of proposals

old name new name"
pushDataToVoSpace importDataClientPush
pullDataToVoSpace importDataServerPull
pushDataFromVoSpace exportDataServerPush
pullDataFromVoSpace exportDataClientPull

which I think are better because

  1. the overall objective of the operation is the first verb
  2. the active party in the transfer is identified
  3. the direction of the transfer is then related to the active party.

-- PaulHarrison - 13 Jun 2006

This is a change to the specification not the WSDL, move this to the specification change page.

-- DaveMorris - 16 Jun 2006

Added:
>
>
I find the newly-proposed names more confusing. It's not obvious to me whether "import" moves data into VOSpace or into the client of VOSpace; similarly with "export".

-- GuyRixon - 04 Jul 2006

 

Votes

name vote comment
PaulHarrison +1
MatthewGraham 0
DaveMorris -1 Change the specification document first
GuyRixon -1  

New GetPropertyKeys operation

This proposal is basically because I am still a little worried about interoperability problems with the completely untyped nature of the property-key pairs - particularly as they are expected to carry some fundamental metadata about the data objects in the current implementation. This call would return the complete list of key names that have been used in the VOSpace, which would then allow clients to attempt to be consistent in the use of key names - it is not much but at least it does provide a mechanism to voluntarily avoid complete anarchy.

Should return a list of the keys with an indication of which ones relate to read-only properties.

-- PaulHarrison - 13 Jun 2006

I agree with adding the method. However, this should be changed in the specification first, not the WSDL. Move this to the specification change page.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris -1 Change the specification document first
GuyRixon -1 Add to spec document first
MatthewGraham +1

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer

Re-assess the semantics of creating a server named data object.

see discussion on mailing list for more background http://www.ivoa.net/forum/vospace/0606/0095.htm

Votes

name vote comment

Rename Status (of Node object) to TransferStatus

The Status member of the Node object is really referring to the transfer status.

Votes

name vote comment
PaulHarrison +1 proposer - benefit is increased clarity

Consider interactions when two clients attempt operations on same data object

use case

client 2 attempts to delete a file that client 1 is currently downloading via a "pull" transfer.

As specified at the moment, Client 1 could have a data transfer cut off with out any real knowledge of why, or client 2 could receive a PermissionDenied fault

Possible solutions

  • add a DataInUse fault to the destructive manipulation methods
  • add a dataBeingRead to the list of Status values

This use case is actually quite an implementation challenge....involves close interaction with the states of the actual data streams for the transports.

Votes

name vote comment
PaulHarrison +1 proposer

Expand "expired" Node status to include "failed"

perhaps failed is a better name can then encompass the case where a partial data transfer has occured and failed half way, for some reason other than expiry

Votes

name vote comment
PaulHarrison +1 proposer

Different type of list operation that returns only the uris...

Efficiency...

Votes

name vote comment
PaulHarrison +1 proposer

Confusing that pullDataToVoSpace, pushDataFromVoSpace return transfer object.

The transfer object contains a location element which is intended to be the uri that pullDataFromVoSpace and pushDataToVoSpace supply as an output parameter, so it is confusing that pullDataToVoSpace, pushDataFromVoSpace return a uri for a transfer when for those methods the uri for transfer is an input parameter.

Recommend that the pullDataToVoSpace, pushDataFromVoSpace return only what is necessary for the client to know - this might mean removing the transfer object from the return list, or refactoring the transfer object - global analysis needed.

Votes

name vote comment
PaulHarrison +1 proposer

Need to have describe how VOSpace should be registered

The registration of VOSpace servers will be crucial their operation - we need some discussion of this and preferably a VOResouce extension schema to accompany it. PaulHarrison - 27 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer


<--  
-->

Revision 162006-07-03 - MatthewGraham

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • -1 if you disagree
  • 0 if you have no particular preference
Added:
>
>
Details and discussion of implementation plans are here.
 

Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Change Requests

Mandate and define at least one transport protocol

without this a compliant VOSpace will not be able to transfer data to another compliant VOSpace.

I recommend http be mandatory.

see detailed discussion in email thread http://www.ivoa.net/forum/vospace/0606/0097.htm

-- PaulHarrison

  • Which version, http-1.0 or http-1.1 ?
  • Which methods http-get, http-put or both ?
    • For a space that stores public images, then http-1.1-get makes sense.
    • For a space that allows upload to sensitive database tables, then http-1.x-put does not have sufficient authentication.

-- DaveMorris - 16 Jun 2006

I think that where VOSpace is acting as a http server then it would be good to mandate http-1.1 compliance - it does fix issues with 1.0 and afterall it is a 7 year old specification now....

As far as a compliance statement goes - how about

(http-1.1-get or https-1.1-get) and (http-1.1-put or https-1.1-put)

-- PaulHarrison - 19 Jun 2006

Making some form of HTTP mandatory makes it easier to write clients and harder to write services. VOSpace fails unless we get useful services, and services are already harder to write than clients.

-- GuyRixon - 03 Jul 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris -1
GuyRixon -1

Specify as optional a small list of well known transport protocols

so that at least implementations will do the same thing for common protocols - this list to include
  • ftp
  • gridftp
  • file - direct access to the file system

-- PaulHarrison

Agree with the list of common protocols.

Defined in a annex to the main specification, including a standard URI and details of the protocol specification (can be just short note and a reference to the external specification).

e.g.

HTTP 1.1 get
URI
    ivo://....vospace/protocols/http-1.1-get
Description :
    Get data using the HTTP-1.1 GET method as defined in RFC2616.
    http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.3

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Specify the key names and meanings for a small group of "essential" property keys

The minimum set which I would say we need mandatory names are
  • vos.Owner
  • vos.ModificationDate
  • vos.Size
  • vos.MimeType

-- PaulHarrison

Agree with the list of common properties.

Defined in a annex to the main specification, including a standard URI and details of what each property means and how it is represented.

e.g.

Data created date
KEY
    vos.data.created.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were originally created.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]
----
Data modified date
KEY
    vos.data.modified.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were last modified.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]

Note - As VOSpace-1.0 does not support append, the only way the created and modified dates will be different is if the server modifies the data underneath.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Clarify the use of the Format parameter in some calls

I believe that the original intention of Format was to specify a possible transformation of the data (mainly on export of table data from RDBMS based stores). In the current version of the document it reads as if this parameter is merely a description of the data - in which case mime-type suffices. In addition I am not sure if this parameter has any meaning for import operations.

Votes

name vote comment
PaulHarrison +1

Change "exception" to "fault"

The document uses the term exception for the "faults" rather than "fault" -- PaulHarrison

Votes

name vote comment
PaulHarrison +1
DaveMorris -1 I can't see a significant benefit from changing this

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer

Consider adding an optional identifier list to the parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a client is currently interacting with a V2.0 VOSpace that has links to a small subset (e.g. 100 data items) - the client needs to make 100 getNodeProperties SOAP calls to the V1.0 space to get the latest metadata about the data objects - with an optional identifier list it can make one call to ListNodes.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris -1 I can't see a significant benefit from changing this

Reconsider getNodeProperties and setNodeProperties

the naming of these operations makes them appear as a pair, where in fact they are not - you get a Node out of the getNodeProperties and have to put a PropertyPairList into the setNodeProperties - If the change to ListNodes were made as above, then getNodeProperties would be redundant anyway. -- PaulHarrison

Votes

name vote comment
PaulHarrison +1 the benefit is improved clarity - in addition there is the slightly irritating issue (see below) that you cannot take the property list from the "get" and use it in the "set" because of read-only properties
DaveMorris -1 I can't see a significant benefit from changing this

Semantics not clear for setNodeProperties

It is not clear
  • how to delete a property - does a null value of the property pair denote this?
  • does the whole set of node properties given as an argument replace the whole set for the node, or is a union operation performed.
  • interaction with the "read-only" properties in these scenarios...
-- PaulHarrison

Agree, need to make this clearer.

  • Null value deletes property.
  • Operation is union not replace.
  • Server throws PermissionDenined for read-only properties.
    • what if only one of the list of properties is read-only? - need to signal which are the bad properties in the exception - also PermissionDenied could be confusing as this normally refers to the permissions on the data object which could potentially be changed in future versions of VOSpace - certain properties are fundamentally readOnly and so would

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Change names of parameters for moveNode and copyNode

Target is a confusing name for the "source" part of a move operation it sounds more like "destination" - recommend use "source" and "destination" -- PaulHarrison

Yep, source and destination are probably better. As long as we make it clear that these are internal locations, not references to external locations.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Rename bulk data transfer operations

  • I stilll find the data transfer operations confusing- I think that the basic problem is that the push and pull verbs are opposite in meaning and viewed from opposite perspectives. I have a new set of proposals

old name new name"
pushDataToVoSpace importDataClientPush
pullDataToVoSpace importDataServerPull
pushDataFromVoSpace exportDataServerPush
pullDataFromVoSpace exportDataClientPull

which I think are better because

  1. the overall objective of the operation is the first verb
  2. the active party in the transfer is identified
  3. the direction of the transfer is then related to the active party.

-- PaulHarrison - 13 Jun 2006

This is a change to the specification not the WSDL, move this to the specification change page.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
MatthewGraham 0
DaveMorris -1 Change the specification document first
GuyRixon -1  

New GetPropertyKeys operation

This proposal is basically because I am still a little worried about interoperability problems with the completely untyped nature of the property-key pairs - particularly as they are expected to carry some fundamental metadata about the data objects in the current implementation. This call would return the complete list of key names that have been used in the VOSpace, which would then allow clients to attempt to be consistent in the use of key names - it is not much but at least it does provide a mechanism to voluntarily avoid complete anarchy.

Should return a list of the keys with an indication of which ones relate to read-only properties.

-- PaulHarrison - 13 Jun 2006

I agree with adding the method. However, this should be changed in the specification first, not the WSDL. Move this to the specification change page.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris -1 Change the specification document first
GuyRixon -1 Add to spec document first
MatthewGraham +1

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer

Re-assess the semantics of creating a server named data object.

see discussion on mailing list for more background http://www.ivoa.net/forum/vospace/0606/0095.htm

Votes

name vote comment

Rename Status (of Node object) to TransferStatus

The Status member of the Node object is really referring to the transfer status.

Votes

name vote comment
PaulHarrison +1 proposer - benefit is increased clarity

Consider interactions when two clients attempt operations on same data object

use case

client 2 attempts to delete a file that client 1 is currently downloading via a "pull" transfer.

As specified at the moment, Client 1 could have a data transfer cut off with out any real knowledge of why, or client 2 could receive a PermissionDenied fault

Possible solutions

  • add a DataInUse fault to the destructive manipulation methods
  • add a dataBeingRead to the list of Status values

This use case is actually quite an implementation challenge....involves close interaction with the states of the actual data streams for the transports.

Votes

name vote comment
PaulHarrison +1 proposer

Expand "expired" Node status to include "failed"

perhaps failed is a better name can then encompass the case where a partial data transfer has occured and failed half way, for some reason other than expiry

Votes

name vote comment
PaulHarrison +1 proposer

Different type of list operation that returns only the uris...

Efficiency...

Votes

name vote comment
PaulHarrison +1 proposer

Confusing that pullDataToVoSpace, pushDataFromVoSpace return transfer object.

The transfer object contains a location element which is intended to be the uri that pullDataFromVoSpace and pushDataToVoSpace supply as an output parameter, so it is confusing that pullDataToVoSpace, pushDataFromVoSpace return a uri for a transfer when for those methods the uri for transfer is an input parameter.

Recommend that the pullDataToVoSpace, pushDataFromVoSpace return only what is necessary for the client to know - this might mean removing the transfer object from the return list, or refactoring the transfer object - global analysis needed.

Votes

name vote comment
PaulHarrison +1 proposer

Need to have describe how VOSpace should be registered

The registration of VOSpace servers will be crucial their operation - we need some discussion of this and preferably a VOResouce extension schema to accompany it. PaulHarrison - 27 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer


<--  
-->

Revision 152006-07-03 - GuyRixon

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • -1 if you disagree
  • 0 if you have no particular preference



Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Change Requests

Mandate and define at least one transport protocol

without this a compliant VOSpace will not be able to transfer data to another compliant VOSpace.

I recommend http be mandatory.

see detailed discussion in email thread http://www.ivoa.net/forum/vospace/0606/0097.htm

-- PaulHarrison

  • Which version, http-1.0 or http-1.1 ?
  • Which methods http-get, http-put or both ?
    • For a space that stores public images, then http-1.1-get makes sense.
    • For a space that allows upload to sensitive database tables, then http-1.x-put does not have sufficient authentication.

-- DaveMorris - 16 Jun 2006

I think that where VOSpace is acting as a http server then it would be good to mandate http-1.1 compliance - it does fix issues with 1.0 and afterall it is a 7 year old specification now....

As far as a compliance statement goes - how about

(http-1.1-get or https-1.1-get) and (http-1.1-put or https-1.1-put)

Changed:
<
<
-- PaulHarrison - 19 Jun 2006
>
>
-- PaulHarrison - 19 Jun 2006
Added:
>
>
Making some form of HTTP mandatory makes it easier to write clients and harder to write services. VOSpace fails unless we get useful services, and services are already harder to write than clients.

-- GuyRixon - 03 Jul 2006

 

Votes

name vote comment
PaulHarrison +1
DaveMorris -1
Added:
>
>
GuyRixon -1
 

Specify as optional a small list of well known transport protocols

so that at least implementations will do the same thing for common protocols - this list to include
  • ftp
  • gridftp
  • file - direct access to the file system

-- PaulHarrison

Agree with the list of common protocols.

Defined in a annex to the main specification, including a standard URI and details of the protocol specification (can be just short note and a reference to the external specification).

e.g.

HTTP 1.1 get
URI
    ivo://....vospace/protocols/http-1.1-get
Description :
    Get data using the HTTP-1.1 GET method as defined in RFC2616.
    http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.3

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Specify the key names and meanings for a small group of "essential" property keys

The minimum set which I would say we need mandatory names are
  • vos.Owner
  • vos.ModificationDate
  • vos.Size
  • vos.MimeType

-- PaulHarrison

Agree with the list of common properties.

Defined in a annex to the main specification, including a standard URI and details of what each property means and how it is represented.

e.g.

Data created date
KEY
    vos.data.created.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were originally created.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]
----
Data modified date
KEY
    vos.data.modified.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were last modified.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]

Note - As VOSpace-1.0 does not support append, the only way the created and modified dates will be different is if the server modifies the data underneath.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Clarify the use of the Format parameter in some calls

I believe that the original intention of Format was to specify a possible transformation of the data (mainly on export of table data from RDBMS based stores). In the current version of the document it reads as if this parameter is merely a description of the data - in which case mime-type suffices. In addition I am not sure if this parameter has any meaning for import operations.

Votes

name vote comment
PaulHarrison +1

Change "exception" to "fault"

The document uses the term exception for the "faults" rather than "fault" -- PaulHarrison

Votes

name vote comment
PaulHarrison +1
DaveMorris -1 I can't see a significant benefit from changing this

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer

Consider adding an optional identifier list to the parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a client is currently interacting with a V2.0 VOSpace that has links to a small subset (e.g. 100 data items) - the client needs to make 100 getNodeProperties SOAP calls to the V1.0 space to get the latest metadata about the data objects - with an optional identifier list it can make one call to ListNodes.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris -1 I can't see a significant benefit from changing this

Reconsider getNodeProperties and setNodeProperties

the naming of these operations makes them appear as a pair, where in fact they are not - you get a Node out of the getNodeProperties and have to put a PropertyPairList into the setNodeProperties - If the change to ListNodes were made as above, then getNodeProperties would be redundant anyway. -- PaulHarrison

Votes

name vote comment
PaulHarrison +1 the benefit is improved clarity - in addition there is the slightly irritating issue (see below) that you cannot take the property list from the "get" and use it in the "set" because of read-only properties
DaveMorris -1 I can't see a significant benefit from changing this

Semantics not clear for setNodeProperties

It is not clear
  • how to delete a property - does a null value of the property pair denote this?
  • does the whole set of node properties given as an argument replace the whole set for the node, or is a union operation performed.
  • interaction with the "read-only" properties in these scenarios...
-- PaulHarrison

Agree, need to make this clearer.

  • Null value deletes property.
  • Operation is union not replace.
  • Server throws PermissionDenined for read-only properties.
    • what if only one of the list of properties is read-only? - need to signal which are the bad properties in the exception - also PermissionDenied could be confusing as this normally refers to the permissions on the data object which could potentially be changed in future versions of VOSpace - certain properties are fundamentally readOnly and so would

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Change names of parameters for moveNode and copyNode

Target is a confusing name for the "source" part of a move operation it sounds more like "destination" - recommend use "source" and "destination" -- PaulHarrison

Yep, source and destination are probably better. As long as we make it clear that these are internal locations, not references to external locations.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Rename bulk data transfer operations

  • I stilll find the data transfer operations confusing- I think that the basic problem is that the push and pull verbs are opposite in meaning and viewed from opposite perspectives. I have a new set of proposals

old name new name"
pushDataToVoSpace importDataClientPush
pullDataToVoSpace importDataServerPull
pushDataFromVoSpace exportDataServerPush
pullDataFromVoSpace exportDataClientPull

which I think are better because

  1. the overall objective of the operation is the first verb
  2. the active party in the transfer is identified
  3. the direction of the transfer is then related to the active party.

-- PaulHarrison - 13 Jun 2006

This is a change to the specification not the WSDL, move this to the specification change page.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
MatthewGraham 0
DaveMorris -1 Change the specification document first
GuyRixon -1  

New GetPropertyKeys operation

This proposal is basically because I am still a little worried about interoperability problems with the completely untyped nature of the property-key pairs - particularly as they are expected to carry some fundamental metadata about the data objects in the current implementation. This call would return the complete list of key names that have been used in the VOSpace, which would then allow clients to attempt to be consistent in the use of key names - it is not much but at least it does provide a mechanism to voluntarily avoid complete anarchy.

Should return a list of the keys with an indication of which ones relate to read-only properties.

-- PaulHarrison - 13 Jun 2006

I agree with adding the method. However, this should be changed in the specification first, not the WSDL. Move this to the specification change page.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris -1 Change the specification document first
GuyRixon -1 Add to spec document first
MatthewGraham +1

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer

Re-assess the semantics of creating a server named data object.

see discussion on mailing list for more background http://www.ivoa.net/forum/vospace/0606/0095.htm

Votes

name vote comment

Rename Status (of Node object) to TransferStatus

The Status member of the Node object is really referring to the transfer status.

Votes

name vote comment
PaulHarrison +1 proposer - benefit is increased clarity

Consider interactions when two clients attempt operations on same data object

use case

client 2 attempts to delete a file that client 1 is currently downloading via a "pull" transfer.

As specified at the moment, Client 1 could have a data transfer cut off with out any real knowledge of why, or client 2 could receive a PermissionDenied fault

Possible solutions

  • add a DataInUse fault to the destructive manipulation methods
  • add a dataBeingRead to the list of Status values

This use case is actually quite an implementation challenge....involves close interaction with the states of the actual data streams for the transports.

Votes

name vote comment
PaulHarrison +1 proposer

Expand "expired" Node status to include "failed"

perhaps failed is a better name can then encompass the case where a partial data transfer has occured and failed half way, for some reason other than expiry

Votes

name vote comment
PaulHarrison +1 proposer

Different type of list operation that returns only the uris...

Efficiency...

Votes

name vote comment
PaulHarrison +1 proposer

Confusing that pullDataToVoSpace, pushDataFromVoSpace return transfer object.

The transfer object contains a location element which is intended to be the uri that pullDataFromVoSpace and pushDataToVoSpace supply as an output parameter, so it is confusing that pullDataToVoSpace, pushDataFromVoSpace return a uri for a transfer when for those methods the uri for transfer is an input parameter.

Recommend that the pullDataToVoSpace, pushDataFromVoSpace return only what is necessary for the client to know - this might mean removing the transfer object from the return list, or refactoring the transfer object - global analysis needed.

Votes

name vote comment
PaulHarrison +1 proposer

Need to have describe how VOSpace should be registered

The registration of VOSpace servers will be crucial their operation - we need some discussion of this and preferably a VOResouce extension schema to accompany it. PaulHarrison - 27 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer


<--  
-->

Revision 142006-06-27 - PaulHarrison

 
META TOPICPARENT name="IvoaGridAndWebServices"

Discussion of the VOSpace 1.0 specification Document

This is a discussion page for the VOSpace-1.0 service specification document.

This is somewhere where we can post proposals and to enable interested parties to discuss the different versions.

For each version there is a Change request section - please add to this and vote on other suggestions

  • +1 if you agree
  • -1 if you disagree
  • 0 if you have no particular preference



Version 0.21

This document was produced as a result of the discussions that occured at the Victoria Interop meeting.

Change Requests

Mandate and define at least one transport protocol

without this a compliant VOSpace will not be able to transfer data to another compliant VOSpace.

I recommend http be mandatory.

see detailed discussion in email thread http://www.ivoa.net/forum/vospace/0606/0097.htm

-- PaulHarrison

  • Which version, http-1.0 or http-1.1 ?
  • Which methods http-get, http-put or both ?
    • For a space that stores public images, then http-1.1-get makes sense.
    • For a space that allows upload to sensitive database tables, then http-1.x-put does not have sufficient authentication.

-- DaveMorris - 16 Jun 2006

I think that where VOSpace is acting as a http server then it would be good to mandate http-1.1 compliance - it does fix issues with 1.0 and afterall it is a 7 year old specification now....

As far as a compliance statement goes - how about

(http-1.1-get or https-1.1-get) and (http-1.1-put or https-1.1-put)

-- PaulHarrison - 19 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris -1

Specify as optional a small list of well known transport protocols

so that at least implementations will do the same thing for common protocols - this list to include
  • ftp
  • gridftp
  • file - direct access to the file system

-- PaulHarrison

Agree with the list of common protocols.

Defined in a annex to the main specification, including a standard URI and details of the protocol specification (can be just short note and a reference to the external specification).

e.g.

HTTP 1.1 get
URI
    ivo://....vospace/protocols/http-1.1-get
Description :
    Get data using the HTTP-1.1 GET method as defined in RFC2616.
    http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.3

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Specify the key names and meanings for a small group of "essential" property keys

The minimum set which I would say we need mandatory names are
  • vos.Owner
  • vos.ModificationDate
  • vos.Size
  • vos.MimeType

-- PaulHarrison

Agree with the list of common properties.

Defined in a annex to the main specification, including a standard URI and details of what each property means and how it is represented.

e.g.

Data created date
KEY
    vos.data.created.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were originally created.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]
----
Data modified date
KEY
    vos.data.modified.date
Description :
    A read-only property generated by the server.
    Indicating when the data contents were last modified.
    Formatted as a ISO 8601 date-time [yyyy-mm-ddTHH:MM:SS.SSS]

Note - As VOSpace-1.0 does not support append, the only way the created and modified dates will be different is if the server modifies the data underneath.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Clarify the use of the Format parameter in some calls

I believe that the original intention of Format was to specify a possible transformation of the data (mainly on export of table data from RDBMS based stores). In the current version of the document it reads as if this parameter is merely a description of the data - in which case mime-type suffices. In addition I am not sure if this parameter has any meaning for import operations.

Votes

name vote comment
PaulHarrison +1

Change "exception" to "fault"

The document uses the term exception for the "faults" rather than "fault" -- PaulHarrison

Votes

name vote comment
PaulHarrison +1
DaveMorris -1 I can't see a significant benefit from changing this

Consider adding an optional wildcard matching identifier to parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix, with typical simle shell wildcard semantics.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a workflow step is writing results into the VOSpace using a common prefix. The next step in the workflow wants to process all of the files produced, but only knows the prefix - without wild card matching the whole of the VOSpace needs to be listed to find the files.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer

Consider adding an optional identifier list to the parameters for ListNodes

This would allow the client to specifly a subset of the VOSpace to be listed - in effect the behaviour would be similar to the "ls" command in unix.

Reason for change: improved efficiency - if ListNodes always has to list the whole VOSpace then it is a pretty blunt instrument, especially as the number of data objects in the space increases.

Use Case

Suppose that there is a 1.0 VOSpace containing 5000 data items and a client is currently interacting with a V2.0 VOSpace that has links to a small subset (e.g. 100 data items) - the client needs to make 100 getNodeProperties SOAP calls to the V1.0 space to get the latest metadata about the data objects - with an optional identifier list it can make one call to ListNodes.

-- PaulHarrison 19 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris -1 I can't see a significant benefit from changing this

Reconsider getNodeProperties and setNodeProperties

the naming of these operations makes them appear as a pair, where in fact they are not - you get a Node out of the getNodeProperties and have to put a PropertyPairList into the setNodeProperties - If the change to ListNodes were made as above, then getNodeProperties would be redundant anyway. -- PaulHarrison

Votes

name vote comment
PaulHarrison +1 the benefit is improved clarity - in addition there is the slightly irritating issue (see below) that you cannot take the property list from the "get" and use it in the "set" because of read-only properties
DaveMorris -1 I can't see a significant benefit from changing this

Semantics not clear for setNodeProperties

It is not clear
  • how to delete a property - does a null value of the property pair denote this?
  • does the whole set of node properties given as an argument replace the whole set for the node, or is a union operation performed.
  • interaction with the "read-only" properties in these scenarios...
-- PaulHarrison

Agree, need to make this clearer.

  • Null value deletes property.
  • Operation is union not replace.
  • Server throws PermissionDenined for read-only properties.
    • what if only one of the list of properties is read-only? - need to signal which are the bad properties in the exception - also PermissionDenied could be confusing as this normally refers to the permissions on the data object which could potentially be changed in future versions of VOSpace - certain properties are fundamentally readOnly and so would

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Change names of parameters for moveNode and copyNode

Target is a confusing name for the "source" part of a move operation it sounds more like "destination" - recommend use "source" and "destination" -- PaulHarrison

Yep, source and destination are probably better. As long as we make it clear that these are internal locations, not references to external locations.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
DaveMorris +1

Rename bulk data transfer operations

  • I stilll find the data transfer operations confusing- I think that the basic problem is that the push and pull verbs are opposite in meaning and viewed from opposite perspectives. I have a new set of proposals

old name new name"
pushDataToVoSpace importDataClientPush
pullDataToVoSpace importDataServerPull
pushDataFromVoSpace exportDataServerPush
pullDataFromVoSpace exportDataClientPull

which I think are better because

  1. the overall objective of the operation is the first verb
  2. the active party in the transfer is identified
  3. the direction of the transfer is then related to the active party.

-- PaulHarrison - 13 Jun 2006

This is a change to the specification not the WSDL, move this to the specification change page.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1
MatthewGraham 0
DaveMorris -1 Change the specification document first
GuyRixon -1  

New GetPropertyKeys operation

This proposal is basically because I am still a little worried about interoperability problems with the completely untyped nature of the property-key pairs - particularly as they are expected to carry some fundamental metadata about the data objects in the current implementation. This call would return the complete list of key names that have been used in the VOSpace, which would then allow clients to attempt to be consistent in the use of key names - it is not much but at least it does provide a mechanism to voluntarily avoid complete anarchy.

Should return a list of the keys with an indication of which ones relate to read-only properties.

-- PaulHarrison - 13 Jun 2006

I agree with adding the method. However, this should be changed in the specification first, not the WSDL. Move this to the specification change page.

-- DaveMorris - 16 Jun 2006

Votes

name vote comment
PaulHarrison +1 proposer
DaveMorris -1 Change the specification document first
GuyRixon -1 Add to spec document first
MatthewGraham +1

Re-assess what metadata should be returned with various faults

Some faults need to return metadata beyond simply their type to convery meaningful information about exactly what has gone wrong. e.g. for a setDataNodeProperties call could specify several read-only properties amongst a larger number of properties, and the client would not know which were the properties in error without a list being returned in the fault.

Votes

name vote comment
PaulHarrison +1 proposer

Re-assess the semantics of creating a server named data object.

see discussion on mailing list for more background http://www.ivoa.net/forum/vospace/0606/0095.htm

Votes

name vote