IVOA Grid & Web Services: VOSpace

Contents


Overview

Users want to be able to easily publish and share their own data as well as have data reside close to computation nodes when large-scale processing or dynamic caches for intermediate data products in workflows are involved. Three modes of data transfer need to be supported: client-server (the client pushes data to, or pulls data from, the server), server-client (the server pushes data to, or pulls data from, the client) and server-server (one server pushes data to, or pulls data from, another server in response to a client request) also known as third-party transfer.

There are many solutions already to these problems so instead of inventing our own we specify a lightweight interface, VOSpace, that sits on top of existing storage solutions, providing a common interface to an underlying heterogeneity of implementations from local file systems to bulk data grids. VOSpace can distinguish between unstructured data, which is just a collection of bytes, and structured data where there is knowledge about how the bytes are arranged and so additional operations are possible. Individual VOSpace instances can also be federated into a peer network so that the user just interacts with a single data space.

VOSpace 1.x specifications relate to a SOAP-based web service.

VOSpace 1.0

VOSpace 1.0 defines an unconnected flat file space and is now an IVOA Recommendation:

Specification

VOSpace 1.1

VOSpace 1.1 extends the existing 1.0 specification to support containers, links between individual VOSpace instances, third party APIs and a "find" mechanism.

Specification

Standard URIs

The VOSpace specification refers to a number of standard URIs for properties, views and protocols.

The following URIs will be used to represent the standard protocols:

  • ivo://ivoa.net/vospace/core#httpget Will be used as the protocol URI for a HTTP GET transfer.
  • ivo://ivoa.net/vospace/core#httpput Will be used as the protocol URI for a HTTP PUT transfer.
The following URIs will be used to represent the standard views:

  • ivo://ivoa.net/vospace/core#anyview Will be used as the view URI to indicate that a service will accept any view for an import operation.
  • ivo://ivoa.net/vospace/core#binaryview Will be used as the view URI to import or export data as a binary file.
  • ivo://ivoa.net/vospace/core#defaultview Will be used by a client to indicate that the service should choose the most appropriate view for a data export.
In order to register VOSpace services in an IVOA registry, the service capabilities need to identified using a standard URI.

The following URIs will be used to represent the service capabilities:

  • ivo://ivoa.net/vospace/core#vospace-1.0 Will be used as the capability URI for a VOSpace 1.0 service.
  • ivo://ivoa.net/vospace/core#vospace-1.1 Will be used as the capability URI for a VOSpace 1.1 service.
  • ivo://ivoa.net/vospace/core#vospace-2.0 Will be used as the capability URI for a VOSpace 2.0 service.
If a service implementation supports more than one version of the VOSpace API, then these capability URIs can also be used within a VOSpace service to identify different VOSpace capabilities for a node.

One use case for this would be a VOSpace-1.1 client talking to a service that implements both VOSpace-1.0 and VOSpace-1.1, where the client is acting on behalf of a third party agent that only understands VOSpace-1.0. In which case, the client can use the information in the VOSpace-1.0 capability to direct the third party agent to the VOSpace-1.0 endpoint.

Discussion

History

VOSpace 2.0

VOSpace 2.0 defines a RESTful interface.

Specification

The proposed recommendation is available here.

Discussion

VOSpace 2.1

VOSpace 2.1 contains minor additions and corrections to the RESTful version 2.0.

Specification

The current Working draft is available here.

Discussion

Custom URIs

The VOSpace specification allows 3rd parties to define custom properties, protocols and views simply by choosing a unique URI to represent them.

The following URLs link to some examples of custom properties, protocols and views.

Note - this list is for information only, and is not intended to be a definitive list of approved properties, protocols or views.

It is possibly to use custom properties, protocols and views in VOSpace messages that are not in this list. However, if you want other people to use your custom URIs, then adding them to this list will enable people to find out about them.

VOSpace 2.2 or 3.0

A list of future potential VOSpace changes, enhancements and discussion items are listed here:

  • New parameter on getNode the sets the sort field on the node children? Support reverse sorting too?
  • Change StandardIDs to match the pattern of TAP 1.1:
    • Make use of the UWSRegExt interface type fields: type="uws:Sync" and type="uws:Async"
    • Single standardID for the specification
  • Remove the internal-facing VOSpace standard capabilities, as noted by DaveMorris here: VOSpace 2.1 RFC
  • Changes from François Bonnarel from the 2.1 RFC VOSpace 2.1 RFC
    • A model diagram like the one in fig 2 with all the classes, not only "node". Relationships should be shown but no need to do a full UML diagram.
    • A couple of diagrams showing where Web service operations occur (VOSPACE to client. Client to VOSPACE. Inside a VOSPACE. From: VOspace to another One etc...)
  • Ability to sort on alternate columns in both ascending and descending order (eg lastModfied, contentLength). CADC has implemented this.
  • In the transfer object, the 'direction' can conflict with the protocol URI. For example, the direction can be 'pullFromVoSpace' and the protocol can be 'HTTP-PUT'. This could be cleaned up to remove error cases.
  • Section 4, Access Control: Version 3.0 should state access control policies at the Node level, or should this be considered orthogonal? * Custom views: views that clients create and use to do work/data reduction at the physical storage location.
  • JPEG2000 Interactive Protocol (JPIP) and VOSpace? A number of articles on Astronomy and Computing
  • move /views and /properties to a registry extension?
  • From Pierre Fernique on the VOSpace 2.1 RFC page:
    • There are many OPTIONAL elements in the this standard. It is not so good for the interoperability (and concretelly we have very few VOSPace clients). I would suggest in the next standard version to avoid (if possible) too many optional features (either remove it, or impose it)
    • The auto generation of a unique URI is an important function. Presently it is implemented (optional) as a kind of "hack" via a reserved suffix keywork ".auto" to the URI. I am thinking that this important feature could be implemented in a better way, as a creatorNode parameter or something like this.

Implementations

groupSorted ascending version endpoint VOSpace root Notes
Astrogrid
CADC v2.0 http://www.canfar.phys.uvic.ca/vospace vos://cadc.nrc.ca!vospace v2.1 in progress
Caltech v1.1 testnvo.cacr.caltech.edu vos://nvo.caltech!vospace/ HTTP transports, supports structured nodes and archive formats (tar, tar.gz)
ESO v2.0 vops1.hq.eso.org vos://org.eso!vospacetest/ HTTP transports

Mailing list

The mailing list for discussion of the VOSpace specifications and related issues is: grid@ivoa.net which is archived at: http://mail.ivoa.net/pipermail/grid

VOSpace 1.0 history

Discussion

The discussion pages that lead up to the V1.0 working draft are available here :

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatodt VOSpace-Usage.odt r1 manage 201.5 K 2007-10-01 - 16:18 PaulHarrison vospace usage note version 0.1
Unknown file formatwsdl VOSpaceContract-v1.1-rc3.wsdl r1 manage 55.2 K 2008-10-06 - 20:35 DaveMorris VOSPace 1.1-rc3 wsdl
Unknown file formatxsd VOSpaceTypes-v1.1-rc3.xsd r1 manage 53.5 K 2008-10-06 - 20:35 DaveMorris VOSPace 1.1-rc3 schema
Unknown file formatxsd VOSpaceTypes-v1.1rc1.xsd r1 manage 52.2 K 2008-03-11 - 16:59 MatthewGraham VOSpace 1.1 RC1 Message Schema
Unknown file formatwsdl vospace.wsdl r1 manage 50.5 K 2008-03-11 - 16:58 MatthewGraham VOSpace 1.1 RC1 WSDL
Edit | Attach | Watch | Print version | History: r28 < r27 < r26 < r25 < r24 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r28 - 2018-06-20 - BrianMajor
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback