VOSpace service specification: Replies to Request for Comments

This page adds additional details to some of the replies to the questions raised on the VOSpace RCF comments page.

---

Comments from MarkTaylor

This is a way for the service to declare that it will, interpret specific properties to have specific meanings.

As an example :
By default, all properties are treated as string name:value pairs, or more accurately as uri:value pairs. So a client can set a generic property, and the service will just store it as a string.

If we have a property URI that reprepresents 'mime type', then a service that provided HTTP GET access could add the property value to the relevant header field of the HTTP GET response.

A basic service might not implement support for adding the mime type header to a HTTP GET response. In which case, it would not list the 'mime type' property in its 'accepts' or 'provides' list. A client can still set the property, but the service would just treat it as a string value, and service would not try to interpret any meaning from the value.

A more complex service might understand that the 'mime type' property should be added to the headers of a HTTP GET response. By including the property in its 'accepts' and 'provides' list the service is declaring that it will allow clients to modify the value, and it will add it to the relevant header field of a HTTP GET response.

If we have a property URI that represents 'data size', then the value will depend on the actual data itself, and logically it should not be possible to set this from outside.

By including the 'data size' property in its 'provides' list, the service is declaring that it 'understands' the property, and will generate the property value from the data.

(DaveMorris)

Comments from DougTody

Once we have registered the core properties, protocols and views then these will all become fixed URIs. Although the URIs would refer to descriptive registry resources, in practice simple clients and services would not need to dereference these via the registry, they just become constant strings and can be hard coded into both service and client.

During normal operation neither the service or client should need to dereference any of the URI identifiers. They can all be treated as opaque identifier strings.

However I appreciate that this isn't clear in the way that the specification is worded.

A number the questions raised in the RFC comments suggest that although the specification may be technically correct, it isn't clear about how thing would actually work in a deployed service.

We plan to create a supplementary document which will include one or more concrete examples of how a get and put transfer would work.

In the mean time, the following use cases may help to explain a bit more :

Use case #1 :

A client tries to access data in a service that only supports encrypted or authenticated protocols. The client does not understand these protocols, but it can still list the protocol (unknown) URIs in its error message :

    Unable to find common transfer protocol.
    Client supports :
        ivo://aaaa
        ivo://bbbb
    Service provides :
        ivo://xxxx
        ivo://yyyy

A slightly more complex client may dereference these URIs and display the display name for the unknown protocols :

    Unable to find common transfer protocol.
    Client supports :
        "Simple http put" (ivo://aaaa)
        "Simple ftp  put" (ivo://bbbb)
    Service provides :
        "Secure http put" (ivo://xxxx)
        "Secure ftp  put" (ivo://yyyy)

The user can then look for a different client tool that implements one of the sercure protocols (by searching the registry for applications that support the protocol URI).

Or, they can send a request to the application developer asking them if they plan to support the new protocol. All the user would need to pass to the developer is the URI of the new protocol ivo://xxxx. The corresponding description in the registry should provide enough information to enable the developer to implement the new protocol. The simplest case would be to include a URL in the registry description that pointed to an external resource containing the full protocol specification.

Use case #2 :

A service implementation begins to see a lot of transfer requests using a new protocol listed in their server logs.

If their logs indicate that a lot of client applications are capable of using the new protocol, then they may decide that it is time to updated their service to support it. The service provider can pass the protocol URI to their developer team, who can then lookup the description in the registry.

Use case #3 :

A complex GUI client may dereference the protocol URIs and use the display name and description to present the options to the user in a more user friendly way.

To speed things up, the client implementation may hard code the most common ones and cache the unusual ones.

The GUI tool may also use the descriptive names and text from the registry resource to populate select lists, tool tips or help boxes.

Summary

• The specification only defines the identifiers as xsi:anyURI

• Implementations are free to use proprietary URN style identifiers urn:file-size.
• If an identifier points to a resource in the registry that contains things like display name and description, then it gives the community a common reference point to describe the property, view or protocol.

• Server and client implementations can treat the URI identifiers as opaque strings.
• If a client or service understands a particular property, then the corresponding URI can be treated as a fixed string.
• If a client or service encounters a new URI that it does not understand, then it can just treat it as an unknown string.

• A more complex implementation may use the display name and description in the registry resource to display additional information to the user.

(DaveMorris)

"... I suspect we might be able to use string equality on such a URI to test for these things ..." - Yes

"... but in principle we would need to look the URI up in the registry, and parse the XML data structure which comes back, before we can determine basic information about what a service can do ..." - No

Once the core set of properties, protocols and views have been registered, client and server implementations can treat the URIs as hard coded string constants.

There is no need to dereference the registry URIs unless an application such as a complex GUI client wants to use the information to display user friendly names and descriptions.

(DaveMorris)

This was intentional. We plan to get v1.1 agreed as soon as possible, and that would support hierarchical data structure, with '/' as the path delimitor. This means that '/' will have a specific meaning in vospace 1.1, so to avoid confusion between the two we want to exclude it from use in vospace 1.0.

It may be possible that a vospace 1.1 service publishes a sub directory as a vospace 1.0 service, allowing vospace 1.0 clients to access data within just that sub directory. In which case, we want the URI identifiers to make sense between the two systems.

If we allow '/' in vospace 1.0 names (and just treat them as normal strings without interpreting the '/'), then there would be an 'odd' different in behaviour between vospace 1.0 and vospace 1.1 services.

• If '/' was not excluded, the you could create something called 'a/b/c/d' in vospace 1.0 even though the impled parent 'a/b/c' didn't exist.
• In vospace 1.1, creating 'a/b/c/d' would fail if 'a/b/c' didn't exist because it treated the '/' as a path delimitor.

We wanted to avoid this discrepancy by explicitly excluding '/' in vospace 1.0.

(DaveMorris)

Support for StructuredData was added to the specification to compliment the DAL services, not to replace them. One target application of VOSpace is to provide a way of importing data into these services.

Some specific examples might help :

Images in SIA

The SIA specification provides tools for finding and getting images, but as yet, no API for importing images. In order to provide an import mechanism, a SIA service could also provide a VOSpace API. The user could use the vospace API to transfer images into the service, using the StructuredData node to identify the files as FITS images.

• If the user imported file as UnstructuredData containing FITS images, the service would just store them as files on disk.

• If the user imported them as StructuredData containing FITS images, then the service would attempt to interpret the contents of the files.
  • If the files did not contain valid data, then the service would reject them.
  • If file did contain valid data, then the service would process the FITS headers and add them to the SIA database.
  • The images would then be available for access via the SIA interface.

VOTable in TAP

The TAP interface provides tools for searching and querying the database, but as yet, no API for importing data. The VOSpace API would enable the user to transfer tabular data into the service, using the StructuredData node to identify the files as tabular data.

• Again, the user may import UnstructuredData containing VOTable data, and the service would just store them as files on disk.

• If the user imported them as StructuredData containing VOTable data, then the service would attempt to interpret the contents of the files.
  • If the files did not contain valid data, then the service would reject them.
  • If file did contain valid data, then the service would process the VOTable metadata and create new database tables to contain the data.
  • The new tables would then be available for access via the TAP interface.

As a specific example of how this could be used, AstroGrid plans to provide a database service that contains static data from a large survey co-located with user data imported via VOSpace. Users would be able to use the VOSpace service to upload their own VOTable files into the user area of the service, and then use the TAP interface to create ADQL join queries to cross reference data from the large survey with their own data.

(DaveMorris)

Yes, the list of protocol options represent different endpoint(s) which provide access to the same data.

These could be alternative mirror services with the same protocol, or different protocol URLs.
If asked to provide access to the data via http or ftp, the service response could contain :

• two separate http URLs for different http mirrors
• plus a ftp URL

All of which would provide access to the same data.

The client is free to use them in any order it wants to, but it can only reliably use each one once (some of the endpoints may be one-shot URLs with cookies embedded in the URL itself).

The client does not have to parse the URL, it only has to recognise the protocol URI. Again, these URIs can be hard coded into the client as part of the code that implements that protocol.

(DaveMorris)

Yes - this was one of the use cases we had in mind when designing the interface.

• A VOSpace client can handle the transfer negotiation, requesting access using 'standard HTTP GET' or 'standard FTP GET' in the list of protocols.
• The VOSpace response would contain standard endpoint URLs for the requested protocols.
• The vospace client can then pass these URLs to a separate application which does the data transfer.

(DaveMorris)