(r6) VOSpace11Formats < IVOA

IVOA Web>IvoaGridAndWebServices>VOSpaceHome>VOSpace11Formats (revision 6) (raw view)~~EditAttach~~
[[http://www.ivoa.net/twiki/bin/view/IVOA/VOSpaceHome][VOSpace home page]]
---+ VO and VOSpace data formats

This is a discussion page looking at defining and registering a list of standard VO data formats.

Although the VOSpace team will probably want to register VOStandard resources for the VO data formats, this discussion is not intended to be about using registry URIs vs MIME types to describe data formats.

All I am suggesting at the moment is that we work together to produce a list of the core VO data formats.
As far as I know, we don't have a list of the common data formats used within the VO. If we do already have one, then please let me know.

Initially, this can be just a wiki page that describes the formats, provides links to the relevant RFC documents,
lists the common MIME types associated with each format, specifies the VO standard MIME type(s) we want to use
and describes how the formats are used within astronomy and the VO.

The information collected on the wiki may form the basis for a VOStandard resource that registers the common data formats,
but the initial step is to collect the information.

-- IVOA.DaveMorris - 29 May 2007

----
---++ Concepts

The recent discussion [[http://www.ivoa.net/forum/dal/0705/0608.htm][thread]] about data formats and MIME types on the DAL mailing list started with a question about "gzipped images in SIAP 1.0" and has openned up into a wider discussion about MIME types and HTTP headers.

The thread has highlighted the fact that there are at least three different concepts that we need to represent when transferring data.

   * _What it contains_ : tabular data, an image etc
   * _How it is represented_ : the primary data format, FITS, VOTable etc (plus modifiers e.g. binary VOTable and tiled FITS)
   * _How it is transferred_ : e.g. zip, tar, gzip, targzip compressed files or streams
 
The discussion has looked at various options for representing these concepts in the HTTP header fields :

   * _Content-type_
   * _Content-encoding_
   * _Content-disposition_

We have been struggling to solve similar problems within the VOSpace group.
The problems we are trying to solve within VOSpace may be slightly more complex for a number of reasons :

   * VOSpace supports asynchronous 3rd party transfers, where the client arranges for data to be sent from one VOSpace service to another. This means that the data transfer occurs outside the scope of the SOAP or REST service call, and so the client cannot use the HTTP request fields to indicate what format it would like the data to be transferred in.
   * VOSpace needs to be able to support other data transfer protocols, like Ftp and GridFtp, which may not have the equivalent metadata fields for _Content-type_ and _Content-encoding_ etc.
   * VOSpace needs to be able to distinguish between
      * "I am going to send you a zip file containing a set of FITS images, and I want you to store it as a zipfile"
      * "I am going to send you a zip file containing a set of FITS images, and I want you to unpack it and store|process the individual FITS images"

However, the core problems of deciding which formats to use, and how to refer to them are the same.

The current VOSpace specification introduces the concept of registered URIs to refer to 'views' which unfortunately attempts to represent at least two of the concepts _what it contains_ and _how it is represented_ using a single URI.
We know this is not sufficient for what we need, and plan to revisit this fairly soon.

Using registered URIs to identify content and formats has proved to be useful, certainly within the context of VOSpace.
However, I appreciate that other groups may prefer to use MIME types or something similar, particularly for HTTP based access.

I think we should work together with the other groups to produce a list of the main VO data formats, their corresponding MIME types and
see if we can identify an inheritance hierarchy where it is appropriate.

For VOSpace services, we would probably want to register these formats in a VOStandard registry resource, enabling us to
use registry URIs to refer to them.

Even if we eventually decide not to use the registered URIs in services and applications, creating a list of the main VO data formats and their corresponding MIME types will provide a useful resource for developers working on VO projects.

The first step is to make an initial list of the formats, define the corresponding MIME types and the inheritance hierarchy.

----
---++ Data, container and transport formats
My initial guess is that we need two (possibly three) types of format, a data format, a container format and possibly a transport format.

#DataFormat
---+++ Data formats

   * A data format is a file format that contains a representation of the data, e.g. FITS or VOTable

Many of the common data formats have multiple MIME types associated with them, and the meaning may depend on the context. 

It would be useful to bring together the information about the different data formats and MIME types from the existing VO standards.

This may make it easier for new VO standards and services to refer to, and preferably re-use, information about MIME types and data formats from the existing VO standards.

One example of this is that we may already have three or four different recomended MIME type strings that mean _ASCII VOTable_.
Which MIME type string is used depends on what type of service you are using, and what the VOTable file contains (metadata about a set of results or the actual data itself).


#ContainerFormat
---+++ Container formats
   * A container format is a file format that contains other files, e.g. zip or tar

To meet the design goals of VOSpace, we need to be able to represent the following concepts to the client application :
   * The selected images are available as individual _FITS/image_ files.
   * The selected images are available as a _zip file_ containing the individual _FITS/image_ files

In this model FITS may be a special case. From my admittedly basic understanding of FITS, it is possible for FITS
to be both a data format (a FITS file containing tabular or image data)
and a container format (one FITS file containing multiple images or tables).

I hope others who have a much better understanding of the FITS format and its useage within astronomy will be able to define this in more detail.

Ideally, we would want to be able to distinguishes between the following concepts :
   * The selected images are available as individual _FITS/image_ files.
   * The selected images are available as a single FITS file containing multiple images

#TransportFormat
---+++ Transport formats
   * A transport format is additional encoding, or compression, that is applied to the data stream itself e.g. gzip content-encoding a HTTP response.

Transport formats will be specific to the particular tranport protocol used. The most common examples are the content-encoding options that can be applied to HTTP data streams.
I don't know if any other tranport protocols support equivalent options for processing or compressing the data stream.

However, it may be useful to at least list the main content-encoding options that we expect VO services to be able to apply to HTTP data streams and identify the standard MIME type strings that VO services should use to refer to them.

---++ Specialization and inheritance
#ExtendedFormat

Some data formats may be specializations of existing formats, and may inherit the MIME type or other details from their parent format.

---+++ Java JAR format
#JarFormat

An example of this is the Java Archive format.
From the JAR file [[http://java.sun.com/j2se/1.4.2/docs/guide/jar/jar.html][specification]] :
   * _A JAR file is essentially a ZIP file that contains an optional META-INF directory ... [that contains] specific files and directories .... that are recognized and interpreted by the Java 2 Platform._

This implies that the JAR format extends the ZIP specification, with additional metadata in the META-INF directory.
As far as I know, there isn't a specific MIME-type defined for Java JAR files, so the JAR format would 'inherit' the MIME type string from the ZIP format.

This particular example may not be useful for astronomers, but I listed it here because it is an already established extension format that developers will be familiar with.

---+++ VOSpace archive format
#VOSpaceFormat

What I think will be useful for astronomers is to be able to say
   * _Archive this branch of my VOSpace as a zip or gzip file and save it to backup space_
then at a later date
   * _Load this backup into VOSpace and restore all the VOSpace metadata_
in order to support this, we may need to define a similar extension to the zip or gzip format that includes VOSpace metadata.

---+++ Data set specific FITS format
#DataSetFormat

Having talked about this with some of our astronomers, one thing they suggested would be useful is be to be able to define
types that represent data from specific data sets.

If we had a VO data type that represented _FITS image_, then they would like to be able to define a new type
that represented a _FITS image from a specific data set_.

This new type would extend the definition of _FITS image_, inheriting all of the properties and behaviour defined fo a generic _FITS image_, and add details of the specific FITS header fields that that particular data set used in their files.

Defining this as an extension type means that tools and applications that don't understand the specific type could handle the data
as _generic FITS_, but tools that did understand the more specific type would be able to make use of the additional metadata
that described what the data set specific header fields meant.

At this point I don't know if this is a GoodIdea or not, or whether this information should be encoded in the content type or
in a separate property.
However, our astronomers seemed to think that the ability to distinguish between a _generic FITS file_ and a _FITS file from a specific data set_
was important.

_How_ we enable them to make this distinction is up for discussion.

<br/>
<!--
      * Set ALLOWTOPICRENAME = %MAINWEB%.TWikiAdminGroup
-->
Topic revision: r6 - 2007-05-29 - DaveMorris
IVOA
Log in or Register
IVOA.net
Wiki Home
WebChanges
WebTopicList
WebStatistics
Twiki Meta & Help
IVOA
Know
Main
Sandbox
TWiki
TWiki intro
TWiki tutorial
User registration
Notify me
Working Groups
Interest Groups
Time Domain
Committees
Stds&Procs
www.ivoa.net
Documents
Events
Members
XML Schema