Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the Validation Tools to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at VOEvent, Getting Started for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, a "yellow page" or directory of service web locations.

There are 2 methods for publishing your service into an IVOA registry.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.0

March 2014

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it SCS, SIAP, SSAP; TAP (using a different application) Dec. 2012 VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed.
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, OAI-PMH May 2014 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repository for implementing IVOA standards: UWS, TAP, TAPRegExt, CDP, VOSpace2, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
DSA AstroGrid     Layer on top of existing database providing Cone Search, (proto) TAP, and other capabilities, including IVOA Single Sign On authentication. However, TAP implementation is out of date, and software is no longer maintained.


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 12/2012 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 3/2013 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
SAVOT AndreSchaaff 4/2010 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
CDS UWS Library CDS cds-question (at) unistra.fr   A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library CDS cds-question (at) unistra.fr 4/2014 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10,5) with a radius of 2. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile



Topic revision: r36 - 2014-05-19 - MarkusDemleitner
 
This site is powered by the TWiki collaboration platformCopyright © 2008-2014 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback