Difference: PublishingInTheVO (1 vs. 32)

Revision 322024-05-27 - GillesLandais

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish your data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending on your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it.
If, having consulted the resources described above or the FAQ below, you are still unsure how to proceed, please send a message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Changed:
<
<
Q: How do I publish images?
>
>
Added:
>
>
Q: How do I publish images?
  A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

Changed:
<
<
If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
>
>
If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  Technical documentation: Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Changed:
<
<
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
>
>
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
Added:
>
>
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
 
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, the "yellow pages" of the VO, queried by user applications (and other things) when looking for services.

To make this work, you need to provide a standardised service description (that's called “resource record” in VO language) and then push that into the big pool of existing registry records. Depending on how many data collections you publish and how often they change, there are various ways to do that, and GettingIntoTheRegistry discusses them.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

csic.es

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec -MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
Changed:
<
<
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated 1.0 dev
Apr 2014
The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
>
>
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated 1.0 dev
Apr 2014
The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
 
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, SLAP, OAI-PMH, SIAv2, Datalink December 2022 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces). For recent developments, see DaCHS on the GAVO blog
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repositories for implementing IVOA standards: SSO, CDP, UWS, SIA, TAP, TAPRegExt, VOSpace, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.
Daiquiri Jochen Klar or E-Science AIP Potsdam TAP, Cone Search Oct 2019 Daiquiri - Python based framework for the publication of scientific databases - Daiquiri on Github. It is deployed on Gaia@AIP or APPLAUSE - Archives of Photographic PLates for Astronomical USE


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 2022 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2022 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B.
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA ( http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
Changed:
<
<
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
>
>
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
 
CDS TAP Library GregoryMantelet 2/2018 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
Changed:
<
<
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
>
>
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
(previous version available here)
 
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile. Old, but still working in 2022.
Changed:
<
<
MocInfo Pierre Fernique 2017 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
>
>
MocInfo Pierre Fernique 2017 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
 
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools
Hipsgen Pierre Fernique 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for image or cube collection. See "How to make your HiPS in 10 steps" and HiPS CDS page.
Hipsgen-cat François Xavier 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for catalogs (see HiPS CDS page)
CDS FITS validator Gilles Landais 2019 A web service based on the Saada API to check the FITS header ObsCore compatibility.

Changed:
<
<

>
>
-->
 

Revision 312023-11-11 - TimJenness

 

Publishing Data into the VO



0. Introduction

Changed:
<
<
The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.
>
>
The purpose of this page is to provide practical information about how to publish your data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.
 
Changed:
<
<
Depending of your needs and your technical experience, there are several ways to publish you data into the VO:
>
>
Depending on your needs and your technical experience, there are several ways to publish you data into the VO:
 
  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.
Changed:
<
<
>
>
 When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
Changed:
<
<
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.
>
>
  • remove your service from the registry if you stop running it.
If, having consulted the resources described above or the FAQ below, you are still unsure how to proceed, please send a message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.
 

1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, the "yellow pages" of the VO, queried by user applications (and other things) when looking for services.

To make this work, you need to provide a standardised service description (that's called “resource record” in VO language) and then push that into the big pool of existing registry records. Depending on how many data collections you publish and how often they change, there are various ways to do that, and GettingIntoTheRegistry discusses them.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

csic.es

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec -MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated 1.0 dev
Apr 2014
The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, SLAP, OAI-PMH, SIAv2, Datalink December 2022 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces). For recent developments, see DaCHS on the GAVO blog
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repositories for implementing IVOA standards: SSO, CDP, UWS, SIA, TAP, TAPRegExt, VOSpace, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.
Daiquiri Jochen Klar or E-Science AIP Potsdam TAP, Cone Search Oct 2019 Daiquiri - Python based framework for the publication of scientific databases - Daiquiri on Github. It is deployed on Gaia@AIP or APPLAUSE - Archives of Photographic PLates for Astronomical USE


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 2022 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2022 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B.
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA ( http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 2/2018 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile. Old, but still working in 2022.
MocInfo Pierre Fernique 2017 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools
Hipsgen Pierre Fernique 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for image or cube collection. See "How to make your HiPS in 10 steps" and HiPS CDS page.
Hipsgen-cat François Xavier 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for catalogs (see HiPS CDS page)
CDS FITS validator Gilles Landais 2019 A web service based on the Saada API to check the FITS header ObsCore compatibility.


<--  
-->

Revision 302023-01-17 - MarkusDemleitner

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, the "yellow pages" of the VO, queried by user applications (and other things) when looking for services.

Changed:
<
<
To make this work, you need to provide a standardised service description (that's called “resource record” in VO language) and then push that into the big pool of existing registry records. Depending on how many data collections you publish and how often they change, there are various ways to do that, and GettingIntoTheRegistry discusses them.
>
>
To make this work, you need to provide a standardised service description (that's called “resource record” in VO language) and then push that into the big pool of existing registry records. Depending on how many data collections you publish and how often they change, there are various ways to do that, and GettingIntoTheRegistry discusses them.
 

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
Changed:
<
<
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
>
>
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

csic.es

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec -MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
 
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
Changed:
<
<
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
>
>
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated 1.0 dev
Apr 2014
The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
 
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
Changed:
<
<
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, SLAP, OAI-PMH, SIAv2, Datalink November 2021 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces). For recent developments, see DaCHS on the GAVO blog
>
>
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, SLAP, OAI-PMH, SIAv2, Datalink December 2022 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces). For recent developments, see DaCHS on the GAVO blog
 
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repositories for implementing IVOA standards: SSO, CDP, UWS, SIA, TAP, TAPRegExt, VOSpace, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
Changed:
<
<
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.
Daiquiri Jochen Klar or E-Science AIP Potsdam TAP, Cone Search Oct 2019 Daiquiri - Python based framework for the publication of scientific databases - Daiquiri on Github. It is deployed on Gaia@AIP or APPLAUSE - Archives of Photographic PLates for Astronomical USE
>
>
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.
Daiquiri Jochen Klar or E-Science AIP Potsdam TAP, Cone Search Oct 2019 Daiquiri - Python based framework for the publication of scientific databases - Daiquiri on Github. It is deployed on Gaia@AIP or APPLAUSE - Archives of Photographic PLates for Astronomical USE
 

4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 2022 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2022 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
Changed:
<
<
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
>
>
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B.
 
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
Changed:
<
<
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
>
>
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA ( http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
 
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 2/2018 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile. Old, but still working in 2022.
MocInfo Pierre Fernique 2017 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools
Hipsgen Pierre Fernique 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for image or cube collection. See "How to make your HiPS in 10 steps" and HiPS CDS page.
Changed:
<
<
Hipsgen-cat François Xavier 2018

HiPS generator tool allowing to create a Hierarchical Progressive Survey for catalogs (see HiPS CDS page)

>
>
Hipsgen-cat François Xavier 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for catalogs (see HiPS CDS page)
 
CDS FITS validator Gilles Landais 2019 A web service based on the Saada API to check the FITS header ObsCore compatibility.


<--  
-->

Revision 292022-09-05 - MarkTaylor

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, the "yellow pages" of the VO, queried by user applications (and other things) when looking for services.

To make this work, you need to provide a standardised service description (that's called “resource record” in VO language) and then push that into the big pool of existing registry records. Depending on how many data collections you publish and how often they change, there are various ways to do that, and GettingIntoTheRegistry discusses them.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, SLAP, OAI-PMH, SIAv2, Datalink November 2021 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces). For recent developments, see DaCHS on the GAVO blog
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repositories for implementing IVOA standards: SSO, CDP, UWS, SIA, TAP, TAPRegExt, VOSpace, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.
Daiquiri Jochen Klar or E-Science AIP Potsdam TAP, Cone Search Oct 2019 Daiquiri - Python based framework for the publication of scientific databases - Daiquiri on Github. It is deployed on Gaia@AIP or APPLAUSE - Archives of Photographic PLates for Astronomical USE


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
Changed:
<
<
STIL MarkTaylor 2021 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2021 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
>
>
STIL MarkTaylor 2022 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2022 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
 
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 2/2018 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
Changed:
<
<
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile
>
>
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile. Old, but still working in 2022.
 
MocInfo Pierre Fernique 2017 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools
Hipsgen Pierre Fernique 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for image or cube collection. See "How to make your HiPS in 10 steps" and HiPS CDS page.
Hipsgen-cat François Xavier 2018

HiPS generator tool allowing to create a Hierarchical Progressive Survey for catalogs (see HiPS CDS page)

CDS FITS validator Gilles Landais 2019 A web service based on the Saada API to check the FITS header ObsCore compatibility.


<--  
-->

Revision 282022-09-02 - GillesLandais

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, the "yellow pages" of the VO, queried by user applications (and other things) when looking for services.

To make this work, you need to provide a standardised service description (that's called “resource record” in VO language) and then push that into the big pool of existing registry records. Depending on how many data collections you publish and how often they change, there are various ways to do that, and GettingIntoTheRegistry discusses them.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, SLAP, OAI-PMH, SIAv2, Datalink November 2021 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces). For recent developments, see DaCHS on the GAVO blog
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repositories for implementing IVOA standards: SSO, CDP, UWS, SIA, TAP, TAPRegExt, VOSpace, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.
Daiquiri Jochen Klar or E-Science AIP Potsdam TAP, Cone Search Oct 2019 Daiquiri - Python based framework for the publication of scientific databases - Daiquiri on Github. It is deployed on Gaia@AIP or APPLAUSE - Archives of Photographic PLates for Astronomical USE


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 2021 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2021 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 2/2018 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile
MocInfo Pierre Fernique 2017 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools
Hipsgen Pierre Fernique 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for image or cube collection. See "How to make your HiPS in 10 steps" and HiPS CDS page.
Hipsgen-cat François Xavier 2018

HiPS generator tool allowing to create a Hierarchical Progressive Survey for catalogs (see HiPS CDS page)

Changed:
<
<
CDS FITS validator Gilles Landais 2019 A web service base on the Saada API to check the FITS header ObsCore compatibility.
>
>
CDS FITS validator Gilles Landais 2019 A web service based on the Saada API to check the FITS header ObsCore compatibility.
 


<--  
-->

Revision 272022-03-02 - AdaNebot

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.
Changed:
<
<
>
>
 When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, the "yellow pages" of the VO, queried by user applications (and other things) when looking for services.

To make this work, you need to provide a standardised service description (that's called “resource record” in VO language) and then push that into the big pool of existing registry records. Depending on how many data collections you publish and how often they change, there are various ways to do that, and GettingIntoTheRegistry discusses them.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, SLAP, OAI-PMH, SIAv2, Datalink November 2021 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces). For recent developments, see DaCHS on the GAVO blog
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repositories for implementing IVOA standards: SSO, CDP, UWS, SIA, TAP, TAPRegExt, VOSpace, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.
Daiquiri Jochen Klar or E-Science AIP Potsdam TAP, Cone Search Oct 2019 Daiquiri - Python based framework for the publication of scientific databases - Daiquiri on Github. It is deployed on Gaia@AIP or APPLAUSE - Archives of Photographic PLates for Astronomical USE


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 2021 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2021 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 2/2018 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile
MocInfo Pierre Fernique 2017 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools
Hipsgen Pierre Fernique 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for image or cube collection. See "How to make your HiPS in 10 steps" and HiPS CDS page.
Hipsgen-cat François Xavier 2018

HiPS generator tool allowing to create a Hierarchical Progressive Survey for catalogs (see HiPS CDS page)

CDS FITS validator Gilles Landais 2019 A web service base on the Saada API to check the FITS header ObsCore compatibility.


<--  
-->

Revision 262022-02-23 - MarkusDemleitner

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, the "yellow pages" of the VO, queried by user applications (and other things) when looking for services.

To make this work, you need to provide a standardised service description (that's called “resource record” in VO language) and then push that into the big pool of existing registry records. Depending on how many data collections you publish and how often they change, there are various ways to do that, and GettingIntoTheRegistry discusses them.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
Changed:
<
<
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, SLAP, OAI-PMH, SIAv2, Datalink October 2018 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces). For recent developments, see DaCHS on the GAVO blog
>
>
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, SLAP, OAI-PMH, SIAv2, Datalink November 2021 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces). For recent developments, see DaCHS on the GAVO blog
 
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repositories for implementing IVOA standards: SSO, CDP, UWS, SIA, TAP, TAPRegExt, VOSpace, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.
Daiquiri Jochen Klar or E-Science AIP Potsdam TAP, Cone Search Oct 2019 Daiquiri - Python based framework for the publication of scientific databases - Daiquiri on Github. It is deployed on Gaia@AIP or APPLAUSE - Archives of Photographic PLates for Astronomical USE
Deleted:
<
<
 

4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 2021 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2021 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 2/2018 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile
MocInfo Pierre Fernique 2017 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools
Hipsgen Pierre Fernique 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for image or cube collection. See "How to make your HiPS in 10 steps" and HiPS CDS page.
Hipsgen-cat François Xavier 2018

HiPS generator tool allowing to create a Hierarchical Progressive Survey for catalogs (see HiPS CDS page)

CDS FITS validator Gilles Landais 2019 A web service base on the Saada API to check the FITS header ObsCore compatibility.


<--  
-->

Revision 252021-12-16 - MarkTaylor

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, the "yellow pages" of the VO, queried by user applications (and other things) when looking for services.

To make this work, you need to provide a standardised service description (that's called “resource record” in VO language) and then push that into the big pool of existing registry records. Depending on how many data collections you publish and how often they change, there are various ways to do that, and GettingIntoTheRegistry discusses them.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, SLAP, OAI-PMH, SIAv2, Datalink October 2018 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces). For recent developments, see DaCHS on the GAVO blog
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repositories for implementing IVOA standards: SSO, CDP, UWS, SIA, TAP, TAPRegExt, VOSpace, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.
Daiquiri Jochen Klar or E-Science AIP Potsdam TAP, Cone Search Oct 2019 Daiquiri - Python based framework for the publication of scientific databases - Daiquiri on Github. It is deployed on Gaia@AIP or APPLAUSE - Archives of Photographic PLates for Astronomical USE


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
Changed:
<
<
STIL MarkTaylor 2018 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2018 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
>
>
STIL MarkTaylor 2021 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2021 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
 
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 2/2018 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile
MocInfo Pierre Fernique 2017 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools
Hipsgen Pierre Fernique 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for image or cube collection. See "How to make your HiPS in 10 steps" and HiPS CDS page.
Hipsgen-cat François Xavier 2018

HiPS generator tool allowing to create a Hierarchical Progressive Survey for catalogs (see HiPS CDS page)

CDS FITS validator Gilles Landais 2019 A web service base on the Saada API to check the FITS header ObsCore compatibility.


<--  
-->

Revision 242019-11-11 - MarkTaylor

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, the "yellow pages" of the VO, queried by user applications (and other things) when looking for services.

To make this work, you need to provide a standardised service description (that's called “resource record” in VO language) and then push that into the big pool of existing registry records. Depending on how many data collections you publish and how often they change, there are various ways to do that, and GettingIntoTheRegistry discusses them.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, SLAP, OAI-PMH, SIAv2, Datalink October 2018 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces). For recent developments, see DaCHS on the GAVO blog
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repositories for implementing IVOA standards: SSO, CDP, UWS, SIA, TAP, TAPRegExt, VOSpace, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.
Daiquiri Jochen Klar or E-Science AIP Potsdam TAP, Cone Search Oct 2019 Daiquiri - Python based framework for the publication of scientific databases - Daiquiri on Github. It is deployed on Gaia@AIP or APPLAUSE - Archives of Photographic PLates for Astronomical USE
Deleted:
<
<
CDS FITS validator Gilles landais (CDS) ObsCore Nov 2019 A web service base on the Saada API to check the FITS header VO-compatibility.
 
Deleted:
<
<
(VO).
 

4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 2018 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2018 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 2/2018 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile
MocInfo Pierre Fernique 2017 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools
Hipsgen Pierre Fernique 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for image or cube collection. See "How to make your HiPS in 10 steps" and HiPS CDS page.
Hipsgen-cat François Xavier 2018

HiPS generator tool allowing to create a Hierarchical Progressive Survey for catalogs (see HiPS CDS page)

Changed:
<
<
>
>
CDS FITS validator Gilles Landais 2019 A web service base on the Saada API to check the FITS header ObsCore compatibility.
 


<--  
-->

Revision 232019-11-08 - GillesLandais

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, the "yellow pages" of the VO, queried by user applications (and other things) when looking for services.

To make this work, you need to provide a standardised service description (that's called “resource record” in VO language) and then push that into the big pool of existing registry records. Depending on how many data collections you publish and how often they change, there are various ways to do that, and GettingIntoTheRegistry discusses them.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, SLAP, OAI-PMH, SIAv2, Datalink October 2018 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces). For recent developments, see DaCHS on the GAVO blog
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repositories for implementing IVOA standards: SSO, CDP, UWS, SIA, TAP, TAPRegExt, VOSpace, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.
Daiquiri Jochen Klar or E-Science AIP Potsdam TAP, Cone Search Oct 2019 Daiquiri - Python based framework for the publication of scientific databases - Daiquiri on Github. It is deployed on Gaia@AIP or APPLAUSE - Archives of Photographic PLates for Astronomical USE
Added:
>
>
CDS FITS validator Gilles landais (CDS) ObsCore Nov 2019 A web service base on the Saada API to check the FITS header VO-compatibility.
  (VO).


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 2018 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2018 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 2/2018 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile
MocInfo Pierre Fernique 2017 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools
Hipsgen Pierre Fernique 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for image or cube collection. See "How to make your HiPS in 10 steps" and HiPS CDS page.
Changed:
<
<
Hipsgen-cat François Xavier 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for catalogs (see HiPS CDS page)
>
>
Hipsgen-cat François Xavier 2018

HiPS generator tool allowing to create a Hierarchical Progressive Survey for catalogs (see HiPS CDS page)

 


<--  
-->

Revision 222019-10-12 - AnastasiaGalkin

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, the "yellow pages" of the VO, queried by user applications (and other things) when looking for services.

To make this work, you need to provide a standardised service description (that's called “resource record” in VO language) and then push that into the big pool of existing registry records. Depending on how many data collections you publish and how often they change, there are various ways to do that, and GettingIntoTheRegistry discusses them.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
Changed:
<
<
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, SLAP, OAI-PMH, SIAv2, Datalink October 2018 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces). For recent developments, see DaCHS on the GAVO blog
>
>
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, SLAP, OAI-PMH, SIAv2, Datalink October 2018 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces). For recent developments, see DaCHS on the GAVO blog
 
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repositories for implementing IVOA standards: SSO, CDP, UWS, SIA, TAP, TAPRegExt, VOSpace, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.
Added:
>
>
Daiquiri Jochen Klar or E-Science AIP Potsdam TAP, Cone Search Oct 2019 Daiquiri - Python based framework for the publication of scientific databases - Daiquiri on Github. It is deployed on Gaia@AIP or APPLAUSE - Archives of Photographic PLates for Astronomical USE
  (VO).


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 2018 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2018 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 2/2018 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile
MocInfo Pierre Fernique 2017 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools
Hipsgen Pierre Fernique 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for image or cube collection. See "How to make your HiPS in 10 steps" and HiPS CDS page.
Hipsgen-cat François Xavier 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for catalogs (see HiPS CDS page)



<--  
-->

Revision 212018-11-19 - MarkusDemleitner

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, the "yellow pages" of the VO, queried by user applications (and other things) when looking for services.

To make this work, you need to provide a standardised service description (that's called “resource record” in VO language) and then push that into the big pool of existing registry records. Depending on how many data collections you publish and how often they change, there are various ways to do that, and GettingIntoTheRegistry discusses them.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
Changed:
<
<
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, OAI-PMH March 2018 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)
>
>
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, SLAP, OAI-PMH, SIAv2, Datalink October 2018 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces). For recent developments, see DaCHS on the GAVO blog
 
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repositories for implementing IVOA standards: SSO, CDP, UWS, SIA, TAP, TAPRegExt, VOSpace, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.

(VO).


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 2018 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2018 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 2/2018 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile
MocInfo Pierre Fernique 2017 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools
Hipsgen Pierre Fernique 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for image or cube collection. See "How to make your HiPS in 10 steps" and HiPS CDS page.
Hipsgen-cat François Xavier 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for catalogs (see HiPS CDS page)



<--  
-->

Revision 202018-05-28 - PatrickDowler

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, the "yellow pages" of the VO, queried by user applications (and other things) when looking for services.

Changed:
<
<
To make this work, you need to provide a standardised service description (that's called “resource record” in VO language) and then push that into the big pool of existing registry records. Depending on how many data collections you publish and how often they change, there are various ways to do that, and GettingIntoTheRegistry discusses them.
>
>
To make this work, you need to provide a standardised service description (that's called “resource record” in VO language) and then push that into the big pool of existing registry records. Depending on how many data collections you publish and how often they change, there are various ways to do that, and GettingIntoTheRegistry discusses them.
 

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, OAI-PMH March 2018 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)
Changed:
<
<
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repository for implementing IVOA standards: UWS, TAP, TAPRegExt, CDP, VOSpace2, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
>
>
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repositories for implementing IVOA standards: SSO, CDP, UWS, SIA, TAP, TAPRegExt, VOSpace, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
 
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.

(VO).


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 2018 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2018 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 2/2018 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile
MocInfo Pierre Fernique 2017 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools
Hipsgen Pierre Fernique 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for image or cube collection. See "How to make your HiPS in 10 steps" and HiPS CDS page.
Hipsgen-cat François Xavier 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for catalogs (see HiPS CDS page)



<--  
-->

Revision 192018-05-28 - MarkusDemleitner

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

Changed:
<
<
A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, a "yellow page" or directory of service web locations.
>
>
A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, the "yellow pages" of the VO, queried by user applications (and other things) when looking for services.
 
Changed:
<
<
There are 2 methods for publishing your service into an IVOA registry.
  • The most common form is to enter your service information in a Fully Searchable Registry. Two systems are available:
>
>
To make this work, you need to provide a standardised service description (that's called “resource record” in VO language) and then push that into the big pool of existing registry records. Depending on how many data collections you publish and how often they change, there are various ways to do that, and GettingIntoTheRegistry discusses them.
Deleted:
<
<
 

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported
<-- -->
Sorted descending
Date of last Release Short Description
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, OAI-PMH March 2018 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repository for implementing IVOA standards: UWS, TAP, TAPRegExt, CDP, VOSpace2, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.

(VO).


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 2018 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2018 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 2/2018 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile
MocInfo Pierre Fernique 2017 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools
Hipsgen Pierre Fernique 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for image or cube collection. See "How to make your HiPS in 10 steps" and HiPS CDS page.
Hipsgen-cat François Xavier 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for catalogs (see HiPS CDS page)



<--  
-->

Revision 182018-05-27 - PierreFernique

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, a "yellow page" or directory of service web locations.

There are 2 methods for publishing your service into an IVOA registry.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, OAI-PMH March 2018 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repository for implementing IVOA standards: UWS, TAP, TAPRegExt, CDP, VOSpace2, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.

(VO).


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 2018 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2018 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 2/2018 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile
Changed:
<
<
MocInfo Pierre Fernique 2015 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
>
>
MocInfo Pierre Fernique 2017 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
 
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools
Added:
>
>
Hipsgen Pierre Fernique 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for image or cube collection. See "How to make your HiPS in 10 steps" and HiPS CDS page.
Hipsgen-cat François Xavier 2018 HiPS generator tool allowing to create a Hierarchical Progressive Survey for catalogs (see HiPS CDS page)
 


<--  
-->

Revision 172018-05-27 - DaveMorris

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, a "yellow page" or directory of service web locations.

There are 2 methods for publishing your service into an IVOA registry.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, OAI-PMH March 2018 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repository for implementing IVOA standards: UWS, TAP, TAPRegExt, CDP, VOSpace2, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
Deleted:
<
<
DSA AstroGrid     Layer on top of existing database providing Cone Search, (proto) TAP, and other capabilities, including IVOA Single Sign On authentication. However, TAP implementation is out of date, and software is no longer maintained.
 
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.

(VO).


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 2018 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2018 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 2/2018 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile
MocInfo Pierre Fernique 2015 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools



<--  
-->

Revision 162018-05-27 - TomDonaldson

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, a "yellow page" or directory of service web locations.

There are 2 methods for publishing your service into an IVOA registry.

Changed:
<
<
>
>
 
  • A secondary method is available via creation of your own Publishing Registry. This option is typically used for archive data centers which have many services. DaCHS (see Toolkit listing) has a publishing registry built in.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, OAI-PMH March 2018 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repository for implementing IVOA standards: UWS, TAP, TAPRegExt, CDP, VOSpace2, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
DSA AstroGrid     Layer on top of existing database providing Cone Search, (proto) TAP, and other capabilities, including IVOA Single Sign On authentication. However, TAP implementation is out of date, and software is no longer maintained.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.

(VO).


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 2018 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2018 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 2/2018 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile
MocInfo Pierre Fernique 2015 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools



<--  
-->

Revision 152018-05-27 - MarkTaylor

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, a "yellow page" or directory of service web locations.

There are 2 methods for publishing your service into an IVOA registry.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, OAI-PMH March 2018 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repository for implementing IVOA standards: UWS, TAP, TAPRegExt, CDP, VOSpace2, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
DSA AstroGrid     Layer on top of existing database providing Cone Search, (proto) TAP, and other capabilities, including IVOA Single Sign On authentication. However, TAP implementation is out of date, and software is no longer maintained.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.

(VO).


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
Changed:
<
<
STIL MarkTaylor 2017 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2017 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
>
>
STIL MarkTaylor 2018 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2018 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
 
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
Changed:
<
<
CDS UWS Library GregoryMantelet 5/2015 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 5/2015 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 5/2015 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
>
>
CDS UWS Library GregoryMantelet 2/2018 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 2/2018 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 2/2018 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
 
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile
MocInfo Pierre Fernique 2015 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools



<--  
-->

Revision 142018-04-26 - MarkusDemleitner

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, a "yellow page" or directory of service web locations.

There are 2 methods for publishing your service into an IVOA registry.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
Changed:
<
<
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, OAI-PMH August 2016 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)
>
>
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, OAI-PMH March 2018 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)
 
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repository for implementing IVOA standards: UWS, TAP, TAPRegExt, CDP, VOSpace2, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
DSA AstroGrid     Layer on top of existing database providing Cone Search, (proto) TAP, and other capabilities, including IVOA Single Sign On authentication. However, TAP implementation is out of date, and software is no longer maintained.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.

(VO).


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 2017 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2017 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
CDS UWS Library GregoryMantelet 5/2015 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 5/2015 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
CDS TAP Library GregoryMantelet 5/2015 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile
MocInfo Pierre Fernique 2015 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools



<--  
-->

Revision 132017-11-06 - GregoryMantelet

 

Publishing Data into the VO



0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.

  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.

  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.

When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.


1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, a "yellow page" or directory of service web locations.

There are 2 methods for publishing your service into an IVOA registry.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, OAI-PMH August 2016 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repository for implementing IVOA standards: UWS, TAP, TAPRegExt, CDP, VOSpace2, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
DSA AstroGrid     Layer on top of existing database providing Cone Search, (proto) TAP, and other capabilities, including IVOA Single Sign On authentication. However, TAP implementation is out of date, and software is no longer maintained.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.

(VO).


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 2017 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2017 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
Changed:
<
<
CDS UWS Library CDS cds-question (at) unistra.fr   A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library CDS cds-question (at) unistra.fr 4/2014 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
>
>
CDS UWS Library GregoryMantelet 5/2015 A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library GregoryMantelet 5/2015 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
Added:
>
>
CDS TAP Library GregoryMantelet 5/2015 Java framework to quickly and easily set up a TAP service. A such service lets query an astronomical database with ADQL queries. This library uses the CDS UWS Library and the CDS ADQL Library.
UCiDy GregoryMantelet 2/2017 Lightweight Java library to validate UCDs (Unified Content Descriptors). A documentation is provided on the Wiki of this GitHub repository. Stable releases are available in the Releases page.
 
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile
MocInfo Pierre Fernique 2015 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools



<--  
-->

Revision 122017-05-14 - MarcoMolinaro

Changed:
<
<

Publishing Data into the VO

>
>

Publishing Data into the VO

 
Deleted:
<
<
 
Added:
>
>
 
Changed:
<
<

IVOA Architecture

>
>

0. Introduction

 
Changed:
<
<
The IVOA Architecture Note (23 Nov 2010) can be a good starting point to explore the list of IVOA Standards, their function and the relationships between them.
>
>
The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.
 
Added:
>
>
Depending of your needs and your technical experience, there are several ways to publish you data into the VO:
 
Changed:
<
<

Workshops and other links about Publishing Data into the VO

>
>
  • if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the list of national VO projects and the selection of open data centers offering publication as a service.
 
Changed:
<
<
Euro-VO DCA and then Euro-VO AIDA used to run workshops oriented towards Data Providers to help them publishing their data into the VO. The latest one took place at ESAC in June 2009 and one can find all information at the following link. It included hands on tutorial on DALToolkit, SAADA and DSA. Previous similar workshop was run in 2008.
>
>
  • if you would like to find your VO services in client applications, please publish your service in a VO Registry.
 
Changed:
<
<
Euro-VO has a web page to some further useful links about how to publish data into the VO.
>
>
  • if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing VO Publishing toolkits.
Deleted:
<
<
 
Changed:
<
<

Data VO Publishing Tools

>
>
Added:
>
>
When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:
 
Changed:
<
<
Name Author Short Description
SAADA Laurent Michel Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer USVAO The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. This project is currently in the process of being moved to VAO; the current version is an older one from NVO.
>
>
  • please use the tools listed in IvoaValidatorsSummary to ensure your services actually behave according to the standards, and
  • remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.
Deleted:
<
<
DALToolkit ESA-VO  
DSA AstroGrid Layer on top of existing database providing Cone Search, (proto) TAP, and other capabilities, including IVOA Single Sign On authentication. However, TAP implementation is out of date, and software is no longer maintained.
VO-DANCE VObs.it  
DaCHS GAVO The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)
OpenCADC Patrick Dowler CADC Open source software (Java, Python) repository for implementing IVOA standards: UWS, TAP, TAPRegExt, CDP, VOSpace2, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.

<--  
-->
 
Added:
>
>

1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

Q: How do I publish images?

A: Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the list of toolkits below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: SIAP Specification, Observation Data Model Core Components

Q: How do I publish spectra?

A: Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the list of toolkits below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on ObsTAP in the section on publishing images above.

Technical documentation: SSAP Specification

Q: How do I publish catalogues or generic data tables?

A: Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the list of toolkits below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See Developer's corner for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.

Technical documentation:

Q: I want to publish alerts of celestial transient events. How can I do that?

A: Use VOEvent. Look at the notes on VOEvent for details.

Technical documentation:

Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?

A: Quite a few of the publication toolkits discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?

A: Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several publication toolkits (e.g., Saada or DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?

A: Several publication toolkits are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, DSA, and DaCHS..

Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?

A: See Section 3 below.

Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?

A: Saada comes fairly close to a point-and-click publication infrastructure. If you have a Debian/Ubuntu machine, DaCHS lets you quickly install the infrastructure, although metadata definition requires a bit documentation-reading.

Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!

A: Some services are designed to reduce the effort of publishing data to the bare minimum:

  • VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at VO-Dance web page at IA2 for more information).
Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
  • VizieR is a reference service providing access to catalogues and tables, with a simple standardized description of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email cds-question (at) unistra.fr.
  • The GAVO Data Center publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
  • Also, the IA2 data center can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
  • The CAB-SVO Data Center also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to your national VO initiative for support.


2. Registry for VO Data Service Discovery

Q: How does my service appear in Client Applications?

A: Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, a "yellow page" or directory of service web locations.

There are 2 methods for publishing your service into an IVOA registry.

3. Toolkits to publish data into the VO

Toolkit Name Contact Point DAL Services Supported Date of last Release Short Description
SVOCat

Enrique Solano

SVO

ConeSearch (CS)

v1.0

Apr 2014

SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required.
MySpec-MyImg

Enrique Solano

SVO

Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP)

v1.0

Apr 2014

MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required.
SAADA Laurent Michel SIAP/SSAP/CSP/TAP

1.8.build15b

November 2015

Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer Doug Tody, USVAO SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated

1.0 dev
Apr 2014

The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData.
VO-Dance VObs.it, IA2 SCS, SIAP [v1.0], SSAP; TAP (using a different application) May 2016 (update) VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed and supported by a TAP_SCHEMA GUI manager. Please note the this publishing infrastructure is under revision, thus it is suggested to contact Marco Molinaro for further and more fresh information.
DaCHS MarkusDemleitner, GAVO SCS, SIAP, SSAP, TAP, OAI-PMH August 2016 The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)
OpenCADC Patrick Dowler CADC     Open source software (Java, Python) repository for implementing IVOA standards: UWS, TAP, TAPRegExt, CDP, VOSpace2, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
DALToolkit ESA-VO SIAP, SSAP June 2009 The ESAVO DALToolkit publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found here. Unfortunately, the DALToolkit has not been maintained since then.
DSA AstroGrid     Layer on top of existing database providing Cone Search, (proto) TAP, and other capabilities, including IVOA Single Sign On authentication. However, TAP implementation is out of date, and software is no longer maintained.
VAPE VObs.it SIAP [edu] 0.97 June 2014 VAPE is a web application for the publication of educational data in the Virtual Observatory. Use the link on first column to reach its main page if you want to register.
CASDA VO Tools
Binary
JamesDempsey, CSIRO TAP, SCS, SIAP [v2.0] 1.2
May 2016
A public use implementation of the TAP, cone search, datalink and SIAP virtual observatory protocols as used in the CSIRO ASKAP Science Data Archive. This Java application can be deployed against an existing database and configured interactively or via text files to allow access to catalogues and image metadata. See CASDA Data Access for a SODA implementation provided access to image files.

(VO).


4. Developer's corner : other useful software tools and libraries for VO development

Tool / Library Name Contact Point Date of last Release Short Description
STIL MarkTaylor 2017 Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable.
STILTS MarkTaylor 2017 Table command-line tools suite. Includes votlint (VOTable validator) and taplint (TAP service validator).
SAVOT AndreSchaaff 5/2014 Simple Access To VOTable, parsing / editing / writing VOTable / N.B. From 4.0, on GitHub
votable.js AndreSchaaff 6/2015 VOTable parsing (including base64) in Javascript
CDS UWS Library CDS cds-question (at) unistra.fr   A UWS (Universal Worker Service) is defined by IVOA (http://www.ivoa.net/Documents/UWS) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library.
CDS ADQL Library CDS cds-question (at) unistra.fr 4/2014 ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.
UCD assignation tool sebastien.derriere (at) astro.unistra.fr 10/2008 The interface allows to find the UCD corresponding to a description in natural langage
samp.js MarkTaylor 2013 Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the SAMP Web Profile
MocInfo Pierre Fernique 2015 MOC libraries. MOC (MultiOrder Coverage map) allows one to specify arbitrary sky regions. The goal is to provide a very fast comparison mechanism between coverage maps
JSAMP Mark Taylor 2015 Java library for incorporating SAMP into java applications. Also contains SAMP utilities and diagnostic tools



<--  
-->
 

Revision 112012-06-26 - root

 

Publishing Data into the VO



IVOA Architecture

The IVOA Architecture Note (23 Nov 2010) can be a good starting point to explore the list of IVOA Standards, their function and the relationships between them.

Workshops and other links about Publishing Data into the VO

Euro-VO DCA and then Euro-VO AIDA used to run workshops oriented towards Data Providers to help them publishing their data into the VO. The latest one took place at ESAC in June 2009 and one can find all information at the following link. It included hands on tutorial on DALToolkit, SAADA and DSA. Previous similar workshop was run in 2008.

Euro-VO has a web page to some further useful links about how to publish data into the VO.

Data VO Publishing Tools

Name Author Short Description
SAADA Laurent Michel Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer USVAO The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. This project is currently in the process of being moved to VAO; the current version is an older one from NVO.
DALToolkit ESA-VO  
DSA AstroGrid Layer on top of existing database providing Cone Search, (proto) TAP, and other capabilities, including IVOA Single Sign On authentication. However, TAP implementation is out of date, and software is no longer maintained.
VO-DANCE VObs.it  
DaCHS GAVO The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)
OpenCADC Patrick Dowler CADC Open source software (Java, Python) repository for implementing IVOA standards: UWS, TAP, TAPRegExt, CDP, VOSpace2, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.

Revision 102011-10-24 - MarkTaylor

 

Publishing Data into the VO



IVOA Architecture

The IVOA Architecture Note (23 Nov 2010) can be a good starting point to explore the list of IVOA Standards, their function and the relationships between them.

Workshops and other links about Publishing Data into the VO

Euro-VO DCA and then Euro-VO AIDA used to run workshops oriented towards Data Providers to help them publishing their data into the VO. The latest one took place at ESAC in June 2009 and one can find all information at the following link. It included hands on tutorial on DALToolkit, SAADA and DSA. Previous similar workshop was run in 2008.

Euro-VO has a web page to some further useful links about how to publish data into the VO.

Data VO Publishing Tools

Name Author Short Description
SAADA Laurent Michel Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer USVAO The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. This project is currently in the process of being moved to VAO; the current version is an older one from NVO.
DALToolkit ESA-VO  
Changed:
<
<
DSA AstroGrid  
>
>
DSA AstroGrid Layer on top of existing database providing Cone Search, (proto) TAP, and other capabilities, including IVOA Single Sign On authentication. However, TAP implementation is out of date, and software is no longer maintained.
 
VO-DANCE VObs.it  
DaCHS GAVO The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)
OpenCADC Patrick Dowler CADC Open source software (Java, Python) repository for implementing IVOA standards: UWS, TAP, TAPRegExt, CDP, VOSpace2, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.

<--  
-->

Revision 92011-10-16 - PatrickDowler

 

Publishing Data into the VO



IVOA Architecture

The IVOA Architecture Note (23 Nov 2010) can be a good starting point to explore the list of IVOA Standards, their function and the relationships between them.

Workshops and other links about Publishing Data into the VO

Euro-VO DCA and then Euro-VO AIDA used to run workshops oriented towards Data Providers to help them publishing their data into the VO. The latest one took place at ESAC in June 2009 and one can find all information at the following link. It included hands on tutorial on DALToolkit, SAADA and DSA. Previous similar workshop was run in 2008.

Euro-VO has a web page to some further useful links about how to publish data into the VO.

Data VO Publishing Tools

Name Author Short Description
SAADA Laurent Michel Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALServer USVAO The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. This project is currently in the process of being moved to VAO; the current version is an older one from NVO.
DALToolkit ESA-VO  
DSA AstroGrid  
VO-DANCE VObs.it  
DaCHS GAVO The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)
Changed:
<
<
>
>
OpenCADC Patrick Dowler CADC Open source software (Java, Python) repository for implementing IVOA standards: UWS, TAP, TAPRegExt, CDP, VOSpace2, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.
 
<--  
-->

Revision 82011-06-15 - DougTody

 

Publishing Data into the VO



IVOA Architecture

The IVOA Architecture Note (23 Nov 2010) can be a good starting point to explore the list of IVOA Standards, their function and the relationships between them.

Workshops and other links about Publishing Data into the VO

Euro-VO DCA and then Euro-VO AIDA used to run workshops oriented towards Data Providers to help them publishing their data into the VO. The latest one took place at ESAC in June 2009 and one can find all information at the following link. It included hands on tutorial on DALToolkit, SAADA and DSA. Previous similar workshop was run in 2008.

Euro-VO has a web page to some further useful links about how to publish data into the VO.

Data VO Publishing Tools

Name Author Short Description
SAADA Laurent Michel Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
Added:
>
>
DALServer USVAO The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. This project is currently in the process of being moved to VAO; the current version is an older one from NVO.
 
DALToolkit ESA-VO  
DSA AstroGrid  
VO-DANCE VObs.it  
DaCHS GAVO The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)


<--  
-->

Revision 72011-06-06 - MarkusDemleitner

 

Publishing Data into the VO



IVOA Architecture

The IVOA Architecture Note (23 Nov 2010) can be a good starting point to explore the list of IVOA Standards, their function and the relationships between them.

Workshops and other links about Publishing Data into the VO

Euro-VO DCA and then Euro-VO AIDA used to run workshops oriented towards Data Providers to help them publishing their data into the VO. The latest one took place at ESAC in June 2009 and one can find all information at the following link. It included hands on tutorial on DALToolkit, SAADA and DSA. Previous similar workshop was run in 2008.

Euro-VO has a web page to some further useful links about how to publish data into the VO.

Data VO Publishing Tools

Name Author Short Description
SAADA Laurent Michel Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALToolkit ESA-VO  
DSA AstroGrid  
VO-DANCE VObs.it  
Changed:
<
<
DaCHS GAVO  
>
>
DaCHS GAVO The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)
 
<--  
-->

Revision 62011-05-25 - MarkTaylor

 

Publishing Data into the VO



IVOA Architecture

The IVOA Architecture Note (23 Nov 2010) can be a good starting point to explore the list of IVOA Standards, their function and the relationships between them.

Workshops and other links about Publishing Data into the VO

Euro-VO DCA and then Euro-VO AIDA used to run workshops oriented towards Data Providers to help them publishing their data into the VO. The latest one took place at ESAC in June 2009 and one can find all information at the following link. It included hands on tutorial on DALToolkit, SAADA and DSA. Previous similar workshop was run in 2008.

Euro-VO has a web page to some further useful links about how to publish data into the VO.

Data VO Publishing Tools

Name Author Short Description
SAADA Laurent Michel Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
DALToolkit ESA-VO  
DSA AstroGrid  
VO-DANCE VObs.it  
Added:
>
>
DaCHS GAVO  
 
<--  
-->

Revision 52011-05-20 - LaurentMichel

 

Publishing Data into the VO



IVOA Architecture

The IVOA Architecture Note (23 Nov 2010) can be a good starting point to explore the list of IVOA Standards, their function and the relationships between them.

Workshops and other links about Publishing Data into the VO

Euro-VO DCA and then Euro-VO AIDA used to run workshops oriented towards Data Providers to help them publishing their data into the VO. The latest one took place at ESAC in June 2009 and one can find all information at the following link. It included hands on tutorial on DALToolkit, SAADA and DSA. Previous similar workshop was run in 2008.

Euro-VO has a web page to some further useful links about how to publish data into the VO.

Data VO Publishing Tools

Name Author Short Description
Changed:
<
<
SAADA Laurent Michel Regularly maintained
>
>
SAADA Laurent Michel Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.
 
DALToolkit ESA-VO  
DSA AstroGrid  
VO-DANCE VObs.it  


<--  
-->

Revision 42011-05-19 - RiccardoSmareglia

 

Publishing Data into the VO



IVOA Architecture

The IVOA Architecture Note (23 Nov 2010) can be a good starting point to explore the list of IVOA Standards, their function and the relationships between them.

Workshops and other links about Publishing Data into the VO

Euro-VO DCA and then Euro-VO AIDA used to run workshops oriented towards Data Providers to help them publishing their data into the VO. The latest one took place at ESAC in June 2009 and one can find all information at the following link. It included hands on tutorial on DALToolkit, SAADA and DSA. Previous similar workshop was run in 2008.

Euro-VO has a web page to some further useful links about how to publish data into the VO.

Data VO Publishing Tools

Name Author Short Description
SAADA Laurent Michel Regularly maintained
DALToolkit ESA-VO  
DSA AstroGrid  
Changed:
<
<
VO-DANCE VObs.it  
>
>
VO-DANCE VObs.it  
 
<--  
-->

Revision 32011-05-19 - ChristopheArviset

 

Publishing Data into the VO



IVOA Architecture

The IVOA Architecture Note (23 Nov 2010) can be a good starting point to explore the list of IVOA Standards, their function and the relationships between them.

Workshops and other links about Publishing Data into the VO

Euro-VO DCA and then Euro-VO AIDA used to run workshops oriented towards Data Providers to help them publishing their data into the VO. The latest one took place at ESAC in June 2009 and one can find all information at the following link. It included hands on tutorial on DALToolkit, SAADA and DSA. Previous similar workshop was run in 2008.

Euro-VO has a web page to some further useful links about how to publish data into the VO.

Data VO Publishing Tools

Name Author Short Description
Changed:
<
<
SAADA CDS, Laurent Michel Regularly maintained
>
>
SAADA Laurent Michel Regularly maintained
 
DALToolkit ESA-VO  
DSA AstroGrid  
Changed:
<
<
>
>
VO-DANCE VObs.it  
 
<--  
-->

Revision 22011-05-19 - ChristopheArviset

 

Publishing Data into the VO



IVOA Architecture

The IVOA Architecture Note (23 Nov 2010) can be a good starting point to explore the list of IVOA Standards, their function and the relationships between them.

Changed:
<
<

Workshops about Publishing Data into the VO

>
>

Workshops and other links about Publishing Data into the VO

 
Changed:
<
<
Euro-VO DCA and then Euro-VO AIDA used to run workshops oriented towards Data Providers to help them publishing their data into the VO. The latest one took place at ESAC in June 2009 and one can find all information at the following link. It included hands on tutorial on DALToolkit, SAADA and DSA.
>
>
Euro-VO DCA and then Euro-VO AIDA used to run workshops oriented towards Data Providers to help them publishing their data into the VO. The latest one took place at ESAC in June 2009 and one can find all information at the following link. It included hands on tutorial on DALToolkit, SAADA and DSA. Previous similar workshop was run in 2008.
Added:
>
>
Euro-VO has a web page to some further useful links about how to publish data into the VO.
 

Data VO Publishing Tools

Name Author Short Description
Changed:
<
<
SAADA CDS, Laurent Michel  
DALToolkit ESA-VO  
DSA AstroGrid  
>
>
SAADA CDS, Laurent Michel Regularly maintained
DALToolkit ESA-VO  
DSA AstroGrid  
 


<--  
-->

Revision 12011-05-19 - ChristopheArviset

 

Publishing Data into the VO



IVOA Architecture

The IVOA Architecture Note (23 Nov 2010) can be a good starting point to explore the list of IVOA Standards, their function and the relationships between them.

Workshops about Publishing Data into the VO

Euro-VO DCA and then Euro-VO AIDA used to run workshops oriented towards Data Providers to help them publishing their data into the VO. The latest one took place at ESAC in June 2009 and one can find all information at the following link. It included hands on tutorial on DALToolkit, SAADA and DSA.

Data VO Publishing Tools

Name Author Short Description
SAADA CDS, Laurent Michel  
DALToolkit ESA-VO  
DSA AstroGrid  


<--  
-->
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback