PublishingInTheVONew < IVOA

---+ Publishing Data into the VO

---

%TOC%
---

---++ 0. Introduction

The purpose of this page is to provide practical information about how to publish you data holdings into the VO. The content of this page is user contributed by members of the IVOA community and its content will evolve as IVOA standards and implementations evolve.

Depending of your needs and your technical experience, there are several ways to publish you data into the VO:

   * if you have very little technical expertise, you might want to contact your national VO projects who might already have the infrastructure to host your data and make it available to the VO. See the [[http://www.ivoa.net/about/member-organizations.html][list of national VO projects]] and the [[#PubService][selection of open data centers]] offering publication as a service.

   * if you would like to find your VO services in client applications, please publish your service in a [[#VOServiceDiscover][VO Registry]].

   * if you have some technical expertise but don't want to spend time to development, you might want to use directly some existing [[#VOPubToolkits][VO Publishing toolkits]].

   * if you prefer VO interfaces to your data yourself, you might find some useful [[#ToolsLibraries][VO software tools and libraries]]. A good start would be to determine what [[#QuestionsAnswers][type of data]] you want to publish (images, catalogues, spectra, ...). Reading through the [[http://www.ivoa.net/Documents/Notes/IVOAArchitecture][IVOA Architecture document]] might also be useful before going into the more details of the specific [[http://www.ivoa.net/Documents/][IVOA standards]] that you might need to use.
When publishing, please keep in mind that once your service is in the registry, it is part of a large distributed information system -- the VO. When your service fails or misbehaves, the user experience for the whole VO is degraded. So:

   * please use the [[http://wiki.ivoa.net/twiki/bin/view/IVOA/ServiceValidation][Validation Tools]] to ensure your services actually behave according to the standards, and
   * remove your service from the registry if you stop running it
If having consulted the resources described above or the FAQ below and you are still unsure on how to proceed, please send message to datapub (at) ivoa.net and we will endeavour to answer as soon as possible.

---

#QuestionsAnswers
---++ 1. Questions & Answers about what type of data to be published into the VO

The following section should help you figuring out what tools you need in order to publish your data in the Virtual Observatory. Since there are several tools that can be useful in different situations, we arranged this section to be requirements driven.

For completeness, you can also find links to the technical specifications of the different protocols involved. However, many of the tools presented in this page aim at abstracting the data publisher from the technicalities of the protocols.

#PublishImages *Q: How do I publish images?*

*A:* Images are handled in the VO using a protocol called Simple Image Access Protocol (SIAP). To publish such data, look for packages speaking SIAP in the [[#VOPubToolkits][list of toolkits]] below.

A more involved alternative -- suitable for complex data sets like data cubes, but right now with much less support in user tools -- is a publication using !ObsTAP, a fairly new standard that lets clients query metadata tables using SQL. To run an !ObsTAP service, you need a server speaking the Table Access Protocol TAP, and you need to understand the VO's Observation Data Model.

Technical documentation: [[http://www.ivoa.net/Documents/SIA/][SIAP Specification]], [[http://www.ivoa.net/Documents/ObsCore/index.html][Observation Data Model Core Components]]

*Q: How do I publish spectra?*

*A:* Spectra are handled in the VO using a protocol called Simple Spectral Access Protocol (SSAP). To publish such data, look for packages supporting SSAP in the [[#VOPubToolkits][list of toolkits]] below.

If you have data cubes rather than individual spectra, SSAP may not quite fit your bill. See the remarks on !ObsTAP in [[#PublishImages][the section on publishing images]] above.

Technical documentation: [[http://www.ivoa.net/Documents/SSA/][SSAP Specification]]

*Q: How do I publish catalogues or generic data tables?*

*A:* Simple tables containing data for celestial objects can be published to the VO using a protocol called Simple Cone Search (SCS). See the [[#VOPubToolkits][list of toolkits]] below for ready-made servers already supporting SCS. Also, it is fairly straightforward to write a simple SCS server, in particular when VOTable generation is already taken care of. See [[#ToolsLibraries][Developer's corner]] for leads.

A more complex interface capable of performing arbitrary queries using the Astronomical Data Query Language (ADQL) is called TAP.

Being flexible and generic, TAP is much more powerful but also more complex, while SCS has a more narrow scope and is thus much more simple.

If you do not want to install any code and just want to publish tables in the VO, contact the <nop>VizieR team as explained in the relevant question, at at cats (at) cdsarc.u-strasbg.fr or at the general CDS email !cds-question (at) !unistra.fr.

Technical documentation:
   * [[http://www.ivoa.net/Documents/latest/ConeSearch.html][Simple Cone Search Specification]]
   * [[http://www.ivoa.net/Documents/TAP/][TAP Specification]]
*Q: I want to publish alerts of celestial transient events. How can I do that?*

*A:* Use VOEvent. Look at [[http://wiki.ivoa.net/twiki/bin/view/IVOA/IvoaVOEvent#GETTING_STARTED][VOEvent, Getting Started]] for details.

Technical documentation:
   * [[http://www.ivoa.net/Documents/VOEvent/][VOEvent Specification]]
*Q: I want to publish a combination of generic data tables, spectra and images, possibly with links to each other. Are there toolkits/services for doing so?*

*A:* Quite a few of the [[#VOPubToolkits][publication toolkits]] discussed below support multiple protocols in one package. Since the protocols have been designed so they can run alongside each other, in all likelihood you will only have to run a single server.

*Q: I have data files (e.g. FITS files representing catalogues, spectra and/or images) and I want to publish them using VO protocols. What tools can I use to publish this data to the VO?*

*A:* Most of the tools build their services around metadata tables kept in relational databases. Getting this metadata (e.g., date of observation, wavelength range, area observed, etc) into such a database is called ingestion, and several [[#VOPubToolkits][publication toolkits]] (e.g., Saada or !DaCHS) have "ingestion components", i.e., programs that understand, say, FITS and let you map your headers to VO-compliant descriptions.

*Q: I have a working database and I want to publish the data therein in the VO. What tools can I use?*

*A:* Several [[#VOPubToolkits][publication toolkits]] are designed so as to work without modifications on existing tables or at least requiring no schema changes. In particular, look for VO-Dance, DALToolkit, and DSA.

*Q: I want to develop my own code to create VO services, but I would like to leverage existing libraries. Where can I find them?*

*A:* See Section 3 below.

*Q: I don't want to write any code. Are there applications I can install and configure in order to serve my data?*

*A:* Saada comes fairly close to a point-and-click publication infrastructure.

*Q: I don't want to write any code and I don't want to install anything. I just want to publish my data!*

*A:* Some services are designed to reduce the effort of publishing data to the bare minimum:
   * VO-Dance is able to connect to remote DBs and archives (once given proper authorizations) and publish VO services from those resources (have a look at [[http://ia2.oats.inaf.it/index.php/vodance][VO-Dance web page]] at IA2 for more information).
#PubService Also, some data centers ingest data from published papers or data provided by authors and create standard services that serve it:
   * [[http://vizier.u-strasbg.fr/][VizieR]] is a reference service providing access to catalogues and tables, with a [[http://cds.u-strasbg.fr/doc/submit.htx][simple standardized description]] of the table(s). More complex data items such as images, spectra, time series, can be added. You can contact the CDS team at cats (at) cdsarc.u-strasbg.fr or at the general CDS email !cds-question (at) <nop>unistra.fr.
   * The [[http://dc.g-vo.org][GAVO Data Center]] publishes data of all kinds of data (images, spectra, tables, cubes) for third parties; it also cares about the registration of your data. For more information, contact gavo (at) ari.uni-heidelberg.de
   * Also, the [[http://ia2.oats.inaf.it][IA2 data center]] can host and publish data from third parties (through VO-Dance and a Distributed Archiving System application). If you are interested contact ia2 (at) oats.inaf.it
   * The [[http://svo.cab.inta-csic.es][CAB-SVO Data Center]] also hosts and published data provided by the Spanish community. Should you are interested, contact svo-support@cab.inta-csic.es
You can also refer to [[http://www.ivoa.net/pub/members/][your national VO initiative]] for support.

---

#VOServiceDiscover
---++ 2. Registry for VO Data Service Discovery

*Q: How does my service appear in Client Applications?*

*A:* Once you have a valid VO data compliant service (see section 1), you have to publish your service in a IVOA registry, a "yellow page" or directory of service web locations.

There are 2 methods for publishing your service into an IVOA registry.
   * The most common form is to enter your service information in a Fully Searchable Registry. Two systems are available: 
      * [[ http://registry.euro-vo.org/insert.jsp ][EURO VO Registry resource insert page]]
      * [[ http://vao.stsci.edu/publishing/ ][VAO Registry publishing interface]]
   * A secondary method is available via creation of your own Publishing Registry. This option is typically used for archive data centers which have many services. !DaCHS (see [[#VOPubToolkits][Toolkit listing]]) has a publishing registry built in.
#VOPubToolkits
---++ 3. Toolkits to publish data into the VO

|  *Toolkit Name*  |  *Contact Point*  |  *DAL Services Supported*  |  *Date of last Release*  |  *Short Description*  |
|  [[http://svo.cab.inta-csic.es/docs/index.php?pagename=Projects/Astronomical_Data_Centres/Publishing][SVOCat]]  |  <p>Enrique Solano</p> <p>[[http://svo.cab.inta-csic.es][SVO]]</p>  | ConeSearch (CS) | <p>v1.0</p> <p>Apr 2014</p> | SVOCat is a tool developed by the Spanish VO to publish tables and catalogues. Through a series of configuration files, a web interface as well as a ConeSearch access is created. No code to write and little technical expertise is required. |
|  [[http://svo.cab.inta-csic.es/docs/index.php?pagename=Projects/Astronomical_Data_Centres/Publishing][MySpec-MyImg]]  |  <p>Enrique Solano</p> <p>[[http://svo.cab.inta- csic.es][SVO]]</p>  | Simple Spectral Access Protocol (SSAP) and Simple Image Access Protocol (SIAP) | <p>v1.0</p> <p>Apr 2014</p> | MySpec-MyImg is a tool developed by the Spanish VO to publish spectra and images. Through a series of configuration files, a web interface as well as a SIAP (images) or SSAP (spectra) access is created. No code to write and little technical expertise is required. |
|  [[http://saada.unistra.fr/saada/][SAADA]]  |  Laurent Michel  | SIAP/SSAP/CSP/TAP | <p>1.8.0</p> <p>March 2014</p> | Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API.  |
|  [[http://dev.usvao.org/vao/wiki/Products/DALServer][DALServer]]  |  Doug Tody, [[http://www.usvao.org][USVAO]]  | SIA/SSA/SCS/SLAP; SIAV2 prototype; TAP implemented but not yet integrated | <p>1.0 dev<br /><span style="background-color: transparent;">Apr 2014 </span></p> | The DALServer framework is intended to provide support for all VO data services within a common, easily user configurable or extensible framework implemented in Java. A production version of DALServer is currently under development within VAO; an older version was released by NVO. The current development version of DALServer (May2014) includes a prototype SIAV2 (and SIAV1) implementation supporting the draft ImageDM and advanced image access via AccessData. |
|  [[http://ia2.oats.inaf.it/index.php?option=com_content&view=section&id=20&Itemid=110][VO-Dance]]  |  [[http://vobs.astro.it/][VObs.it]]  |  SCS, SIAP, SSAP; TAP (using a different application)  | Dec. 2012 | VODance is a tool for rapid Virtual Observatory compliant services deployment. Data Access Layer services are created on the fly and published to VO without the need to move your data or DB metadata. A TAP dedicated application has been developed.  |
|  [[http://soft.g-vo.org/dachs][DaCHS]]  |  MarkusDemleitner, [[http://www.g-vo.org/][GAVO]]  | SCS, SIAP, SSAP, TAP, OAI-PMH | May 2014 | The Data Center Helper Suite is intended as an all-round data center software, including data ingestion (from, e.g., FITS, text files, binary data), unified metadata handling (including an OAI-PMH registry interface), and multi-protocol publication (including TAP, SCS, SIAP, SSAP, and web interfaces)  |
|  [[http://code.google.com/p/opencadc/][OpenCADC]]  |  Patrick Dowler [[http://www.cadc.hia.nrc.gc.ca/][CADC]]  | | | Open source software (Java, Python) repository for implementing IVOA standards: UWS, TAP, TAPRegExt, CDP, VOSpace2, VOSI. The repository also includes test code to test conformance of a service with the specification; these are generally designed to be used as unit or integration tests.  |
|  [[http://www.sciops.esa.int/index.php?project=SAT&page=dal_proj][DALToolkit]]  |  ESA-VO  |  SIAP, SSAP  |  June 2009  | The ESAVO [[http://www.sciops.esa.int/index.php?project=SAT&page=dal_proj][DALToolkit]] publishing software is a Java based server allowing you to publish astronomical images and spectra into the VO through the SIAP and SSAP Data Access Layer (DAL) protocols. Specific tutorials can be found [[http://cds.u-strasbg.fr/twikiAIDA/bin/view/EuroVOAIDA/AidaVOWS2009/AidaVOWS2009EsaDALToolKit][here]]. Unfortunately, the DALToolkit has not been maintained since then.  |
|  [[http://cds.u-strasbg.fr/twikiAIDA/bin/view/EuroVOAIDA/AidaVOWS2009/AidaVOWS2009AstrogridDsa][DSA]]  |  AstroGrid  | | | Layer on top of existing database providing Cone Search, (proto) TAP, and other capabilities, including IVOA Single Sign On authentication. However, TAP implementation is out of date, and software is no longer maintained. |

---

#ToolsLibraries
---++ 4. Developer's corner : other useful software tools and libraries for VO development

|  *Tool / Library Name*  |  *Contact Point*  |  *Date of last Release*  |  *Short Description*  |
| [[http://www.starlink.ac.uk/stil/][STIL]] | IVOA.MarkTaylor | 12/2012 | Table I/O and processing library in Java, including full VOTable I/O. Fast and scalable. |
| [[http://www.starlink.ac.uk/stilts/][STILTS]] | IVOA.MarkTaylor | 3/2013 | Table command-line tools suite. Includes [[http://www.starlink.ac.uk/stilts/sun256/votlint.html][votlint]] (VOTable validator) and [[http://www.starlink.ac.uk/stilts/sun256/taplint.html][taplint]] (TAP service validator). |
| [[http://cds.u-strasbg.fr/resources/doku.php?id=savot_v3.0][SAVOT]] | IVOA.AndreSchaaff | 4/2010 | Simple Access To VOTable, parsing / editing / writing VOTable / N.B. [[https://github.com/aschaaff/savot ][From 4.0, on GitHub]] |
| [[http://cds.u-strasbg.fr/resources/doku.php?id=uwslib][CDS UWS Library]] | CDS !cds-question (at) !unistra.fr | | A UWS (Universal Worker Service) is defined by IVOA ([[http://www.ivoa.net/Documents/UWS]]) as a web-service which lets execute one or several jobs in an asynchronous manner. To sum up, a UWS is a set of jobs lists. By sending requests, a user can create, start, stop or delete a job. It can also ask a summary of a job, get its result(s) or get the content of a jobs lists. This library is a convenient implementation of a UWS which already implements all behaviors and functionalities described by the IVOA recommendation v1.0, and has a complete set of documentation. It is designed to be as quick and easy to use as possible so that the developer of a UWS has not to worry with the UWS management. Basically, a developer has firstly to define jobs to use, and then, to create a HTTP servlet in which he creates its UWS and forwards it all requests. The library is implemented in the CDS TAP Library. |
| [[http://cds.u-strasbg.fr/resources/doku.php?id=adqllib][CDS ADQL Library]] | CDS !cds-question (at) unistra.fr | 4/2014 | ADQL (Astronomical Data Query Language) is a SQL-like language specifically designed to query VO services and particularly TAP. It is defined by the IVOA in the following document: http://www.ivoa.net/Documents/latest/ADQL.html. To sum up ADQL adds to the basic SQL syntax some functions to select sky regions. For instance: CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 10, 5, 2) = 1 corresponds to a cone search around (10°,5°) with a radius of 2°. This library lets parsing, editing and translating queries expressed in ADQL 2.0.  |
| [[http://cdsweb.u-strasbg.fr/UCD/cgi-bin/descr2ucd][UCD assignation tool]] | sebastien.derriere (at) astro.unistra.fr | 10/2008 | The interface allows to find the UCD corresponding to a description in natural langage  |
| [[http://astrojs.github.io/sampjs/][samp.js]] | MarkTaylor | 2013 | Small javascript library to allow web pages/web applications (for instance ones associated with data providers) to interact with client-side tools using the [[http://www.ivoa.net/documents/SAMP/][SAMP]] Web Profile |

---

<br /> <!--
      * Set ALLOWTOPICRENAME = IVOA.TWikiAdminGroup
-->
This topic: IVOA > PublishingInTheVONew
Topic revision: r35 - 2014-05-19 - LaurentMichel