VAO SIAv2 Prototype Effort

This page shares information on the VAO Project's effort to prototype an SIAv2 service. The goal is to demostrate key features and uncover practical lessons that can be folded back into the SIAv2 standardization process.

Summer 2013 Prototype

In the spring 2013, the VAO, responding to feedback from the Spring 2013 Interop, developed a plan to build an SIAv2 prototype service by the Fall 2013 Interop. In particular, feedback from invited panelists representing leading archives resulted in a focus on basic dataset discovery and retrieval (over more advanced features, such as cutouts or reprojections). The VAO plan has four major phases:

  1. Updating the relevant specifications, namely the Image Data Model WD and the SIAv2 protocol WD.
  2. Defining scope and schedule for the delivered prototype
  3. Implementation using the VAO DalServer code base as the platform
  4. Testing and evaluation of the deployed prototype

Regarding the Documents Used as Baseline Specifications

It was assumed from the start that the relevant IVOA Working Drafts, namely the SIAv2 prototol specficiation and the Image Data Model specification it is based on--would continue to evolve throughout the development period. For project management, this raise two concerns

  • the scope of the effort would be apt to "creep" to keep up with the evolving document
  • it would be difficult to evaluate our prototype if the specification had evolved away from the version we started with.
To address this problem, it was clear that we needed to "freeze" the pair of documents at a reasonable intermediate version, and then carefully control further changes only as needed to address specific goals and defined scope of our prototype. Meanwhile, the official IVOA WDs can continue to evolve. Our so-called "VAO Baseline" versions are listed below; they are specially named and labelled to emphasize that these baseline versions do not represent the current versions by the official IVOA SIAv2 editors. Nor are they intended to represent alternative or competing versions. Instead, they should be considered a close approximation to a subset of the SIAv2 capabilties sufficient for gaining experience and insight into implementing a real SIAv2 service.

Details

*See note above Regarding the Documents Used as Baseline Specifications

Summary of Scope

This prototype is focused on data discovery and delivery via the queryData operation.

  • No cutout or data extraction will be supported; only the download of full single datasets (via the Access.Reference attribute) will be enabled.
    • Assume MODE=archival
  • Synchronous queries are supported only (/sync).
  • Assume INTERSECT=OVERLAYS (any image overlaping query region is returned)
  • The focus of our testing will be on the required query parameters
    • POS, SIZE, BAND, TIME, POL, FORMAT
    • no support for the optional parameters was planned; support for some optional parameters was included but not formally tested.
  • Output columns: required and recommended columns supported
    • Note use of VOTable GROUPs to associate related columns

Datasets Used

a plot of the locations of the image centers for the datasets in the test collectiona plot of the locations of the image centers for the datasets in the test collection

Our test collection is a diverse mix of 2-D and N-D images.

  • from ALMA, VLA, JVLA, Heracles, Califa, JCMT, JWST, CARMA
  • includes traditional 2D images, 3D radio cubes, moment maps, sparse images

We extracted and transformed the FITS metadata and loaded them into a MySQL database. This was supplemented with both researched and "made up" data. We noted where we needed to "make things up" (mainly identifiers) to help assess the requirements on data providers.

Results

Prototype service is deploy at http://vaosa-vm1.aoc.nrao.edu/ivoa-dal/siapv2/

  • IVOA colleagues are invited to submit queries to explore behavior
  • start with test queries below, edit query in your browser to try variations.
Source code:

The project and our assessment was summarized in our VAO Presentation at the IVOA 2013 Fall Interop. A more detailed report is expected here.

We assess our demo in light of the CSP-recommended use cases.

Test Queries

General Coverage Queries

Title return all images in a square region
Query /sync?REQUEST=querydata&POS=202.48416,+47.23055&SIZE=0.033
Should Return VOTable listing 4 heracles images of NGC 5194
Status Passed.

Title return all images in a rectangular region
Query /siapv2/sync?REQUEST=querydata&POS=244,10&SIZE=6,4
Should Return VOTable listing 3 califa images
Status Passed

Title return all images in a region with supported frame
Query /sync?REQUEST=querydata&POS=244,10;ICRS&SIZE=6,4
Should Return VOTable listing 3 califa images
Status Passed

Title restrict to spectral window
Query /sync?REQUEST=querydata&BAND=0.21/0.22
Should Return VOTable listing 5 NED images and 5 VLA images
Status Note: service complies with spec, but additional images returned due to null value handling

ID 3.1.5
Title restrict to sky region and spectral window
Query /sync?REQUEST=querydata&POS=21,+3.6;ICRS&SIZE=0.5&BAND=0.21/0.22
Should Return VOTable listing one NED image and 1 VLA image
Status Open Issue (?)

Title Time Coverage
Query /sync?REQUEST=querydata&TIME=1992-01-01/1997-01-01
Should Return VOTable listing two NED image and 2 VLA image
Status Note: service complies with spec, but additional images returned due to null value handling

Polarization

Title Stokes V coverage
Query /sync?REQUEST=querydata&POL=V
Should Return VOTable listing only images with Stokes V coverage (This may be a null set)
Status Passed

Title Any polarization
Query /sync?REQUEST=querydata&POL=any
Should Return VOTable listing all images
Status Passed

Title Stokes I polarization
Query /sync?REQUEST=querydata&POL=I
Should Return VOTable listing all images that represent measurements of total intensity. (Should include any images that do not explicitly measure polarization?)
Status  

General Protocol Testing

Title Erroneous Query
Query /sync?REQUEST=querydata&POS=asdfl&SIZE=0.1
Should Return VOTable with properly set INFO specifying error
Status Passed

Title return all images in a region with unsupported frame
Query /sync?REQUEST=querydata&POS=245,10;GALACTIC&SIZE=2,4
Should Return VOTable with error contents indicating unsupported frame
Status Query attempting an unsupported frame does not trigger notification that the frame is unsupported.

Title Metadata Query
Query /sync?REQUEST=querydata&FORMAT=metadata
/sync?REQUEST=querydata&maxrec=0
Should Return a VOTable with a fully specified table header but no rows
Status Passed

Title FITS Format Query
Query /sync?REQUEST=querydata&FORMAT=image/fits
Should Return a VOTable listing all our images
Status Passed

Title Unsupported Format Query
Query /sync?REQUEST=querydata&FORMAT=image/jpeg
/sync?REQUEST=querydata&FORMAT=graphic
Should Return a VOTable listing no images (since we are not supporting this format)
Status Passed

Progress

03-Sep-2013

We have collected approximately a dozen FITS image cube datasets (including ones from JVLA, CARMA, JCMT, Chandra, and JWST) representing a variety of important types of features. We determined the set of metadata that we will capture for this data and expose through the service, and we have an initial version of the extracted metadata. The service is in development within the DALServer framework, set to be ready for testing by Sept. 6. Currently, the software currently supports creation of the output VOTable response to a queryData request; while the table is empty, the table columns are fully specified with the proper UTypes defined by the ImageDM.

13-Sep-2013

The completed software was delivered into our SVN repository on Sept. 9, and the service instance for evaluation has been deployed at http://vaosa-vm1.aoc.nrao.edu/ivoa-dal/siapv2/sync on Sept. 10. Testing is now underway, with bug reports resulting in code updates and redeployment of the service.

27-Sep-2013

The deployed service and the underlying software was "officially" released today (see details above). An incomplete list of issues to feed back into the standards development was summarized in our Presentation to the IVOA. A more complete report is planned.


Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf CubeSchedule-v2.pdf r1 manage 198.2 K 2013-09-03 - 16:24 RayPlante  
PDFpdf IVOASep13-VAO-SIAv2Prototype-S.pdf r1 manage 424.4 K 2013-09-27 - 22:14 RayPlante  
PDFpdf VAO-SIAP-2.0-20130830.pdf r1 manage 927.0 K 2013-09-03 - 16:26 RayPlante  
PDFpdf WD-ImageDM-20130811.pdf r1 manage 1120.8 K 2013-09-03 - 16:26 RayPlante  
PNGpng alldata-collections-dist.png r1 manage 7.9 K 2013-09-27 - 16:24 RayPlante rendering of distribution of datasets
PNGpng alldata-distribution.png r1 manage 16.3 K 2013-09-27 - 16:24 RayPlante rendering of distribution of datasets
Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r6 - 2013-09-30 - RayPlante
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback