VAO SIAv2 Prototype Effort

This page shares information on the VAO Project's effort to prototype an SIAv2 service. The goal is to demostrate key features and uncover practical lessons that can be folded back into the SIAv2 standardization process.

Summer 2013 Prototype

In the spring 2013, the VAO, responding to feedback from the Spring 2013 Interop, developed a plan to build an SIAv2 prototype service by the Fall 2013 Interop. In particular, feedback from invited panelists representing leading archives resulted in a focus on basic dataset discovery and retrieval (over more advanced features, such as cutouts or reprojections). The VAO plan has four major phases:

1. Updating the relevant specifications, namely the Image Data Model WD and the SIAv2 protocol WD.
2. Defining scope and schedule for the delivered prototype
3. Implementation using the VAO DalServer code base as the platform
4. Testing and evaluation of the deployed prototype

Regarding the Documents Used as Baseline Specifications

It was assumed from the start that the relevant IVOA Working Drafts, namely the SIAv2 prototol specficiation and the Image Data Model specification it is based on--would continue to evolve throughout the development period. For project management, this raise two concerns

• the scope of the effort would be apt to "creep" to keep up with the evolving document
• it would be difficult to evaluate our prototype if the specification had evolved away from the version we started with.

To address this problem, it was clear that we needed to "freeze" the pair of documents at a reasonable intermediate version, and then carefully control further changes only as needed to address specific goals and defined scope of our prototype. Meanwhile, the official IVOA WDs can continue to evolve. Our so-called "VAO Baseline" versions are listed below; they are specially named and labelled to emphasize that these baseline versions do not represent the current versions by the official IVOA SIAv2 editors. Nor are they intended to represent alternative or competing versions. Instead, they should be considered a close approximation to a subset of the SIAv2 capabilties sufficient for gaining experience and insight into implementing a real SIAv2 service.

Details

• Requirements, Scope, and Schedule
• VAO Baseline Specifications*:
  • ImageDM
  • SIAv2
• Results:
  • VAO Presentation slides
  • see pointers listed below
• Test Queries: enumerated below

*See note above Regarding the Documents Used as Baseline Specifications

Summary of Scope

This prototype is focused on data discovery and delivery via the queryData operation.

• No cutout or data extraction will be supported; only the download of full single datasets (via the Access.Reference attribute) will be enabled.
  • Assume MODE=archival
• Synchronous queries are supported only (/sync).
• Assume INTERSECT=OVERLAYS (any image overlaping query region is returned)
• The focus of our testing will be on the required query parameters
  • POS, SIZE, BAND, TIME, POL, FORMAT
  • no support for the optional parameters was planned; support for some optional parameters was included but not formally tested.
• Output columns: required and recommended columns supported
  • Note use of VOTable GROUPs to associate related columns

Datasets Used

Our test collection is a diverse mix of 2-D and N-D images.

• from ALMA, VLA, JVLA, Heracles, Califa, JCMT, JWST, CARMA
• includes traditional 2D images, 3D radio cubes, moment maps, sparse images

We extracted and transformed the FITS metadata and loaded them into a MySQL database. This was supplemented with both researched and "made up" data. We noted where we needed to "make things up" (mainly identifiers) to help assess the requirements on data providers.

Results

Prototype service is deploy at http://vaosa-vm1.aoc.nrao.edu/ivoa-dal/siapv2/

• IVOA colleagues are invited to submit queries to explore behavior
• start with test queries below, edit query in your browser to try variations.

Source code:

• Browse on-line or download distribution tar-ball: dalserver-siav2proto.1.tar.gz
• "Friendly IVOAers" may build and deploy service on own (Linux) platform
  • Requires Apache, Java, Tomcat, MySQL, Python
  • Data files not included (thus, downloads wil not work)
  • see prototype/README
• A web page form is also available

The project and our assessment was summarized in our VAO Presentation at the IVOA 2013 Fall Interop. A more detailed report is expected here.

We assess our demo in light of the CSP-recommended use cases.

Test Queries

General Coverage Queries

Title	return all images in a square region
Query	/sync?REQUEST=querydata&POS=202.48416,+47.23055&SIZE=0.033
Should Return	VOTable listing 4 heracles images of NGC 5194
Status	Passed.

Title	return all images in a rectangular region
Query	/siapv2/sync?REQUEST=querydata&POS=244,10&SIZE=6,4
Should Return	VOTable listing 3 califa images
Status	Passed

Title	return all images in a region with supported frame
Query	/sync?REQUEST=querydata&POS=244,10;ICRS&SIZE=6,4
Should Return	VOTable listing 3 califa images
Status	Passed

Title	restrict to spectral window
Query	/sync?REQUEST=querydata&BAND=0.21/0.22
Should Return	VOTable listing 5 NED images and 5 VLA images
Status	Note: service complies with spec, but additional images returned due to null value handling

ID	3.1.5
Title	restrict to sky region and spectral window
Query	/sync?REQUEST=querydata&POS=21,+3.6;ICRS&SIZE=0.5&BAND=0.21/0.22
Should Return	VOTable listing one NED image and 1 VLA image
Status	Open Issue (?)

Title	Time Coverage
Query	/sync?REQUEST=querydata&TIME=1992-01-01/1997-01-01
Should Return	VOTable listing two NED image and 2 VLA image
Status	Note: service complies with spec, but additional images returned due to null value handling

Polarization

Title	Stokes V coverage
Query	/sync?REQUEST=querydata&POL=V
Should Return	VOTable listing only images with Stokes V coverage (This may be a null set)
Status	Passed

Title	Any polarization
Query	/sync?REQUEST=querydata&POL=any
Should Return	VOTable listing all images
Status	Passed

Title	Stokes I polarization
Query	/sync?REQUEST=querydata&POL=I
Should Return	VOTable listing all images that represent measurements of total intensity. (Should include any images that do not explicitly measure polarization?)
Status

General Protocol Testing

Title	Erroneous Query
Query	/sync?REQUEST=querydata&POS=asdfl&SIZE=0.1
Should Return	VOTable with properly set INFO specifying error
Status	Passed

Title	return all images in a region with unsupported frame
Query	/sync?REQUEST=querydata&POS=245,10;GALACTIC&SIZE=2,4
Should Return	VOTable with error contents indicating unsupported frame
Status	Query attempting an unsupported frame does not trigger notification that the frame is unsupported.

Title	Metadata Query
Query	/sync?REQUEST=querydata&FORMAT=metadata /sync?REQUEST=querydata&maxrec=0
Should Return	a VOTable with a fully specified table header but no rows
Status	Passed

Title	FITS Format Query
Query	/sync?REQUEST=querydata&FORMAT=image/fits
Should Return	a VOTable listing all our images
Status	Passed

Title	Unsupported Format Query
Query	/sync?REQUEST=querydata&FORMAT=image/jpeg /sync?REQUEST=querydata&FORMAT=graphic
Should Return	a VOTable listing no images (since we are not supporting this format)
Status	Passed

Progress

03-Sep-2013

We have collected approximately a dozen FITS image cube datasets (including ones from JVLA, CARMA, JCMT, Chandra, and JWST) representing a variety of important types of features. We determined the set of metadata that we will capture for this data and expose through the service, and we have an initial version of the extracted metadata. The service is in development within the DALServer framework, set to be ready for testing by Sept. 6. Currently, the software currently supports creation of the output VOTable response to a queryData request; while the table is empty, the table columns are fully specified with the proper UTypes defined by the ImageDM.

13-Sep-2013

The completed software was delivered into our SVN repository on Sept. 9, and the service instance for evaluation has been deployed at http://vaosa-vm1.aoc.nrao.edu/ivoa-dal/siapv2/sync on Sept. 10. Testing is now underway, with bug reports resulting in code updates and redeployment of the service.

27-Sep-2013

The deployed service and the underlying software was "officially" released today (see details above). An incomplete list of issues to feed back into the standards development was summarized in our Presentation to the IVOA. A more complete report is planned.