DAL Future - discussion page

This page is meant to gather opinions, feedback and proposals for DAL future (see Trieste 2016 Fall Interop presentation).

The underlying question is what will be the main evolution of the DAL protocols in a near future (SIA, SODA, TAP, ADQL, DataLink, etc.).

The main drivers are:

1. CSP priorities : what is achieved and what is not achieved.
2. Feedback from implementation of recently recommended protocols (SIA, DataLink, SODA, TAP new version)
3. Multidimensional data access roadmap as established in october 2013 in Hawaï (Fall Interop + ADASS).

• Data Type
• Functionalities
• Software Design
• Position in IVOA landscape

Contribution can take the usual forms: linked pages to topics listed below (or added ones), mailing list follow up from the topics.

(please, when creating a page to link from a topic here, use a DALFutureTopic TWiki name for the new page)

Data Type

Add Time Domain ( HIGHEST Scientific priority: see CSP Time Series use cases)

• use cases
• ideas for solutions
• attempts and prototypes reports

Functionalities

• Feedback : what is missing/inadequate in current protocols
• Extended metadata (for Cubes and Time Domain)
• Extended server side data services (regridding, deconvolution, etc.)

Software Design

Interface design

• Feedback on current interfaces

Recently I discovered that calls to datalink+SODA, outside ObsCore and SIAv2 context are available in VizieR, but also in GAVO. Markus is now using in SIA 1.0 responses (see http://dc.zah.uni-heidelberg.de/lswscans/res/positions/siap/siap.xml?POS=311.41098458333335,30.723715361111108&SIZE=2.64&FORMAT=image/fits) It sounds reasonable and I can imagine we wil find more of that in the future. But here there are already fields with utypes Access.Reference in le SIA mai n part, it is now difficult/impossible to recognize that a given link will return Datalink or something else.

On the other side, the Datalink VOTable response doesn't contain any signature allowing to recognize a posteriori that it has a datalink content. That's a pity. This would have been harmless and could simplify everything.

This has direct consequence on Aladin v10 (or any other client), which has the code necessary to perform specific actions for DataLink., cannot benefit from all these standardisation efforts and is limited to display the DataLink result as a simple,VOTable without coordinates , or still worse in a Web navigator.

The good point is that we are now close to a real usage and no more in prototyping. THis kind of usage is probably wider than what was imagined initially for datalink Aladin is close to make something of it there is now a couple of servers which deliver these things. JUST sad that everybody is using it his own way with the consequency that it remains unusable, while nearly nothing is missing to fix that.

So, I would recommend two changes:

-A simple method to identify the links and to discover what they are supposed to return

solution 1: a specific utype for datalink URL (ex: utype=Access.Reference.Datalink.1.xxx")

solution 2: an appropriate LINK in the FIELD definition (ex: LINK content-type="application-stream/votable;datalink" ...)

-A signature in the VOTable, for example using an INFO tag or as an attribute of RESOURCE , RESOURCE type="result;datalink"...

These "solutions" are just examples to illustrate the idea. WE have to check their validity with regard with the standard evolution. -- PierreFernique - 2017-03-07

• Driving extended functionalities

Current DAL situation for dataset discovery and access

We have a core of 4 protocols SIAV2, ObsTAP, DataLink and SODA ?

ObsTAP is controled via ADQL and its response is an Obszcore table. SIAV2 is a parameter driven service, doesn't requite TAP infrastructure . In some respects it is a PQL ObsTAP (actually parameter language allows more evolution towards virtual data)

DataSets can be searched by any of the Spatial, time, band and polarization criteria.

Access is managed by SODA / currently only doing cutouts on ND cubes.

DataLink provides gluing facility between all these protocols responses and with other services

Older protocol "lost "functionalities

SSA allows to discover spectra by Spatial and BAND positions / response in discovery mo

de standardized with SSA response (some kind of pre-Obscore)

SIAV1.0 allows to discover anything with a 2D signature on space, but only Spatial axes are standardized.

SSA and mostly SIAV1 have a "virtual data discovery mode" IN that case the retrieval is performing "Server side operations for data access"

Possible evolution to extend the functionalities of the new protocol and tackle the TimeSeries CSP priority

SIAV2 interface could allow discovery of TimeSeries and Spectra with little extensions (time frequencies characterisation for example) Some functionalities available in SSA/SIAV1.0 have to be added to SIAV2

Virtual data discovery : Basically the service arbitrates the discovery query parameters and propose a matching SODA URL. Specially usefull for TimeSeries where many time the TimeSeries is built from the data content.

We could extend the parameters in ObsCore to tackle TimeSeries and spectra, add input paramaters to constrain that and add virtual data functionality

This will be both an extension of SIAV2.0 and a new overall "DataSet

SODA = add spectra and Time Series functionalities

Provide Extended metadata consistent with NDcube DM : is this a work for SIAV2 or for SODA 1.1

-- FrancoisBonnarel - 2017-05-15

• Standard definition of custom services

Full knowledge of custom services is in the hands of these custom services developpers !!! Currently service descriptors are in the hands of data centers operating DAL discovery and/or DataLink services.

They may not know all the details on the service parameters

DataLink service operfors may not know the dataset metadata in detail

It could be usefull to add in DataLink the feature that services autodescribe

-- FrancoisBonnarel - 2017-05-15

Pushing code to the data

• Science cases
• Attempts and prototypes

Formats and Languages

• Json, PQL, PDL, etc.
• What to use next ?

TAP evolution

• From relational databases to...
• Relaxing ADQL ?

Position in IVOA landscape

Merging DAL protocols, HiPS and MOC

• HiPS is also a discovery and access mode: How to marry service approach and HiPS Approach in both ways ?
• Querying services by MOC : is that necessary ?

Data Models

• ObsCore
• DataSet Metadata
• ND-Cube
• SparseCube
• STC
• other...

Updating SCS

SCS still requires VOTable 1.1 and has some other quirks that makes it needlessly incompatible with the rest of the DAL landscape (in particular, it's totally DALI-incompatible). We should figure out how we can evolve it to be less odd with minimal disruption to existing services and clients (e.g., relaxing VOTable requirements, support for DALI MAXREC, RESPONSEFORMAT, metadata discovery). -- MarkusDemleitner - 2017-03-01

In SCS there are three things which are causing problems for me:

1) The UCD's that are required by the spec are rather outdated. 2) VOTable is required to be 1.0 or 1.1, which is far behind the current 1.3. 3) It requires a column with ucd="ID_MAIN". In UCD1+, this would be meta.id. We do not always have a column with that ucd, but we do have one with ucd=meta.record. So I would propose that the table must have one of meta.id or meta.record.

-- WalterLandry - 2017-04-15