<--
DAL Future - discussion pageThis page is meant to gather opinions, feedback and proposals for DAL future (see Trieste 2016 Fall Interop presentation). The underlying question is what will be the main evolution of the DAL protocols in a near future (SIA, SODA, TAP, ADQL, DataLink, etc.). The main drivers are:
Data TypeAdd Time Domain ( HIGHEST Scientific priority: see CSP Time Series use cases)
Functionalities
Software DesignInterface design
| ||||||||
Added: | ||||||||
> > | Recently I discovered that calls to datalink+SODA, outside ObsCore and SIAv2 context are available in VizieR, but also in GAVO. Markus is now using in SIA 1.0 responses (see http://dc.zah.uni-heidelberg.de/lswscans/res/positions/siap/siap.xml?POS=311.41098458333335,30.723715361111108&SIZE=2.64&FORMAT=image/fits) It sounds reasonable and I can imagine we wil find more of that in the future. But here there are already fields with utypes Access.Reference in le SIA mai n part, it is now difficult/impossible to recognize that a given link will return Datalink or something else. | |||||||
Changed: | ||||||||
< < | Recently I discovered that calls to datalink+SODA, outside ObsCore and SIAv2 context are available in VizieR, but also in GAVO. Markus is now using in SIA 1.0 responses (see http://dc.zah.uni-heidelberg.de/lswscans/res/positions/siap/siap.xml?POS=311.41098458333335,30.723715361111108&SIZE=2.64&FORMAT=image/fits) It sounds reasonable and I can imagine we wil find more of that in the future. But here there are already fields with utypes Access.Reference in le SIA mai n part, it is now difficult/impossible to recognize that a given link will return Datalink or something else. | |||||||
> > | On the other side, the Datalink VOTable response doesn't contain any signature allowing to recognize a posteriori that it has a datalink content. That's a pity. This would have been harmless and could simplify everything. | |||||||
Changed: | ||||||||
< < | On the other side, the Datalink VOTable response doesn't contain any signature allowing to recognize a posteriori that it has a datalink content. That's a pity. This would have been harmless and could simplify everything. | |||||||
> > | This has direct consequence on Aladin v10 (or any other client), which has the code necessary to perform specific actions for DataLink., cannot benefit from all these standardisation efforts and is limited to display the DataLink result as a simple,VOTable without coordinates , or still worse in a Web navigator. | |||||||
Changed: | ||||||||
< < | This has direct consequence on Aladin v10 (or any other client), which has the code necessary to perform specific actions for DataLink., cannot benefit from all these standardisation efforts and is limited to display the DataLink result as a simple,VOTable without coordinates , or still worse in a Web navigator. | |||||||
> > | The good point is that we are now close to a real usage and no more in prototyping. THis kind of usage is probably wider than what was imagined initially for datalink Aladin is close to make something of it there is now a couple of servers which deliver these things. JUST sad that everybody is using it his own way with the consequency that it remains unusable, while nearly nothing is missing to fix that. | |||||||
Deleted: | ||||||||
< < | The good point is that we are now close to a real usage and no more in prototyping. THis kind of usage is probably wider than what was imagined initially for datalink Aladin is close to make something of it there is now a couple of servers which deliver these things. JUST sad that everybody is using it his own way with the consequency that it remains unusable, while nearly nothing is missing to fix that. | |||||||
So, I would recommend two changes: | ||||||||
Changed: | ||||||||
< < | -A simple method to identify the links and to discover what they are supposed to return | |||||||
> > | -A simple method to identify the links and to discover what they are supposed to return | |||||||
Changed: | ||||||||
< < | solution 1: a specific utype for datalink URL (ex: utype=Access.Reference.Datalink.1.xxx") | |||||||
> > | solution 1: a specific utype for datalink URL (ex: utype=Access.Reference.Datalink.1.xxx") | |||||||
Changed: | ||||||||
< < | solution 2: an appropriate LINK in the FIELD definition (ex: LINK content-type="application-stream/votable;datalink" ...) | |||||||
> > | solution 2: an appropriate LINK in the FIELD definition (ex: LINK content-type="application-stream/votable;datalink" ...) | |||||||
Changed: | ||||||||
< < | -A signature in the VOTable, for example using an INFO tag or as an attribute of RESOURCE , RESOURCE type="result;datalink"... | |||||||
> > | -A signature in the VOTable, for example using an INFO tag or as an attribute of RESOURCE , RESOURCE type="result;datalink"... | |||||||
These "solutions" are just examples to illustrate the idea. WE have to check their validity with regard with the standard evolution. -- PierreFernique - 2017-03-07
| ||||||||
Deleted: | ||||||||
< < | ||||||||
Current DAL situation for dataset discovery and access | ||||||||
Changed: | ||||||||
< < | We have a core of 4 protocols SIAV2, ObsTAP, DataLink and SODA ? | |||||||
> > | We have a core of 4 protocols SIAV2, ObsTAP, DataLink and SODA ? | |||||||
Changed: | ||||||||
< < | ObsTAP is controled via ADQL and its response is an Obszcore table. SIAV2 is a parameter driven service, doesn't requite TAP infrastructure . In some respects it is a PQL ObsTAP (actually parameter language allows more evolution towards virtual data) | |||||||
> > | ObsTAP is controled via ADQL and its response is an Obszcore table. SIAV2 is a parameter driven service, doesn't requite TAP infrastructure . In some respects it is a PQL ObsTAP (actually parameter language allows more evolution towards virtual data) | |||||||
Changed: | ||||||||
< < | DataSets can be searched by any of the Spatial, time, band and polarization criteria. | |||||||
> > | DataSets can be searched by any of the Spatial, time, band and polarization criteria. | |||||||
Changed: | ||||||||
< < | Access is managed by SODA / currently only doing cutouts on ND cubes. | |||||||
> > | Access is managed by SODA / currently only doing cutouts on ND cubes. | |||||||
Changed: | ||||||||
< < | DataLink provides gluing facility between all these protocols responses and with other services | |||||||
> > | DataLink provides gluing facility between all these protocols responses and with other services | |||||||
Older protocol "lost "functionalities | ||||||||
Changed: | ||||||||
< < | SSA allows to discover spectra by Spatial and BAND positions / response in discovery mo | |||||||
> > | SSA allows to discover spectra by Spatial and BAND positions / response in discovery mo | |||||||
de standardized with SSA response (some kind of pre-Obscore) | ||||||||
Changed: | ||||||||
< < | SIAV1.0 allows to discover anything with a 2D signature on space, but only Spatial axes are standardized. | |||||||
> > | SIAV1.0 allows to discover anything with a 2D signature on space, but only Spatial axes are standardized. | |||||||
Changed: | ||||||||
< < | SSA and mostly SIAV1 have a "virtual data discovery mode" IN that case the retrieval is performing "Server side operations for data access" | |||||||
> > | SSA and mostly SIAV1 have a "virtual data discovery mode" IN that case the retrieval is performing "Server side operations for data access" | |||||||
Deleted: | ||||||||
< < | ||||||||
Possible evolution to extend the functionalities of the new protocol and tackle the TimeSeries CSP priority | ||||||||
Changed: | ||||||||
< < | SIAV2 interface could allow discovery of TimeSeries and Spectra with little extensions (time frequencies characterisation for example) | |||||||
> > | SIAV2 interface could allow discovery of TimeSeries and Spectra with little extensions (time frequencies characterisation for example) Some functionalities available in SSA/SIAV1.0 have to be added to SIAV2 | |||||||
Deleted: | ||||||||
< < | Some functionalities available in SSA/SIAV1.0 have to be added to SIAV2 | |||||||
Changed: | ||||||||
< < | Virtual data discovery : Basically the service arbitrates the discovery query parameters and propose a matching SODA URL. Specially usefull for TimeSeries where many time the TimeSeries is built from the data content. | |||||||
> > | Virtual data discovery : Basically the service arbitrates the discovery query parameters and propose a matching SODA URL. Specially usefull for TimeSeries where many time the TimeSeries is built from the data content. | |||||||
Changed: | ||||||||
< < | We could extend the parameters in ObsCore to tackle TimeSeries and spectra, add input paramaters to constrain that and add virtual data functionality | |||||||
> > | We could extend the parameters in ObsCore to tackle TimeSeries and spectra, add input paramaters to constrain that and add virtual data functionality | |||||||
Changed: | ||||||||
< < | This will be both an extension of SIAV2.0 and a new overall "DataSet | |||||||
> > | This will be both an extension of SIAV2.0 and a new overall "DataSet | |||||||
Changed: | ||||||||
< < | SODA = add spectra and Time Series functionalities | |||||||
> > | SODA = add spectra and Time Series functionalities | |||||||
Changed: | ||||||||
< < | Provide Extended metadata consistent with NDcube DM : is this a work for SIAV2 or for SODA 1.1 | |||||||
> > | Provide Extended metadata consistent with NDcube DM : is this a work for SIAV2 or for SODA 1.1 | |||||||
-- FrancoisBonnarel - 2017-05-15 | ||||||||
Deleted: | ||||||||
< < | ||||||||
| ||||||||
Added: | ||||||||
> > | Full knowledge of custom services is in the hands of these custom services developpers !!! Currently service descriptors are in the hands of data centers operating DAL discovery and/or DataLink services. | |||||||
Changed: | ||||||||
< < | Full knowledge of custom services is in the hands of these custom services developpers !!! | |||||||
> > | They may not know all the details on the service parameters | |||||||
Deleted: | ||||||||
< < | Currently service descriptors are in the hands of data centers operating DAL discovery and/or DataLink services. | |||||||
Changed: | ||||||||
< < | They may not know all the details on the service parameters | |||||||
> > | DataLink service operfors may not know the dataset metadata in detail | |||||||
Changed: | ||||||||
< < | DataLink service operfors may not know the dataset metadata in detail | |||||||
> > | It could be usefull to add in DataLink the feature that services autodescribe | |||||||
Deleted: | ||||||||
< < | It could be usefull to add in DataLink the feature that services autodescribe | |||||||
-- FrancoisBonnarel - 2017-05-15
Pushing code to the data
Formats and Languages
TAP evolution
| ||||||||
Added: | ||||||||
> > | TAP & Healpix[This part is a sum-up of the talk Bringing Healpix and MOC in TAP by G. Mantelet presented at the IVOA Interoperability meeting in May 2017 in Shanghai.] In TAP, with few extensions, it could be possible to get Healpix information and/or to add constraints on Healpix information. Proposed new features:
SELECT ivo_healpix_index(7, POINT(’’, ra, dec)) AS hpx_index, COUNT(*) AS density FROM tycho2 GROUP BY hpx_index
SELECT moc_agg(7, POINT(’’, ra, dec)) AS mymoc FROM tycho2 ...
SELECT * FROM tycho2 WHERE ivo_healpix_index(7, POINT(’’, ra, dec)) IN (12,23,68,69,70)
SELECT * FROM tycho2 WHERE 1= CONTAINS(POINT(’’, ra, dec), REGION(’2/12-20 5/60’))
SELECT t.* FROM tycho2 AS t JOIN TAP_UPLOAD.mymoc AS m ON 1=CONTAINS(POINT(’’, t.ra, t.dec), m.moc1)TAP_UPLOAD.mymoc is a normal uploaded VOTable table with a column named moc1 of type ‘VARCHAR’ and xtype 'MOC'.
SELECT t.* FROM tycho2 AS t JOIN TAP_UPLOAD.mymoc AS m ON 1=CONTAINS(POINT(’’, t.ra, t.dec), m.moc)Instead of uploading a VOTable, a FITS file would be uploaded (the TAP implementation has to allow that). The uploaded FITS file has special headers specifying that it represents neither an image nor a table, but a MOC. Then, it should be considered as such while used in the ADQL query. But TAP allows only the upload of table. So, in order to use the uploaded MOC, the TAP service has to create a table of only one cell containing the uploaded MOC (so, one cell for the entire FITS file). As for a "normal" upload, the name of the table is provided in the HTTP parameter UPLOAD, but there is no name for the column containing the single MOC and that we need to refer to in the ADQL query. To solve this issue, we could agree on a standard name for this column: let's say "moc". So, on the above example we had UPLOAD="mymoc,param:moc.fits" which has been uploaded as the table TAP_UPLOAD.mymoc with only row and one column named "moc". -- GregoryMantelet - 2017-05-29 | |||||||
Position in IVOA landscapeMerging DAL protocols, HiPS and MOC
Data Models
Updating SCS | ||||||||
Changed: | ||||||||
< < | SCS still requires VOTable 1.1 and has some other quirks that makes it needlessly incompatible with the rest of the DAL landscape (in particular, it's totally DALI-incompatible). We should figure out how we can evolve it to be less odd with minimal disruption to existing services and clients (e.g., relaxing VOTable requirements, support for DALI MAXREC, RESPONSEFORMAT, metadata discovery). -- MarkusDemleitner - 2017-03-01 | |||||||
> > | SCS still requires VOTable 1.1 and has some other quirks that makes it needlessly incompatible with the rest of the DAL landscape (in particular, it's totally DALI-incompatible). We should figure out how we can evolve it to be less odd with minimal disruption to existing services and clients (e.g., relaxing VOTable requirements, support for DALI MAXREC, RESPONSEFORMAT, metadata discovery). -- MarkusDemleitner - 2017-03-01 | |||||||
In SCS there are three things which are causing problems for me: | ||||||||
Changed: | ||||||||
< < | 1) The UCD's that are required by the spec are rather outdated. | |||||||
> > | 1) The UCD's that are required by the spec are rather outdated. 2) VOTable is required to be 1.0 or 1.1, which is far behind the current 1.3. 3) It requires a column with ucd="ID_MAIN". In UCD1+, this would be meta.id. We do not always have a column with that ucd, but we do have one with ucd=meta.record. So I would propose that the table must have one of meta.id or meta.record. | |||||||
Deleted: | ||||||||
< < | 2) VOTable is required to be 1.0 or 1.1, which is far behind the current 1.3. 3) It requires a column with ucd="ID_MAIN". In UCD1+, this would be meta.id. We do not always have a column with that ucd, but we do have one with ucd=meta.record. So I would propose that the table must have one of meta.id or meta.record. | |||||||
-- WalterLandry - 2017-04-15 | ||||||||
Added: | ||||||||
> > |