<--
DAL Future - discussion pageThis page is meant to gather opinions, feedback and proposals for DAL future (see Trieste 2016 Fall Interop presentation). The underlying question is what will be the main evolution of the DAL protocols in a near future (SIA, SODA, TAP, ADQL, DataLink, etc.). The main drivers are:
Data TypeAdd Time Domain ( HIGHEST Scientific priority: see CSP Time Series use cases)
Functionalities
Software DesignInterface design
| ||||||||
Changed: | ||||||||
< < | ObsTAP is controled via ADQL and its response is an Obszcore table. SIAV2 is a parameter driven service, doesn't requite TAP infrastructure . In some respects it is a PQL ObsTAP (actually parameter language allows more evolution towards virtual data) | |||||||
> > | ObsTAP is controled via ADQL and its response is an Obscore table. SIAV2 is a parameter driven service, doesn't requite TAP infrastructure . In some respects it is a PQL ObsTAP (actually parameter language allows more evolution towards virtual data) | |||||||
DataSets can be searched by any of the Spatial, time, band and polarization criteria. Access is managed by SODA / currently only doing cutouts on ND cubes. DataLink provides gluing facility between all these protocols responses and with other services Older protocol "lost "functionalities SSA allows to discover spectra by Spatial and BAND positions / response in discovery mo de standardized with SSA response (some kind of pre-Obscore) SIAV1.0 allows to discover anything with a 2D signature on space, but only Spatial axes are standardized. SSA and mostly SIAV1 have a "virtual data discovery mode" IN that case the retrieval is performing "Server side operations for data access" Possible evolution to extend the functionalities of the new protocol and tackle the TimeSeries CSP priority SIAV2 interface could allow discovery of TimeSeries and Spectra with little extensions (time frequencies characterisation for example) Some functionalities available in SSA/SIAV1.0 have to be added to SIAV2 Virtual data discovery : Basically the service arbitrates the discovery query parameters and propose a matching SODA URL. Specially usefull for TimeSeries where many time the TimeSeries is built from the data content. We could extend the parameters in ObsCore to tackle TimeSeries and spectra, add input paramaters to constrain that and add virtual data functionality | ||||||||
Changed: | ||||||||
< < | This will be both an extension of SIAV2.0 and a new overall "DataSet | |||||||
> > | This will be both an extension of SIAV2.0 and a new overall "DataSet discovery" protocol. | |||||||
SODA = add spectra and Time Series functionalities | ||||||||
Changed: | ||||||||
< < | Provide Extended metadata consistent with NDcube DM : is this a work for SIAV2 or for SODA 1.1 | |||||||
> > | Provide Extended metadata consistent with NDcube DM : is this a work for SIAV2 or for SODA 1.1 ? | |||||||
Added: | ||||||||
> > | Extended metadata retrieval (any kind of full serialization of datamodels for a given dataproductype) is very close to retrieval of the dataset themselves (or excerpt/transformations of datasets). So it seems that this functionnality is more a SODA one than SIAV2 one... | |||||||
-- FrancoisBonnarel - 2017-05-15
Pushing code to the data
Formats and Languages
TAP evolution
TAP & Healpix[This part is a sum-up of the talk Bringing Healpix and MOC in TAP by G. Mantelet presented at the IVOA Interoperability meeting in May 2017 in Shanghai.] In TAP, with few extensions, it could be possible to get Healpix information and/or to add constraints on Healpix information. Proposed new features:
SELECT ivo_healpix_index(7, POINT(’’, ra, dec)) AS hpx_index, COUNT(*) AS density FROM tycho2 GROUP BY hpx_index
SELECT moc_agg(7, POINT(’’, ra, dec)) AS mymoc FROM tycho2 ...
SELECT * FROM tycho2 WHERE ivo_healpix_index(7, POINT(’’, ra, dec)) IN (12,23,68,69,70)
SELECT * FROM tycho2 WHERE 1= CONTAINS(POINT(’’, ra, dec), REGION(’2/12-20 5/60’))
SELECT t.* FROM tycho2 AS t JOIN TAP_UPLOAD.mymoc AS m ON 1=CONTAINS(POINT(’’, t.ra, t.dec), m.moc1)TAP_UPLOAD.mymoc is a normal uploaded VOTable table with a column named moc1 of type ‘VARCHAR’ and xtype 'MOC'.
SELECT t.* FROM tycho2 AS t JOIN TAP_UPLOAD.mymoc AS m ON 1=CONTAINS(POINT(’’, t.ra, t.dec), m.moc)Instead of uploading a VOTable, a FITS file would be uploaded (the TAP implementation has to allow that). The uploaded FITS file has special headers specifying that it represents neither an image nor a table, but a MOC. Then, it should be considered as such while used in the ADQL query. But TAP allows only the upload of table. So, in order to use the uploaded MOC, the TAP service has to create a table of only one cell containing the uploaded MOC (so, one cell for the entire FITS file). As for a "normal" upload, the name of the table is provided in the HTTP parameter UPLOAD, but there is no name for the column containing the single MOC and that we need to refer to in the ADQL query. To solve this issue, we could agree on a standard name for this column: let's say "moc". So, on the above example we had UPLOAD="mymoc,param:moc.fits" which has been uploaded as the table TAP_UPLOAD.mymoc with only row and one column named "moc". -- GregoryMantelet - 2017-05-29 Position in IVOA landscapeMerging DAL protocols, HiPS and MOC
Data Models
Updating SCSSCS still requires VOTable 1.1 and has some other quirks that makes it needlessly incompatible with the rest of the DAL landscape (in particular, it's totally DALI-incompatible). We should figure out how we can evolve it to be less odd with minimal disruption to existing services and clients (e.g., relaxing VOTable requirements, support for DALI MAXREC, RESPONSEFORMAT, metadata discovery). -- MarkusDemleitner - 2017-03-01 In SCS there are three things which are causing problems for me: 1) The UCD's that are required by the spec are rather outdated. 2) VOTable is required to be 1.0 or 1.1, which is far behind the current 1.3. 3) It requires a column with ucd="ID_MAIN". In UCD1+, this would be meta.id. We do not always have a column with that ucd, but we do have one with ucd=meta.record. So I would propose that the table must have one of meta.id or meta.record. -- WalterLandry - 2017-04-15 |