*IVOA November 2020 Interoperability Meeting
*Data Access Layer - session 3
*Thursday 19 November 2020 - 13:30 UTC
Participants: 72

*François Bonnarel
*DataLink status report & discussion
Towards version 1.1.
Summary since May '20. (see slides for details, w/ links)

1. link_auth & link_authorized columns for authZ required on link and identity authZ (if authN).

2. OVERFLOW text made consistent w/ DALI (can be used for batch queries).

3. dataproduct_type:
    cannot be a parameter to media type, it's against RFC
    proposed new optional content_qualifier
	    using IVOA dataproduct_type vocabulary
		but also other vocabs if use cases require it

3 in discussion, all of the above 3 have a Pull Request in GitHub.

Templating mechanism for service descriptors: using RFC-6750 (https://tools.ietf.org/html/rfc6570 ), allows tackling ReST like services templating.
Also this has a PR on GitHub.

Issues raised at this Interop and earlier: (all under discussion)
    - Trey R.: porting DataLink FIELDS response in the service descriptor
    - Gregory D.-F.: move service desc. to VOTable (could solve the double specification inside DataLink) Paul H. +1 on moving serv.desc out of spec.
		    G. D.-F.: optional vs. required params are sort of trust into service
			it's inerintely entangled to VOTable. If we keep in tight to DataLink could be less flexible for changes. A new schema for descriptors could be useful. Back-compatibility has to be checked.
			[Paul H] could I point out  https://www.ivoa.net/documents/PDL/20140523/index.html it has not got much traction, but it was intended to solve this problem...
			A new version without the bits that people do not like could be thought about - I think that should be separate from the datalink standard. The lack of success of PDL suggests that putting this sort of stuff into the DataLink standard would mean the version with it in would take years to get to REC.

    - EPN-TAP Stéphane E.: behaviour issues (?)
    - Ada. N: new model of links semantics -> major change. Could model mapping strategy (to VOTable) solve it? [LM] Serv Desc in VOTable can use xsd@import, so that this schema can be used in other contexts.

Put here other issues if you feel they're missing.

Praise to review and comment the 4 PR above, then coordination to publish after merge.

Laurent M.: content_qualifier How do you make a difference between IVOA vocab and other VOCAB? Markus: In Datalink, the same way as for semantics: URIs are interpreted relative to http://www.ivoa.net/rdf/product-type.  Hence, #image is from there, http://foo.bar.baz/voc#oddity is from another vocabulary (though, personally, I'd discourage that).
François B.: dataproduct_type is the first use case, but the link can be independent of the source in the main table. When you have metadata records or views of provenance the dataproduct_type semantics could be not enough. Usage of full URI (see Markus above) could solve.
LM: same issue is for vocab in MANGO (distinguishing int./ext. vocabulary)
G. D.-F.: has use cases too for mixed int/ext vocabs. URI prefixing.

Trey R.: optional/required params (in building the URL) difficult to express, is the descriptor expressing that.
Templating would be useful to Firefly.
A URL taking a URL and both having parameters... how to
PD: it's currently not specified -> please add an issue on GitHub

Trey R: Maybe I missed this but it is still not clear to me if there is a way to tell the difference between the service descriptor specifying the it is defining a data product or it is defining another web page that should be opened by the client.

Trey R: I created two issue on GitHub

*Gregory Mantelet
*ADQL status report & discussion
Very few issues left to go new-PRec + RFC.

Work done:
    
    boolean type: grammar complexity for =1 & =True -> postponed (and PEG grammar will help)
    
    geometries coosys arg. (now optional) & multiple signatures for CIRCLE and POLYGON -> solved through Erratum-3 to ADQL-2.0
    
    UDF catalogue: PEN currently in RFC, when endorsed (EN) ADQL-2.1 will simply link it. Will also solve the coordinate system management.
    
Work in progress: fixing definition of set operations. Pull Request already contains a proposal (and examples). Example solution for MySQL and SQLServer (PostgreSQL behaves like ADQL, or viceversa): SQLServer solution suggested restriction of CTEs to root level of the statement only (also because use cases against this restriction couldn't be found).

TODO:
    CAST vs. Constructors. (TIMESTAMP, SMALLINT, POINT) - discussion after interop
    Intervals overlap: if possible in 2.1 otherwise move this to 2.2+
    UPPER: currently ILIKE and LOWER in 2.1. We should have it probably, issue to be raised
    
Implementations: CADC has started, DaCHS & VOLLT/ADQL-Lib has 2.1 (to be checked)

Appendix includes changes and statuses, plus future work

Igor C.: question on xmatch differences in the exposed examples for the boolean (slide 6).
Gregory: solution by parse/translate query
Markus: distance doesn't make it a lot worse than it already is with all the rewriting that happens in the geometry department (DaCHS has both q3c and pgsphere indices on top).

Theresa D.: SQLServer implementation at MAST has a lot of similar rewrites, seems good to move to 2.1, can test how these changes work together

[DaveM] Yay - good progress on this.
Regarding testing - does anyoine have a useable deployment of Oracle in Docker we could use for validation testing?

Long time since we worked on this, but if anyone wants to bring these containers  up to date happy to recieve PRs.
https://github.com/ivoa/cosmopterix

*François Bonnarel
*SIAv2 & SODA (evolution status)
Since May (slides w/ links and details)
SIAv2 & SODA will be ported to GitHub (in progress, SODA nearly there, SIAv2 might take longer)

10 SIA issues (mainly details and inconsistencies)
- Query by MOC could be a nice add to next version.
- extend it to other "obscore" datatypes

7 SODA issues
- pixel cutouts
- it's currently for a single datatype: cannot "SODA-out" an image from a cube, e.g.

Enhancements:
    - virtual data discovery in SIAv2
    - extension of SIA2 <<camp>> to whole obscore or to specialise extensions and have new query parameters. SSA-next as a SIA extension?

SODA:
    extraction of partial MS for visibility data?
V. Galluzzi: The possibility to extract at least given targets and/or spectral windows would satisfy most use cases.
	[Marco M.] what would be the query params there? positional + energy axis, plus ...?
V. Galluzzi: Yes, positional (or target name), energy and possibly correlation products (only XX and YY for those interested in total intensity, or all the four products in case you want to assess polarimetry )/polarization parameters (I, Q,...) 
		position, band, time, polarisation are already there, XX/YY et al. not. I suggest an issue on https://github.com/ivoa-std/SODA
		Ok, but it sounds strange, since polarization parameters (Stokes types) are actually defined from correlation products, e.g. I= (XX+YY)/2, Q=(XX-YY)/2 ( and so on, in the linear basis). Similarly, for circular basis it is: I=(RR+LL)/2 , Q=(RL+LR)/2... 
		Could be, but the SODA REC doesn't speak of XX&al. currently, need to check history records
		Just to finalize the comment: it is quite common to convert visibility cubes into Stokes only after the calibration, e.g. for VLASS calibrated data cubes; raw visibilities are mostly in the form XX, YY, ... or RR, LL...
			Marco M.: (addition), after quick read of SODA: all needed polarisation states are available, directly taken from ObsCore.
		
progress: finish porting, write PRs, discuss, provide new WDs?

G. D-F.: an all-sky cube. would be happy for SODA enhancement for operations there.