Difference: SIAP-2_0-Next (1 vs. 11)

Revision 112024-03-15 - FrancoisBonnarel

 
META TOPICPARENT name="IvoaDAL"

Common SIAP-2.0-Next and DAP1.0 page.

  • Previous material before May 2020 (most of it common with SODA-Next)
Old "DAL future" page : https://wiki.ivoa.net/twiki/bin/view/IVOA/DALFuture

IVOA note : http://www.ivoa.net/documents/Notes/RecentDALProtocolsFeedback/index.html

IVOA interop meetings presentation :

Paris : https://wiki.ivoa.net/internal/IVOA/InterOpMay2019DAL/SIA2-SODA-next.pdf

College Park : https://wiki.ivoa.net/internal/IVOA/InterOpNov2018DAL/HiPStoFits_Prototype_IVOA.pdf

Victoria : https://wiki.ivoa.net/internal/IVOA/InterOpMayy2018DAL/DAL-Feedback.pdf

  • Additional material since May 2020 (IVOA interop meetings and Running meetings - most of it common with SODA-Next)

We list presentations also look at Notes and discussions.

May 2020 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpMay2020DAL/SIA2-SODA-nextPyVO-support.pdf

November 2020 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpNov2020DAL/SIA2-SODA-next-status.pdf

January 2021 running meeting #7: https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/IVOA_DAL_RM7_etherpad.txt

February 2021 running meeting #8 : https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/IVOA_DAL_RM8_etherpad.txt

March 2021 running meeting #9 : https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/IVOA_DAL_RM9_etherpad.txt

May 2021 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpMay2021DAL/SIA2-nextDataSetDiscov.pdf

November 2021 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpNov2021DAL/minoc-soda-objectstore.pdf

April 2022 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpApr2022DAL/SIA2SODAUpgradesExtendingSIA2toDataSetSAP.pdf

September 2022 running meeting #14 : https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/DAP-SODA-DALRunningmeeting.pdf

October 2022 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpApr2022DAL/SIA2SODAUpgradesExtendingSIA2toDataSetSAP.pdf

March 2022 running meeting #15 : https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/DalRunning10-DL-DAP-SODA.pdf

May 2023 Bologna interop : https://wiki.ivoa.net/internal/IVOA/InterOpMay2023DAL/SIA-DAPSODA.pdf

Added:
>
>
November 2023 Tucson interop: https://wiki.ivoa.net/internal/IVOA/InterOpNov2023DAL/DAP-SODA-status.pdf
 

SIAP-2.0 feedback, issues and proposals (status in May 2020)

On GitHub: https://github.com/ivoa-std/SIA

Several SIAP2.0 parameters have a limited list of possible values – Some have lists limited by protocol (and obscore)

POL (Stokes, LINEAR, etc..) DPTYPE (image, cube, visibility, timeseries ;..) CALIB : levels FORMAT : fts, jpeg , png, etc..

– Some have free string values

COLLECTION (HST, WISE, etc...), FACILITY (VLT, Keck, Chandra), INSTRUMENT (ACS, MEGACAM, etc.)

– PARAMETERs less useful if we have no prior idea of their possible values. It often happens that services do not provide these lists of parameters although section 2.1.1 However Section 2.1.20 of SIAP2.0 states : Service PARAMETER self description Any service may include a DataLink service descriptor in the vOTable output to describe itself. this descriptor would describe the supported query parameters (standard and custom) including list of values for those with a fixed list (eg. COLLECTION INSTRUMENT FACILITY DPTYPE CALIB and FORMAT) *

This will allow to discover "possibilities" of the service prior to usage and optimize the queries

The "MAY" is obviously too weak. Version 2.1 could state SHOULD or MUST

Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)

ObsCore has an optional parameter "obs_release_date", but some projects complain no SIA PARAMETER allows to select on this.

Solution : create an OPTIONAL parameter to select on that ? What is the policy for OPTIONAL parameters ?

Done with SIA Pull request #16 . Merged in may 2022. (FrancoisBonnarel - 2023-05-31)

SIAP2 PyVO interface developers and CADC complain that no wild-carding or incomplete strong value can be admitted in string parameter. Only exact matching.

What is the price to pay to add this functionality and how do we allow it ?

Basically not admitted to avoid encoding issues. Ad hoc solution proposed for ivo identifiers. Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)

What is the price to pay to allow case-insensitive and how do we do that ?

Several string values PARAMETERS may be case insensitive, except where the "data model" forbids it (COLLECTION, FACILITY, INSTRUMENT). Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)

The spec clearly states that RA is in [0,360] and DEC is in [-90,90] but some POS=RANGE ... examples use -Inf and +Inf (probably copied the idea from use of open ended intervals in other params).

I consider this a bug in the example that should result in a fix and erratum because I recall we explicitly decided to not have services doing range reduction.

Done with SIA2.0 erratum 1 . Accepted 2023-04-05. (FrancoisBonnarel - 2023-05-31)

  • 1 shot discovery (and then access) to cutouts was possible in SIA1 but no more in SIA2 (github SIA issue #6)
SIAP1 had « cut out » and « mosaic » modes besides « archive » mode → 1 shot before access but only spatial We now have : SIAP2.0 or ObsTAP which deal with all axes

SODA : for cutouts only (all axes) +DataLink (Service descriptor and/or {links} table) → 2 shots before access (instead of 1)

Although it is pretty useful to have the 2 shots sometimes (allow the user to choose the actual limits according to ObsCore content and general context) there have been some complains (SkyView people for example also others) that the one-shot functionality disappeared. It would be perfectly possible to provide it by replacing the full retrieval or datalink url in access_url by a SODA url. SODA URL parameters are common with SIA. Where SIA Parameters values constrain the discovery, SODA parameters force the cutout dimensions.

A capability "virtual data generation" would have to be added to the service. If it exists how do we choose to provide these cutouts by a new "VIRTUAL" boolean parameter? Or by providing two lines in the response with the same obs_id ? The obscore values will be different, encluding publisher_did, but the observation will be the same

Two different ways of implementing RETRIEVEMODE PARAMETER (=VIRTUAL) can be considered. Using full SODA Queriy URL in access_url or using service descriptor with ref for all input parameters. Text change with two alternative solutions proposed in SIA Pull request #23 . Result can be seen in internal draft below. Still to be discussed. (FrancoisBonnarel - 2023-05-31)

Alberto Micol 2019/03/&4 :

SIA specifies:

"The FORMAT parameter specifies response format(s) (the access_format column from the ObsCore [7] data model). This column describes the format of the response from the access_url (see 3.1.3) so the values could be data file types (e.g. application/fits) or they could be the DataLink [8] MIME type."

The initial part of the sentence mentions "response format(s)", which that greatly overlaps,

and generates confusion (at least in my mind), with the following DALI specification:

"The RESPONSEFORMAT parameter is used so the client can specify the format of the response (e.g. the output of the job)."

The SIA FORMAT and the DALI RESPONSEFORMAT are two different things, and I think that should be better clarified

in the text of the next SIA version. Could that be done?

Done with SIA2.0 erratum 2 . Accepted 2023-04-05. (FrancoisBonnarel - 2023-05-31)

Extending VIrtual SIAP2 as described in issue re-gridding/rebinning along a specific WCS can be done by adding this functionality to the virtual data discovery functionality described in issue #6. The access URL would be a SODA URL with the additional parameters proposed in SODA issue #4

An alternative is allowing SIA1 parameters of the MOSAIC mode to SIA2 interface

This is related to a change in SODA. Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)

MOC (as well as STMOC, and maybe STEMOC) has proven to be a very efficient way to describe coverage of data and is used for discovery of data. It has been often discussed and also during last interop that it should be useful to query SIA services by MOC ? It should probably be with a new parameter and not via POS considering that it is probably more useful to have STMOC or STEMOC, than spatial MOC alone. This is also valid for SODA of course (SODA Issue #5). I wonder if this new parameter should not be standardized by DALI

Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)

There was a demand to be able to use a parameter based interface "à la" SIAP2 for other dataproduct_types than cube and image DPTYPE constraint should be relaxed eg: measurements time-series etc.... CADC actually created a service like this. What could be the name? SIA is no more appropriate For time-series this has been discussed in the following ivoa note: http://www.ivoa.net/documents/Notes/TimeSeriesDiscoveryAndAccess/index.html For some dataproduct_types a specific ObsCOre extension may be needed; For TimeSeries this was discussed in recent past. see for example:

https://wiki.ivoa.net/internal/IVOA/InterOpOct2017TDIG/IVOA2017Santiago-DiscussionTDIG-DM-DAL-Session1.pdf

or

http://volute.g-vo.org/svn/trunk/projects/time-domain/time-series/note/TSSerializationNote.pdf

Long discussion on this led to a new specification called DAP (see below) and result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)

* Update matching in-progress DataLink update ( GregoryDubois-Felsmann -2023-03-10)

The DataLink recommendation for how a self-description service descriptor should be written is changing, and this affects SIA because an example of such a self-description was included in the standard. (see SIA PR 22 ).

The proposed change looks needed. I suggest to repost this PR on the new DAP github repository. FrancoisBonnarel - 2023-05-31

DAP 1.0 proposal _(starting from may 2022 - FrancoisBonnarel - 2023-05-31)

Basically it was decided at the May 2022 interop to move to a new specification of a parameter based interface called 'Dataset Access Protocol" or "DAP".

The new github project starting from SIA2.0 is there : https://github.com/ivoa-std/DAP

All the changes in text from SIA2.0 to extend the scope of such a specification can be found in DAP PR #3. This PR also contains all the changes for SIA2.0 described above. The https://github.com/ivoa-std/SIA project is now idle and is there for legacy of issues and discussions. Result of all the changes can be seen in internal draft below.

FrancoisBonnarel - 2023-05-31

SODA-1.0 feedback, issues and proposals

Now available here : https://wiki.ivoa.net/twiki/bin/view/IVOA/SODA-1_0-Next


<--  
-->

META FILEATTACHMENT attachment="DAP.pdf" attr="" comment="first attempt transition SIA -- DAP" date="1684361972" name="DAP.pdf" path="DAP.pdf" size="646011" user="FrancoisBonnarel" version="2"

Revision 102024-03-13 - FrancoisBonnarel

 
META TOPICPARENT name="IvoaDAL"

Common SIAP-2.0-Next and DAP1.0 page.

  • Previous material before May 2020 (most of it common with SODA-Next)
Old "DAL future" page : https://wiki.ivoa.net/twiki/bin/view/IVOA/DALFuture

IVOA note : http://www.ivoa.net/documents/Notes/RecentDALProtocolsFeedback/index.html

IVOA interop meetings presentation :

Paris : https://wiki.ivoa.net/internal/IVOA/InterOpMay2019DAL/SIA2-SODA-next.pdf

College Park : https://wiki.ivoa.net/internal/IVOA/InterOpNov2018DAL/HiPStoFits_Prototype_IVOA.pdf

Victoria : https://wiki.ivoa.net/internal/IVOA/InterOpMayy2018DAL/DAL-Feedback.pdf

  • Additional material since May 2020 (IVOA interop meetings and Running meetings - most of it common with SODA-Next)

We list presentations also look at Notes and discussions.

May 2020 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpMay2020DAL/SIA2-SODA-nextPyVO-support.pdf

November 2020 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpNov2020DAL/SIA2-SODA-next-status.pdf

January 2021 running meeting #7: https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/IVOA_DAL_RM7_etherpad.txt

February 2021 running meeting #8 : https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/IVOA_DAL_RM8_etherpad.txt

March 2021 running meeting #9 : https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/IVOA_DAL_RM9_etherpad.txt

May 2021 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpMay2021DAL/SIA2-nextDataSetDiscov.pdf

November 2021 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpNov2021DAL/minoc-soda-objectstore.pdf

April 2022 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpApr2022DAL/SIA2SODAUpgradesExtendingSIA2toDataSetSAP.pdf

September 2022 running meeting #14 : https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/DAP-SODA-DALRunningmeeting.pdf

October 2022 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpApr2022DAL/SIA2SODAUpgradesExtendingSIA2toDataSetSAP.pdf

March 2022 running meeting #15 : https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/DalRunning10-DL-DAP-SODA.pdf

May 2023 Bologna interop : https://wiki.ivoa.net/internal/IVOA/InterOpMay2023DAL/SIA-DAPSODA.pdf

SIAP-2.0 feedback, issues and proposals (status in May 2020)

On GitHub: https://github.com/ivoa-std/SIA

Several SIAP2.0 parameters have a limited list of possible values – Some have lists limited by protocol (and obscore)

POL (Stokes, LINEAR, etc..) DPTYPE (image, cube, visibility, timeseries ;..) CALIB : levels FORMAT : fts, jpeg , png, etc..

– Some have free string values

COLLECTION (HST, WISE, etc...), FACILITY (VLT, Keck, Chandra), INSTRUMENT (ACS, MEGACAM, etc.)

– PARAMETERs less useful if we have no prior idea of their possible values. It often happens that services do not provide these lists of parameters although section 2.1.1 However Section 2.1.20 of SIAP2.0 states : Service PARAMETER self description Any service may include a DataLink service descriptor in the vOTable output to describe itself. this descriptor would describe the supported query parameters (standard and custom) including list of values for those with a fixed list (eg. COLLECTION INSTRUMENT FACILITY DPTYPE CALIB and FORMAT) *

This will allow to discover "possibilities" of the service prior to usage and optimize the queries

The "MAY" is obviously too weak. Version 2.1 could state SHOULD or MUST

Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)

ObsCore has an optional parameter "obs_release_date", but some projects complain no SIA PARAMETER allows to select on this.

Solution : create an OPTIONAL parameter to select on that ? What is the policy for OPTIONAL parameters ?

Done with SIA Pull request #16 . Merged in may 2022. (FrancoisBonnarel - 2023-05-31)

SIAP2 PyVO interface developers and CADC complain that no wild-carding or incomplete strong value can be admitted in string parameter. Only exact matching.

What is the price to pay to add this functionality and how do we allow it ?

Basically not admitted to avoid encoding issues. Ad hoc solution proposed for ivo identifiers. Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)

What is the price to pay to allow case-insensitive and how do we do that ?

Several string values PARAMETERS may be case insensitive, except where the "data model" forbids it (COLLECTION, FACILITY, INSTRUMENT). Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)

The spec clearly states that RA is in [0,360] and DEC is in [-90,90] but some POS=RANGE ... examples use -Inf and +Inf (probably copied the idea from use of open ended intervals in other params).

I consider this a bug in the example that should result in a fix and erratum because I recall we explicitly decided to not have services doing range reduction.

Done with SIA2.0 erratum 1 . Accepted 2023-04-05. (FrancoisBonnarel - 2023-05-31)

  • 1 shot discovery (and then access) to cutouts was possible in SIA1 but no more in SIA2 (github SIA issue #6)
SIAP1 had « cut out » and « mosaic » modes besides « archive » mode → 1 shot before access but only spatial We now have : SIAP2.0 or ObsTAP which deal with all axes

SODA : for cutouts only (all axes) +DataLink (Service descriptor and/or {links} table) → 2 shots before access (instead of 1)

Although it is pretty useful to have the 2 shots sometimes (allow the user to choose the actual limits according to ObsCore content and general context) there have been some complains (SkyView people for example also others) that the one-shot functionality disappeared. It would be perfectly possible to provide it by replacing the full retrieval or datalink url in access_url by a SODA url. SODA URL parameters are common with SIA. Where SIA Parameters values constrain the discovery, SODA parameters force the cutout dimensions.

A capability "virtual data generation" would have to be added to the service. If it exists how do we choose to provide these cutouts by a new "VIRTUAL" boolean parameter? Or by providing two lines in the response with the same obs_id ? The obscore values will be different, encluding publisher_did, but the observation will be the same

Two different ways of implementing RETRIEVEMODE PARAMETER (=VIRTUAL) can be considered. Using full SODA Queriy URL in access_url or using service descriptor with ref for all input parameters. Text change with two alternative solutions proposed in SIA Pull request #23 . Result can be seen in internal draft below. Still to be discussed. (FrancoisBonnarel - 2023-05-31)

Alberto Micol 2019/03/&4 :

SIA specifies:

"The FORMAT parameter specifies response format(s) (the access_format column from the ObsCore [7] data model). This column describes the format of the response from the access_url (see 3.1.3) so the values could be data file types (e.g. application/fits) or they could be the DataLink [8] MIME type."

The initial part of the sentence mentions "response format(s)", which that greatly overlaps,

and generates confusion (at least in my mind), with the following DALI specification:

"The RESPONSEFORMAT parameter is used so the client can specify the format of the response (e.g. the output of the job)."

The SIA FORMAT and the DALI RESPONSEFORMAT are two different things, and I think that should be better clarified

in the text of the next SIA version. Could that be done?

Done with SIA2.0 erratum 2 . Accepted 2023-04-05. (FrancoisBonnarel - 2023-05-31)

Extending VIrtual SIAP2 as described in issue re-gridding/rebinning along a specific WCS can be done by adding this functionality to the virtual data discovery functionality described in issue #6. The access URL would be a SODA URL with the additional parameters proposed in SODA issue #4

An alternative is allowing SIA1 parameters of the MOSAIC mode to SIA2 interface

This is related to a change in SODA. Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)

MOC (as well as STMOC, and maybe STEMOC) has proven to be a very efficient way to describe coverage of data and is used for discovery of data. It has been often discussed and also during last interop that it should be useful to query SIA services by MOC ? It should probably be with a new parameter and not via POS considering that it is probably more useful to have STMOC or STEMOC, than spatial MOC alone. This is also valid for SODA of course (SODA Issue #5). I wonder if this new parameter should not be standardized by DALI

Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)

There was a demand to be able to use a parameter based interface "à la" SIAP2 for other dataproduct_types than cube and image DPTYPE constraint should be relaxed eg: measurements time-series etc.... CADC actually created a service like this. What could be the name? SIA is no more appropriate For time-series this has been discussed in the following ivoa note: http://www.ivoa.net/documents/Notes/TimeSeriesDiscoveryAndAccess/index.html For some dataproduct_types a specific ObsCOre extension may be needed; For TimeSeries this was discussed in recent past. see for example:

https://wiki.ivoa.net/internal/IVOA/InterOpOct2017TDIG/IVOA2017Santiago-DiscussionTDIG-DM-DAL-Session1.pdf

or

http://volute.g-vo.org/svn/trunk/projects/time-domain/time-series/note/TSSerializationNote.pdf

Long discussion on this led to a new specification called DAP (see below) and result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)

* Update matching in-progress DataLink update ( GregoryDubois-Felsmann -2023-03-10)

The DataLink recommendation for how a self-description service descriptor should be written is changing, and this affects SIA because an example of such a self-description was included in the standard. (see SIA PR 22 ).

The proposed change looks needed. I suggest to repost this PR on the new DAP github repository. FrancoisBonnarel - 2023-05-31

DAP 1.0 proposal _(starting from may 2022 - FrancoisBonnarel - 2023-05-31)

Changed:
<
<
Basically it was decided at the May 2022 interop to move to a new specification of a parameter based interface called 'Dataset Access Protocol" or 3DAP".
>
>
Basically it was decided at the May 2022 interop to move to a new specification of a parameter based interface called 'Dataset Access Protocol" or "DAP".
 
Changed:
<
<
The new github project strating from SIA2.9 is there : https://github.com/ivoa-std/DAP
>
>
The new github project starting from SIA2.0 is there : https://github.com/ivoa-std/DAP
 
Changed:
<
<
All the changes in text from SIA2.0 to extend the scope of such a specification can be found in DAP PR #3. This PR also contains also all the changes for SIA2.0 described above. The https://github.com/ivoa-std/SIA project is now idel and is there for legacy if issues and discussions.
>
>
All the changes in text from SIA2.0 to extend the scope of such a specification can be found in DAP PR #3. This PR also contains all the changes for SIA2.0 described above. The https://github.com/ivoa-std/SIA project is now idle and is there for legacy of issues and discussions.
 Result of all the changes can be seen in internal draft below.

FrancoisBonnarel - 2023-05-31

Deleted:
<
<
 

SODA-1.0 feedback, issues and proposals

Now available here : https://wiki.ivoa.net/twiki/bin/view/IVOA/SODA-1_0-Next


<--  
-->

META FILEATTACHMENT attachment="DAP.pdf" attr="" comment="first attempt transition SIA -- DAP" date="1684361972" name="DAP.pdf" path="DAP.pdf" size="646011" user="FrancoisBonnarel" version="2"

Revision 92023-05-31 - FrancoisBonnarel

 
META TOPICPARENT name="IvoaDAL"

Common SIAP-2.0-Next and DAP1.0 page.

  • Previous material before May 2020 (most of it common with SODA-Next)
Old "DAL future" page : https://wiki.ivoa.net/twiki/bin/view/IVOA/DALFuture

IVOA note : http://www.ivoa.net/documents/Notes/RecentDALProtocolsFeedback/index.html

IVOA interop meetings presentation :

Paris : https://wiki.ivoa.net/internal/IVOA/InterOpMay2019DAL/SIA2-SODA-next.pdf

College Park : https://wiki.ivoa.net/internal/IVOA/InterOpNov2018DAL/HiPStoFits_Prototype_IVOA.pdf

Victoria : https://wiki.ivoa.net/internal/IVOA/InterOpMayy2018DAL/DAL-Feedback.pdf

  • Additional material since May 2020 (IVOA interop meetings and Running meetings - most of it common with SODA-Next)

We list presentations also look at Notes and discussions.

May 2020 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpMay2020DAL/SIA2-SODA-nextPyVO-support.pdf

November 2020 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpNov2020DAL/SIA2-SODA-next-status.pdf

January 2021 running meeting #7: https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/IVOA_DAL_RM7_etherpad.txt

February 2021 running meeting #8 : https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/IVOA_DAL_RM8_etherpad.txt

March 2021 running meeting #9 : https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/IVOA_DAL_RM9_etherpad.txt

May 2021 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpMay2021DAL/SIA2-nextDataSetDiscov.pdf

November 2021 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpNov2021DAL/minoc-soda-objectstore.pdf

April 2022 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpApr2022DAL/SIA2SODAUpgradesExtendingSIA2toDataSetSAP.pdf

September 2022 running meeting #14 : https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/DAP-SODA-DALRunningmeeting.pdf

October 2022 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpApr2022DAL/SIA2SODAUpgradesExtendingSIA2toDataSetSAP.pdf

March 2022 running meeting #15 : https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/DalRunning10-DL-DAP-SODA.pdf

May 2023 Bologna interop : https://wiki.ivoa.net/internal/IVOA/InterOpMay2023DAL/SIA-DAPSODA.pdf

SIAP-2.0 feedback, issues and proposals (status in May 2020)

On GitHub: https://github.com/ivoa-std/SIA

Several SIAP2.0 parameters have a limited list of possible values – Some have lists limited by protocol (and obscore)

POL (Stokes, LINEAR, etc..) DPTYPE (image, cube, visibility, timeseries ;..) CALIB : levels FORMAT : fts, jpeg , png, etc..

– Some have free string values

COLLECTION (HST, WISE, etc...), FACILITY (VLT, Keck, Chandra), INSTRUMENT (ACS, MEGACAM, etc.)

– PARAMETERs less useful if we have no prior idea of their possible values. It often happens that services do not provide these lists of parameters although section 2.1.1 However Section 2.1.20 of SIAP2.0 states : Service PARAMETER self description Any service may include a DataLink service descriptor in the vOTable output to describe itself. this descriptor would describe the supported query parameters (standard and custom) including list of values for those with a fixed list (eg. COLLECTION INSTRUMENT FACILITY DPTYPE CALIB and FORMAT) *

This will allow to discover "possibilities" of the service prior to usage and optimize the queries

The "MAY" is obviously too weak. Version 2.1 could state SHOULD or MUST

Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)

ObsCore has an optional parameter "obs_release_date", but some projects complain no SIA PARAMETER allows to select on this.

Solution : create an OPTIONAL parameter to select on that ? What is the policy for OPTIONAL parameters ?

Done with SIA Pull request #16 . Merged in may 2022. (FrancoisBonnarel - 2023-05-31)

SIAP2 PyVO interface developers and CADC complain that no wild-carding or incomplete strong value can be admitted in string parameter. Only exact matching.

What is the price to pay to add this functionality and how do we allow it ?

Basically not admitted to avoid encoding issues. Ad hoc solution proposed for ivo identifiers. Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)

What is the price to pay to allow case-insensitive and how do we do that ?

Several string values PARAMETERS may be case insensitive, except where the "data model" forbids it (COLLECTION, FACILITY, INSTRUMENT). Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)

The spec clearly states that RA is in [0,360] and DEC is in [-90,90] but some POS=RANGE ... examples use -Inf and +Inf (probably copied the idea from use of open ended intervals in other params).

I consider this a bug in the example that should result in a fix and erratum because I recall we explicitly decided to not have services doing range reduction.

Done with SIA2.0 erratum 1 . Accepted 2023-04-05. (FrancoisBonnarel - 2023-05-31)

  • 1 shot discovery (and then access) to cutouts was possible in SIA1 but no more in SIA2 (github SIA issue #6)
SIAP1 had « cut out » and « mosaic » modes besides « archive » mode → 1 shot before access but only spatial We now have : SIAP2.0 or ObsTAP which deal with all axes

SODA : for cutouts only (all axes) +DataLink (Service descriptor and/or {links} table) → 2 shots before access (instead of 1)

Although it is pretty useful to have the 2 shots sometimes (allow the user to choose the actual limits according to ObsCore content and general context) there have been some complains (SkyView people for example also others) that the one-shot functionality disappeared. It would be perfectly possible to provide it by replacing the full retrieval or datalink url in access_url by a SODA url. SODA URL parameters are common with SIA. Where SIA Parameters values constrain the discovery, SODA parameters force the cutout dimensions.

A capability "virtual data generation" would have to be added to the service. If it exists how do we choose to provide these cutouts by a new "VIRTUAL" boolean parameter? Or by providing two lines in the response with the same obs_id ? The obscore values will be different, encluding publisher_did, but the observation will be the same

Two different ways of implementing RETRIEVEMODE PARAMETER (=VIRTUAL) can be considered. Using full SODA Queriy URL in access_url or using service descriptor with ref for all input parameters. Text change with two alternative solutions proposed in SIA Pull request #23 . Result can be seen in internal draft below. Still to be discussed. (FrancoisBonnarel - 2023-05-31)

Alberto Micol 2019/03/&4 :

SIA specifies:

"The FORMAT parameter specifies response format(s) (the access_format column from the ObsCore [7] data model). This column describes the format of the response from the access_url (see 3.1.3) so the values could be data file types (e.g. application/fits) or they could be the DataLink [8] MIME type."

The initial part of the sentence mentions "response format(s)", which that greatly overlaps,

and generates confusion (at least in my mind), with the following DALI specification:

"The RESPONSEFORMAT parameter is used so the client can specify the format of the response (e.g. the output of the job)."

The SIA FORMAT and the DALI RESPONSEFORMAT are two different things, and I think that should be better clarified

in the text of the next SIA version. Could that be done?

Done with SIA2.0 erratum 2 . Accepted 2023-04-05. (FrancoisBonnarel - 2023-05-31)

Extending VIrtual SIAP2 as described in issue re-gridding/rebinning along a specific WCS can be done by adding this functionality to the virtual data discovery functionality described in issue #6. The access URL would be a SODA URL with the additional parameters proposed in SODA issue #4

An alternative is allowing SIA1 parameters of the MOSAIC mode to SIA2 interface

This is related to a change in SODA. Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)

MOC (as well as STMOC, and maybe STEMOC) has proven to be a very efficient way to describe coverage of data and is used for discovery of data. It has been often discussed and also during last interop that it should be useful to query SIA services by MOC ? It should probably be with a new parameter and not via POS considering that it is probably more useful to have STMOC or STEMOC, than spatial MOC alone. This is also valid for SODA of course (SODA Issue #5). I wonder if this new parameter should not be standardized by DALI

Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)

There was a demand to be able to use a parameter based interface "à la" SIAP2 for other dataproduct_types than cube and image DPTYPE constraint should be relaxed eg: measurements time-series etc.... CADC actually created a service like this. What could be the name? SIA is no more appropriate For time-series this has been discussed in the following ivoa note: http://www.ivoa.net/documents/Notes/TimeSeriesDiscoveryAndAccess/index.html For some dataproduct_types a specific ObsCOre extension may be needed; For TimeSeries this was discussed in recent past. see for example:

https://wiki.ivoa.net/internal/IVOA/InterOpOct2017TDIG/IVOA2017Santiago-DiscussionTDIG-DM-DAL-Session1.pdf

or

http://volute.g-vo.org/svn/trunk/projects/time-domain/time-series/note/TSSerializationNote.pdf

Long discussion on this led to a new specification called DAP (see below) and result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)

Added:
>
>
* Update matching in-progress DataLink update ( GregoryDubois-Felsmann -2023-03-10)

The DataLink recommendation for how a self-description service descriptor should be written is changing, and this affects SIA because an example of such a self-description was included in the standard. (see SIA PR 22 ).

The proposed change looks needed. I suggest to repost this PR on the new DAP github repository. FrancoisBonnarel - 2023-05-31

 

DAP 1.0 proposal _(starting from may 2022 - FrancoisBonnarel - 2023-05-31)

Basically it was decided at the May 2022 interop to move to a new specification of a parameter based interface called 'Dataset Access Protocol" or 3DAP".

The new github project strating from SIA2.9 is there : https://github.com/ivoa-std/DAP

All the changes in text from SIA2.0 to extend the scope of such a specification can be found in DAP PR #3. This PR also contains also all the changes for SIA2.0 described above. The https://github.com/ivoa-std/SIA project is now idel and is there for legacy if issues and discussions. Result of all the changes can be seen in internal draft below.

FrancoisBonnarel - 2023-05-31

Added:
>
>
 

SODA-1.0 feedback, issues and proposals

Now available here : https://wiki.ivoa.net/twiki/bin/view/IVOA/SODA-1_0-Next


<--  
-->

META FILEATTACHMENT attachment="DAP.pdf" attr="" comment="first attempt transition SIA -- DAP" date="1684361972" name="DAP.pdf" path="DAP.pdf" size="646011" user="FrancoisBonnarel" version="2"

Revision 82023-05-31 - FrancoisBonnarel

 
META TOPICPARENT name="IvoaDAL"
Changed:
<
<

SODA-1.0 cumulative"> SIAP-2.0 and SODA-1.0 cumulative "-Next" page.

>
>

Common SIAP-2.0-Next and DAP1.0 page.

 
Changed:
<
<
  • Previous material
>
>
  • Previous material before May 2020 (most of it common with SODA-Next)
 Old "DAL future" page : https://wiki.ivoa.net/twiki/bin/view/IVOA/DALFuture

IVOA note : http://www.ivoa.net/documents/Notes/RecentDALProtocolsFeedback/index.html

IVOA interop meetings presentation :

Paris : https://wiki.ivoa.net/internal/IVOA/InterOpMay2019DAL/SIA2-SODA-next.pdf

College Park : https://wiki.ivoa.net/internal/IVOA/InterOpNov2018DAL/HiPStoFits_Prototype_IVOA.pdf

Victoria : https://wiki.ivoa.net/internal/IVOA/InterOpMayy2018DAL/DAL-Feedback.pdf

Added:
>
>
  • Additional material since May 2020 (IVOA interop meetings and Running meetings - most of it common with SODA-Next)

We list presentations also look at Notes and discussions.

May 2020 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpMay2020DAL/SIA2-SODA-nextPyVO-support.pdf

November 2020 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpNov2020DAL/SIA2-SODA-next-status.pdf

January 2021 running meeting #7: https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/IVOA_DAL_RM7_etherpad.txt

February 2021 running meeting #8 : https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/IVOA_DAL_RM8_etherpad.txt

March 2021 running meeting #9 : https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/IVOA_DAL_RM9_etherpad.txt

May 2021 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpMay2021DAL/SIA2-nextDataSetDiscov.pdf

November 2021 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpNov2021DAL/minoc-soda-objectstore.pdf

April 2022 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpApr2022DAL/SIA2SODAUpgradesExtendingSIA2toDataSetSAP.pdf

September 2022 running meeting #14 : https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/DAP-SODA-DALRunningmeeting.pdf

October 2022 virtual interop : https://wiki.ivoa.net/internal/IVOA/InterOpApr2022DAL/SIA2SODAUpgradesExtendingSIA2toDataSetSAP.pdf

March 2022 running meeting #15 : https://wiki.ivoa.net/internal/IVOA/IvoaDAL_RunningMeetings/DalRunning10-DL-DAP-SODA.pdf

May 2023 Bologna interop : https://wiki.ivoa.net/internal/IVOA/InterOpMay2023DAL/SIA-DAPSODA.pdf

 

SIAP-2.0 feedback, issues and proposals (status in May 2020)

On GitHub: https://github.com/ivoa-std/SIA

Several SIAP2.0 parameters have a limited list of possible values – Some have lists limited by protocol (and obscore)

POL (Stokes, LINEAR, etc..) DPTYPE (image, cube, visibility, timeseries ;..) CALIB : levels FORMAT : fts, jpeg , png, etc..

– Some have free string values

COLLECTION (HST, WISE, etc...), FACILITY (VLT, Keck, Chandra), INSTRUMENT (ACS, MEGACAM, etc.)

– PARAMETERs less useful if we have no prior idea of their possible values. It often happens that services do not provide these lists of parameters although section 2.1.1 However Section 2.1.20 of SIAP2.0 states : Service PARAMETER self description Any service may include a DataLink service descriptor in the vOTable output to describe itself. this descriptor would describe the supported query parameters (standard and custom) including list of values for those with a fixed list (eg. COLLECTION INSTRUMENT FACILITY DPTYPE CALIB and FORMAT) *

This will allow to discover "possibilities" of the service prior to usage and optimize the queries

The "MAY" is obviously too weak. Version 2.1 could state SHOULD or MUST

Added:
>
>
Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)
  ObsCore has an optional parameter "obs_release_date", but some projects complain no SIA PARAMETER allows to select on this.

Solution : create an OPTIONAL parameter to select on that ? What is the policy for OPTIONAL parameters ?

Changed:
<
<
Done with https://github.com/ivoa-std/SIA/pull/16 . Merged in may 2022. (FrancoisBonnarel - 2023-05-31)
>
>
Done with SIA Pull request #16 . Merged in may 2022. (FrancoisBonnarel - 2023-05-31)
 
Changed:
<
<
  • No Wild-carding of the input PARAMETERS values exists
>
>
 SIAP2 PyVO interface developers and CADC complain that no wild-carding or incomplete strong value can be admitted in string parameter. Only exact matching.

What is the price to pay to add this functionality and how do we allow it ?

Added:
>
>
Basically not admitted to avoid encoding issues. Ad hoc solution proposed for ivo identifiers. Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)
  What is the price to pay to allow case-insensitive and how do we do that ?
Added:
>
>
Several string values PARAMETERS may be case insensitive, except where the "data model" forbids it (COLLECTION, FACILITY, INSTRUMENT). Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)
  The spec clearly states that RA is in [0,360] and DEC is in [-90,90] but some POS=RANGE ... examples use -Inf and +Inf (probably copied the idea from use of open ended intervals in other params).

I consider this a bug in the example that should result in a fix and erratum because I recall we explicitly decided to not have services doing range reduction.

Added:
>
>
Done with SIA2.0 erratum 1 . Accepted 2023-04-05. (FrancoisBonnarel - 2023-05-31)
 
  • 1 shot discovery (and then access) to cutouts was possible in SIA1 but no more in SIA2 (github SIA issue #6)
SIAP1 had « cut out » and « mosaic » modes besides « archive » mode → 1 shot before access but only spatial We now have : SIAP2.0 or ObsTAP which deal with all axes

SODA : for cutouts only (all axes) +DataLink (Service descriptor and/or {links} table) → 2 shots before access (instead of 1)

Although it is pretty useful to have the 2 shots sometimes (allow the user to choose the actual limits according to ObsCore content and general context) there have been some complains (SkyView people for example also others) that the one-shot functionality disappeared. It would be perfectly possible to provide it by replacing the full retrieval or datalink url in access_url by a SODA url. SODA URL parameters are common with SIA. Where SIA Parameters values constrain the discovery, SODA parameters force the cutout dimensions.

A capability "virtual data generation" would have to be added to the service. If it exists how do we choose to provide these cutouts by a new "VIRTUAL" boolean parameter? Or by providing two lines in the response with the same obs_id ? The obscore values will be different, encluding publisher_did, but the observation will be the same

Added:
>
>
Two different ways of implementing RETRIEVEMODE PARAMETER (=VIRTUAL) can be considered. Using full SODA Queriy URL in access_url or using service descriptor with ref for all input parameters. Text change with two alternative solutions proposed in SIA Pull request #23 . Result can be seen in internal draft below. Still to be discussed. (FrancoisBonnarel - 2023-05-31)
  Alberto Micol 2019/03/&4 :

SIA specifies:

"The FORMAT parameter specifies response format(s) (the access_format column from the ObsCore [7] data model). This column describes the format of the response from the access_url (see 3.1.3) so the values could be data file types (e.g. application/fits) or they could be the DataLink [8] MIME type."

The initial part of the sentence mentions "response format(s)", which that greatly overlaps,

and generates confusion (at least in my mind), with the following DALI specification:

"The RESPONSEFORMAT parameter is used so the client can specify the format of the response (e.g. the output of the job)."

The SIA FORMAT and the DALI RESPONSEFORMAT are two different things, and I think that should be better clarified

in the text of the next SIA version. Could that be done?

Added:
>
>
Done with SIA2.0 erratum 2 . Accepted 2023-04-05. (FrancoisBonnarel - 2023-05-31)
  Extending VIrtual SIAP2 as described in issue re-gridding/rebinning along a specific WCS can be done by adding this functionality to the virtual data discovery functionality described in issue #6. The access URL would be a SODA URL with the additional parameters proposed in SODA issue #4

An alternative is allowing SIA1 parameters of the MOSAIC mode to SIA2 interface

Added:
>
>
This is related to a change in SODA. Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)
 

MOC (as well as STMOC, and maybe STEMOC) has proven to be a very efficient way to describe coverage of data and is used for discovery of data. It has been often discussed and also during last interop that it should be useful to query SIA services by MOC ? It should probably be with a new parameter and not via POS considering that it is probably more useful to have STMOC or STEMOC, than spatial MOC alone. This is also valid for SODA of course (SODA Issue #5). I wonder if this new parameter should not be standardized by DALI

Added:
>
>
Text change proposed in SIA Pull request #23 . Result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)
 

There was a demand to be able to use a parameter based interface "à la" SIAP2 for other dataproduct_types than cube and image DPTYPE constraint should be relaxed eg: measurements time-series etc.... CADC actually created a service like this. What could be the name? SIA is no more appropriate For time-series this has been discussed in the following ivoa note: http://www.ivoa.net/documents/Notes/TimeSeriesDiscoveryAndAccess/index.html For some dataproduct_types a specific ObsCOre extension may be needed; For TimeSeries this was discussed in recent past. see for example:

https://wiki.ivoa.net/internal/IVOA/InterOpOct2017TDIG/IVOA2017Santiago-DiscussionTDIG-DM-DAL-Session1.pdf

or

http://volute.g-vo.org/svn/trunk/projects/time-domain/time-series/note/TSSerializationNote.pdf

Changed:
<
<

SODA-1.0 feedback, issues and proposals

>
>
Long discussion on this led to a new specification called DAP (see below) and result can be seen in internal draft below. (FrancoisBonnarel - 2023-05-31)
 
Changed:
<
<
On GitHub: https://github.com/ivoa-std/SODA
>
>

DAP 1.0 proposal _(starting from may 2022 - FrancoisBonnarel - 2023-05-31)

 
Changed:
<
<
  • Wrong VOTAble syntax in polarization example #1
>
>
Basically it was decided at the May 2022 interop to move to a new specification of a parameter based interface called 'Dataset Access Protocol" or 3DAP".
Deleted:
<
<
ON January the 16th 2020 Alberto Micol discovered :
 
Changed:
<
<
Page 19 of the SODA standard shows:
>
>
The new github project strating from SIA2.9 is there : https://github.com/ivoa-std/DAP
 
Changed:
<
<
'Polarization states to be extracted.
>
>
All the changes in text from SIA2.0 to extend the scope of such a specification can be found in DAP PR #3. This PR also contains also all the changes for SIA2.0 described above. The https://github.com/ivoa-std/SIA project is now idel and is there for legacy if issues and discussions.
Added:
>
>
Result of all the changes can be seen in internal draft below.
 
Changed:
<
<
>
>
FrancoisBonnarel - 2023-05-31
 
Deleted:
<
<
 
Deleted:
<
<
 
Changed:
<
<
My understanding of the VOTable standard is that the correct way should instead be
>
>

SODA-1.0 feedback, issues and proposals

 
Changed:
<
<
(see the sentence
>
>
Now available here : https://wiki.ivoa.net/twiki/bin/view/IVOA/SODA-1_0-Next
Deleted:
<
<
"All three MIN, MAX and OPTION sub-elements store their value corresponding to the minimum, maximum, or ``special value'' in a value attribute."

In the paragraph at: http://www.ivoa.net/documents/VOTable/20191021/REC-VOTable-1.4-20191021.html#sec:values ):

Polarization states to be extracted.

Alberto is perfectly right !!!

  • Wrong example for BAND interval MIN and MAX values #2

This appeared in a discussion between Alberto Micol and Markus on the dal Mailing list

MIN and MAX cannot be single values as in the example It should be an array. But an array of what ?

An array of minimal values and an array of maximal values (Markus)

An array of minimal length interval and an array of maximal length interval (Alberto)

  • pixel cutouts (instead of world coordinates) on all axes are missing #3

Considered for version 1.0 but delayed to stay simple (due to syntax problems) Asked by CASDA , others Ranges can be given by SODA service descriptor or computed from ObsCore

Which syntax ?

  • No possibility to control output WCS by regridding/rebinning exists in SODA. #4

This was possible in SIA1.0 for the spatial axis (only) could be done by ADDing standard parameters to SODA for Spatial resolution : SPATRES Rotation : ROTA Sky Projection : PROJ or Alternatively by providing a WCS = (full text of WCS part of FITS header)

Skyview + CDS (SIA on top of Hips2FITS) have this functionality

Should we stabdardize it ? And If yes should we extend it to SIA2 in virtual mode (SIA issue #8) ? Or should we let it as a mixture of SIAP2 and SIAP1 parameters ?

  • It is not possible to query SODA services by MOC #5

MOC (as well as STMOC, and maybe STEMOC) has proven to be a very efficient way to describe the coverage of data and is used for discovery of data. It has been often discussed and also during last interop that it should be useful to query SODA services by MOC? It should probably be with a new parameter and not via POS considering that it is probably more useful to have STMOC or STEMOC, than spatial MOC alone. This is also valid for SIA of course (SIA Issue #9). Of course, the behaviour for SODA will be very different than the one with SIA. SODA-(ST)MOC will extract a subset matching the (ST)MOC while SIA-MOC will only select datasets intersecting with the MOC. I wonder if this new parameter should not be standardized by DALI

  • Managing multiple PARAMS queries responses #6

When several of the PARAM queries (POS, BAND, etc...) are multiple we are facing a cartesian product issue: how do we retrieve all these results. Seems to be easy in async mode which is built to manage multiple results? But in sync? MEF formats? archive files? VOTable containing pointers? What about using UPLOAD functionality (DALI, 3.4.5 UPLOAD) for this?

  • SODA spec doesn't tell us how to provide dataproduct_type transformation #7

First implementations already provide various flavours of cutouts from cubes. Some cutout interface can extract subcubes from a large cube. Some others could create spectra or images (dimension reduction). An idea could be to use the DPTYPE parameter designed for SIAP2 to force this product_type conversion. This has been discussed for TimeSeries there: http://www.ivoa.net/documents/Notes/TimeSeriesDiscoveryAndAccess/index.html

 
<--  
-->

META FILEATTACHMENT attachment="DAP.pdf" attr="" comment="first attempt transition SIA -- DAP" date="1684361972" name="DAP.pdf" path="DAP.pdf" size="646011" user="FrancoisBonnarel" version="2"

Revision 72023-05-31 - FrancoisBonnarel

 
META TOPICPARENT name="IvoaDAL"

SODA-1.0 cumulative"> SIAP-2.0 and SODA-1.0 cumulative "-Next" page.

  • Previous material
Old "DAL future" page : https://wiki.ivoa.net/twiki/bin/view/IVOA/DALFuture

IVOA note : http://www.ivoa.net/documents/Notes/RecentDALProtocolsFeedback/index.html

IVOA interop meetings presentation :

Paris : https://wiki.ivoa.net/internal/IVOA/InterOpMay2019DAL/SIA2-SODA-next.pdf

College Park : https://wiki.ivoa.net/internal/IVOA/InterOpNov2018DAL/HiPStoFits_Prototype_IVOA.pdf

Victoria : https://wiki.ivoa.net/internal/IVOA/InterOpMayy2018DAL/DAL-Feedback.pdf

Changed:
<
<

SIAP-2.0 feedback, issues and proposals

>
>

SIAP-2.0 feedback, issues and proposals (status in May 2020)

  On GitHub: https://github.com/ivoa-std/SIA
Changed:
<
<
  • input PARAMETERS with limited list of values : better description #1
>
>
 Several SIAP2.0 parameters have a limited list of possible values – Some have lists limited by protocol (and obscore)

POL (Stokes, LINEAR, etc..) DPTYPE (image, cube, visibility, timeseries ;..) CALIB : levels FORMAT : fts, jpeg , png, etc..

– Some have free string values

COLLECTION (HST, WISE, etc...), FACILITY (VLT, Keck, Chandra), INSTRUMENT (ACS, MEGACAM, etc.)

– PARAMETERs less useful if we have no prior idea of their possible values. It often happens that services do not provide these lists of parameters although section 2.1.1 However Section 2.1.20 of SIAP2.0 states : Service PARAMETER self description Any service may include a DataLink service descriptor in the vOTable output to describe itself. this descriptor would describe the supported query parameters (standard and custom) including list of values for those with a fixed list (eg. COLLECTION INSTRUMENT FACILITY DPTYPE CALIB and FORMAT) *

This will allow to discover "possibilities" of the service prior to usage and optimize the queries

The "MAY" is obviously too weak. Version 2.1 could state SHOULD or MUST

Changed:
<
<
  • No input PARAMETER exists to select the RELEASE DATE #2
>
>
 ObsCore has an optional parameter "obs_release_date", but some projects complain no SIA PARAMETER allows to select on this.

Solution : create an OPTIONAL parameter to select on that ? What is the policy for OPTIONAL parameters ?

Added:
>
>
Done with https://github.com/ivoa-std/SIA/pull/16 . Merged in may 2022. (FrancoisBonnarel - 2023-05-31)
 
  • No Wild-carding of the input PARAMETERS values exists
SIAP2 PyVO interface developers and CADC complain that no wild-carding or incomplete strong value can be admitted in string parameter. Only exact matching.

What is the price to pay to add this functionality and how do we allow it ?

Changed:
<
<
  • input PARAMETERS values are case sensitive #4
>
>
 What is the price to pay to allow case-insensitive and how do we do that ?
Changed:
<
<
  • POS=RANGE examples inconsistent with spec #5 (From Pat Dowler)
>
>
 The spec clearly states that RA is in [0,360] and DEC is in [-90,90] but some POS=RANGE ... examples use -Inf and +Inf (probably copied the idea from use of open ended intervals in other params).

I consider this a bug in the example that should result in a fix and erratum because I recall we explicitly decided to not have services doing range reduction.

Changed:
<
<
  • 1 shot discovery (and then access) to cutouts was possible in SIA1 but no more in SIA2 #6
>
>
  • 1 shot discovery (and then access) to cutouts was possible in SIA1 but no more in SIA2 (github SIA issue #6)
 SIAP1 had « cut out » and « mosaic » modes besides « archive » mode → 1 shot before access but only spatial We now have : SIAP2.0 or ObsTAP which deal with all axes

SODA : for cutouts only (all axes) +DataLink (Service descriptor and/or {links} table) → 2 shots before access (instead of 1)

Although it is pretty useful to have the 2 shots sometimes (allow the user to choose the actual limits according to ObsCore content and general context) there have been some complains (SkyView people for example also others) that the one-shot functionality disappeared. It would be perfectly possible to provide it by replacing the full retrieval or datalink url in access_url by a SODA url. SODA URL parameters are common with SIA. Where SIA Parameters values constrain the discovery, SODA parameters force the cutout dimensions.

A capability "virtual data generation" would have to be added to the service. If it exists how do we choose to provide these cutouts by a new "VIRTUAL" boolean parameter? Or by providing two lines in the response with the same obs_id ? The obscore values will be different, encluding publisher_did, but the observation will be the same

Changed:
<
<
  • Possible confusion between FORMAT and RESPONSEFORMAT parameter #7
>
>
 Alberto Micol 2019/03/&4 :

SIA specifies:

"The FORMAT parameter specifies response format(s) (the access_format column from the ObsCore [7] data model). This column describes the format of the response from the access_url (see 3.1.3) so the values could be data file types (e.g. application/fits) or they could be the DataLink [8] MIME type."

The initial part of the sentence mentions "response format(s)", which that greatly overlaps,

and generates confusion (at least in my mind), with the following DALI specification:

"The RESPONSEFORMAT parameter is used so the client can specify the format of the response (e.g. the output of the job)."

The SIA FORMAT and the DALI RESPONSEFORMAT are two different things, and I think that should be better clarified

in the text of the next SIA version. Could that be done?

Changed:
<
<
  • SIA2 cannot discover rebinned data like SIA1 was able to do #8
>
>
 Extending VIrtual SIAP2 as described in issue re-gridding/rebinning along a specific WCS can be done by adding this functionality to the virtual data discovery functionality described in issue #6. The access URL would be a SODA URL with the additional parameters proposed in SODA issue #4

An alternative is allowing SIA1 parameters of the MOSAIC mode to SIA2 interface

Changed:
<
<
  • It is not possible to query SIA services by MOC #9
>
>
  MOC (as well as STMOC, and maybe STEMOC) has proven to be a very efficient way to describe coverage of data and is used for discovery of data. It has been often discussed and also during last interop that it should be useful to query SIA services by MOC ? It should probably be with a new parameter and not via POS considering that it is probably more useful to have STMOC or STEMOC, than spatial MOC alone. This is also valid for SODA of course (SODA Issue #5). I wonder if this new parameter should not be standardized by DALI
Changed:
<
<
  • extension of SIA-style protocol usage outside the image/cube "camp" #10
>
>
  There was a demand to be able to use a parameter based interface "à la" SIAP2 for other dataproduct_types than cube and image DPTYPE constraint should be relaxed eg: measurements time-series etc.... CADC actually created a service like this. What could be the name? SIA is no more appropriate For time-series this has been discussed in the following ivoa note: http://www.ivoa.net/documents/Notes/TimeSeriesDiscoveryAndAccess/index.html For some dataproduct_types a specific ObsCOre extension may be needed; For TimeSeries this was discussed in recent past. see for example:

https://wiki.ivoa.net/internal/IVOA/InterOpOct2017TDIG/IVOA2017Santiago-DiscussionTDIG-DM-DAL-Session1.pdf

or

http://volute.g-vo.org/svn/trunk/projects/time-domain/time-series/note/TSSerializationNote.pdf

SODA-1.0 feedback, issues and proposals

On GitHub: https://github.com/ivoa-std/SODA

  • Wrong VOTAble syntax in polarization example #1
ON January the 16th 2020 Alberto Micol discovered :

Page 19 of the SODA standard shows:

'Polarization states to be extracted.

My understanding of the VOTable standard is that the correct way should instead be

(see the sentence

"All three MIN, MAX and OPTION sub-elements store their value corresponding to the minimum, maximum, or ``special value'' in a value attribute."

In the paragraph at: http://www.ivoa.net/documents/VOTable/20191021/REC-VOTable-1.4-20191021.html#sec:values ):

Polarization states to be extracted.

Alberto is perfectly right !!!

  • Wrong example for BAND interval MIN and MAX values #2

This appeared in a discussion between Alberto Micol and Markus on the dal Mailing list

MIN and MAX cannot be single values as in the example It should be an array. But an array of what ?

An array of minimal values and an array of maximal values (Markus)

An array of minimal length interval and an array of maximal length interval (Alberto)

  • pixel cutouts (instead of world coordinates) on all axes are missing #3

Considered for version 1.0 but delayed to stay simple (due to syntax problems) Asked by CASDA , others Ranges can be given by SODA service descriptor or computed from ObsCore

Which syntax ?

  • No possibility to control output WCS by regridding/rebinning exists in SODA. #4

This was possible in SIA1.0 for the spatial axis (only) could be done by ADDing standard parameters to SODA for Spatial resolution : SPATRES Rotation : ROTA Sky Projection : PROJ or Alternatively by providing a WCS = (full text of WCS part of FITS header)

Skyview + CDS (SIA on top of Hips2FITS) have this functionality

Should we stabdardize it ? And If yes should we extend it to SIA2 in virtual mode (SIA issue #8) ? Or should we let it as a mixture of SIAP2 and SIAP1 parameters ?

  • It is not possible to query SODA services by MOC #5

MOC (as well as STMOC, and maybe STEMOC) has proven to be a very efficient way to describe the coverage of data and is used for discovery of data. It has been often discussed and also during last interop that it should be useful to query SODA services by MOC? It should probably be with a new parameter and not via POS considering that it is probably more useful to have STMOC or STEMOC, than spatial MOC alone. This is also valid for SIA of course (SIA Issue #9). Of course, the behaviour for SODA will be very different than the one with SIA. SODA-(ST)MOC will extract a subset matching the (ST)MOC while SIA-MOC will only select datasets intersecting with the MOC. I wonder if this new parameter should not be standardized by DALI

  • Managing multiple PARAMS queries responses #6

When several of the PARAM queries (POS, BAND, etc...) are multiple we are facing a cartesian product issue: how do we retrieve all these results. Seems to be easy in async mode which is built to manage multiple results? But in sync? MEF formats? archive files? VOTable containing pointers? What about using UPLOAD functionality (DALI, 3.4.5 UPLOAD) for this?

  • SODA spec doesn't tell us how to provide dataproduct_type transformation #7

First implementations already provide various flavours of cutouts from cubes. Some cutout interface can extract subcubes from a large cube. Some others could create spectra or images (dimension reduction). An idea could be to use the DPTYPE parameter designed for SIAP2 to force this product_type conversion. This has been discussed for TimeSeries there: http://www.ivoa.net/documents/Notes/TimeSeriesDiscoveryAndAccess/index.html


<--  
-->

META FILEATTACHMENT attachment="DAP.pdf" attr="" comment="first attempt transition SIA -- DAP" date="1684361972" name="DAP.pdf" path="DAP.pdf" size="646011" user="FrancoisBonnarel" version="2"

Revision 62023-05-17 - FrancoisBonnarel

 
META TOPICPARENT name="IvoaDAL"

SODA-1.0 cumulative"> SIAP-2.0 and SODA-1.0 cumulative "-Next" page.

  • Previous material
Old "DAL future" page : https://wiki.ivoa.net/twiki/bin/view/IVOA/DALFuture

IVOA note : http://www.ivoa.net/documents/Notes/RecentDALProtocolsFeedback/index.html

IVOA interop meetings presentation :

Paris : https://wiki.ivoa.net/internal/IVOA/InterOpMay2019DAL/SIA2-SODA-next.pdf

College Park : https://wiki.ivoa.net/internal/IVOA/InterOpNov2018DAL/HiPStoFits_Prototype_IVOA.pdf

Victoria : https://wiki.ivoa.net/internal/IVOA/InterOpMayy2018DAL/DAL-Feedback.pdf

SIAP-2.0 feedback, issues and proposals

On GitHub: https://github.com/ivoa-std/SIA

  • input PARAMETERS with limited list of values : better description #1
Several SIAP2.0 parameters have a limited list of possible values – Some have lists limited by protocol (and obscore)

POL (Stokes, LINEAR, etc..) DPTYPE (image, cube, visibility, timeseries ;..) CALIB : levels FORMAT : fts, jpeg , png, etc..

– Some have free string values

COLLECTION (HST, WISE, etc...), FACILITY (VLT, Keck, Chandra), INSTRUMENT (ACS, MEGACAM, etc.)

– PARAMETERs less useful if we have no prior idea of their possible values. It often happens that services do not provide these lists of parameters although section 2.1.1 However Section 2.1.20 of SIAP2.0 states : Service PARAMETER self description Any service may include a DataLink service descriptor in the vOTable output to describe itself. this descriptor would describe the supported query parameters (standard and custom) including list of values for those with a fixed list (eg. COLLECTION INSTRUMENT FACILITY DPTYPE CALIB and FORMAT) *

This will allow to discover "possibilities" of the service prior to usage and optimize the queries

The "MAY" is obviously too weak. Version 2.1 could state SHOULD or MUST

  • No input PARAMETER exists to select the RELEASE DATE #2
ObsCore has an optional parameter "obs_release_date", but some projects complain no SIA PARAMETER allows to select on this.

Solution : create an OPTIONAL parameter to select on that ? What is the policy for OPTIONAL parameters ?

  • No Wild-carding of the input PARAMETERS values exists
SIAP2 PyVO interface developers and CADC complain that no wild-carding or incomplete strong value can be admitted in string parameter. Only exact matching.

What is the price to pay to add this functionality and how do we allow it ?

  • input PARAMETERS values are case sensitive #4
What is the price to pay to allow case-insensitive and how do we do that ?

  • POS=RANGE examples inconsistent with spec #5 (From Pat Dowler)
The spec clearly states that RA is in [0,360] and DEC is in [-90,90] but some POS=RANGE ... examples use -Inf and +Inf (probably copied the idea from use of open ended intervals in other params).

I consider this a bug in the example that should result in a fix and erratum because I recall we explicitly decided to not have services doing range reduction.

  • 1 shot discovery (and then access) to cutouts was possible in SIA1 but no more in SIA2 #6
SIAP1 had « cut out » and « mosaic » modes besides « archive » mode → 1 shot before access but only spatial We now have : SIAP2.0 or ObsTAP which deal with all axes

SODA : for cutouts only (all axes) +DataLink (Service descriptor and/or {links} table) → 2 shots before access (instead of 1)

Although it is pretty useful to have the 2 shots sometimes (allow the user to choose the actual limits according to ObsCore content and general context) there have been some complains (SkyView people for example also others) that the one-shot functionality disappeared. It would be perfectly possible to provide it by replacing the full retrieval or datalink url in access_url by a SODA url. SODA URL parameters are common with SIA. Where SIA Parameters values constrain the discovery, SODA parameters force the cutout dimensions.

A capability "virtual data generation" would have to be added to the service. If it exists how do we choose to provide these cutouts by a new "VIRTUAL" boolean parameter? Or by providing two lines in the response with the same obs_id ? The obscore values will be different, encluding publisher_did, but the observation will be the same

  • Possible confusion between FORMAT and RESPONSEFORMAT parameter #7
Alberto Micol 2019/03/&4 :

SIA specifies:

"The FORMAT parameter specifies response format(s) (the access_format column from the ObsCore [7] data model). This column describes the format of the response from the access_url (see 3.1.3) so the values could be data file types (e.g. application/fits) or they could be the DataLink [8] MIME type."

The initial part of the sentence mentions "response format(s)", which that greatly overlaps,

and generates confusion (at least in my mind), with the following DALI specification:

"The RESPONSEFORMAT parameter is used so the client can specify the format of the response (e.g. the output of the job)."

The SIA FORMAT and the DALI RESPONSEFORMAT are two different things, and I think that should be better clarified

in the text of the next SIA version. Could that be done?

  • SIA2 cannot discover rebinned data like SIA1 was able to do #8
Extending VIrtual SIAP2 as described in issue re-gridding/rebinning along a specific WCS can be done by adding this functionality to the virtual data discovery functionality described in issue #6. The access URL would be a SODA URL with the additional parameters proposed in SODA issue #4

An alternative is allowing SIA1 parameters of the MOSAIC mode to SIA2 interface

  • It is not possible to query SIA services by MOC #9

MOC (as well as STMOC, and maybe STEMOC) has proven to be a very efficient way to describe coverage of data and is used for discovery of data. It has been often discussed and also during last interop that it should be useful to query SIA services by MOC ? It should probably be with a new parameter and not via POS considering that it is probably more useful to have STMOC or STEMOC, than spatial MOC alone. This is also valid for SODA of course (SODA Issue #5). I wonder if this new parameter should not be standardized by DALI

  • extension of SIA-style protocol usage outside the image/cube "camp" #10

There was a demand to be able to use a parameter based interface "à la" SIAP2 for other dataproduct_types than cube and image DPTYPE constraint should be relaxed eg: measurements time-series etc.... CADC actually created a service like this. What could be the name? SIA is no more appropriate For time-series this has been discussed in the following ivoa note: http://www.ivoa.net/documents/Notes/TimeSeriesDiscoveryAndAccess/index.html For some dataproduct_types a specific ObsCOre extension may be needed; For TimeSeries this was discussed in recent past. see for example:

https://wiki.ivoa.net/internal/IVOA/InterOpOct2017TDIG/IVOA2017Santiago-DiscussionTDIG-DM-DAL-Session1.pdf

or

http://volute.g-vo.org/svn/trunk/projects/time-domain/time-series/note/TSSerializationNote.pdf

SODA-1.0 feedback, issues and proposals

On GitHub: https://github.com/ivoa-std/SODA

  • Wrong VOTAble syntax in polarization example #1
ON January the 16th 2020 Alberto Micol discovered :

Page 19 of the SODA standard shows:

'Polarization states to be extracted.

My understanding of the VOTable standard is that the correct way should instead be

(see the sentence

"All three MIN, MAX and OPTION sub-elements store their value corresponding to the minimum, maximum, or ``special value'' in a value attribute."

In the paragraph at: http://www.ivoa.net/documents/VOTable/20191021/REC-VOTable-1.4-20191021.html#sec:values ):

Polarization states to be extracted.

Alberto is perfectly right !!!

  • Wrong example for BAND interval MIN and MAX values #2

This appeared in a discussion between Alberto Micol and Markus on the dal Mailing list

MIN and MAX cannot be single values as in the example It should be an array. But an array of what ?

An array of minimal values and an array of maximal values (Markus)

An array of minimal length interval and an array of maximal length interval (Alberto)

  • pixel cutouts (instead of world coordinates) on all axes are missing #3

Considered for version 1.0 but delayed to stay simple (due to syntax problems) Asked by CASDA , others Ranges can be given by SODA service descriptor or computed from ObsCore

Which syntax ?

  • No possibility to control output WCS by regridding/rebinning exists in SODA. #4

This was possible in SIA1.0 for the spatial axis (only) could be done by ADDing standard parameters to SODA for Spatial resolution : SPATRES Rotation : ROTA Sky Projection : PROJ or Alternatively by providing a WCS = (full text of WCS part of FITS header)

Skyview + CDS (SIA on top of Hips2FITS) have this functionality

Should we stabdardize it ? And If yes should we extend it to SIA2 in virtual mode (SIA issue #8) ? Or should we let it as a mixture of SIAP2 and SIAP1 parameters ?

  • It is not possible to query SODA services by MOC #5

MOC (as well as STMOC, and maybe STEMOC) has proven to be a very efficient way to describe the coverage of data and is used for discovery of data. It has been often discussed and also during last interop that it should be useful to query SODA services by MOC? It should probably be with a new parameter and not via POS considering that it is probably more useful to have STMOC or STEMOC, than spatial MOC alone. This is also valid for SIA of course (SIA Issue #9). Of course, the behaviour for SODA will be very different than the one with SIA. SODA-(ST)MOC will extract a subset matching the (ST)MOC while SIA-MOC will only select datasets intersecting with the MOC. I wonder if this new parameter should not be standardized by DALI

  • Managing multiple PARAMS queries responses #6

When several of the PARAM queries (POS, BAND, etc...) are multiple we are facing a cartesian product issue: how do we retrieve all these results. Seems to be easy in async mode which is built to manage multiple results? But in sync? MEF formats? archive files? VOTable containing pointers? What about using UPLOAD functionality (DALI, 3.4.5 UPLOAD) for this?

  • SODA spec doesn't tell us how to provide dataproduct_type transformation #7

First implementations already provide various flavours of cutouts from cubes. Some cutout interface can extract subcubes from a large cube. Some others could create spectra or images (dimension reduction). An idea could be to use the DPTYPE parameter designed for SIAP2 to force this product_type conversion. This has been discussed for TimeSeries there: http://www.ivoa.net/documents/Notes/TimeSeriesDiscoveryAndAccess/index.html


<--  
-->
Added:
>
>
META FILEATTACHMENT attachment="DAP.pdf" attr="" comment="first attempt transition SIA -- DAP" date="1684361972" name="DAP.pdf" path="DAP.pdf" size="646011" user="FrancoisBonnarel" version="2"
 

Revision 52020-05-13 - FrancoisBonnarel

 
META TOPICPARENT name="IvoaDAL"

SODA-1.0 cumulative"> SIAP-2.0 and SODA-1.0 cumulative "-Next" page.

  • Previous material
Changed:
<
<
Old DAL future page : https://wiki.ivoa.net/twiki/bin/view/IVOA/DALFuture
>
>
Old "DAL future" page : https://wiki.ivoa.net/twiki/bin/view/IVOA/DALFuture
  IVOA note : http://www.ivoa.net/documents/Notes/RecentDALProtocolsFeedback/index.html

IVOA interop meetings presentation :

Paris : https://wiki.ivoa.net/internal/IVOA/InterOpMay2019DAL/SIA2-SODA-next.pdf

College Park : https://wiki.ivoa.net/internal/IVOA/InterOpNov2018DAL/HiPStoFits_Prototype_IVOA.pdf

Victoria : https://wiki.ivoa.net/internal/IVOA/InterOpMayy2018DAL/DAL-Feedback.pdf

SIAP-2.0 feedback, issues and proposals

On GitHub: https://github.com/ivoa-std/SIA

  • input PARAMETERS with limited list of values : better description #1
Several SIAP2.0 parameters have a limited list of possible values – Some have lists limited by protocol (and obscore)

POL (Stokes, LINEAR, etc..) DPTYPE (image, cube, visibility, timeseries ;..) CALIB : levels FORMAT : fts, jpeg , png, etc..

– Some have free string values

COLLECTION (HST, WISE, etc...), FACILITY (VLT, Keck, Chandra), INSTRUMENT (ACS, MEGACAM, etc.)

– PARAMETERs less useful if we have no prior idea of their possible values. It often happens that services do not provide these lists of parameters although section 2.1.1 However Section 2.1.20 of SIAP2.0 states : Service PARAMETER self description Any service may include a DataLink service descriptor in the vOTable output to describe itself. this descriptor would describe the supported query parameters (standard and custom) including list of values for those with a fixed list (eg. COLLECTION INSTRUMENT FACILITY DPTYPE CALIB and FORMAT) *

This will allow to discover "possibilities" of the service prior to usage and optimize the queries

The "MAY" is obviously too weak. Version 2.1 could state SHOULD or MUST

  • No input PARAMETER exists to select the RELEASE DATE #2
ObsCore has an optional parameter "obs_release_date", but some projects complain no SIA PARAMETER allows to select on this.

Solution : create an OPTIONAL parameter to select on that ? What is the policy for OPTIONAL parameters ?

  • No Wild-carding of the input PARAMETERS values exists
SIAP2 PyVO interface developers and CADC complain that no wild-carding or incomplete strong value can be admitted in string parameter. Only exact matching.

What is the price to pay to add this functionality and how do we allow it ?

  • input PARAMETERS values are case sensitive #4
What is the price to pay to allow case-insensitive and how do we do that ?

  • POS=RANGE examples inconsistent with spec #5 (From Pat Dowler)
The spec clearly states that RA is in [0,360] and DEC is in [-90,90] but some POS=RANGE ... examples use -Inf and +Inf (probably copied the idea from use of open ended intervals in other params).

I consider this a bug in the example that should result in a fix and erratum because I recall we explicitly decided to not have services doing range reduction.

  • 1 shot discovery (and then access) to cutouts was possible in SIA1 but no more in SIA2 #6
SIAP1 had « cut out » and « mosaic » modes besides « archive » mode → 1 shot before access but only spatial We now have : SIAP2.0 or ObsTAP which deal with all axes

SODA : for cutouts only (all axes) +DataLink (Service descriptor and/or {links} table) → 2 shots before access (instead of 1)

Although it is pretty useful to have the 2 shots sometimes (allow the user to choose the actual limits according to ObsCore content and general context) there have been some complains (SkyView people for example also others) that the one-shot functionality disappeared. It would be perfectly possible to provide it by replacing the full retrieval or datalink url in access_url by a SODA url. SODA URL parameters are common with SIA. Where SIA Parameters values constrain the discovery, SODA parameters force the cutout dimensions.

A capability "virtual data generation" would have to be added to the service. If it exists how do we choose to provide these cutouts by a new "VIRTUAL" boolean parameter? Or by providing two lines in the response with the same obs_id ? The obscore values will be different, encluding publisher_did, but the observation will be the same

  • Possible confusion between FORMAT and RESPONSEFORMAT parameter #7
Alberto Micol 2019/03/&4 :

SIA specifies:

"The FORMAT parameter specifies response format(s) (the access_format column from the ObsCore [7] data model). This column describes the format of the response from the access_url (see 3.1.3) so the values could be data file types (e.g. application/fits) or they could be the DataLink [8] MIME type."

The initial part of the sentence mentions "response format(s)", which that greatly overlaps,

and generates confusion (at least in my mind), with the following DALI specification:

"The RESPONSEFORMAT parameter is used so the client can specify the format of the response (e.g. the output of the job)."

The SIA FORMAT and the DALI RESPONSEFORMAT are two different things, and I think that should be better clarified

in the text of the next SIA version. Could that be done?

  • SIA2 cannot discover rebinned data like SIA1 was able to do #8
Extending VIrtual SIAP2 as described in issue re-gridding/rebinning along a specific WCS can be done by adding this functionality to the virtual data discovery functionality described in issue #6. The access URL would be a SODA URL with the additional parameters proposed in SODA issue #4

An alternative is allowing SIA1 parameters of the MOSAIC mode to SIA2 interface

  • It is not possible to query SIA services by MOC #9

MOC (as well as STMOC, and maybe STEMOC) has proven to be a very efficient way to describe coverage of data and is used for discovery of data. It has been often discussed and also during last interop that it should be useful to query SIA services by MOC ? It should probably be with a new parameter and not via POS considering that it is probably more useful to have STMOC or STEMOC, than spatial MOC alone. This is also valid for SODA of course (SODA Issue #5). I wonder if this new parameter should not be standardized by DALI

Added:
>
>
  • extension of SIA-style protocol usage outside the image/cube "camp" #10

There was a demand to be able to use a parameter based interface "à la" SIAP2 for other dataproduct_types than cube and image DPTYPE constraint should be relaxed eg: measurements time-series etc.... CADC actually created a service like this. What could be the name? SIA is no more appropriate For time-series this has been discussed in the following ivoa note: http://www.ivoa.net/documents/Notes/TimeSeriesDiscoveryAndAccess/index.html For some dataproduct_types a specific ObsCOre extension may be needed; For TimeSeries this was discussed in recent past. see for example:

https://wiki.ivoa.net/internal/IVOA/InterOpOct2017TDIG/IVOA2017Santiago-DiscussionTDIG-DM-DAL-Session1.pdf

or

http://volute.g-vo.org/svn/trunk/projects/time-domain/time-series/note/TSSerializationNote.pdf

 

SODA-1.0 feedback, issues and proposals

On GitHub: https://github.com/ivoa-std/SODA

  • Wrong VOTAble syntax in polarization example #1
ON January the 16th 2020 Alberto Micol discovered :

Page 19 of the SODA standard shows:

'Polarization states to be extracted.

My understanding of the VOTable standard is that the correct way should instead be

(see the sentence

"All three MIN, MAX and OPTION sub-elements store their value corresponding to the minimum, maximum, or ``special value'' in a value attribute."

In the paragraph at: http://www.ivoa.net/documents/VOTable/20191021/REC-VOTable-1.4-20191021.html#sec:values ):

Polarization states to be extracted.

Alberto is perfectly right !!!

  • Wrong example for BAND interval MIN and MAX values #2

This appeared in a discussion between Alberto Micol and Markus on the dal Mailing list

MIN and MAX cannot be single values as in the example It should be an array. But an array of what ?

An array of minimal values and an array of maximal values (Markus)

An array of minimal length interval and an array of maximal length interval (Alberto)

  • pixel cutouts (instead of world coordinates) on all axes are missing #3

Considered for version 1.0 but delayed to stay simple (due to syntax problems) Asked by CASDA , others Ranges can be given by SODA service descriptor or computed from ObsCore

Which syntax ?

  • No possibility to control output WCS by regridding/rebinning exists in SODA. #4

This was possible in SIA1.0 for the spatial axis (only) could be done by ADDing standard parameters to SODA for Spatial resolution : SPATRES Rotation : ROTA Sky Projection : PROJ or Alternatively by providing a WCS = (full text of WCS part of FITS header)

Skyview + CDS (SIA on top of Hips2FITS) have this functionality

Should we stabdardize it ? And If yes should we extend it to SIA2 in virtual mode (SIA issue #8) ? Or should we let it as a mixture of SIAP2 and SIAP1 parameters ?

  • It is not possible to query SODA services by MOC #5
Changed:
<
<
MOC (as well as STMOC, and maybe STEMOC) has proven to be a very efficient way to describe coverage of data and is used for discovery of data. It has been often discussed and also during last interop that it should be useful to query SODA services by MOC ? It should probably be with a new parameter and not via POS considering that it is probably more useful to have STMOC or STEMOC, than spatial MOC alone. This is also valid for SIA of course (SIA Issue #9). Of course the behavior for SODA will be very different than the one with SIA. SODA-(ST)MOC will extract a subset matching the (ST)MOC while SIA-MOC will only select datasets intersecting with the MOC. I wonder if this new parameter should not be standardized by DALI
>
>
MOC (as well as STMOC, and maybe STEMOC) has proven to be a very efficient way to describe the coverage of data and is used for discovery of data. It has been often discussed and also during last interop that it should be useful to query SODA services by MOC? It should probably be with a new parameter and not via POS considering that it is probably more useful to have STMOC or STEMOC, than spatial MOC alone. This is also valid for SIA of course (SIA Issue #9). Of course, the behaviour for SODA will be very different than the one with SIA. SODA-(ST)MOC will extract a subset matching the (ST)MOC while SIA-MOC will only select datasets intersecting with the MOC. I wonder if this new parameter should not be standardized by DALI
 
  • Managing multiple PARAMS queries responses #6
Changed:
<
<
When several of the PARAM queries (POS, BAND, etc...) are multiple we are facing a cartesian product issue : how do we retrieve all these results. Seems to be easy in async mode which is built to manage multiple results ? But in sync ? MEF formats ? archive files ? VOTable contaning pointers ? What about using UPLOAD functionality (DALI, 3.4.5 UPLOAD) for this ?
>
>
When several of the PARAM queries (POS, BAND, etc...) are multiple we are facing a cartesian product issue: how do we retrieve all these results. Seems to be easy in async mode which is built to manage multiple results? But in sync? MEF formats? archive files? VOTable containing pointers? What about using UPLOAD functionality (DALI, 3.4.5 UPLOAD) for this?
Added:
>
>
  • SODA spec doesn't tell us how to provide dataproduct_type transformation #7

First implementations already provide various flavours of cutouts from cubes. Some cutout interface can extract subcubes from a large cube. Some others could create spectra or images (dimension reduction). An idea could be to use the DPTYPE parameter designed for SIAP2 to force this product_type conversion. This has been discussed for TimeSeries there: http://www.ivoa.net/documents/Notes/TimeSeriesDiscoveryAndAccess/index.html

 
<--  
-->

Revision 42020-05-13 - FrancoisBonnarel

 
META TOPICPARENT name="IvoaDAL"

SODA-1.0 cumulative"> SIAP-2.0 and SODA-1.0 cumulative "-Next" page.

  • Previous material
Old DAL future page : https://wiki.ivoa.net/twiki/bin/view/IVOA/DALFuture

IVOA note : http://www.ivoa.net/documents/Notes/RecentDALProtocolsFeedback/index.html

IVOA interop meetings presentation :

Paris : https://wiki.ivoa.net/internal/IVOA/InterOpMay2019DAL/SIA2-SODA-next.pdf

College Park : https://wiki.ivoa.net/internal/IVOA/InterOpNov2018DAL/HiPStoFits_Prototype_IVOA.pdf

Victoria : https://wiki.ivoa.net/internal/IVOA/InterOpMayy2018DAL/DAL-Feedback.pdf

SIAP-2.0 feedback, issues and proposals

On GitHub: https://github.com/ivoa-std/SIA

  • input PARAMETERS with limited list of values : better description #1
Several SIAP2.0 parameters have a limited list of possible values – Some have lists limited by protocol (and obscore)

POL (Stokes, LINEAR, etc..) DPTYPE (image, cube, visibility, timeseries ;..) CALIB : levels FORMAT : fts, jpeg , png, etc..

– Some have free string values

COLLECTION (HST, WISE, etc...), FACILITY (VLT, Keck, Chandra), INSTRUMENT (ACS, MEGACAM, etc.)

– PARAMETERs less useful if we have no prior idea of their possible values. It often happens that services do not provide these lists of parameters although section 2.1.1 However Section 2.1.20 of SIAP2.0 states : Service PARAMETER self description Any service may include a DataLink service descriptor in the vOTable output to describe itself. this descriptor would describe the supported query parameters (standard and custom) including list of values for those with a fixed list (eg. COLLECTION INSTRUMENT FACILITY DPTYPE CALIB and FORMAT) *

This will allow to discover "possibilities" of the service prior to usage and optimize the queries

The "MAY" is obviously too weak. Version 2.1 could state SHOULD or MUST

  • No input PARAMETER exists to select the RELEASE DATE #2
ObsCore has an optional parameter "obs_release_date", but some projects complain no SIA PARAMETER allows to select on this.

Solution : create an OPTIONAL parameter to select on that ? What is the policy for OPTIONAL parameters ?

  • No Wild-carding of the input PARAMETERS values exists
SIAP2 PyVO interface developers and CADC complain that no wild-carding or incomplete strong value can be admitted in string parameter. Only exact matching.

What is the price to pay to add this functionality and how do we allow it ?

  • input PARAMETERS values are case sensitive #4
What is the price to pay to allow case-insensitive and how do we do that ?

  • POS=RANGE examples inconsistent with spec #5 (From Pat Dowler)
The spec clearly states that RA is in [0,360] and DEC is in [-90,90] but some POS=RANGE ... examples use -Inf and +Inf (probably copied the idea from use of open ended intervals in other params).

I consider this a bug in the example that should result in a fix and erratum because I recall we explicitly decided to not have services doing range reduction.

  • 1 shot discovery (and then access) to cutouts was possible in SIA1 but no more in SIA2 #6
SIAP1 had « cut out » and « mosaic » modes besides « archive » mode → 1 shot before access but only spatial We now have : SIAP2.0 or ObsTAP which deal with all axes

SODA : for cutouts only (all axes) +DataLink (Service descriptor and/or {links} table) → 2 shots before access (instead of 1)

Although it is pretty useful to have the 2 shots sometimes (allow the user to choose the actual limits according to ObsCore content and general context) there have been some complains (SkyView people for example also others) that the one-shot functionality disappeared. It would be perfectly possible to provide it by replacing the full retrieval or datalink url in access_url by a SODA url. SODA URL parameters are common with SIA. Where SIA Parameters values constrain the discovery, SODA parameters force the cutout dimensions.

A capability "virtual data generation" would have to be added to the service. If it exists how do we choose to provide these cutouts by a new "VIRTUAL" boolean parameter? Or by providing two lines in the response with the same obs_id ? The obscore values will be different, encluding publisher_did, but the observation will be the same

  • Possible confusion between FORMAT and RESPONSEFORMAT parameter #7
Alberto Micol 2019/03/&4 :

SIA specifies:

"The FORMAT parameter specifies response format(s) (the access_format column from the ObsCore [7] data model). This column describes the format of the response from the access_url (see 3.1.3) so the values could be data file types (e.g. application/fits) or they could be the DataLink [8] MIME type."

The initial part of the sentence mentions "response format(s)", which that greatly overlaps,

and generates confusion (at least in my mind), with the following DALI specification:

"The RESPONSEFORMAT parameter is used so the client can specify the format of the response (e.g. the output of the job)."

The SIA FORMAT and the DALI RESPONSEFORMAT are two different things, and I think that should be better clarified

in the text of the next SIA version. Could that be done?

  • SIA2 cannot discover rebinned data like SIA1 was able to do #8
Extending VIrtual SIAP2 as described in issue re-gridding/rebinning along a specific WCS can be done by adding this functionality to the virtual data discovery functionality described in issue #6. The access URL would be a SODA URL with the additional parameters proposed in SODA issue #4

An alternative is allowing SIA1 parameters of the MOSAIC mode to SIA2 interface

Added:
>
>
  • It is not possible to query SIA services by MOC #9

MOC (as well as STMOC, and maybe STEMOC) has proven to be a very efficient way to describe coverage of data and is used for discovery of data. It has been often discussed and also during last interop that it should be useful to query SIA services by MOC ? It should probably be with a new parameter and not via POS considering that it is probably more useful to have STMOC or STEMOC, than spatial MOC alone. This is also valid for SODA of course (SODA Issue #5). I wonder if this new parameter should not be standardized by DALI

 

SODA-1.0 feedback, issues and proposals

On GitHub: https://github.com/ivoa-std/SODA

  • Wrong VOTAble syntax in polarization example #1
ON January the 16th 2020 Alberto Micol discovered :

Page 19 of the SODA standard shows:

'Polarization states to be extracted.

My understanding of the VOTable standard is that the correct way should instead be

(see the sentence

"All three MIN, MAX and OPTION sub-elements store their value corresponding to the minimum, maximum, or ``special value'' in a value attribute."

In the paragraph at: http://www.ivoa.net/documents/VOTable/20191021/REC-VOTable-1.4-20191021.html#sec:values ):

Polarization states to be extracted.

Alberto is perfectly right !!!

  • Wrong example for BAND interval MIN and MAX values #2

This appeared in a discussion between Alberto Micol and Markus on the dal Mailing list

MIN and MAX cannot be single values as in the example It should be an array. But an array of what ?

An array of minimal values and an array of maximal values (Markus)

An array of minimal length interval and an array of maximal length interval (Alberto)

  • pixel cutouts (instead of world coordinates) on all axes are missing #3

Considered for version 1.0 but delayed to stay simple (due to syntax problems) Asked by CASDA , others Ranges can be given by SODA service descriptor or computed from ObsCore

Which syntax ?

  • No possibility to control output WCS by regridding/rebinning exists in SODA. #4

This was possible in SIA1.0 for the spatial axis (only) could be done by ADDing standard parameters to SODA for Spatial resolution : SPATRES Rotation : ROTA Sky Projection : PROJ or Alternatively by providing a WCS = (full text of WCS part of FITS header)

Skyview + CDS (SIA on top of Hips2FITS) have this functionality

Should we stabdardize it ? And If yes should we extend it to SIA2 in virtual mode (SIA issue #8) ? Or should we let it as a mixture of SIAP2 and SIAP1 parameters ?

Added:
>
>
  • It is not possible to query SODA services by MOC #5
 
Added:
>
>
MOC (as well as STMOC, and maybe STEMOC) has proven to be a very efficient way to describe coverage of data and is used for discovery of data. It has been often discussed and also during last interop that it should be useful to query SODA services by MOC ? It should probably be with a new parameter and not via POS considering that it is probably more useful to have STMOC or STEMOC, than spatial MOC alone. This is also valid for SIA of course (SIA Issue #9). Of course the behavior for SODA will be very different than the one with SIA. SODA-(ST)MOC will extract a subset matching the (ST)MOC while SIA-MOC will only select datasets intersecting with the MOC. I wonder if this new parameter should not be standardized by DALI

  • Managing multiple PARAMS queries responses #6

When several of the PARAM queries (POS, BAND, etc...) are multiple we are facing a cartesian product issue : how do we retrieve all these results. Seems to be easy in async mode which is built to manage multiple results ? But in sync ? MEF formats ? archive files ? VOTable contaning pointers ? What about using UPLOAD functionality (DALI, 3.4.5 UPLOAD) for this ?

 
<--  
-->

Revision 32020-05-08 - FrancoisBonnarel

 
META TOPICPARENT name="IvoaDAL"
Changed:
<
<

SODA-1.0 cumulative"> SIAP-2.0 and SODA-1.0 cumulative "-Next" page.

>
>

SODA-1.0 cumulative"> SIAP-2.0 and SODA-1.0 cumulative "-Next" page.

 
Changed:
<
<
  • Previous material
>
>
  • Previous material
Added:
>
>
Old DAL future page : https://wiki.ivoa.net/twiki/bin/view/IVOA/DALFuture
 
Changed:
<
<
Old DAL future page : https://wiki.ivoa.net/twiki/bin/view/IVOA/DALFuture
>
>
IVOA note : http://www.ivoa.net/documents/Notes/RecentDALProtocolsFeedback/index.html
 
Changed:
<
<
IVOA note : http://www.ivoa.net/documents/Notes/RecentDALProtocolsFeedback/index.html
>
>
IVOA interop meetings presentation :
 
Changed:
<
<
IVOA interop meetings presentation :
>
>
Paris : https://wiki.ivoa.net/internal/IVOA/InterOpMay2019DAL/SIA2-SODA-next.pdf
 
Changed:
<
<
Paris : https://wiki.ivoa.net/internal/IVOA/InterOpMay2019DAL/SIA2-SODA-next.pdf
>
>
College Park : https://wiki.ivoa.net/internal/IVOA/InterOpNov2018DAL/HiPStoFits_Prototype_IVOA.pdf
 
Changed:
<
<
College Park : https://wiki.ivoa.net/internal/IVOA/InterOpNov2018DAL/HiPStoFits_Prototype_IVOA.pdf
>
>
Victoria : https://wiki.ivoa.net/internal/IVOA/InterOpMayy2018DAL/DAL-Feedback.pdf
 
Changed:
<
<
Victoria : https://wiki.ivoa.net/internal/IVOA/InterOpMayy2018DAL/DAL-Feedback.pdf
>
>

SIAP-2.0 feedback, issues and proposals

Deleted:
<
<
 
Deleted:
<
<

SIAP-2.0 feedback, issues and proposal

 On GitHub: https://github.com/ivoa-std/SIA
Changed:
<
<
  • input PARAMETERS with limited list of values : better description #1
Several SIAP2.0 parameters have a limited list of possible values
>
>
  • input PARAMETERS with limited list of values : better description #1
Several SIAP2.0 parameters have a limited list of possible values – Some have lists limited by protocol (and obscore)
Deleted:
<
<
– Some have lists limited by protocol (and obscore)
 
Changed:
<
<
POL (Stokes, LINEAR, etc..)
>
>
POL (Stokes, LINEAR, etc..) DPTYPE (image, cube, visibility, timeseries ;..) CALIB : levels FORMAT : fts, jpeg , png, etc..
Deleted:
<
<
DPTYPE (image, cube, visibility, timeseries ;..) CALIB : levels FORMAT : fts, jpeg , png, etc..
 
Changed:
<
<
– Some have free string values
>
>
– Some have free string values
 
Changed:
<
<
COLLECTION (HST, WISE, etc...),
>
>
COLLECTION (HST, WISE, etc...), FACILITY (VLT, Keck, Chandra), INSTRUMENT (ACS, MEGACAM, etc.)
Deleted:
<
<
FACILITY (VLT, Keck, Chandra), INSTRUMENT (ACS, MEGACAM, etc.)
 
Changed:
<
<
– PARAMETERs less useful if we have no prior idea of their possible
>
>
– PARAMETERs less useful if we have no prior idea of their possible values. It often happens that services do not provide these lists of parameters although section 2.1.1 However Section 2.1.20 of SIAP2.0 states : Service PARAMETER self description Any service may include a DataLink service descriptor in the vOTable output to describe itself. this descriptor would describe the supported query parameters (standard and custom) including list of values for those with a fixed list (eg. COLLECTION INSTRUMENT FACILITY DPTYPE CALIB and FORMAT) *
Deleted:
<
<
values. It often happens that services do not provide these lists of parameters although section 2.1.1 However Section 2.1.20 of SIAP2.0 states : Service PARAMETER self description Any service may include a DataLink service descriptor in the vOTable output to describe itself. this descriptor would describe the supported query parameters (standard and custom) including list of values for those with a fixed list (eg. COLLECTION INSTRUMENT FACILITY DPTYPE CALIB and FORMAT) *
  This will allow to discover "possibilities" of the service prior to usage and optimize the queries

The "MAY" is obviously too weak. Version 2.1 could state SHOULD or MUST

Added:
>
>
  • No input PARAMETER exists to select the RELEASE DATE #2
ObsCore has an optional parameter "obs_release_date", but some projects complain no SIA PARAMETER allows to select on this.

Solution : create an OPTIONAL parameter to select on that ? What is the policy for OPTIONAL parameters ?

  • No Wild-carding of the input PARAMETERS values exists
SIAP2 PyVO interface developers and CADC complain that no wild-carding or incomplete strong value can be admitted in string parameter. Only exact matching.

What is the price to pay to add this functionality and how do we allow it ?

  • input PARAMETERS values are case sensitive #4
What is the price to pay to allow case-insensitive and how do we do that ?

  • POS=RANGE examples inconsistent with spec #5 (From Pat Dowler)
The spec clearly states that RA is in [0,360] and DEC is in [-90,90] but some POS=RANGE ... examples use -Inf and +Inf (probably copied the idea from use of open ended intervals in other params).

I consider this a bug in the example that should result in a fix and erratum because I recall we explicitly decided to not have services doing range reduction.

  • 1 shot discovery (and then access) to cutouts was possible in SIA1 but no more in SIA2 #6
SIAP1 had « cut out » and « mosaic » modes besides « archive » mode → 1 shot before access but only spatial We now have : SIAP2.0 or ObsTAP which deal with all axes

SODA : for cutouts only (all axes) +DataLink (Service descriptor and/or {links} table) → 2 shots before access (instead of 1)

Although it is pretty useful to have the 2 shots sometimes (allow the user to choose the actual limits according to ObsCore content and general context) there have been some complains (SkyView people for example also others) that the one-shot functionality disappeared. It would be perfectly possible to provide it by replacing the full retrieval or datalink url in access_url by a SODA url. SODA URL parameters are common with SIA. Where SIA Parameters values constrain the discovery, SODA parameters force the cutout dimensions.

A capability "virtual data generation" would have to be added to the service. If it exists how do we choose to provide these cutouts by a new "VIRTUAL" boolean parameter? Or by providing two lines in the response with the same obs_id ? The obscore values will be different, encluding publisher_did, but the observation will be the same

  • Possible confusion between FORMAT and RESPONSEFORMAT parameter #7
Alberto Micol 2019/03/&4 :

SIA specifies:

"The FORMAT parameter specifies response format(s) (the access_format column from the ObsCore [7] data model). This column describes the format of the response from the access_url (see 3.1.3) so the values could be data file types (e.g. application/fits) or they could be the DataLink [8] MIME type."

The initial part of the sentence mentions "response format(s)", which that greatly overlaps,

and generates confusion (at least in my mind), with the following DALI specification:

"The RESPONSEFORMAT parameter is used so the client can specify the format of the response (e.g. the output of the job)."

The SIA FORMAT and the DALI RESPONSEFORMAT are two different things, and I think that should be better clarified

in the text of the next SIA version. Could that be done?

  • SIA2 cannot discover rebinned data like SIA1 was able to do #8
Extending VIrtual SIAP2 as described in issue re-gridding/rebinning along a specific WCS can be done by adding this functionality to the virtual data discovery functionality described in issue #6. The access URL would be a SODA URL with the additional parameters proposed in SODA issue #4

An alternative is allowing SIA1 parameters of the MOSAIC mode to SIA2 interface

SODA-1.0 feedback, issues and proposals

On GitHub: https://github.com/ivoa-std/SODA

  • Wrong VOTAble syntax in polarization example #1
ON January the 16th 2020 Alberto Micol discovered :

Page 19 of the SODA standard shows:

'Polarization states to be extracted.

My understanding of the VOTable standard is that the correct way should instead be

(see the sentence

"All three MIN, MAX and OPTION sub-elements store their value corresponding to the minimum, maximum, or ``special value'' in a value attribute."

In the paragraph at: http://www.ivoa.net/documents/VOTable/20191021/REC-VOTable-1.4-20191021.html#sec:values ):

Polarization states to be extracted.

Alberto is perfectly right !!!

  • Wrong example for BAND interval MIN and MAX values #2

This appeared in a discussion between Alberto Micol and Markus on the dal Mailing list

MIN and MAX cannot be single values as in the example It should be an array. But an array of what ?

An array of minimal values and an array of maximal values (Markus)

An array of minimal length interval and an array of maximal length interval (Alberto)

  • pixel cutouts (instead of world coordinates) on all axes are missing #3

Considered for version 1.0 but delayed to stay simple (due to syntax problems) Asked by CASDA , others Ranges can be given by SODA service descriptor or computed from ObsCore

Which syntax ?

  • No possibility to control output WCS by regridding/rebinning exists in SODA. #4

This was possible in SIA1.0 for the spatial axis (only) could be done by ADDing standard parameters to SODA for Spatial resolution : SPATRES Rotation : ROTA Sky Projection : PROJ or Alternatively by providing a WCS = (full text of WCS part of FITS header)

Skyview + CDS (SIA on top of Hips2FITS) have this functionality

Should we stabdardize it ? And If yes should we extend it to SIA2 in virtual mode (SIA issue #8) ? Or should we let it as a mixture of SIAP2 and SIAP1 parameters ?

 


<--  
-->

Revision 22020-05-07 - FrancoisBonnarel

 
META TOPICPARENT name="IvoaDAL"
Changed:
<
<
SIAP-2.0 and SODA-1.0 cumulative "-Next" page.
>
>

SODA-1.0 cumulative"> SIAP-2.0 and SODA-1.0 cumulative "-Next" page.

Added:
>
>
  • Previous material

Old DAL future page : https://wiki.ivoa.net/twiki/bin/view/IVOA/DALFuture

IVOA note : http://www.ivoa.net/documents/Notes/RecentDALProtocolsFeedback/index.html

IVOA interop meetings presentation :

Paris : https://wiki.ivoa.net/internal/IVOA/InterOpMay2019DAL/SIA2-SODA-next.pdf

College Park : https://wiki.ivoa.net/internal/IVOA/InterOpNov2018DAL/HiPStoFits_Prototype_IVOA.pdf

Victoria : https://wiki.ivoa.net/internal/IVOA/InterOpMayy2018DAL/DAL-Feedback.pdf

SIAP-2.0 feedback, issues and proposal

On GitHub: https://github.com/ivoa-std/SIA

  • input PARAMETERS with limited list of values : better description #1
Several SIAP2.0 parameters have a limited list of possible values – Some have lists limited by protocol (and obscore)

POL (Stokes, LINEAR, etc..) DPTYPE (image, cube, visibility, timeseries ;..) CALIB : levels FORMAT : fts, jpeg , png, etc..

– Some have free string values

COLLECTION (HST, WISE, etc...), FACILITY (VLT, Keck, Chandra), INSTRUMENT (ACS, MEGACAM, etc.)

– PARAMETERs less useful if we have no prior idea of their possible values. It often happens that services do not provide these lists of parameters although section 2.1.1 However Section 2.1.20 of SIAP2.0 states : Service PARAMETER self description Any service may include a DataLink service descriptor in the vOTable output to describe itself. this descriptor would describe the supported query parameters (standard and custom) including list of values for those with a fixed list (eg. COLLECTION INSTRUMENT FACILITY DPTYPE CALIB and FORMAT) *

This will allow to discover "possibilities" of the service prior to usage and optimize the queries

The "MAY" is obviously too weak. Version 2.1 could state SHOULD or MUST

 
<--  
-->

Revision 12019-09-05 - MarcoMolinaro

 
META TOPICPARENT name="IvoaDAL"
SIAP-2.0 and SODA-1.0 cumulative "-Next" page.


<--  
-->
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback