International Virtual Observatory Alliance

Difference: Spectral2RFCr2 (1 vs. 5)

Revision 52016-09-29 - MarkCresitelloDittmar

  META TOPICPARENT 
  name="SpectralDM2"  

 Spectral v2.0 Proposed Recommendation: Request for Comments
- META TOPICPARENT
+  name="SpectralDM2"
->
>
+Work on this Data model has concluded as of Dec 2015. The development of component models DatasetMetadata, NDCube, and STC2 form a basis set of models on which this specification should be built. It was decided that the timescale between the release of this model and the subsequent conversion to the component model basis is too short to warrent moving forward.

Upon the release of the above component models, it is expected that the Spectral model will be revisited, possibly expanding the scope to cover other products such as Eschelle spectra and TimeSeries. This document should serve as a guide for that work. The content of this model satisfies the requirements for which it was generated, so a simple translation of concepts should be a good starting point.

-- MarkCresitelloDittmar - 2016-09-29
 This page records the public discussion for round 2 of the Spectral2.0 Data Model Proposed Recommendation. The initial pass went through RFC to TCG where it received significant input. These were addressed, and it was decided that the document should pass through RFC again.

Initial RFC/TCG review comments can be found here: 

 http://wiki.ivoa.net/twiki/bin/view/IVOA/Spectral2RFC
 
Latest version of Spectral2.0 can be found at: 
 PR-SpectralDM-2.0-20150528
 
NOTE: No new capabilities have been added to the document over what was presented at the IVOA interop in Sesto. The intent is that this document fulfills the requirements of its commissioning. A new project will be initiated to expand the capabilities for unsupported types (eg: Echelle spectra), and to cast the model in terms of the DatasetMetadata/CubeDM models now in progress.

 Reference Interoperable Implementations 

1) Speclib Java library: can be found here.

Java library for interpreting Spectral-2.0 model serializations. 

 loads model definition from ancillary file, with design capable of supporting alternate storage mechanisms (DB, vo-dml, etc)
  interprets data file according to the model definition.
  implements interfaces for model components, supports all primitive datatypes, Quantity, Lists which are used in the Spectral model Note: This library is a work in progress, and is being updated to include:
  support for multiple data models ( Spectrum-1.1; Dataset; NDCube; etc )
  enhanced validation capabilities
 
2) Serializations utilizing Spectral 2.0 model specifications 
 GAVO: GAVO DC Software Distribution DaCHS
  SVO: TSAP - Theorical Spectral Model Service (Not available?)
 
3) Interoperability demo

A demonstration utility (speclist) was generated and demonstrated at the Sesto interop. This utility queries the GAVO database for spectral data and generates a simple plot with metadata browser. Users can explore the metadata space, and zoom on plot details.

 Implementations Validators 

At present, the speclib library is not ready to support 'validation', but is fairly strict in the interpretation of serializations in terms of the data model. Issues and non-compliant elements will produce exceptions or be interpreted as custom elements.
 RFC Review Period: 21 Sep 2015 - 16 Oct 2015 


 Comments from the IVOA Community during RFC period: 21 Sep 2015 - 16 Oct 2015 

In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your Wiki Name so that authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.

Additional discussion about any of the comments or responses can be conducted on the WG mailing list. However, please be sure to enter your initial comments here for full consideration in any future revisions of this document
 

 Sample comment by WikiName  
 Response (by WikiName)
 
 

 Comments by Markus Demleitner 

As the author of one reference implementation cited (DaCHS) I must make the reservation that I am only implementing what was already part of SDM 1; I am essentially mapping SSA to SDM2, which works. I am, however, not implementing photometry points, transformations, or any other advanced feature.

This is true. I think this demonstrates that an SDM1 service can reasonably convert to serving SDM2 files. I will note that the speclib implementation can serialize the full model. ( -- MarkCresitelloDittmar )

Minor details aside, I am still unsure whether we should bother passing this recommendation when it is clear that major use cases (time series, compound spectra) are not covered. I would not mind so much if we expected a smooth evolution (version 2.1) towards these new use cases, but I understand that fairly fundamental changes (alignment with the future image DM, introduction of a principled serialisation) will be necessary to get there, so implementors of this will have no advantage when implementing what will presumably have to be SDM 3.

This is the concern I raised at the meeting, and I share it. The conversion of Spectral-2.0 to fold into the Dataset/Cube/VO-DML family will be significant. On the other hand, we don't know what the timetable for this will be. We need a resource to work it, requirements for the new features (TimeSeries, Echelle, Normalized, etc), and the finalization of Dataset/Cube/STC2/VO-DML. My position is that if providers find the new features of this model useful (which presumably is why it was commissioned), and are OK with the upcoming revision, then we should move it forward. (-- MarkCresitelloDittmar )

In the remainder I list some minor issues, taken from last TCG review where I think they still stand. Note that I don't think any of these are actually showstoppers, and I'll accept "see last review answers" for each of them; I hope a quick overview over these potential issues here, might be useful for other reviewers, though.
 

 Given the number of cross references in the document, it would really help if cross-references could be hyperlinks.  
 I could do this, I just this week made this change in the DatasetDM document. ( -- MarkCresitelloDittmar )
 
  I still believe we should remove TimeSI, FluxSI, and SpectralSI -- applications cannot rely on them anyway, and we've had VOUnits for, what, two years by now (including two implementations serving C, Java, and python), so applications can be expected to migrate there.  
 I would prefer to make this a lein on the next revision, so a proper evaluation of the use case which put them there can be done, and any dependency with SSA can be resolved. ( -- MarkCresitelloDittmar )
 
  I still believe modelling the CoordSys both in characterisation and toplevel is asking for trouble. There may be datasets out there that require this kind of modelling, but if the model is really out to describe those in that level of detail, there needs to be a better explanation, and guidance what to do in the typical (simple) case.  
 I think the model is correct in this regard. The dataset contains 1 or more Coordinate systems (CoordSys), and the characterisation axes need to refer to the frame which they are characterizing. ( -- MarkCresitelloDittmar )
 
  I'd like one of ResolPower.refVal and Resolution to go  
 Again, this is something I'd prefer to put on the action list for the next revision. I'm not sure it's worth pulling resources off the higher priority work to evaluate and update this. ( -- MarkCresitelloDittmar )
 
  I still don't quite get why 4.4 has empty subsections for each of the three following sections (but then I didn't really try hard). Why not pull 4.5 through 4.7 into subsections of 4.4?  
 Sections 4.4.1, 4.4.2, 4.4.3 identify the properties of the object in 4.4, any text description clarifies the particular usage of that object in this context.. The sections are empty because there wasn't any text to add for that context. So the descriptions are in their individual sections (4.5, 4.6, 4.7). ( -- MarkCresitelloDittmar )
 
  Since case-insensitivity is a pain, I'd still like to strike the case-insensitivity of dopplerDefinition strings.  
 I'd have to check, but I don't think this is a change from the current spec, so we aren't adding burden, just not relieving it. I think this might fold better into the next revision, where enumerations are better specified, and as part of the standardizing practice, comparisons with enums could be considered case-sensitive. ( -- MarkCresitelloDittmar )
 
  I still don't believe TimeFrame.zero has a Benefit/Cost ratio even approaching 1. From below.  
 This is a standard property of the STC TimeFrame. ( -- MarkCresitelloDittmar )
 
 
-- MarkusDemleitner - 2015-09-30


 Comments by Jean-Michel Glorian 

Typos in C.1.1 Basic Spectrum Instance 

 <PARAM name="Collection1" .... arraysize="*">
-<
<
+ <PARAM name="Collection2" .... arraysize="*">
->
>
+  <PARAM name="Collection2" .... arraysize="*">
  <PARAM name="Collection3".... arraysize="*">
  <PARAM name="DatasetID" .... arraysize="*">
-<
<
+<PARAM name="CreationDate".... arraysize="*">
 <PARAM name="Logo".... arraysize="*">
->
>
->
>
+ <PARAM name="CreationDate".... arraysize="*">
  <PARAM name="Logo".... arraysize="*">
 <-- 
 
 
 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->

Revision 42015-11-02 - JeanMichelGlorian

META TOPICPARENT	name="SpectralDM2"

Spectral v2.0 Proposed Recommendation: Request for Comments

This page records the public discussion for round 2 of the Spectral2.0 Data Model Proposed Recommendation. The initial pass went through RFC to TCG where it received significant input. These were addressed, and it was decided that the document should pass through RFC again.

Initial RFC/TCG review comments can be found here:

http://wiki.ivoa.net/twiki/bin/view/IVOA/Spectral2RFC

Latest version of Spectral2.0 can be found at:

PR-SpectralDM-2.0-20150528

NOTE: No new capabilities have been added to the document over what was presented at the IVOA interop in Sesto. The intent is that this document fulfills the requirements of its commissioning. A new project will be initiated to expand the capabilities for unsupported types (eg: Echelle spectra), and to cast the model in terms of the DatasetMetadata/CubeDM models now in progress.

Reference Interoperable Implementations

1) Speclib Java library: can be found here.

Java library for interpreting Spectral-2.0 model serializations.

loads model definition from ancillary file, with design capable of supporting alternate storage mechanisms (DB, vo-dml, etc)
interprets data file according to the model definition.
implements interfaces for model components, supports all primitive datatypes, Quantity, Lists which are used in the Spectral model Note: This library is a work in progress, and is being updated to include:
support for multiple data models ( Spectrum-1.1; Dataset; NDCube; etc )
enhanced validation capabilities

2) Serializations utilizing Spectral 2.0 model specifications

GAVO: GAVO DC Software Distribution DaCHS
SVO: TSAP - Theorical Spectral Model Service (Not available?)

3) Interoperability demo

A demonstration utility (speclist) was generated and demonstrated at the Sesto interop. This utility queries the GAVO database for spectral data and generates a simple plot with metadata browser. Users can explore the metadata space, and zoom on plot details.

Implementations Validators

At present, the speclib library is not ready to support 'validation', but is fairly strict in the interpretation of serializations in terms of the data model. Issues and non-compliant elements will produce exceptions or be interpreted as custom elements.

RFC Review Period: 21 Sep 2015 - 16 Oct 2015

Comments from the IVOA Community during RFC period: 21 Sep 2015 - 16 Oct 2015

In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your Wiki Name so that authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.

Additional discussion about any of the comments or responses can be conducted on the WG mailing list. However, please be sure to enter your initial comments here for full consideration in any future revisions of this document

Sample comment by WikiName
- Response (by WikiName)

Comments by Markus Demleitner

As the author of one reference implementation cited (DaCHS) I must make the reservation that I am only implementing what was already part of SDM 1; I am essentially mapping SSA to SDM2, which works. I am, however, not implementing photometry points, transformations, or any other advanced feature.

Changed:

<
<

This is true. I think this demonstrates that an SDM1 service can reasonably convert to serving SDM2 files. I will note that the speclib implementation can serialize the full model. ( -- MarkCresitelloDittmar )

>
>

Minor details aside, I am still unsure whether we should bother passing this recommendation when it is clear that major use cases (time series, compound spectra) are not covered. I would not mind so much if we expected a smooth evolution (version 2.1) towards these new use cases, but I understand that fairly fundamental changes (alignment with the future image DM, introduction of a principled serialisation) will be necessary to get there, so implementors of this will have no advantage when implementing what will presumably have to be SDM 3.

This is the concern I raised at the meeting, and I share it. The conversion of Spectral-2.0 to fold into the Dataset/Cube/VO-DML family will be significant. On the other hand, we don't know what the timetable for this will be. We need a resource to work it, requirements for the new features (TimeSeries, Echelle, Normalized, etc), and the finalization of Dataset/Cube/STC2/VO-DML. My position is that if providers find the new features of this model useful (which presumably is why it was commissioned), and are OK with the upcoming revision, then we should move it forward. (-- MarkCresitelloDittmar )

In the remainder I list some minor issues, taken from last TCG review where I think they still stand. Note that I don't think any of these are actually showstoppers, and I'll accept "see last review answers" for each of them; I hope a quick overview over these potential issues here, might be useful for other reviewers, though.

Given the number of cross references in the document, it would really help if cross-references could be hyperlinks.
- I could do this, I just this week made this change in the DatasetDM document. ( -- MarkCresitelloDittmar )
I still believe we should remove TimeSI, FluxSI, and SpectralSI -- applications cannot rely on them anyway, and we've had VOUnits for, what, two years by now (including two implementations serving C, Java, and python), so applications can be expected to migrate there.
- I would prefer to make this a lein on the next revision, so a proper evaluation of the use case which put them there can be done, and any dependency with SSA can be resolved. ( -- MarkCresitelloDittmar )
I still believe modelling the CoordSys both in characterisation and toplevel is asking for trouble. There may be datasets out there that require this kind of modelling, but if the model is really out to describe those in that level of detail, there needs to be a better explanation, and guidance what to do in the typical (simple) case.
- I think the model is correct in this regard. The dataset contains 1 or more Coordinate systems (CoordSys), and the characterisation axes need to refer to the frame which they are characterizing. ( -- MarkCresitelloDittmar )
I'd like one of ResolPower.refVal and Resolution to go
- Again, this is something I'd prefer to put on the action list for the next revision. I'm not sure it's worth pulling resources off the higher priority work to evaluate and update this. ( -- MarkCresitelloDittmar )
I still don't quite get why 4.4 has empty subsections for each of the three following sections (but then I didn't really try hard). Why not pull 4.5 through 4.7 into subsections of 4.4?
- Sections 4.4.1, 4.4.2, 4.4.3 identify the properties of the object in 4.4, any text description clarifies the particular usage of that object in this context.. The sections are empty because there wasn't any text to add for that context. So the descriptions are in their individual sections (4.5, 4.6, 4.7). ( -- MarkCresitelloDittmar )
Since case-insensitivity is a pain, I'd still like to strike the case-insensitivity of dopplerDefinition strings.
- I'd have to check, but I don't think this is a change from the current spec, so we aren't adding burden, just not relieving it. I think this might fold better into the next revision, where enumerations are better specified, and as part of the standardizing practice, comparisons with enums could be considered case-sensitive. ( -- MarkCresitelloDittmar )
I still don't believe TimeFrame.zero has a Benefit/Cost ratio even approaching 1. From below.
- This is a standard property of the STC TimeFrame. ( -- MarkCresitelloDittmar )

-- MarkusDemleitner - 2015-09-30

Added:

>
>

Comments by Jean-Michel Glorian