SSAP V1.1 Collaborative Page

SSAP/SDM Documents

The draft V1.1 documents can be found here:

• SSA V1.1 draft (pdf) (doc) (Updated April 17 2011)
• SSA V1.1 data model spreadsheet

The current SSA/SDM standards documents can be found here:

• Simple Spectral Access V1.04 (1 Feb 2009)
• Spectrum Data Model (29 Oct 2008)
• SSA/SDM model summary spreadsheet

-- DougTody - 04 Apr 2011

20110118 Draft Discussion

The (ab)use of XML namespace attributes (xmlns:spec="http://www.ivoa.net/xml/SpectrumModel/v1.01") for the "declaration" of utype data model identifiers has been discussed in the context of STC in VOTable; back then, the result was that the data model URI should be transmitted in DataModel.URI. In addition to repairing the mismatch between (fixed) utype DM identifiers and (in principle arbitrary) XML namespace prefixes, this would for the Spectrum data model give the additional advantage that the DM identifier is serialized into FITS as well.

Although very few actual SSA implementations get the XML namespace declaration "right" (i.e., as specified in the existing standard) for ssa and spec utypes and I cannot see how applications would rely on it anyway, dropping it is probably not an option within a minor update (though I'd like a deprecation notice). Still, I'd really appreciate to see a "should" clause as to including DataModel.URI in both SSA and SDM. That's not going to hurt anyone but will help future standards development.

[Oh, and in case you have doubts whether I'm addressing a real problem here: Try following the recommendation as to the "canonical" prefixes in SimpleDALRegExt and then write in ssa utypes in the registration document...]

-- MarkusDemleitner - 11 Mar 2011

Markus - I agree this usage in SSA V1.0/1.1 is incorrect and should be fixed, once we have an agreement on how to handle this within VOTable and for Utype namespaces. Best to leave it as it is until this is settled.

-- DougTody - 04 Apr 2011

SSAP/SDM Document Update Discussion

At the EuroVO AIDA a small group of people interested in updating the SSA and SpectrumDM document gathered. Participants; BrunoRino, KeithNoddle (DAL chair), MireilleLouys (DM chair), AlbertoMicol, IgorChilingarian, JesusSalgado, FrancoisBonnarel.

Separately from this meeting, from the US side, DougTody and JonathanMcDowell (the SSAP and SDM document editors) had also been discussing these proposed changes with the document authors and others via email.

We set forth the following goal: To create 1.1 versions (of SSA and the SpectrumDM) that attempt to fix or clarify what were seen as inconsistencies between the two documents. Mor extensive clarifications or additions could be added in a version 1.2. Big changes, possibly not backwards compatible, are postponed to 2.0 versions.

The rationale for this split is to have version 1.1 approved quickly and replace the current 1.0. A version 1.2 would permit more changes but would take significantly longer, gathering more extensive input from the community.

In both cases (1.1. and 1.2) the assumption is that no existing application should break. This means that when a fix creates potential breakage, the potentially affected applications should be consulted.

Below, the result of our discussion on what must be changed in order to reach version 1.1. We were lucky enough to have both the DM and DAL Working Group leads; we concluded that after a short period of time after circulating these meeting minutes to the relevant lists, Working Drafts should be produced, ahead of the May interop in Victoria.

SSAP/SDM Inconsistencies

1. The SSA data model is derived, but decoupled, from the SpectrumDM

The acknowledged divergences are: - "required" flags (Mandatory, Recommended, Optional) are different - the SSA data model contains service related metadata, that have no meaning for the SpectrumDM - the SpectrumDM contains metadata related to data analysis (Data.*) that are of no interest for data discovery (which is the purpose of SSA) - in SSA, the "Spectrum." prefix found in the SpectrumDM utypes was dropped, and in some cases a "Dataset." prefix was added.

Even if those divergences are discrepancies, they are not going to be fixed in a 1.x document. Instead, SSA Section 2.2 ("Data Model") must be changed to reflect these divergences. It should state explicitly that a reader of the SSA should only refer to the SpectrumDM to seek clarification about the meaning of metadata fields. The specification of required fields, the UCDs, and even the utype syntax for setting up a SSA server are to be read from the SSA document. Metadata defined in the SpectrumDM, but not listed in the SSA, are not relevant for SSA service interface.

This is a compromise towards reaching rapidly a stable revision of the documents. We would much prefer to have a single source for the definition of the datamodel, which the SSA protocol would just extend. But we believe this is too large of a task to achieve while maintaining backwards compatibility, on a reasonable time-scale.

2. The use of "*" and ".." in UCDs

This must be eliminated.

These characters are always used in the context of "em.*" or "em...". Our understanding is that these characters are placeholders, which a data provider must fill in, according to its requirements. A list of all possible values and meanings must be provided instead, using the following primary UCDs:

em.wl em.freq em.energy

3. The use of "*" in utypes

This must be eliminated. The correct utypes to use are the ones in the "Query Response" section of SSA, but without the "*":

Char.SpatialAxis.SamplingPrecision.SampleExtent Char.SpatialAxis.SamplingPrecision.FillFactor Char.SpectralAxis.SamplingPrecision.SampleExtent Char.SpectralAxis.SamplingPrecision.FillFactor Char.TimeAxis.SamplingPrecision.SampleExtent Char.TimeAxis.SamplingPrecision.FillFactor

4. Missing UCDs in SSA The SSA is correct, the SpectrumDM should not have a UCD for the following elements (the UCDs provided in the SpectrumDM on those elements are either wrong or confusing):

5 Misc. typos The SpectrumDM (on the FITS serialization section) should fix the following utypes:

The SSA should fix the following UCD: em;spec.binSize -> em;spect.binSize

Extra spaces in UCDs and utypes are typos and should be removed

6. Dimensional analysis typo

In the SpectrumDM, change (from 10-10 to 1E-10) the way to express exponents within the dimensional analysis elements

Section 3.2 should read:

Pedro Osuna and Jesus Salgado have proposed a representation in the spirit of dimensional analysis, using the symbols M, L, T to signify kg, m, s respectively and omitting the ** for powers, so that 10**3 Jy Hz which is equivalent to 10**-23 kg s**-2 is written compactly as 1E-23MT-2

and the example in section 9.4:

SPECSDIM= '1E-10 L' / Spectral SIDim FLUXSDIM= '1E+7 ML-1T-3' / Flux SDim

7. Wrong UCDs

Spectrum.Char.SpectralAxis.Coverage.Location.Value has a wrong UCD of instr.bandpass. It should become the following list (in accordance to point 2 above):

em.wl;instr.bandpass em.freq;instr.bandpass em.energy;instr.bandpass

8. Inconsistencies within the SSA itself

Add the Dataset.Deleted utype to Appendix D.

Remove the Data.* utypes from Appendix D.

Add the remaining missing utypes present in Appendix D to section 4.2 (the list is too long and too boring to show here)

9. Inconsistencies within the SpectrumDM itself

Add a comment to the FITS serialisation stating that it does not cover the whole of the SpectrumDM utypes

Note: The "consolidation" activities detailed in points 7 and 8 should also make sure the order by which the utypes are listed is consistent throughout all documents.

See also:

SSA and SDM Documents Update (response to above)

I am trying to synthesize these various comments on SSA/SDM. I think I mostly agree with the detailed changes suggested. The comments here are mostly higher level.

At this point it appears that both Jonathan and Bruno agree that the SSA metadata and the SDM, while quite similar, are not quite the same thing (hence where there are differences they are not necessarily "inconsistencies" but are restrictions in units, mandatory fields etc. required by the specific application). SSA defines the data model used in the query response, describing the data which a service can deliver. SDM defines the data model of a spectrum dataset/file/instance.

A key point which has not been adequately discussed is the role of the generic GDS/Obs data model. While the SDM explicitly describes a Spectrum instance, most of what is in the SSA query response is actually GDS/ObsDM metadata and not specific to Spectra at all - at this level the metadata should be the same for an image, for an SED, for a time series, etc. It does not make a whole lot of sense for example to have Spectrum.Target.Name and Image.Target.Name and allow these to be two different things. An application which deals with multiple types of data would like global and generic metadata such as this, which is applicable to any type of data, to be usable as just "Target.Name" (possibly in a future version something such as Obs.Target.Name would be preferable, but we did not have that when SSA was defined). Once we have a full set of DAL2 services one would like to have the common stuff factored out so that it does not have to be customized for every type of data. For the purposes of data discovery and description most of this stuff is the same.

The above is the reason why the "Spectrum" prefix does not carry over into the SSA query response. Rather, essentially all of the SSA query response metadata is either generic dataset metadata, or specific to the service mechanism itself and hence also generic and applicable to all such services. It is not actually from the SDM at all, we merely require them to be consistent. That small part of the query response metadata which is not generic and which is specific to the type of dataset being queried (Spectrum, Image, whatever) is what is in the "Dataset" component model.

In other words, for a DAL2 query response what we have is virtually all generic dataset / Obs metadata, plus a separate "Dataset" element which is specific to the type of dataset being queried. This generic query response is supposed to be the same for all of the "typed" services derived from the generic Dataset. The data model (if any is formally defined) for a specific type of dataset such as Spectrum, Image, etc. also inherits the generic dataset metadata. Ideally we want to have all these be internally consistent for a particular major version of the interface (1.x, 2.x, etc.). Once we have SSA, SIAV2, ObsTAP, etc. it is clear that it could be attractive to standardize the generic metadata - which comprises most of the metadata returned - among all these very similar services.

Now there are some things which could possibly be done differently several years later, for example the "Dataset" component in the query response might instead take on the class name of the specific type of data which it describes, such as Spectrum or Image. Either approach is possible and it doesn't make all that much difference.

The full discussion from which the above was taken is available at: http://www.ivoa.net/forum/dal/1004/1690.htm

SSAP V1.1 Collaborative Page

SSAP/SDM Documents

The draft V1.1 documents can be found here:

• Spectrum DM V1.1 draft (pdf)

• SSA V1.1 data model spreadsheet

The current SSA/SDM standards documents can be found here:

• Simple Spectral Access V1.04 (1 Feb 2009)
• Spectrum Data Model (29 Oct 2008)
• SSA/SDM model summary spreadsheet

-- DougTody - 04 Apr 2011

20110118 Draft Discussion

The (ab)use of XML namespace attributes (xmlns:spec="http://www.ivoa.net/xml/SpectrumModel/v1.01") for the "declaration" of utype data model identifiers has been discussed in the context of STC in VOTable; back then, the result was that the data model URI should be transmitted in DataModel.URI. In addition to repairing the mismatch between (fixed) utype DM identifiers and (in principle arbitrary) XML namespace prefixes, this would for the Spectrum data model give the additional advantage that the DM identifier is serialized into FITS as well.

Although very few actual SSA implementations get the XML namespace declaration "right" (i.e., as specified in the existing standard) for ssa and spec utypes and I cannot see how applications would rely on it anyway, dropping it is probably not an option within a minor update (though I'd like a deprecation notice). Still, I'd really appreciate to see a "should" clause as to including DataModel.URI in both SSA and SDM. That's not going to hurt anyone but will help future standards development.

[Oh, and in case you have doubts whether I'm addressing a real problem here: Try following the recommendation as to the "canonical" prefixes in SimpleDALRegExt and then write in ssa utypes in the registration document...]

-- MarkusDemleitner - 11 Mar 2011

Markus - I agree this usage in SSA V1.0/1.1 is incorrect and should be fixed, once we have an agreement on how to handle this within VOTable and for Utype namespaces. Best to leave it as it is until this is settled.

-- DougTody - 04 Apr 2011

SSAP/SDM Document Update Discussion

At the EuroVO AIDA a small group of people interested in updating the SSA and SpectrumDM document gathered. Participants; BrunoRino, KeithNoddle (DAL chair), MireilleLouys (DM chair), AlbertoMicol, IgorChilingarian, JesusSalgado, FrancoisBonnarel.

Separately from this meeting, from the US side, DougTody and JonathanMcDowell (the SSAP and SDM document editors) had also been discussing these proposed changes with the document authors and others via email.

We set forth the following goal: To create 1.1 versions (of SSA and the SpectrumDM) that attempt to fix or clarify what were seen as inconsistencies between the two documents. Mor extensive clarifications or additions could be added in a version 1.2. Big changes, possibly not backwards compatible, are postponed to 2.0 versions.

The rationale for this split is to have version 1.1 approved quickly and replace the current 1.0. A version 1.2 would permit more changes but would take significantly longer, gathering more extensive input from the community.

In both cases (1.1. and 1.2) the assumption is that no existing application should break. This means that when a fix creates potential breakage, the potentially affected applications should be consulted.

Below, the result of our discussion on what must be changed in order to reach version 1.1. We were lucky enough to have both the DM and DAL Working Group leads; we concluded that after a short period of time after circulating these meeting minutes to the relevant lists, Working Drafts should be produced, ahead of the May interop in Victoria.

SSAP/SDM Inconsistencies

1. The SSA data model is derived, but decoupled, from the SpectrumDM

The acknowledged divergences are: - "required" flags (Mandatory, Recommended, Optional) are different - the SSA data model contains service related metadata, that have no meaning for the SpectrumDM - the SpectrumDM contains metadata related to data analysis (Data.*) that are of no interest for data discovery (which is the purpose of SSA) - in SSA, the "Spectrum." prefix found in the SpectrumDM utypes was dropped, and in some cases a "Dataset." prefix was added.

Even if those divergences are discrepancies, they are not going to be fixed in a 1.x document. Instead, SSA Section 2.2 ("Data Model") must be changed to reflect these divergences. It should state explicitly that a reader of the SSA should only refer to the SpectrumDM to seek clarification about the meaning of metadata fields. The specification of required fields, the UCDs, and even the utype syntax for setting up a SSA server are to be read from the SSA document. Metadata defined in the SpectrumDM, but not listed in the SSA, are not relevant for SSA service interface.

This is a compromise towards reaching rapidly a stable revision of the documents. We would much prefer to have a single source for the definition of the datamodel, which the SSA protocol would just extend. But we believe this is too large of a task to achieve while maintaining backwards compatibility, on a reasonable time-scale.

2. The use of "*" and ".." in UCDs

This must be eliminated.

These characters are always used in the context of "em.*" or "em...". Our understanding is that these characters are placeholders, which a data provider must fill in, according to its requirements. A list of all possible values and meanings must be provided instead, using the following primary UCDs:

em.wl em.freq em.energy

3. The use of "*" in utypes

This must be eliminated. The correct utypes to use are the ones in the "Query Response" section of SSA, but without the "*":

Char.SpatialAxis.SamplingPrecision.SampleExtent Char.SpatialAxis.SamplingPrecision.FillFactor Char.SpectralAxis.SamplingPrecision.SampleExtent Char.SpectralAxis.SamplingPrecision.FillFactor Char.TimeAxis.SamplingPrecision.SampleExtent Char.TimeAxis.SamplingPrecision.FillFactor

4. Missing UCDs in SSA The SSA is correct, the SpectrumDM should not have a UCD for the following elements (the UCDs provided in the SpectrumDM on those elements are either wrong or confusing):

5 Misc. typos The SpectrumDM (on the FITS serialization section) should fix the following utypes:

The SSA should fix the following UCD: em;spec.binSize -> em;spect.binSize

Extra spaces in UCDs and utypes are typos and should be removed

6. Dimensional analysis typo

In the SpectrumDM, change (from 10-10 to 1E-10) the way to express exponents within the dimensional analysis elements

Section 3.2 should read:

Pedro Osuna and Jesus Salgado have proposed a representation in the spirit of dimensional analysis, using the symbols M, L, T to signify kg, m, s respectively and omitting the ** for powers, so that 10**3 Jy Hz which is equivalent to 10**-23 kg s**-2 is written compactly as 1E-23MT-2

and the example in section 9.4:

SPECSDIM= '1E-10 L' / Spectral SIDim FLUXSDIM= '1E+7 ML-1T-3' / Flux SDim

7. Wrong UCDs

Spectrum.Char.SpectralAxis.Coverage.Location.Value has a wrong UCD of instr.bandpass. It should become the following list (in accordance to point 2 above):

em.wl;instr.bandpass em.freq;instr.bandpass em.energy;instr.bandpass

8. Inconsistencies within the SSA itself

Add the Dataset.Deleted utype to Appendix D.

Remove the Data.* utypes from Appendix D.

Add the remaining missing utypes present in Appendix D to section 4.2 (the list is too long and too boring to show here)

9. Inconsistencies within the SpectrumDM itself

Add a comment to the FITS serialisation stating that it does not cover the whole of the SpectrumDM utypes

Note: The "consolidation" activities detailed in points 7 and 8 should also make sure the order by which the utypes are listed is consistent throughout all documents.

See also: http://www.ivoa.net/internal/IVOA/IvoaExecMeetingFM34/SSA_SDM_assessment.xls

SSA and SDM Documents Update (response to above)

I am trying to synthesize these various comments on SSA/SDM. I think I mostly agree with the detailed changes suggested. The comments here are mostly higher level.

At this point it appears that both Jonathan and Bruno agree that the SSA metadata and the SDM, while quite similar, are not quite the same thing (hence where there are differences they are not necessarily "inconsistencies" but are restrictions in units, mandatory fields etc. required by the specific application). SSA defines the data model used in the query response, describing the data which a service can deliver. SDM defines the data model of a spectrum dataset/file/instance.

A key point which has not been adequately discussed is the role of the generic GDS/Obs data model. While the SDM explicitly describes a Spectrum instance, most of what is in the SSA query response is actually GDS/ObsDM metadata and not specific to Spectra at all - at this level the metadata should be the same for an image, for an SED, for a time series, etc. It does not make a whole lot of sense for example to have Spectrum.Target.Name and Image.Target.Name and allow these to be two different things. An application which deals with multiple types of data would like global and generic metadata such as this, which is applicable to any type of data, to be usable as just "Target.Name" (possibly in a future version something such as Obs.Target.Name would be preferable, but we did not have that when SSA was defined). Once we have a full set of DAL2 services one would like to have the common stuff factored out so that it does not have to be customized for every type of data. For the purposes of data discovery and description most of this stuff is the same.

The above is the reason why the "Spectrum" prefix does not carry over into the SSA query response. Rather, essentially all of the SSA query response metadata is either generic dataset metadata, or specific to the service mechanism itself and hence also generic and applicable to all such services. It is not actually from the SDM at all, we merely require them to be consistent. That small part of the query response metadata which is not generic and which is specific to the type of dataset being queried (Spectrum, Image, whatever) is what is in the "Dataset" component model.

In other words, for a DAL2 query response what we have is virtually all generic dataset / Obs metadata, plus a separate "Dataset" element which is specific to the type of dataset being queried. This generic query response is supposed to be the same for all of the "typed" services derived from the generic Dataset. The data model (if any is formally defined) for a specific type of dataset such as Spectrum, Image, etc. also inherits the generic dataset metadata. Ideally we want to have all these be internally consistent for a particular major version of the interface (1.x, 2.x, etc.). Once we have SSA, SIAV2, ObsTAP, etc. it is clear that it could be attractive to standardize the generic metadata - which comprises most of the metadata returned - among all these very similar services.

Now there are some things which could possibly be done differently several years later, for example the "Dataset" component in the query response might instead take on the class name of the specific type of data which it describes, such as Spectrum or Image. Either approach is possible and it doesn't make all that much difference.

The full discussion from which the above was taken is available at: http://www.ivoa.net/forum/dal/1004/1690.htm

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->

SSAP V1.1 Collaborative Page

SSAP/SDM Documents

The draft V1.1 documents can be found here:

• Spectrum DM V1.1 draft (pdf)

• SSA V1.1 data model spreadsheet

The current SSA/SDM standards documents can be found here:

• Simple Spectral Access V1.04 (1 Feb 2009)
• Spectrum Data Model (29 Oct 2008)
• SSA/SDM model summary spreadsheet

20110118 Draft Discussion

The (ab)use of XML namespace attributes (xmlns:spec="http://www.ivoa.net/xml/SpectrumModel/v1.01") for the "declaration" of utype data model identifiers has been discussed in the context of STC in VOTable; back then, the result was that the data model URI should be transmitted in DataModel.URI. In addition to repairing the mismatch between (fixed) utype DM identifiers and (in principle arbitrary) XML namespace prefixes, this would for the Spectrum data model give the additional advantage that the DM identifier is serialized into FITS as well.

Although very few actual SSA implementations get the XML namespace declaration "right" (i.e., as specified in the existing standard) for ssa and spec utypes and I cannot see how applications would rely on it anyway, dropping it is probably not an option within a minor update (though I'd like a deprecation notice). Still, I'd really appreciate to see a "should" clause as to including DataModel.URI in both SSA and SDM. That's not going to hurt anyone but will help future standards development.

[Oh, and in case you have doubts whether I'm addressing a real problem here: Try following the recommendation as to the "canonical" prefixes in SimpleDALRegExt and then write in ssa utypes in the registration document...]

-- MarkusDemleitner - 11 Mar 2011

SSAP/SDM Document Update Discussion

At the EuroVO AIDA a small group of people interested in updating the SSA and SpectrumDM document gathered. Participants; BrunoRino, KeithNoddle (DAL chair), MireilleLouys (DM chair), AlbertoMicol, IgorChilingarian, JesusSalgado, FrancoisBonnarel.

Separately from this meeting, from the US side, DougTody and JonathanMcDowell (the SSAP and SDM document editors) had also been discussing these proposed changes with the document authors and others via email.

We set forth the following goal: To create 1.1 versions (of SSA and the SpectrumDM) that attempt to fix or clarify what were seen as inconsistencies between the two documents. Mor extensive clarifications or additions could be added in a version 1.2. Big changes, possibly not backwards compatible, are postponed to 2.0 versions.

The rationale for this split is to have version 1.1 approved quickly and replace the current 1.0. A version 1.2 would permit more changes but would take significantly longer, gathering more extensive input from the community.

In both cases (1.1. and 1.2) the assumption is that no existing application should break. This means that when a fix creates potential breakage, the potentially affected applications should be consulted.

Below, the result of our discussion on what must be changed in order to reach version 1.1. We were lucky enough to have both the DM and DAL Working Group leads; we concluded that after a short period of time after circulating these meeting minutes to the relevant lists, Working Drafts should be produced, ahead of the May interop in Victoria.

SSAP/SDM Inconsistencies

1. The SSA data model is derived, but decoupled, from the SpectrumDM

The acknowledged divergences are: - "required" flags (Mandatory, Recommended, Optional) are different - the SSA data model contains service related metadata, that have no meaning for the SpectrumDM - the SpectrumDM contains metadata related to data analysis (Data.*) that are of no interest for data discovery (which is the purpose of SSA) - in SSA, the "Spectrum." prefix found in the SpectrumDM utypes was dropped, and in some cases a "Dataset." prefix was added.

Even if those divergences are discrepancies, they are not going to be fixed in a 1.x document. Instead, SSA Section 2.2 ("Data Model") must be changed to reflect these divergences. It should state explicitly that a reader of the SSA should only refer to the SpectrumDM to seek clarification about the meaning of metadata fields. The specification of required fields, the UCDs, and even the utype syntax for setting up a SSA server are to be read from the SSA document. Metadata defined in the SpectrumDM, but not listed in the SSA, are not relevant for SSA service interface.

This is a compromise towards reaching rapidly a stable revision of the documents. We would much prefer to have a single source for the definition of the datamodel, which the SSA protocol would just extend. But we believe this is too large of a task to achieve while maintaining backwards compatibility, on a reasonable time-scale.

2. The use of "*" and ".." in UCDs

This must be eliminated.

These characters are always used in the context of "em.*" or "em...". Our understanding is that these characters are placeholders, which a data provider must fill in, according to its requirements. A list of all possible values and meanings must be provided instead, using the following primary UCDs:

em.wl em.freq em.energy

3. The use of "*" in utypes

This must be eliminated. The correct utypes to use are the ones in the "Query Response" section of SSA, but without the "*":

Char.SpatialAxis.SamplingPrecision.SampleExtent Char.SpatialAxis.SamplingPrecision.FillFactor Char.SpectralAxis.SamplingPrecision.SampleExtent Char.SpectralAxis.SamplingPrecision.FillFactor Char.TimeAxis.SamplingPrecision.SampleExtent Char.TimeAxis.SamplingPrecision.FillFactor

4. Missing UCDs in SSA The SSA is correct, the SpectrumDM should not have a UCD for the following elements (the UCDs provided in the SpectrumDM on those elements are either wrong or confusing):

5 Misc. typos The SpectrumDM (on the FITS serialization section) should fix the following utypes:

The SSA should fix the following UCD: em;spec.binSize -> em;spect.binSize

Extra spaces in UCDs and utypes are typos and should be removed

6. Dimensional analysis typo

In the SpectrumDM, change (from 10-10 to 1E-10) the way to express exponents within the dimensional analysis elements

Section 3.2 should read:

Pedro Osuna and Jesus Salgado have proposed a representation in the spirit of dimensional analysis, using the symbols M, L, T to signify kg, m, s respectively and omitting the ** for powers, so that 10**3 Jy Hz which is equivalent to 10**-23 kg s**-2 is written compactly as 1E-23MT-2

and the example in section 9.4:

SPECSDIM= '1E-10 L' / Spectral SIDim FLUXSDIM= '1E+7 ML-1T-3' / Flux SDim

7. Wrong UCDs

Spectrum.Char.SpectralAxis.Coverage.Location.Value has a wrong UCD of instr.bandpass. It should become the following list (in accordance to point 2 above):

em.wl;instr.bandpass em.freq;instr.bandpass em.energy;instr.bandpass

8. Inconsistencies within the SSA itself

Add the Dataset.Deleted utype to Appendix D.

Remove the Data.* utypes from Appendix D.

Add the remaining missing utypes present in Appendix D to section 4.2 (the list is too long and too boring to show here)

9. Inconsistencies within the SpectrumDM itself

Add a comment to the FITS serialisation stating that it does not cover the whole of the SpectrumDM utypes

Note: The "consolidation" activities detailed in points 7 and 8 should also make sure the order by which the utypes are listed is consistent throughout all documents.

See also: http://www.ivoa.net/internal/IVOA/IvoaExecMeetingFM34/SSA_SDM_assessment.xls

SSA and SDM Documents Update (response to above)

I am trying to synthesize these various comments on SSA/SDM. I think I mostly agree with the detailed changes suggested. The comments here are mostly higher level.

At this point it appears that both Jonathan and Bruno agree that the SSA metadata and the SDM, while quite similar, are not quite the same thing (hence where there are differences they are not necessarily "inconsistencies" but are restrictions in units, mandatory fields etc. required by the specific application). SSA defines the data model used in the query response, describing the data which a service can deliver. SDM defines the data model of a spectrum dataset/file/instance.

A key point which has not been adequately discussed is the role of the generic GDS/Obs data model. While the SDM explicitly describes a Spectrum instance, most of what is in the SSA query response is actually GDS/ObsDM metadata and not specific to Spectra at all - at this level the metadata should be the same for an image, for an SED, for a time series, etc. It does not make a whole lot of sense for example to have Spectrum.Target.Name and Image.Target.Name and allow these to be two different things. An application which deals with multiple types of data would like global and generic metadata such as this, which is applicable to any type of data, to be usable as just "Target.Name" (possibly in a future version something such as Obs.Target.Name would be preferable, but we did not have that when SSA was defined). Once we have a full set of DAL2 services one would like to have the common stuff factored out so that it does not have to be customized for every type of data. For the purposes of data discovery and description most of this stuff is the same.

The above is the reason why the "Spectrum" prefix does not carry over into the SSA query response. Rather, essentially all of the SSA query response metadata is either generic dataset metadata, or specific to the service mechanism itself and hence also generic and applicable to all such services. It is not actually from the SDM at all, we merely require them to be consistent. That small part of the query response metadata which is not generic and which is specific to the type of dataset being queried (Spectrum, Image, whatever) is what is in the "Dataset" component model.

In other words, for a DAL2 query response what we have is virtually all generic dataset / Obs metadata, plus a separate "Dataset" element which is specific to the type of dataset being queried. This generic query response is supposed to be the same for all of the "typed" services derived from the generic Dataset. The data model (if any is formally defined) for a specific type of dataset such as Spectrum, Image, etc. also inherits the generic dataset metadata. Ideally we want to have all these be internally consistent for a particular major version of the interface (1.x, 2.x, etc.). Once we have SSA, SIAV2, ObsTAP, etc. it is clear that it could be attractive to standardize the generic metadata - which comprises most of the metadata returned - among all these very similar services.

Now there are some things which could possibly be done differently several years later, for example the "Dataset" component in the query response might instead take on the class name of the specific type of data which it describes, such as Spectrum or Image. Either approach is possible and it doesn't make all that much difference.

The full discussion from which the above was taken is available at: http://www.ivoa.net/forum/dal/1004/1690.htm

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->

SSAP V1.1 Collaborative Page

SSAP/SDM Documents

The draft V1.1 documents can be found here:

• Spectrum DM V1.1 draft (pdf)

• SSA V1.1 data model spreadsheet

The current SSA/SDM standards documents can be found here:

• Simple Spectral Access V1.04 (1 Feb 2009)
• Spectrum Data Model (29 Oct 2008)
• SSA/SDM model summary spreadsheet

20110118 Draft Discussion

The (ab)use of XML namespace attributes (xmlns:spec="http://www.ivoa.net/xml/SpectrumModel/v1.01") for the "declaration" of utype data model identifiers has been discussed in the context of STC in VOTable; back then, the result was that the data model URI should be transmitted in DataModel.URI. In addition to repairing the mismatch between (fixed) utype DM identifiers and (in principle arbitrary) XML namespace prefixes, this would for the Spectrum data model give the additional advantage that the DM identifier is serialized into FITS as well.

Although very few actual SSA implementations get the XML namespace declaration "right" (i.e., as specified in the existing standard) for ssa and spec utypes and I cannot see how applications would rely on it anyway, dropping it is probably not an option within a minor update (though I'd like a deprecation notice). Still, I'd really appreciate to see a "should" clause as to including DataModel.URI in both SSA and SDM. That's not going to hurt anyone but will help future standards development.

[Oh, and in case you have doubts whether I'm addressing a real problem here: Try following the recommendation as to the "canonical" prefixes in SimpleDALRegExt and then write in ssa utypes in the registration document...]

-- MarkusDemleitner - 11 Mar 2011

SSAP/SDM Document Update Discussion

At the EuroVO AIDA a small group of people interested in updating the SSA and SpectrumDM document gathered. Participants; BrunoRino, KeithNoddle (DAL chair), MireilleLouys (DM chair), AlbertoMicol, IgorChilingarian, JesusSalgado, FrancoisBonnarel.

Separately from this meeting, from the US side, DougTody and JonathanMcDowell (the SSAP and SDM document editors) had also been discussing these proposed changes with the document authors and others via email.

We set forth the following goal: To create 1.1 versions (of SSA and the SpectrumDM) that attempt to fix or clarify what were seen as inconsistencies between the two documents. Mor extensive clarifications or additions could be added in a version 1.2. Big changes, possibly not backwards compatible, are postponed to 2.0 versions.

The rationale for this split is to have version 1.1 approved quickly and replace the current 1.0. A version 1.2 would permit more changes but would take significantly longer, gathering more extensive input from the community.

In both cases (1.1. and 1.2) the assumption is that no existing application should break. This means that when a fix creates potential breakage, the potentially affected applications should be consulted.

Below, the result of our discussion on what must be changed in order to reach version 1.1. We were lucky enough to have both the DM and DAL Working Group leads; we concluded that after a short period of time after circulating these meeting minutes to the relevant lists, Working Drafts should be produced, ahead of the May interop in Victoria.

SSAP/SDM Inconsistencies

1. The SSA data model is derived, but decoupled, from the SpectrumDM

The acknowledged divergences are: - "required" flags (Mandatory, Recommended, Optional) are different - the SSA data model contains service related metadata, that have no meaning for the SpectrumDM - the SpectrumDM contains metadata related to data analysis (Data.*) that are of no interest for data discovery (which is the purpose of SSA) - in SSA, the "Spectrum." prefix found in the SpectrumDM utypes was dropped, and in some cases a "Dataset." prefix was added.

Even if those divergences are discrepancies, they are not going to be fixed in a 1.x document. Instead, SSA Section 2.2 ("Data Model") must be changed to reflect these divergences. It should state explicitly that a reader of the SSA should only refer to the SpectrumDM to seek clarification about the meaning of metadata fields. The specification of required fields, the UCDs, and even the utype syntax for setting up a SSA server are to be read from the SSA document. Metadata defined in the SpectrumDM, but not listed in the SSA, are not relevant for SSA service interface.

This is a compromise towards reaching rapidly a stable revision of the documents. We would much prefer to have a single source for the definition of the datamodel, which the SSA protocol would just extend. But we believe this is too large of a task to achieve while maintaining backwards compatibility, on a reasonable time-scale.

2. The use of "*" and ".." in UCDs

This must be eliminated.

These characters are always used in the context of "em.*" or "em...". Our understanding is that these characters are placeholders, which a data provider must fill in, according to its requirements. A list of all possible values and meanings must be provided instead, using the following primary UCDs:

em.wl em.freq em.energy

3. The use of "*" in utypes

This must be eliminated. The correct utypes to use are the ones in the "Query Response" section of SSA, but without the "*":

Char.SpatialAxis.SamplingPrecision.SampleExtent Char.SpatialAxis.SamplingPrecision.FillFactor Char.SpectralAxis.SamplingPrecision.SampleExtent Char.SpectralAxis.SamplingPrecision.FillFactor Char.TimeAxis.SamplingPrecision.SampleExtent Char.TimeAxis.SamplingPrecision.FillFactor

4. Missing UCDs in SSA The SSA is correct, the SpectrumDM should not have a UCD for the following elements (the UCDs provided in the SpectrumDM on those elements are either wrong or confusing):

5 Misc. typos The SpectrumDM (on the FITS serialization section) should fix the following utypes:

The SSA should fix the following UCD: em;spec.binSize -> em;spect.binSize

Extra spaces in UCDs and utypes are typos and should be removed

6. Dimensional analysis typo

In the SpectrumDM, change (from 10-10 to 1E-10) the way to express exponents within the dimensional analysis elements

Section 3.2 should read:

Pedro Osuna and Jesus Salgado have proposed a representation in the spirit of dimensional analysis, using the symbols M, L, T to signify kg, m, s respectively and omitting the ** for powers, so that 10**3 Jy Hz which is equivalent to 10**-23 kg s**-2 is written compactly as 1E-23MT-2

and the example in section 9.4:

SPECSDIM= '1E-10 L' / Spectral SIDim FLUXSDIM= '1E+7 ML-1T-3' / Flux SDim

7. Wrong UCDs

Spectrum.Char.SpectralAxis.Coverage.Location.Value has a wrong UCD of instr.bandpass. It should become the following list (in accordance to point 2 above):

em.wl;instr.bandpass em.freq;instr.bandpass em.energy;instr.bandpass

8. Inconsistencies within the SSA itself

Add the Dataset.Deleted utype to Appendix D.

Remove the Data.* utypes from Appendix D.

Add the remaining missing utypes present in Appendix D to section 4.2 (the list is too long and too boring to show here)

9. Inconsistencies within the SpectrumDM itself

Add a comment to the FITS serialisation stating that it does not cover the whole of the SpectrumDM utypes

Note: The "consolidation" activities detailed in points 7 and 8 should also make sure the order by which the utypes are listed is consistent throughout all documents.

See also: http://www.ivoa.net/internal/IVOA/IvoaExecMeetingFM34/SSA_SDM_assessment.xls

SSA and SDM Documents Update (response to above)

I am trying to synthesize these various comments on SSA/SDM. I think I mostly agree with the detailed changes suggested. The comments here are mostly higher level.

At this point it appears that both Jonathan and Bruno agree that the SSA metadata and the SDM, while quite similar, are not quite the same thing (hence where there are differences they are not necessarily "inconsistencies" but are restrictions in units, mandatory fields etc. required by the specific application). SSA defines the data model used in the query response, describing the data which a service can deliver. SDM defines the data model of a spectrum dataset/file/instance.

A key point which has not been adequately discussed is the role of the generic GDS/Obs data model. While the SDM explicitly describes a Spectrum instance, most of what is in the SSA query response is actually GDS/ObsDM metadata and not specific to Spectra at all - at this level the metadata should be the same for an image, for an SED, for a time series, etc. It does not make a whole lot of sense for example to have Spectrum.Target.Name and Image.Target.Name and allow these to be two different things. An application which deals with multiple types of data would like global and generic metadata such as this, which is applicable to any type of data, to be usable as just "Target.Name" (possibly in a future version something such as Obs.Target.Name would be preferable, but we did not have that when SSA was defined). Once we have a full set of DAL2 services one would like to have the common stuff factored out so that it does not have to be customized for every type of data. For the purposes of data discovery and description most of this stuff is the same.

The above is the reason why the "Spectrum" prefix does not carry over into the SSA query response. Rather, essentially all of the SSA query response metadata is either generic dataset metadata, or specific to the service mechanism itself and hence also generic and applicable to all such services. It is not actually from the SDM at all, we merely require them to be consistent. That small part of the query response metadata which is not generic and which is specific to the type of dataset being queried (Spectrum, Image, whatever) is what is in the "Dataset" component model.

In other words, for a DAL2 query response what we have is virtually all generic dataset / Obs metadata, plus a separate "Dataset" element which is specific to the type of dataset being queried. This generic query response is supposed to be the same for all of the "typed" services derived from the generic Dataset. The data model (if any is formally defined) for a specific type of dataset such as Spectrum, Image, etc. also inherits the generic dataset metadata. Ideally we want to have all these be internally consistent for a particular major version of the interface (1.x, 2.x, etc.). Once we have SSA, SIAV2, ObsTAP, etc. it is clear that it could be attractive to standardize the generic metadata - which comprises most of the metadata returned - among all these very similar services.

Now there are some things which could possibly be done differently several years later, for example the "Dataset" component in the query response might instead take on the class name of the specific type of data which it describes, such as Spectrum or Image. Either approach is possible and it doesn't make all that much difference.

The full discussion from which the above was taken is available at: http://www.ivoa.net/forum/dal/1004/1690.htm

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->

SSAP V1.1 Collaborative Page

SSAP/SDM Documents

The draft V1.1 documents can be found here:

• Spectrum DM V1.1 draft (pdf)
• SSA V1.1 draft (pdf) (doc)
• SSA V1.1 data model spreadsheet

The current SSA/SDM standards documents can be found here:

• Simple Spectral Access V1.04 (1 Feb 2009)
• Spectrum Data Model (29 Oct 2008)
• SSA/SDM model summary spreadsheet

20110118 Draft Discussion

The (ab)use of XML namespace attributes (xmlns:spec="http://www.ivoa.net/xml/SpectrumModel/v1.01") for the "declaration" of utype data model identifiers has been discussed in the context of STC in VOTable; back then, the result was that the data model URI should be transmitted in DataModel.URI. In addition to repairing the mismatch between (fixed) utype DM identifiers and (in principle arbitrary) XML namespace prefixes, this would for the Spectrum data model give the additional advantage that the DM identifier is serialized into FITS as well.

Although very few actual SSA implementations get the XML namespace declaration "right" (i.e., as specified in the existing standard) for ssa and spec utypes and I cannot see how applications would rely on it anyway, dropping

[Oh, and in case you have doubts whether I'm addressing a real problem here: Try following the recommendation as to the "canonical" prefixes in SimpleDALRegExt and then write in ssa utypes in the registration document...]

-- MarkusDemleitner - 11 Mar 2011

SSAP/SDM Document Update Discussion

At the EuroVO AIDA a small group of people interested in updating the SSA and SpectrumDM document gathered. Participants; BrunoRino, KeithNoddle (DAL chair), MireilleLouys (DM chair), AlbertoMicol, IgorChilingarian, JesusSalgado, FrancoisBonnarel.

Separately from this meeting, from the US side, DougTody and JonathanMcDowell (the SSAP and SDM document editors) had also been discussing these proposed changes with the document authors and others via email.

We set forth the following goal: To create 1.1 versions (of SSA and the SpectrumDM) that attempt to fix or clarify what were seen as inconsistencies between the two documents. Mor extensive clarifications or additions could be added in a version 1.2. Big changes, possibly not backwards compatible, are postponed to 2.0 versions.

The rationale for this split is to have version 1.1 approved quickly and replace the current 1.0. A version 1.2 would permit more changes but would take significantly longer, gathering more extensive input from the community.

In both cases (1.1. and 1.2) the assumption is that no existing application should break. This means that when a fix creates potential breakage, the potentially affected applications should be consulted.

Below, the result of our discussion on what must be changed in order to reach version 1.1. We were lucky enough to have both the DM and DAL Working Group leads; we concluded that after a short period of time after circulating these meeting minutes to the relevant lists, Working Drafts should be produced, ahead of the May interop in Victoria.

SSAP/SDM Inconsistencies

1. The SSA data model is derived, but decoupled, from the SpectrumDM

The acknowledged divergences are: - "required" flags (Mandatory, Recommended, Optional) are different - the SSA data model contains service related metadata, that have no meaning for the SpectrumDM - the SpectrumDM contains metadata related to data analysis (Data.*) that are of no interest for data discovery (which is the purpose of SSA) - in SSA, the "Spectrum." prefix found in the SpectrumDM utypes was dropped, and in some cases a "Dataset." prefix was added.

Even if those divergences are discrepancies, they are not going to be fixed in a 1.x document. Instead, SSA Section 2.2 ("Data Model") must be changed to reflect these divergences. It should state explicitly that a reader of the SSA should only refer to the SpectrumDM to seek clarification about the meaning of metadata fields. The specification of required fields, the UCDs, and even the utype syntax for setting up a SSA server are to be read from the SSA document. Metadata defined in the SpectrumDM, but not listed in the SSA, are not relevant for SSA service interface.

This is a compromise towards reaching rapidly a stable revision of the documents. We would much prefer to have a single source for the definition of the datamodel, which the SSA protocol would just extend. But we believe this is too large of a task to achieve while maintaining backwards compatibility, on a reasonable time-scale.

2. The use of "*" and ".." in UCDs

This must be eliminated.

These characters are always used in the context of "em.*" or "em...". Our understanding is that these characters are placeholders, which a data provider must fill in, according to its requirements. A list of all possible values and meanings must be provided instead, using the following primary UCDs:

em.wl em.freq em.energy

3. The use of "*" in utypes

This must be eliminated. The correct utypes to use are the ones in the "Query Response" section of SSA, but without the "*":

Char.SpatialAxis.SamplingPrecision.SampleExtent Char.SpatialAxis.SamplingPrecision.FillFactor Char.SpectralAxis.SamplingPrecision.SampleExtent Char.SpectralAxis.SamplingPrecision.FillFactor Char.TimeAxis.SamplingPrecision.SampleExtent Char.TimeAxis.SamplingPrecision.FillFactor

4. Missing UCDs in SSA The SSA is correct, the SpectrumDM should not have a UCD for the following elements (the UCDs provided in the SpectrumDM on those elements are either wrong or confusing):

5 Misc. typos The SpectrumDM (on the FITS serialization section) should fix the following utypes:

The SSA should fix the following UCD: em;spec.binSize -> em;spect.binSize

Extra spaces in UCDs and utypes are typos and should be removed

6. Dimensional analysis typo

In the SpectrumDM, change (from 10-10 to 1E-10) the way to express exponents within the dimensional analysis elements

Section 3.2 should read:

Pedro Osuna and Jesus Salgado have proposed a representation in the spirit of dimensional analysis, using the symbols M, L, T to signify kg, m, s respectively and omitting the ** for powers, so that 10**3 Jy Hz which is equivalent to 10**-23 kg s**-2 is written compactly as 1E-23MT-2

and the example in section 9.4:

SPECSDIM= '1E-10 L' / Spectral SIDim FLUXSDIM= '1E+7 ML-1T-3' / Flux SDim

7. Wrong UCDs

Spectrum.Char.SpectralAxis.Coverage.Location.Value has a wrong UCD of instr.bandpass. It should become the following list (in accordance to point 2 above):

em.wl;instr.bandpass em.freq;instr.bandpass em.energy;instr.bandpass

8. Inconsistencies within the SSA itself

Add the Dataset.Deleted utype to Appendix D.

Remove the Data.* utypes from Appendix D.

Add the remaining missing utypes present in Appendix D to section 4.2 (the list is too long and too boring to show here)

9. Inconsistencies within the SpectrumDM itself

Add a comment to the FITS serialisation stating that it does not cover the whole of the SpectrumDM utypes

Note: The "consolidation" activities detailed in points 7 and 8 should also make sure the order by which the utypes are listed is consistent throughout all documents.

See also: http://www.ivoa.net/internal/IVOA/IvoaExecMeetingFM34/SSA_SDM_assessment.xls

SSA and SDM Documents Update (response to above)

I am trying to synthesize these various comments on SSA/SDM. I think I mostly agree with the detailed changes suggested. The comments here are mostly higher level.

At this point it appears that both Jonathan and Bruno agree that the SSA metadata and the SDM, while quite similar, are not quite the same thing (hence where there are differences they are not necessarily "inconsistencies" but are restrictions in units, mandatory fields etc. required by the specific application). SSA defines the data model used in the query response, describing the data which a service can deliver. SDM defines the data model of a spectrum dataset/file/instance.

A key point which has not been adequately discussed is the role of the generic GDS/Obs data model. While the SDM explicitly describes a Spectrum instance, most of what is in the SSA query response is actually GDS/ObsDM metadata and not specific to Spectra at all - at this level the metadata should be the same for an image, for an SED, for a time series, etc. It does not make a whole lot of sense for example to have Spectrum.Target.Name and Image.Target.Name and allow these to be two different things. An application which deals with multiple types of data would like global and generic metadata such as this, which is applicable to any type of data, to be usable as just "Target.Name" (possibly in a future version something such as Obs.Target.Name would be preferable, but we did not have that when SSA was defined). Once we have a full set of DAL2 services one would like to have the common stuff factored out so that it does not have to be customized for every type of data. For the purposes of data discovery and description most of this stuff is the same.

The above is the reason why the "Spectrum" prefix does not carry over into the SSA query response. Rather, essentially all of the SSA query response metadata is either generic dataset metadata, or specific to the service mechanism itself and hence also generic and applicable to all such services. It is not actually from the SDM at all, we merely require them to be consistent. That small part of the query response metadata which is not generic and which is specific to the type of dataset being queried (Spectrum, Image, whatever) is what is in the "Dataset" component model.

In other words, for a DAL2 query response what we have is virtually all generic dataset / Obs metadata, plus a separate "Dataset" element which is specific to the type of dataset being queried. This generic query response is supposed to be the same for all of the "typed" services derived from the generic Dataset. The data model (if any is formally defined) for a specific type of dataset such as Spectrum, Image, etc. also inherits the generic dataset metadata. Ideally we want to have all these be internally consistent for a particular major version of the interface (1.x, 2.x, etc.). Once we have SSA, SIAV2, ObsTAP, etc. it is clear that it could be attractive to standardize the generic metadata - which comprises most of the metadata returned - among all these very similar services.

Now there are some things which could possibly be done differently several years later, for example the "Dataset" component in the query response might instead take on the class name of the specific type of data which it describes, such as Spectrum or Image. Either approach is possible and it doesn't make all that much difference.

The full discussion from which the above was taken is available at: http://www.ivoa.net/forum/dal/1004/1690.htm

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->

SSAP V1.1 Collaborative Page

SSAP/SDM Documents

The draft V1.1 documents can be found here:

• Spectrum DM V1.1 draft (pdf)
• SSA V1.1 draft (pdf) (doc)
• SSA V1.1 data model spreadsheet

The current SSA/SDM standards documents can be found here:

• Simple Spectral Access V1.04 (1 Feb 2009)
• Spectrum Data Model (29 Oct 2008)
• SSA/SDM model summary spreadsheet

SSAP/SDM Document Update Discussion

At the EuroVO AIDA a small group of people interested in updating the SSA and SpectrumDM document gathered. Participants; BrunoRino, KeithNoddle (DAL chair), MireilleLouys (DM chair), AlbertoMicol, IgorChilingarian, JesusSalgado, FrancoisBonnarel.

Separately from this meeting, from the US side, DougTody and JonathanMcDowell (the SSAP and SDM document editors) had also been discussing these proposed changes with the document authors and others via email.

We set forth the following goal: To create 1.1 versions (of SSA and the SpectrumDM) that attempt to fix or clarify what were seen as inconsistencies between the two documents. Mor extensive clarifications or additions could be added in a version 1.2. Big changes, possibly not backwards compatible, are postponed to 2.0 versions.

The rationale for this split is to have version 1.1 approved quickly and replace the current 1.0. A version 1.2 would permit more changes but would take significantly longer, gathering more extensive input from the community.

In both cases (1.1. and 1.2) the assumption is that no existing application should break. This means that when a fix creates potential breakage, the potentially affected applications should be consulted.

Below, the result of our discussion on what must be changed in order to reach version 1.1. We were lucky enough to have both the DM and DAL Working Group leads; we concluded that after a short period of time after circulating these meeting minutes to the relevant lists, Working Drafts should be produced, ahead of the May interop in Victoria.

SSAP/SDM Inconsistencies

1. The SSA data model is derived, but decoupled, from the SpectrumDM

The acknowledged divergences are: - "required" flags (Mandatory, Recommended, Optional) are different - the SSA data model contains service related metadata, that have no meaning for the SpectrumDM - the SpectrumDM contains metadata related to data analysis (Data.*) that are of no interest for data discovery (which is the purpose of SSA) - in SSA, the "Spectrum." prefix found in the SpectrumDM utypes was dropped, and in some cases a "Dataset." prefix was added.

Even if those divergences are discrepancies, they are not going to be fixed in a 1.x document. Instead, SSA Section 2.2 ("Data Model") must be changed to reflect these divergences. It should state explicitly that a reader of the SSA should only refer to the SpectrumDM to seek clarification about the meaning of metadata fields. The specification of required fields, the UCDs, and even the utype syntax for setting up a SSA server are to be read from the SSA document. Metadata defined in the SpectrumDM, but not listed in the SSA, are not relevant for SSA service interface.

This is a compromise towards reaching rapidly a stable revision of the documents. We would much prefer to have a single source for the definition of the datamodel, which the SSA protocol would just extend. But we believe this is too large of a task to achieve while maintaining backwards compatibility, on a reasonable time-scale.

2. The use of "*" and ".." in UCDs

This must be eliminated.

These characters are always used in the context of "em.*" or "em...". Our understanding is that these characters are placeholders, which a data provider must fill in, according to its requirements. A list of all possible values and meanings must be provided instead, using the following primary UCDs:

em.wl em.freq em.energy

3. The use of "*" in utypes

This must be eliminated. The correct utypes to use are the ones in the "Query Response" section of SSA, but without the "*":

Char.SpatialAxis.SamplingPrecision.SampleExtent Char.SpatialAxis.SamplingPrecision.FillFactor Char.SpectralAxis.SamplingPrecision.SampleExtent Char.SpectralAxis.SamplingPrecision.FillFactor Char.TimeAxis.SamplingPrecision.SampleExtent Char.TimeAxis.SamplingPrecision.FillFactor

4. Missing UCDs in SSA The SSA is correct, the SpectrumDM should not have a UCD for the following elements (the UCDs provided in the SpectrumDM on those elements are either wrong or confusing):

5 Misc. typos The SpectrumDM (on the FITS serialization section) should fix the following utypes:

The SSA should fix the following UCD: em;spec.binSize -> em;spect.binSize

Extra spaces in UCDs and utypes are typos and should be removed

6. Dimensional analysis typo

In the SpectrumDM, change (from 10-10 to 1E-10) the way to express exponents within the dimensional analysis elements

Section 3.2 should read:

Pedro Osuna and Jesus Salgado have proposed a representation in the spirit of dimensional analysis, using the symbols M, L, T to signify kg, m, s respectively and omitting the ** for powers, so that 10**3 Jy Hz which is equivalent to 10**-23 kg s**-2 is written compactly as 1E-23MT-2

and the example in section 9.4:

SPECSDIM= '1E-10 L' / Spectral SIDim FLUXSDIM= '1E+7 ML-1T-3' / Flux SDim

7. Wrong UCDs

Spectrum.Char.SpectralAxis.Coverage.Location.Value has a wrong UCD of instr.bandpass. It should become the following list (in accordance to point 2 above):

em.wl;instr.bandpass em.freq;instr.bandpass em.energy;instr.bandpass

8. Inconsistencies within the SSA itself

Add the Dataset.Deleted utype to Appendix D.

Remove the Data.* utypes from Appendix D.

Add the remaining missing utypes present in Appendix D to section 4.2 (the list is too long and too boring to show here)

9. Inconsistencies within the SpectrumDM itself

Add a comment to the FITS serialisation stating that it does not cover the whole of the SpectrumDM utypes

Note: The "consolidation" activities detailed in points 7 and 8 should also make sure the order by which the utypes are listed is consistent throughout all documents.

See also: http://www.ivoa.net/internal/IVOA/IvoaExecMeetingFM34/SSA_SDM_assessment.xls

SSA and SDM Documents Update (response to above)

I am trying to synthesize these various comments on SSA/SDM. I think I mostly agree with the detailed changes suggested. The comments here are mostly higher level.

At this point it appears that both Jonathan and Bruno agree that the SSA metadata and the SDM, while quite similar, are not quite the same thing (hence where there are differences they are not necessarily "inconsistencies" but are restrictions in units, mandatory fields etc. required by the specific application). SSA defines the data model used in the query response, describing the data which a service can deliver. SDM defines the data model of a spectrum dataset/file/instance.

A key point which has not been adequately discussed is the role of the generic GDS/Obs data model. While the SDM explicitly describes a Spectrum instance, most of what is in the SSA query response is actually GDS/ObsDM metadata and not specific to Spectra at all - at this level the metadata should be the same for an image, for an SED, for a time series, etc. It does not make a whole lot of sense for example to have Spectrum.Target.Name and Image.Target.Name and allow these to be two different things. An application which deals with multiple types of data would like global and generic metadata such as this, which is applicable to any type of data, to be usable as just "Target.Name" (possibly in a future version something such as Obs.Target.Name would be preferable, but we did not have that when SSA was defined). Once we have a full set of DAL2 services one would like to have the common stuff factored out so that it does not have to be customized for every type of data. For the purposes of data discovery and description most of this stuff is the same.

The above is the reason why the "Spectrum" prefix does not carry over into the SSA query response. Rather, essentially all of the SSA query response metadata is either generic dataset metadata, or specific to the service mechanism itself and hence also generic and applicable to all such services. It is not actually from the SDM at all, we merely require them to be consistent. That small part of the query response metadata which is not generic and which is specific to the type of dataset being queried (Spectrum, Image, whatever) is what is in the "Dataset" component model.

In other words, for a DAL2 query response what we have is virtually all generic dataset / Obs metadata, plus a separate "Dataset" element which is specific to the type of dataset being queried. This generic query response is supposed to be the same for all of the "typed" services derived from the generic Dataset. The data model (if any is formally defined) for a specific type of dataset such as Spectrum, Image, etc. also inherits the generic dataset metadata. Ideally we want to have all these be internally consistent for a particular major version of the interface (1.x, 2.x, etc.). Once we have SSA, SIAV2, ObsTAP, etc. it is clear that it could be attractive to standardize the generic metadata - which comprises most of the metadata returned - among all these very similar services.

Now there are some things which could possibly be done differently several years later, for example the "Dataset" component in the query response might instead take on the class name of the specific type of data which it describes, such as Spectrum or Image. Either approach is possible and it doesn't make all that much difference.

The full discussion from which the above was taken is available at: http://www.ivoa.net/forum/dal/1004/1690.htm

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->

SSAP V1.1 Collaborative Page

SSAP/SDM Documents

• SSA V1.1 data model spreadsheet

The current SSA/SDM standards documents can be found here:

• Simple Spectral Access V1.04 (1 Feb 2009)
• Spectrum Data Model (29 Oct 2008)
• SSA/SDM model summary spreadsheet

SSAP/SDM Document Update Discussion

At the EuroVO AIDA a small group of people interested in updating the SSA and SpectrumDM document gathered. Participants; BrunoRino, KeithNoddle (DAL chair), MireilleLouys (DM chair), AlbertoMicol, IgorChilingarian, JesusSalgado, FrancoisBonnarel.

Separately from this meeting, from the US side, DougTody and JonathanMcDowell (the SSAP and SDM document editors) had also been discussing these proposed changes with the document authors and others via email.

We set forth the following goal: To create 1.1 versions (of SSA and the SpectrumDM) that attempt to fix or clarify what were seen as inconsistencies between the two documents. Mor extensive clarifications or additions could be added in a version 1.2. Big changes, possibly not backwards compatible, are postponed to 2.0 versions.

The rationale for this split is to have version 1.1 approved quickly and replace the current 1.0. A version 1.2 would permit more changes but would take significantly longer, gathering more extensive input from the community.

In both cases (1.1. and 1.2) the assumption is that no existing application should break. This means that when a fix creates potential breakage, the potentially affected applications should be consulted.

Below, the result of our discussion on what must be changed in order to reach version 1.1. We were lucky enough to have both the DM and DAL Working Group leads; we concluded that after a short period of time after circulating these meeting minutes to the relevant lists, Working Drafts should be produced, ahead of the May interop in Victoria.

SSAP/SDM Inconsistencies

1. The SSA data model is derived, but decoupled, from the SpectrumDM

The acknowledged divergences are: - "required" flags (Mandatory, Recommended, Optional) are different - the SSA data model contains service related metadata, that have no meaning for the SpectrumDM - the SpectrumDM contains metadata related to data analysis (Data.*) that are of no interest for data discovery (which is the purpose of SSA) - in SSA, the "Spectrum." prefix found in the SpectrumDM utypes was dropped, and in some cases a "Dataset." prefix was added.

Even if those divergences are discrepancies, they are not going to be fixed in a 1.x document. Instead, SSA Section 2.2 ("Data Model") must be changed to reflect these divergences. It should state explicitly that a reader of the SSA should only refer to the SpectrumDM to seek clarification about the meaning of metadata fields. The specification of required fields, the UCDs, and even the utype syntax for setting up a SSA server are to be read from the SSA document. Metadata defined in the SpectrumDM, but not listed in the SSA, are not relevant for SSA service interface.

This is a compromise towards reaching rapidly a stable revision of the documents. We would much prefer to have a single source for the definition of the datamodel, which the SSA protocol would just extend. But we believe this is too large of a task to achieve while maintaining backwards compatibility, on a reasonable time-scale.

2. The use of "*" and ".." in UCDs

This must be eliminated.

These characters are always used in the context of "em.*" or "em...". Our understanding is that these characters are placeholders, which a data provider must fill in, according to its requirements. A list of all possible values and meanings must be provided instead, using the following primary UCDs:

em.wl em.freq em.energy

3. The use of "*" in utypes

This must be eliminated. The correct utypes to use are the ones in the "Query Response" section of SSA, but without the "*":

Char.SpatialAxis.SamplingPrecision.SampleExtent Char.SpatialAxis.SamplingPrecision.FillFactor Char.SpectralAxis.SamplingPrecision.SampleExtent Char.SpectralAxis.SamplingPrecision.FillFactor Char.TimeAxis.SamplingPrecision.SampleExtent Char.TimeAxis.SamplingPrecision.FillFactor

4. Missing UCDs in SSA The SSA is correct, the SpectrumDM should not have a UCD for the following elements (the UCDs provided in the SpectrumDM on those elements are either wrong or confusing):

5 Misc. typos The SpectrumDM (on the FITS serialization section) should fix the following utypes:

The SSA should fix the following UCD: em;spec.binSize -> em;spect.binSize

Extra spaces in UCDs and utypes are typos and should be removed

6. Dimensional analysis typo

In the SpectrumDM, change (from 10-10 to 1E-10) the way to express exponents within the dimensional analysis elements

Section 3.2 should read:

Pedro Osuna and Jesus Salgado have proposed a representation in the spirit of dimensional analysis, using the symbols M, L, T to signify kg, m, s respectively and omitting the ** for powers, so that 10**3 Jy Hz which is equivalent to 10**-23 kg s**-2 is written compactly as 1E-23MT-2

and the example in section 9.4:

SPECSDIM= '1E-10 L' / Spectral SIDim FLUXSDIM= '1E+7 ML-1T-3' / Flux SDim

7. Wrong UCDs

Spectrum.Char.SpectralAxis.Coverage.Location.Value has a wrong UCD of instr.bandpass. It should become the following list (in accordance to point 2 above):

em.wl;instr.bandpass em.freq;instr.bandpass em.energy;instr.bandpass

8. Inconsistencies within the SSA itself

Add the Dataset.Deleted utype to Appendix D.

Remove the Data.* utypes from Appendix D.

Add the remaining missing utypes present in Appendix D to section 4.2 (the list is too long and too boring to show here)

9. Inconsistencies within the SpectrumDM itself

Add a comment to the FITS serialisation stating that it does not cover the whole of the SpectrumDM utypes

Note: The "consolidation" activities detailed in points 7 and 8 should also make sure the order by which the utypes are listed is consistent throughout all documents.

See also: http://www.ivoa.net/internal/IVOA/IvoaExecMeetingFM34/SSA_SDM_assessment.xls

SSA and SDM Documents Update (response to above)

I am trying to synthesize these various comments on SSA/SDM. I think I mostly agree with the detailed changes suggested. The comments here are mostly higher level.

At this point it appears that both Jonathan and Bruno agree that the SSA metadata and the SDM, while quite similar, are not quite the same thing (hence where there are differences they are not necessarily "inconsistencies" but are restrictions in units, mandatory fields etc. required by the specific application). SSA defines the data model used in the query response, describing the data which a service can deliver. SDM defines the data model of a spectrum dataset/file/instance.

A key point which has not been adequately discussed is the role of the generic GDS/Obs data model. While the SDM explicitly describes a Spectrum instance, most of what is in the SSA query response is actually GDS/ObsDM metadata and not specific to Spectra at all - at this level the metadata should be the same for an image, for an SED, for a time series, etc. It does not make a whole lot of sense for example to have Spectrum.Target.Name and Image.Target.Name and allow these to be two different things. An application which deals with multiple types of data would like global and generic metadata such as this, which is applicable to any type of data, to be usable as just "Target.Name" (possibly in a future version something such as Obs.Target.Name would be preferable, but we did not have that when SSA was defined). Once we have a full set of DAL2 services one would like to have the common stuff factored out so that it does not have to be customized for every type of data. For the purposes of data discovery and description most of this stuff is the same.

The above is the reason why the "Spectrum" prefix does not carry over into the SSA query response. Rather, essentially all of the SSA query response metadata is either generic dataset metadata, or specific to the service mechanism itself and hence also generic and applicable to all such services. It is not actually from the SDM at all, we merely require them to be consistent. That small part of the query response metadata which is not generic and which is specific to the type of dataset being queried (Spectrum, Image, whatever) is what is in the "Dataset" component model.

In other words, for a DAL2 query response what we have is virtually all generic dataset / Obs metadata, plus a separate "Dataset" element which is specific to the type of dataset being queried. This generic query response is supposed to be the same for all of the "typed" services derived from the generic Dataset. The data model (if any is formally defined) for a specific type of dataset such as Spectrum, Image, etc. also inherits the generic dataset metadata. Ideally we want to have all these be internally consistent for a particular major version of the interface (1.x, 2.x, etc.). Once we have SSA, SIAV2, ObsTAP, etc. it is clear that it could be attractive to standardize the generic metadata - which comprises most of the metadata returned - among all these very similar services.

Now there are some things which could possibly be done differently several years later, for example the "Dataset" component in the query response might instead take on the class name of the specific type of data which it describes, such as Spectrum or Image. Either approach is possible and it doesn't make all that much difference.

The full discussion from which the above was taken is available at: http://www.ivoa.net/forum/dal/1004/1690.htm

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->

SSAP V1.1 Collaborative Page

SSAP/SDM Documents

The current updated version of the Spectrum DM (1.04, 2009-05-15) provided by Jonathan Mc Dowell is available here :

http://hea-www.harvard.edu/~jcm/vo/docs/specrc3.pdf

It is a first step to meet the changes required.

The working draft SSA/SDM V1.1 documents can be found here:

• SSA V1.1 working draft (pdf with markup)
• SSA V1.1 draft (doc)
• SSA V1.1 data model spreadsheet

The current SSA/SDM standards documents can be found here:

• Simple Spectral Access V1.04 (1 Feb 2009)
• Spectrum Data Model (29 Oct 2008)
• SSA/SDM model summary spreadsheet

SSAP/SDM Document Update Discussion

At the EuroVO AIDA a small group of people interested in updating the SSA and SpectrumDM document gathered. Participants; BrunoRino, KeithNoddle (DAL chair), MireilleLouys (DM chair), AlbertoMicol, IgorChilingarian, JesusSalgado, FrancoisBonnarel.

Separately from this meeting, from the US side, DougTody and JonathanMcDowell (the SSAP and SDM document editors) had also been discussing these proposed changes with the document authors and others via email.

We set forth the following goal: To create 1.1 versions (of SSA and the SpectrumDM) that attempt to fix or clarify what were seen as inconsistencies between the two documents. Mor extensive clarifications or additions could be added in a version 1.2. Big changes, possibly not backwards compatible, are postponed to 2.0 versions.

The rationale for this split is to have version 1.1 approved quickly and replace the current 1.0. A version 1.2 would permit more changes but would take significantly longer, gathering more extensive input from the community.

In both cases (1.1. and 1.2) the assumption is that no existing application should break. This means that when a fix creates potential breakage, the potentially affected applications should be consulted.

Below, the result of our discussion on what must be changed in order to reach version 1.1. We were lucky enough to have both the DM and DAL Working Group leads; we concluded that after a short period of time after circulating these meeting minutes to the relevant lists, Working Drafts should be produced, ahead of the May interop in Victoria.

SSAP/SDM Inconsistencies

1. The SSA data model is derived, but decoupled, from the SpectrumDM

The acknowledged divergences are: - "required" flags (Mandatory, Recommended, Optional) are different - the SSA data model contains service related metadata, that have no meaning for the SpectrumDM - the SpectrumDM contains metadata related to data analysis (Data.*) that are of no interest for data discovery (which is the purpose of SSA) - in SSA, the "Spectrum." prefix found in the SpectrumDM utypes was dropped, and in some cases a "Dataset." prefix was added.

Even if those divergences are discrepancies, they are not going to be fixed in a 1.x document. Instead, SSA Section 2.2 ("Data Model") must be changed to reflect these divergences. It should state explicitly that a reader of the SSA should only refer to the SpectrumDM to seek clarification about the meaning of metadata fields. The specification of required fields, the UCDs, and even the utype syntax for setting up a SSA server are to be read from the SSA document. Metadata defined in the SpectrumDM, but not listed in the SSA, are not relevant for SSA service interface.

This is a compromise towards reaching rapidly a stable revision of the documents. We would much prefer to have a single source for the definition of the datamodel, which the SSA protocol would just extend. But we believe this is too large of a task to achieve while maintaining backwards compatibility, on a reasonable time-scale.

2. The use of "*" and ".." in UCDs

This must be eliminated.

These characters are always used in the context of "em.*" or "em...". Our understanding is that these characters are placeholders, which a data provider must fill in, according to its requirements. A list of all possible values and meanings must be provided instead, using the following primary UCDs:

em.wl em.freq em.energy

3. The use of "*" in utypes

This must be eliminated. The correct utypes to use are the ones in the "Query Response" section of SSA, but without the "*":

Char.SpatialAxis.SamplingPrecision.SampleExtent Char.SpatialAxis.SamplingPrecision.FillFactor Char.SpectralAxis.SamplingPrecision.SampleExtent Char.SpectralAxis.SamplingPrecision.FillFactor Char.TimeAxis.SamplingPrecision.SampleExtent Char.TimeAxis.SamplingPrecision.FillFactor

4. Missing UCDs in SSA The SSA is correct, the SpectrumDM should not have a UCD for the following elements (the UCDs provided in the SpectrumDM on those elements are either wrong or confusing):

5 Misc. typos The SpectrumDM (on the FITS serialization section) should fix the following utypes:

The SSA should fix the following UCD: em;spec.binSize -> em;spect.binSize

Extra spaces in UCDs and utypes are typos and should be removed

6. Dimensional analysis typo

In the SpectrumDM, change (from 10-10 to 1E-10) the way to express exponents within the dimensional analysis elements

Section 3.2 should read:

Pedro Osuna and Jesus Salgado have proposed a representation in the spirit of dimensional analysis, using the symbols M, L, T to signify kg, m, s respectively and omitting the ** for powers, so that 10**3 Jy Hz which is equivalent to 10**-23 kg s**-2 is written compactly as 1E-23MT-2

and the example in section 9.4:

SPECSDIM= '1E-10 L' / Spectral SIDim FLUXSDIM= '1E+7 ML-1T-3' / Flux SDim

7. Wrong UCDs

Spectrum.Char.SpectralAxis.Coverage.Location.Value has a wrong UCD of instr.bandpass. It should become the following list (in accordance to point 2 above):

em.wl;instr.bandpass em.freq;instr.bandpass em.energy;instr.bandpass

8. Inconsistencies within the SSA itself

Add the Dataset.Deleted utype to Appendix D.

Remove the Data.* utypes from Appendix D.

Add the remaining missing utypes present in Appendix D to section 4.2 (the list is too long and too boring to show here)

9. Inconsistencies within the SpectrumDM itself

Add a comment to the FITS serialisation stating that it does not cover the whole of the SpectrumDM utypes

Note: The "consolidation" activities detailed in points 7 and 8 should also make sure the order by which the utypes are listed is consistent throughout all documents.

See also: http://www.ivoa.net/internal/IVOA/IvoaExecMeetingFM34/SSA_SDM_assessment.xls

SSA and SDM Documents Update (response to above)

I am trying to synthesize these various comments on SSA/SDM. I think I mostly agree with the detailed changes suggested. The comments here are mostly higher level.

At this point it appears that both Jonathan and Bruno agree that the SSA metadata and the SDM, while quite similar, are not quite the same thing (hence where there are differences they are not necessarily "inconsistencies" but are restrictions in units, mandatory fields etc. required by the specific application). SSA defines the data model used in the query response, describing the data which a service can deliver. SDM defines the data model of a spectrum dataset/file/instance.

A key point which has not been adequately discussed is the role of the generic GDS/Obs data model. While the SDM explicitly describes a Spectrum instance, most of what is in the SSA query response is actually GDS/ObsDM metadata and not specific to Spectra at all - at this level the metadata should be the same for an image, for an SED, for a time series, etc. It does not make a whole lot of sense for example to have Spectrum.Target.Name and Image.Target.Name and allow these to be two different things. An application which deals with multiple types of data would like global and generic metadata such as this, which is applicable to any type of data, to be usable as just "Target.Name" (possibly in a future version something such as Obs.Target.Name would be preferable, but we did not have that when SSA was defined). Once we have a full set of DAL2 services one would like to have the common stuff factored out so that it does not have to be customized for every type of data. For the purposes of data discovery and description most of this stuff is the same.

The above is the reason why the "Spectrum" prefix does not carry over into the SSA query response. Rather, essentially all of the SSA query response metadata is either generic dataset metadata, or specific to the service mechanism itself and hence also generic and applicable to all such services. It is not actually from the SDM at all, we merely require them to be consistent. That small part of the query response metadata which is not generic and which is specific to the type of dataset being queried (Spectrum, Image, whatever) is what is in the "Dataset" component model.

In other words, for a DAL2 query response what we have is virtually all generic dataset / Obs metadata, plus a separate "Dataset" element which is specific to the type of dataset being queried. This generic query response is supposed to be the same for all of the "typed" services derived from the generic Dataset. The data model (if any is formally defined) for a specific type of dataset such as Spectrum, Image, etc. also inherits the generic dataset metadata. Ideally we want to have all these be internally consistent for a particular major version of the interface (1.x, 2.x, etc.). Once we have SSA, SIAV2, ObsTAP, etc. it is clear that it could be attractive to standardize the generic metadata - which comprises most of the metadata returned - among all these very similar services.

Now there are some things which could possibly be done differently several years later, for example the "Dataset" component in the query response might instead take on the class name of the specific type of data which it describes, such as Spectrum or Image. Either approach is possible and it doesn't make all that much difference.

The full discussion from which the above was taken is available at: http://www.ivoa.net/forum/dal/1004/1690.htm

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->

SSAP V1.1 Collaborative Page

SSAP/SDM Documents

• SSA V1.1 working draft (pdf with markup)
• SSA V1.1 draft (doc)
• SSA V1.1 data model spreadsheet

The current SSA/SDM standards documents can be found here:

• Simple Spectral Access V1.04 (1 Feb 2009)
• Spectrum Data Model (29 Oct 2008)
• SSA/SDM model summary spreadsheet

SSAP/SDM Document Update Discussion

At the EuroVO AIDA a small group of people interested in updating the SSA and SpectrumDM document gathered. Participants; BrunoRino, KeithNoddle (DAL chair), MireilleLouys (DM chair), AlbertoMicol, IgorChilingarian, JesusSalgado, FrancoisBonnarel.

Separately from this meeting, from the US side, DougTody and JonathanMcDowell (the SSAP and SDM document editors) had also been discussing these proposed changes with the document authors and others via email.

We set forth the following goal: To create 1.1 versions (of SSA and the SpectrumDM) that attempt to fix or clarify what were seen as inconsistencies between the two documents. Mor extensive clarifications or additions could be added in a version 1.2. Big changes, possibly not backwards compatible, are postponed to 2.0 versions.

The rationale for this split is to have version 1.1 approved quickly and replace the current 1.0. A version 1.2 would permit more changes but would take significantly longer, gathering more extensive input from the community.

In both cases (1.1. and 1.2) the assumption is that no existing application should break. This means that when a fix creates potential breakage, the potentially affected applications should be consulted.

Below, the result of our discussion on what must be changed in order to reach version 1.1. We were lucky enough to have both the DM and DAL Working Group leads; we concluded that after a short period of time after circulating these meeting minutes to the relevant lists, Working Drafts should be produced, ahead of the May interop in Victoria.

SSAP/SDM Inconsistencies

1. The SSA data model is derived, but decoupled, from the SpectrumDM

The acknowledged divergences are: - "required" flags (Mandatory, Recommended, Optional) are different - the SSA data model contains service related metadata, that have no meaning for the SpectrumDM - the SpectrumDM contains metadata related to data analysis (Data.*) that are of no interest for data discovery (which is the purpose of SSA) - in SSA, the "Spectrum." prefix found in the SpectrumDM utypes was dropped, and in some cases a "Dataset." prefix was added.

Even if those divergences are discrepancies, they are not going to be fixed in a 1.x document. Instead, SSA Section 2.2 ("Data Model") must be changed to reflect these divergences. It should state explicitly that a reader of the SSA should only refer to the SpectrumDM to seek clarification about the meaning of metadata fields. The specification of required fields, the UCDs, and even the utype syntax for setting up a SSA server are to be read from the SSA document. Metadata defined in the SpectrumDM, but not listed in the SSA, are not relevant for SSA service interface.

This is a compromise towards reaching rapidly a stable revision of the documents. We would much prefer to have a single source for the definition of the datamodel, which the SSA protocol would just extend. But we believe this is too large of a task to achieve while maintaining backwards compatibility, on a reasonable time-scale.

2. The use of "*" and ".." in UCDs

This must be eliminated.

These characters are always used in the context of "em.*" or "em...". Our understanding is that these characters are placeholders, which a data provider must fill in, according to its requirements. A list of all possible values and meanings must be provided instead, using the following primary UCDs:

em.wl em.freq em.energy

3. The use of "*" in utypes

This must be eliminated. The correct utypes to use are the ones in the "Query Response" section of SSA, but without the "*":

Char.SpatialAxis.SamplingPrecision.SampleExtent Char.SpatialAxis.SamplingPrecision.FillFactor Char.SpectralAxis.SamplingPrecision.SampleExtent Char.SpectralAxis.SamplingPrecision.FillFactor Char.TimeAxis.SamplingPrecision.SampleExtent Char.TimeAxis.SamplingPrecision.FillFactor

4. Missing UCDs in SSA The SSA is correct, the SpectrumDM should not have a UCD for the following elements (the UCDs provided in the SpectrumDM on those elements are either wrong or confusing):

5 Misc. typos The SpectrumDM (on the FITS serialization section) should fix the following utypes:

Extra spaces in UCDs and utypes are typos and should be removed

6. Dimensional analysis typo

In the SpectrumDM, change (from 10-10 to 1E-10) the way to express exponents within the dimensional analysis elements

Section 3.2 should read:

Pedro Osuna and Jesus Salgado have proposed a representation in the spirit of dimensional analysis, using the symbols M, L, T to signify kg, m, s respectively and omitting the ** for powers, so that 10**3 Jy Hz which is equivalent to 10**-23 kg s**-2 is written compactly as 1E-23MT-2

and the example in section 9.4:

SPECSDIM= '1E-10 L' / Spectral SIDim FLUXSDIM= '1E+7 ML-1T-3' / Flux SDim

7. Wrong UCDs

Spectrum.Char.SpectralAxis.Coverage.Location.Value has a wrong UCD of instr.bandpass. It should become the following list (in accordance to point 2 above):

em.wl;instr.bandpass em.freq;instr.bandpass em.energy;instr.bandpass

8. Inconsistencies within the SSA itself

Add the Dataset.Deleted utype to Appendix D.

Remove the Data.* utypes from Appendix D.

Add the remaining missing utypes present in Appendix D to section 4.2 (the list is too long and too boring to show here)

9. Inconsistencies within the SpectrumDM itself

Add a comment to the FITS serialisation stating that it does not cover the whole of the SpectrumDM utypes

Note: The "consolidation" activities detailed in points 7 and 8 should also make sure the order by which the utypes are listed is consistent throughout all documents.

See also: http://www.ivoa.net/internal/IVOA/IvoaExecMeetingFM34/SSA_SDM_assessment.xls

SSA and SDM Documents Update (response to above)

I am trying to synthesize these various comments on SSA/SDM. I think I mostly agree with the detailed changes suggested. The comments here are mostly higher level.

At this point it appears that both Jonathan and Bruno agree that the SSA metadata and the SDM, while quite similar, are not quite the same thing (hence where there are differences they are not necessarily "inconsistencies" but are restrictions in units, mandatory fields etc. required by the specific application). SSA defines the data model used in the query response, describing the data which a service can deliver. SDM defines the data model of a spectrum dataset/file/instance.

A key point which has not been adequately discussed is the role of the generic GDS/Obs data model. While the SDM explicitly describes a Spectrum instance, most of what is in the SSA query response is actually GDS/ObsDM metadata and not specific to Spectra at all - at this level the metadata should be the same for an image, for an SED, for a time series, etc. It does not make a whole lot of sense for example to have Spectrum.Target.Name and Image.Target.Name and allow these to be two different things. An application which deals with multiple types of data would like global and generic metadata such as this, which is applicable to any type of data, to be usable as just "Target.Name" (possibly in a future version something such as Obs.Target.Name would be preferable, but we did not have that when SSA was defined). Once we have a full set of DAL2 services one would like to have the common stuff factored out so that it does not have to be customized for every type of data. For the purposes of data discovery and description most of this stuff is the same.

The above is the reason why the "Spectrum" prefix does not carry over into the SSA query response. Rather, essentially all of the SSA query response metadata is either generic dataset metadata, or specific to the service mechanism itself and hence also generic and applicable to all such services. It is not actually from the SDM at all, we merely require them to be consistent. That small part of the query response metadata which is not generic and which is specific to the type of dataset being queried (Spectrum, Image, whatever) is what is in the "Dataset" component model.

In other words, for a DAL2 query response what we have is virtually all generic dataset / Obs metadata, plus a separate "Dataset" element which is specific to the type of dataset being queried. This generic query response is supposed to be the same for all of the "typed" services derived from the generic Dataset. The data model (if any is formally defined) for a specific type of dataset such as Spectrum, Image, etc. also inherits the generic dataset metadata. Ideally we want to have all these be internally consistent for a particular major version of the interface (1.x, 2.x, etc.). Once we have SSA, SIAV2, ObsTAP, etc. it is clear that it could be attractive to standardize the generic metadata - which comprises most of the metadata returned - among all these very similar services.

Now there are some things which could possibly be done differently several years later, for example the "Dataset" component in the query response might instead take on the class name of the specific type of data which it describes, such as Spectrum or Image. Either approach is possible and it doesn't make all that much difference.

The full discussion from which the above was taken is available at: http://www.ivoa.net/forum/dal/1004/1690.htm

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->

SSAP V1.1 Collaborative Page

SSAP/SDM Documents

The working draft SSA/SDM V1.1 documents can be found here:

• Simple Spectral Access V1.04 (1 Feb 2009)
• Spectrum Data Model (29 Oct 2008)
• SSA/SDM model summary spreadsheet

SSAP/SDM Document Update Discussion

At the EuroVO AIDA a small group of people interested in updating the SSA and SpectrumDM document gathered. Participants; BrunoRino, KeithNoddle (DAL chair), MireilleLouys (DM chair), AlbertoMicol, IgorChilingarian, JesusSalgado, FrancoisBonnarel.

Separately from this meeting, from the US side, DougTody and JonathanMcDowell (the SSAP and SDM document editors) had also been discussing these proposed changes with the document authors and others via email.

We set forth the following goal: To create 1.1 versions (of SSA and the SpectrumDM) that attempt to fix or clarify what were seen as inconsistencies between the two documents. Mor extensive clarifications or additions could be added in a version 1.2. Big changes, possibly not backwards compatible, are postponed to 2.0 versions.

The rationale for this split is to have version 1.1 approved quickly and replace the current 1.0. A version 1.2 would permit more changes but would take significantly longer, gathering more extensive input from the community.

In both cases (1.1. and 1.2) the assumption is that no existing application should break. This means that when a fix creates potential breakage, the potentially affected applications should be consulted.

Below, the result of our discussion on what must be changed in order to reach version 1.1. We were lucky enough to have both the DM and DAL Working Group leads; we concluded that after a short period of time after circulating these meeting minutes to the relevant lists, Working Drafts should be produced, ahead of the May interop in Victoria.

SSAP/SDM Inconsistencies

1. The SSA data model is derived, but decoupled, from the SpectrumDM

The acknowledged divergences are: - "required" flags (Mandatory, Recommended, Optional) are different - the SSA data model contains service related metadata, that have no meaning for the SpectrumDM - the SpectrumDM contains metadata related to data analysis (Data.*) that are of no interest for data discovery (which is the purpose of SSA) - in SSA, the "Spectrum." prefix found in the SpectrumDM utypes was dropped, and in some cases a "Dataset." prefix was added.

Even if those divergences are discrepancies, they are not going to be fixed in a 1.x document. Instead, SSA Section 2.2 ("Data Model") must be changed to reflect these divergences. It should state explicitly that a reader of the SSA should only refer to the SpectrumDM to seek clarification about the meaning of metadata fields. The specification of required fields, the UCDs, and even the utype syntax for setting up a SSA server are to be read from the SSA document. Metadata defined in the SpectrumDM, but not listed in the SSA, are not relevant for SSA service interface.

This is a compromise towards reaching rapidly a stable revision of the documents. We would much prefer to have a single source for the definition of the datamodel, which the SSA protocol would just extend. But we believe this is too large of a task to achieve while maintaining backwards compatibility, on a reasonable time-scale.

2. The use of "*" and ".." in UCDs

This must be eliminated.

These characters are always used in the context of "em.*" or "em...". Our understanding is that these characters are placeholders, which a data provider must fill in, according to its requirements. A list of all possible values and meanings must be provided instead, using the following primary UCDs:

em.wl em.freq em.energy

3. The use of "*" in utypes

This must be eliminated. The correct utypes to use are the ones in the "Query Response" section of SSA, but without the "*":

Char.SpatialAxis.SamplingPrecision.SampleExtent Char.SpatialAxis.SamplingPrecision.FillFactor Char.SpectralAxis.SamplingPrecision.SampleExtent Char.SpectralAxis.SamplingPrecision.FillFactor Char.TimeAxis.SamplingPrecision.SampleExtent Char.TimeAxis.SamplingPrecision.FillFactor

4. Missing UCDs in SSA The SSA is correct, the SpectrumDM should not have a UCD for the following elements (the UCDs provided in the SpectrumDM on those elements are either wrong or confusing):

utype UCD to be removed Dataset.TimeSI time;arith.zp Char.SpatialAxis.Name meta.id Char.SpatialAxis.Ucd meta.ucd Char.SpatialAxis.Unit meta.unit

5 Misc. typos The SpectrumDM (on the FITS serialization section) should fix the following utypes:

Spectrum.Curation.ContactName -> Spectrum.Curation.Contact.Name Spectrum.Curation.ContactEmail -> Spectrum.Curation.Contact.Email Spectrum.Char.SpatialAxis.Accuracy.StatErr -> Spectrum.Char.SpatialAxis.Accuracy.StatError

The SSA should fix the following UCD: em;spec.binSize -> em;spect.binSize

Extra spaces in UCDs and utypes are typos and should be removed

6. Dimensional analysis typo

In the SpectrumDM, change (from 10-10 to 1E-10) the way to express exponents within the dimensional analysis elements

Section 3.2 should read:

Pedro Osuna and Jesus Salgado have proposed a representation in the spirit of dimensional analysis, using the symbols M, L, T to signify kg, m, s respectively and omitting the ** for powers, so that 10**3 Jy Hz which is equivalent to 10**-23 kg s**-2 is written compactly as 1E-23MT-2

and the example in section 9.4:

SPECSDIM= '1E-10 L' / Spectral SIDim FLUXSDIM= '1E+7 ML-1T-3' / Flux SDim

7. Wrong UCDs

Spectrum.Char.SpectralAxis.Coverage.Location.Value has a wrong UCD of instr.bandpass. It should become the following list (in accordance to point 2 above):

em.wl;instr.bandpass em.freq;instr.bandpass em.energy;instr.bandpass

8. Inconsistencies within the SSA itself

Add the Dataset.Deleted utype to Appendix D.

Remove the Data.* utypes from Appendix D.

Add the remaining missing utypes present in Appendix D to section 4.2 (the list is too long and too boring to show here)

9. Inconsistencies within the SpectrumDM itself

Add a comment to the FITS serialisation stating that it does not cover the whole of the SpectrumDM utypes

Note: The "consolidation" activities detailed in points 7 and 8 should also make sure the order by which the utypes are listed is consistent throughout all documents.

See also: http://www.ivoa.net/internal/IVOA/IvoaExecMeetingFM34/SSA_SDM_assessment.xls

SSA and SDM Documents Update (response to above)

I am trying to synthesize these various comments on SSA/SDM. I think I mostly agree with the detailed changes suggested. The comments here are mostly higher level.

At this point it appears that both Jonathan and Bruno agree that the SSA metadata and the SDM, while quite similar, are not quite the same thing (hence where there are differences they are not necessarily "inconsistencies" but are restrictions in units, mandatory fields etc. required by the specific application). SSA defines the data model used in the query response, describing the data which a service can deliver. SDM defines the data model of a spectrum dataset/file/instance.

A key point which has not been adequately discussed is the role of the generic GDS/Obs data model. While the SDM explicitly describes a Spectrum instance, most of what is in the SSA query response is actually GDS/ObsDM metadata and not specific to Spectra at all - at this level the metadata should be the same for an image, for an SED, for a time series, etc. It does not make a whole lot of sense for example to have Spectrum.Target.Name and Image.Target.Name and allow these to be two different things. An application which deals with multiple types of data would like global and generic metadata such as this, which is applicable to any type of data, to be usable as just "Target.Name" (possibly in a future version something such as Obs.Target.Name would be preferable, but we did not have that when SSA was defined). Once we have a full set of DAL2 services one would like to have the common stuff factored out so that it does not have to be customized for every type of data. For the purposes of data discovery and description most of this stuff is the same.

The above is the reason why the "Spectrum" prefix does not carry over into the SSA query response. Rather, essentially all of the SSA query response metadata is either generic dataset metadata, or specific to the service mechanism itself and hence also generic and applicable to all such services. It is not actually from the SDM at all, we merely require them to be consistent. That small part of the query response metadata which is not generic and which is specific to the type of dataset being queried (Spectrum, Image, whatever) is what is in the "Dataset" component model.

In other words, for a DAL2 query response what we have is virtually all generic dataset / Obs metadata, plus a separate "Dataset" element which is specific to the type of dataset being queried. This generic query response is supposed to be the same for all of the "typed" services derived from the generic Dataset. The data model (if any is formally defined) for a specific type of dataset such as Spectrum, Image, etc. also inherits the generic dataset metadata. Ideally we want to have all these be internally consistent for a particular major version of the interface (1.x, 2.x, etc.). Once we have SSA, SIAV2, ObsTAP, etc. it is clear that it could be attractive to standardize the generic metadata - which comprises most of the metadata returned - among all these very similar services.

Now there are some things which could possibly be done differently several years later, for example the "Dataset" component in the query response might instead take on the class name of the specific type of data which it describes, such as Spectrum or Image. Either approach is possible and it doesn't make all that much difference.

The full discussion from which the above was taken is available at: http://www.ivoa.net/forum/dal/1004/1690.htm

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->

SSAP V1.1 Collaborative Page

SSAP/SDM Documents

The working draft SSA/SDM V1.1 documents can be found here:

[to be added]

The current SSA/SDM standards documents can be found here:

• Simple Spectral Access V1.04 (1 Feb 2009)
• Spectrum Data Model (29 Oct 2008)
• SSA/SDM model summary spreadsheet

SSAP/SDM Document Update Discussion

At the EuroVO AIDA a small group of people interested in updating the SSA and SpectrumDM document gathered. Participants; BrunoRino, KeithNoddle (DAL chair), MireilleLouys (DM chair), AlbertoMicol, IgorChilingarian, JesusSalgado, FrancoisBonnarel.

Separately from this meeting, from the US side, DougTody and JonathanMcDowell (the SSAP and SDM document editors) had also been discussing these proposed changes with the document authors and others via email.

We set forth the following goal: To create 1.1 versions (of SSA and the SpectrumDM) that attempt to fix or clarify what were seen as inconsistencies between the two documents. Mor extensive clarifications or additions could be added in a version 1.2. Big changes, possibly not backwards compatible, are postponed to 2.0 versions.

The rationale for this split is to have version 1.1 approved quickly and replace the current 1.0. A version 1.2 would permit more changes but would take significantly longer, gathering more extensive input from the community.

In both cases (1.1. and 1.2) the assumption is that no existing application should break. This means that when a fix creates potential breakage, the potentially affected applications should be consulted.

Below, the result of our discussion on what must be changed in order to reach version 1.1. We were lucky enough to have both the DM and DAL Working Group leads; we concluded that after a short period of time after circulating these meeting minutes to the relevant lists, Working Drafts should be produced, ahead of the May interop in Victoria.

SSAP/SDM Inconsistencies

1. The SSA data model is derived, but decoupled, from the SpectrumDM

The acknowledged divergences are: - "required" flags (Mandatory, Recommended, Optional) are different - the SSA data model contains service related metadata, that have no meaning for the SpectrumDM - the SpectrumDM contains metadata related to data analysis (Data.*) that are of no interest for data discovery (which is the purpose of SSA) - in SSA, the "Spectrum." prefix found in the SpectrumDM utypes was dropped, and in some cases a "Dataset." prefix was added.

Even if those divergences are discrepancies, they are not going to be fixed in a 1.x document. Instead, SSA Section 2.2 ("Data Model") must be changed to reflect these divergences. It should state explicitly that a reader of the SSA should only refer to the SpectrumDM to seek clarification about the meaning of metadata fields. The specification of required fields, the UCDs, and even the utype syntax for setting up a SSA server are to be read from the SSA document. Metadata defined in the SpectrumDM, but not listed in the SSA, are not relevant for SSA service interface.

This is a compromise towards reaching rapidly a stable revision of the documents. We would much prefer to have a single source for the definition of the datamodel, which the SSA protocol would just extend. But we believe this is too large of a task to achieve while maintaining backwards compatibility, on a reasonable time-scale.

2. The use of "*" and ".." in UCDs

This must be eliminated.

These characters are always used in the context of "em.*" or "em...". Our understanding is that these characters are placeholders, which a data provider must fill in, according to its requirements. A list of all possible values and meanings must be provided instead, using the following primary UCDs:

em.wl em.freq em.energy

3. The use of "*" in utypes

This must be eliminated. The correct utypes to use are the ones in the "Query Response" section of SSA, but without the "*":

Char.SpatialAxis.SamplingPrecision.SampleExtent Char.SpatialAxis.SamplingPrecision.FillFactor Char.SpectralAxis.SamplingPrecision.SampleExtent Char.SpectralAxis.SamplingPrecision.FillFactor Char.TimeAxis.SamplingPrecision.SampleExtent Char.TimeAxis.SamplingPrecision.FillFactor

4. Missing UCDs in SSA The SSA is correct, the SpectrumDM should not have a UCD for the following elements (the UCDs provided in the SpectrumDM on those elements are either wrong or confusing):

utype UCD to be removed Dataset.TimeSI time;arith.zp Char.SpatialAxis.Name meta.id Char.SpatialAxis.Ucd meta.ucd Char.SpatialAxis.Unit meta.unit

5 Misc. typos The SpectrumDM (on the FITS serialization section) should fix the following utypes:

Spectrum.Curation.ContactName -> Spectrum.Curation.Contact.Name Spectrum.Curation.ContactEmail -> Spectrum.Curation.Contact.Email Spectrum.Char.SpatialAxis.Accuracy.StatErr -> Spectrum.Char.SpatialAxis.Accuracy.StatError

The SSA should fix the following UCD: em;spec.binSize -> em;spect.binSize

Extra spaces in UCDs and utypes are typos and should be removed

6. Dimensional analysis typo

In the SpectrumDM, change (from 10-10 to 1E-10) the way to express exponents within the dimensional analysis elements

Section 3.2 should read:

Pedro Osuna and Jesus Salgado have proposed a representation in the spirit of dimensional analysis, using the symbols M, L, T to signify kg, m, s respectively and omitting the ** for powers, so that 10**3 Jy Hz which is equivalent to 10**-23 kg s**-2 is written compactly as 1E-23MT-2

and the example in section 9.4:

SPECSDIM= '1E-10 L' / Spectral SIDim FLUXSDIM= '1E+7 ML-1T-3' / Flux SDim

7. Wrong UCDs

Spectrum.Char.SpectralAxis.Coverage.Location.Value has a wrong UCD of instr.bandpass. It should become the following list (in accordance to point 2 above):

em.wl;instr.bandpass em.freq;instr.bandpass em.energy;instr.bandpass

8. Inconsistencies within the SSA itself

Add the Dataset.Deleted utype to Appendix D.

Remove the Data.* utypes from Appendix D.

Add the remaining missing utypes present in Appendix D to section 4.2 (the list is too long and too boring to show here)

9. Inconsistencies within the SpectrumDM itself

Add a comment to the FITS serialisation stating that it does not cover the whole of the SpectrumDM utypes

Note: The "consolidation" activities detailed in points 7 and 8 should also make sure the order by which the utypes are listed is consistent throughout all documents.

See also: http://www.ivoa.net/internal/IVOA/IvoaExecMeetingFM34/SSA_SDM_assessment.xls

SSA and SDM Documents Update (response to above)

I am trying to synthesize these various comments on SSA/SDM. I think I mostly agree with the detailed changes suggested. The comments here are mostly higher level.

At this point it appears that both Jonathan and Bruno agree that the SSA metadata and the SDM, while quite similar, are not quite the same thing (hence where there are differences they are not necessarily "inconsistencies" but are restrictions in units, mandatory fields etc. required by the specific application). SSA defines the data model used in the query response, describing the data which a service can deliver. SDM defines the data model of a spectrum dataset/file/instance.

A key point which has not been adequately discussed is the role of the generic GDS/Obs data model. While the SDM explicitly describes a Spectrum instance, most of what is in the SSA query response is actually GDS/ObsDM metadata and not specific to Spectra at all - at this level the metadata should be the same for an image, for an SED, for a time series, etc. It does not make a whole lot of sense for example to have Spectrum.Target.Name and Image.Target.Name and allow these to be two different things. An application which deals with multiple types of data would like global and generic metadata such as this, which is applicable to any type of data, to be usable as just "Target.Name" (possibly in a future version something such as Obs.Target.Name would be preferable, but we did not have that when SSA was defined). Once we have a full set of DAL2 services one would like to have the common stuff factored out so that it does not have to be customized for every type of data. For the purposes of data discovery and description most of this stuff is the same.

The above is the reason why the "Spectrum" prefix does not carry over into the SSA query response. Rather, essentially all of the SSA query response metadata is either generic dataset metadata, or specific to the service mechanism itself and hence also generic and applicable to all such services. It is not actually from the SDM at all, we merely require them to be consistent. That small part of the query response metadata which is not generic and which is specific to the type of dataset being queried (Spectrum, Image, whatever) is what is in the "Dataset" component model.

In other words, for a DAL2 query response what we have is virtually all generic dataset / Obs metadata, plus a separate "Dataset" element which is specific to the type of dataset being queried. This generic query response is supposed to be the same for all of the "typed" services derived from the generic Dataset. The data model (if any is formally defined) for a specific type of dataset such as Spectrum, Image, etc. also inherits the generic dataset metadata. Ideally we want to have all these be internally consistent for a particular major version of the interface (1.x, 2.x, etc.). Once we have SSA, SIAV2, ObsTAP, etc. it is clear that it could be attractive to standardize the generic metadata - which comprises most of the metadata returned - among all these very similar services.

Now there are some things which could possibly be done differently several years later, for example the "Dataset" component in the query response might instead take on the class name of the specific type of data which it describes, such as Spectrum or Image. Either approach is possible and it doesn't make all that much difference.

The full discussion from which the above was taken is available at: http://www.ivoa.net/forum/dal/1004/1690.htm

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->

SSAP V1.1 Collaborative Page

SSAP/SDM Documents

The working draft SSA/SDM V1.1 documents can be found here:

[to be added]

The current SSA/SDM standards documents can be found here:

SSAP/SDM Document Update Discussion

At the EuroVO AIDA a small group of people interested in updating the SSA and SpectrumDM document gathered. Participants; BrunoRino, KeithNoddle (DAL chair), MireilleLouys (DM chair), AlbertoMicol, IgorChilingarian, JesusSalgado, FrancoisBonnarel.

Separately from this meeting, from the US side, DougTody and JonathanMcDowell (the SSAP and SDM document editors) had also been discussing these proposed changes with the document authors and others via email.

We set forth the following goal: To create 1.1 versions (of SSA and the SpectrumDM) that attempt to fix or clarify what were seen as inconsistencies between the two documents. Mor extensive clarifications or additions could be added in a version 1.2. Big changes, possibly not backwards compatible, are postponed to 2.0 versions.

The rationale for this split is to have version 1.1 approved quickly and replace the current 1.0. A version 1.2 would permit more changes but would take significantly longer, gathering more extensive input from the community.

In both cases (1.1. and 1.2) the assumption is that no existing application should break. This means that when a fix creates potential breakage, the potentially affected applications should be consulted.

Below, the result of our discussion on what must be changed in order to reach version 1.1. We were lucky enough to have both the DM and DAL Working Group leads; we concluded that after a short period of time after circulating these meeting minutes to the relevant lists, Working Drafts should be produced, ahead of the May interop in Victoria.

SSAP/SDM Inconsistencies

1. The SSA data model is derived, but decoupled, from the SpectrumDM

The acknowledged divergences are: - "required" flags (Mandatory, Recommended, Optional) are different - the SSA data model contains service related metadata, that have no meaning for the SpectrumDM - the SpectrumDM contains metadata related to data analysis (Data.*) that are of no interest for data discovery (which is the purpose of SSA) - in SSA, the "Spectrum." prefix found in the SpectrumDM utypes was dropped, and in some cases a "Dataset." prefix was added.

Even if those divergences are discrepancies, they are not going to be fixed in a 1.x document. Instead, SSA Section 2.2 ("Data Model") must be changed to reflect these divergences. It should state explicitly that a reader of the SSA should only refer to the SpectrumDM to seek clarification about the meaning of metadata fields. The specification of required fields, the UCDs, and even the utype syntax for setting up a SSA server are to be read from the SSA document. Metadata defined in the SpectrumDM, but not listed in the SSA, are not relevant for SSA service interface.

This is a compromise towards reaching rapidly a stable revision of the documents. We would much prefer to have a single source for the definition of the datamodel, which the SSA protocol would just extend. But we believe this is too large of a task to achieve while maintaining backwards compatibility, on a reasonable time-scale.

2. The use of "*" and ".." in UCDs

This must be eliminated.

These characters are always used in the context of "em.*" or "em...". Our understanding is that these characters are placeholders, which a data provider must fill in, according to its requirements. A list of all possible values and meanings must be provided instead, using the following primary UCDs:

em.wl em.freq em.energy

3. The use of "*" in utypes

This must be eliminated. The correct utypes to use are the ones in the "Query Response" section of SSA, but without the "*":

Char.SpatialAxis.SamplingPrecision.SampleExtent Char.SpatialAxis.SamplingPrecision.FillFactor Char.SpectralAxis.SamplingPrecision.SampleExtent Char.SpectralAxis.SamplingPrecision.FillFactor Char.TimeAxis.SamplingPrecision.SampleExtent Char.TimeAxis.SamplingPrecision.FillFactor

4. Missing UCDs in SSA The SSA is correct, the SpectrumDM should not have a UCD for the following elements (the UCDs provided in the SpectrumDM on those elements are either wrong or confusing):

utype UCD to be removed Dataset.TimeSI time;arith.zp Char.SpatialAxis.Name meta.id Char.SpatialAxis.Ucd meta.ucd Char.SpatialAxis.Unit meta.unit

5 Misc. typos The SpectrumDM (on the FITS serialization section) should fix the following utypes:

Spectrum.Curation.ContactName -> Spectrum.Curation.Contact.Name Spectrum.Curation.ContactEmail -> Spectrum.Curation.Contact.Email Spectrum.Char.SpatialAxis.Accuracy.StatErr -> Spectrum.Char.SpatialAxis.Accuracy.StatError

The SSA should fix the following UCD: em;spec.binSize -> em;spect.binSize

Extra spaces in UCDs and utypes are typos and should be removed

6. Dimensional analysis typo

In the SpectrumDM, change (from 10-10 to 1E-10) the way to express exponents within the dimensional analysis elements

Section 3.2 should read:

Pedro Osuna and Jesus Salgado have proposed a representation in the spirit of dimensional analysis, using the symbols M, L, T to signify kg, m, s respectively and omitting the ** for powers, so that 10**3 Jy Hz which is equivalent to 10**-23 kg s**-2 is written compactly as 1E-23MT-2

and the example in section 9.4:

SPECSDIM= '1E-10 L' / Spectral SIDim FLUXSDIM= '1E+7 ML-1T-3' / Flux SDim

7. Wrong UCDs

Spectrum.Char.SpectralAxis.Coverage.Location.Value has a wrong UCD of instr.bandpass. It should become the following list (in accordance to point 2 above):

em.wl;instr.bandpass em.freq;instr.bandpass em.energy;instr.bandpass

8. Inconsistencies within the SSA itself

Add the Dataset.Deleted utype to Appendix D.

Remove the Data.* utypes from Appendix D.

Add the remaining missing utypes present in Appendix D to section 4.2 (the list is too long and too boring to show here)

9. Inconsistencies within the SpectrumDM itself

Add a comment to the FITS serialisation stating that it does not cover the whole of the SpectrumDM utypes

Note: The "consolidation" activities detailed in points 7 and 8 should also make sure the order by which the utypes are listed is consistent throughout all documents.

See also: http://www.ivoa.net/internal/IVOA/IvoaExecMeetingFM34/SSA_SDM_assessment.xls

SSA and SDM Documents Update (response to above)

I am trying to synthesize these various comments on SSA/SDM. I think I mostly agree with the detailed changes suggested. The comments here are mostly higher level.

At this point it appears that both Jonathan and Bruno agree that the SSA metadata and the SDM, while quite similar, are not quite the same thing (hence where there are differences they are not necessarily "inconsistencies" but are restrictions in units, mandatory fields etc. required by the specific application). SSA defines the data model used in the query response, describing the data which a service can deliver. SDM defines the data model of a spectrum dataset/file/instance.

A key point which has not been adequately discussed is the role of the generic GDS/Obs data model. While the SDM explicitly describes a Spectrum instance, most of what is in the SSA query response is actually GDS/ObsDM metadata and not specific to Spectra at all - at this level the metadata should be the same for an image, for an SED, for a time series, etc. It does not make a whole lot of sense for example to have Spectrum.Target.Name and Image.Target.Name and allow these to be two different things. An application which deals with multiple types of data would like global and generic metadata such as this, which is applicable to any type of data, to be usable as just "Target.Name" (possibly in a future version something such as Obs.Target.Name would be preferable, but we did not have that when SSA was defined). Once we have a full set of DAL2 services one would like to have the common stuff factored out so that it does not have to be customized for every type of data. For the purposes of data discovery and description most of this stuff is the same.

The above is the reason why the "Spectrum" prefix does not carry over into the SSA query response. Rather, essentially all of the SSA query response metadata is either generic dataset metadata, or specific to the service mechanism itself and hence also generic and applicable to all such services. It is not actually from the SDM at all, we merely require them to be consistent. That small part of the query response metadata which is not generic and which is specific to the type of dataset being queried (Spectrum, Image, whatever) is what is in the "Dataset" component model.

In other words, for a DAL2 query response what we have is virtually all generic dataset / Obs metadata, plus a separate "Dataset" element which is specific to the type of dataset being queried. This generic query response is supposed to be the same for all of the "typed" services derived from the generic Dataset. The data model (if any is formally defined) for a specific type of dataset such as Spectrum, Image, etc. also inherits the generic dataset metadata. Ideally we want to have all these be internally consistent for a particular major version of the interface (1.x, 2.x, etc.). Once we have SSA, SIAV2, ObsTAP, etc. it is clear that it could be attractive to standardize the generic metadata - which comprises most of the metadata returned - among all these very similar services.

Now there are some things which could possibly be done differently several years later, for example the "Dataset" component in the query response might instead take on the class name of the specific type of data which it describes, such as Spectrum or Image. Either approach is possible and it doesn't make all that much difference.

The full discussion from which the above was taken is available at: http://www.ivoa.net/forum/dal/1004/1690.htm

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->

Separately from this meeting, from the US side, DougTody and JonathanMcDowell (the SSAP and SDM document editors) had also been discussing these proposed changes with the document authors and others via email.

We set forth the following goal: To create 1.1 versions (of SSA and the SpectrumDM) that attempt to fix or clarify what were seen as inconsistencies between the two documents. Mor extensive clarifications or additions could be added in a version 1.2. Big changes, possibly not backwards compatible, are postponed to 2.0 versions.

The rationale for this split is to have version 1.1 approved quickly and replace the current 1.0. A version 1.2 would permit more changes but would take significantly longer, gathering more extensive input from the community.

In both cases (1.1. and 1.2) the assumption is that no existing application should break. This means that when a fix creates potential breakage, the potentially affected applications should be consulted.

Below, the result of our discussion on what must be changed in order to reach version 1.1. We were lucky enough to have both the DM and DAL Working Group leads; we concluded that after a short period of time after circulating these meeting minutes to the relevant lists, Working Drafts should be produced, ahead of the May interop in Victoria.

1. The SSA data model is derived, but decoupled, from the SpectrumDM

The acknowledged divergences are: - "required" flags (Mandatory, Recommended, Optional) are different - the SSA data model contains service related metadata, that have no meaning for the SpectrumDM - the SpectrumDM contains metadata related to data analysis (Data.*) that are of no interest for data discovery (which is the purpose of SSA) - in SSA, the "Spectrum." prefix found in the SpectrumDM utypes was dropped, and in some cases a "Dataset." prefix was added.

Even if those divergences are discrepancies, they are not going to be fixed in a 1.x document. Instead, SSA Section 2.2 ("Data Model") must be changed to reflect these divergences. It should state explicitly that a reader of the SSA should only refer to the SpectrumDM to seek clarification about the meaning of metadata fields. The specification of required fields, the UCDs, and even the utype syntax for setting up a SSA server are to be read from the SSA document. Metadata defined in the SpectrumDM, but not listed in the SSA, are not relevant for SSA service interface.

This is a compromise towards reaching rapidly a stable revision of the documents. We would much prefer to have a single source for the definition of the datamodel, which the SSA protocol would just extend. But we believe this is too large of a task to achieve while maintaining backwards compatibility, on a reasonable time-scale.

2. The use of "*" and ".." in UCDs

This must be eliminated.

These characters are always used in the context of "em.*" or "em...". Our understanding is that these characters are placeholders, which a data provider must fill in, according to its requirements. A list of all possible values and meanings must be provided instead, using the following primary UCDs:

em.wl em.freq em.energy

3. The use of "*" in utypes

This must be eliminated. The correct utypes to use are the ones in the "Query Response" section of SSA, but without the "*":

Char.SpatialAxis.SamplingPrecision.SampleExtent Char.SpatialAxis.SamplingPrecision.FillFactor Char.SpectralAxis.SamplingPrecision.SampleExtent Char.SpectralAxis.SamplingPrecision.FillFactor Char.TimeAxis.SamplingPrecision.SampleExtent Char.TimeAxis.SamplingPrecision.FillFactor

4. Missing UCDs in SSA The SSA is correct, the SpectrumDM should not have a UCD for the following elements (the UCDs provided in the SpectrumDM on those elements are either wrong or confusing):

utype UCD to be removed Dataset.TimeSI time;arith.zp Char.SpatialAxis.Name meta.id Char.SpatialAxis.Ucd meta.ucd Char.SpatialAxis.Unit meta.unit

5 Misc. typos The SpectrumDM (on the FITS serialization section) should fix the following utypes:

Spectrum.Curation.ContactName -> Spectrum.Curation.Contact.Name Spectrum.Curation.ContactEmail -> Spectrum.Curation.Contact.Email Spectrum.Char.SpatialAxis.Accuracy.StatErr -> Spectrum.Char.SpatialAxis.Accuracy.StatError

The SSA should fix the following UCD: em;spec.binSize -> em;spect.binSize

Extra spaces in UCDs and utypes are typos and should be removed

6. Dimensional analysis typo

In the SpectrumDM, change (from 10-10 to 1E-10) the way to express exponents within the dimensional analysis elements

Section 3.2 should read:

Pedro Osuna and Jesus Salgado have proposed a representation in the spirit of dimensional analysis, using the symbols M, L, T to signify kg, m, s respectively and omitting the ** for powers, so that 10**3 Jy Hz which is equivalent to 10**-23 kg s**-2 is written compactly as 1E-23MT-2

and the example in section 9.4:

SPECSDIM= '1E-10 L' / Spectral SIDim FLUXSDIM= '1E+7 ML-1T-3' / Flux SDim

7. Wrong UCDs

Spectrum.Char.SpectralAxis.Coverage.Location.Value has a wrong UCD of instr.bandpass. It should become the following list (in accordance to point 2 above):

em.wl;instr.bandpass em.freq;instr.bandpass em.energy;instr.bandpass

8. Inconsistencies within the SSA itself

Add the Dataset.Deleted utype to Appendix D.

Remove the Data.* utypes from Appendix D.

Add the remaining missing utypes present in Appendix D to section 4.2 (the list is too long and too boring to show here)

9. Inconsistencies within the SpectrumDM itself

Add a comment to the FITS serialisation stating that it does not cover the whole of the SpectrumDM utypes

Note: The "consolidation" activities detailed in points 7 and 8 should also make sure the order by which the utypes are listed is consistent throughout all documents.

See also: http://www.ivoa.net/internal/IVOA/IvoaExecMeetingFM34/SSA_SDM_assessment.xls

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->

Even if those divergences are discrepancies, they are not going to be

This must be eliminated.

These characters are always used in the context of "em.*" or "em...". Our understanding is that these characters are placeholders, which a

3. The use of "*" in utypes

This must be eliminated. The correct utypes to use are the ones in the "Query Response" section of SSA, but without the "*":

5 Misc. typos

Extra spaces in UCDs and utypes are typos and should be removed

6. Dimensional analysis typo

Section 3.2 should read:

Pedro Osuna and Jesus Salgado have proposed a representation in the spirit of dimensional analysis, using the symbols M, L, T to signify kg, m, s respectively and omitting the ** for powers, so that 10**3 Jy Hz which is equivalent to 10**-23 kg s**-2 is written compactly as 1E-23MT-2

and the example in section 9.4:

SPECSDIM= '1E-10 L' / Spectral SIDim FLUXSDIM= '1E+7 ML-1T-3' / Flux SDim

7. Wrong UCDs

8. Inconsistencies within the SSA itself

Add the Dataset.Deleted utype to Appendix D.

Remove the Data.* utypes from Appendix D.

Add the remaining missing utypes present in Appendix D to section 4.2 (the list is too long and too boring to show here)

Note: The "consolidation" activities detailed in points 7 and 8 should also make sure the order by which the utypes are listed is consistent throughout all documents.

See also: http://www.ivoa.net/internal/IVOA/IvoaExecMeetingFM34/SSA_SDM_assessment.xls

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->