Spectrum Data Model RFC
This document will act as
RFC centre for the
Spectrum Data Model 1.01 Proposed Recommendation.
Review period: 16 May 2007 to 12 Jun 2007 (still open)
In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your
WikiName so authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.
Discussion about any of the comments or responses should be conducted on the data model mailing list,
dm@ivoa.net.
Comments
- First sample comment (by BrunoRino): ...
- Response (by authorname): ...
- AndreaPreiteMartinez - 19 Jun 2007
- I have revised the UCDs in Tables 1, 2 and 3. The revised tables can be found here. Note that rows with changed UCDs are in bold+italic.
- Pag.23: erase sentence "My solution for brightness... "
- Pag.23 last line: change with "Note: we propose for the next version of the IVOA REC UCD-list to (a) change the flag of pos.eq from S to Q, (b) add two new words em.start, em.end to indicate the lower/start and upper/end boundaries of a spectral interval (as already done for the time axis)"
Andrea,
I don't like the fact that you propose the same UCD for energy flux
density and photon number flux density, and the same UCD for polarized
flux and total flux, and the same UCD for observed flux and for source
intensity per unit emission solid angle. These are different physical
concepts, in some cases they have the same units and I REALLY need
different UCDs to describe them to users - this is exactly what UCDs are
for. I want to avoid using units to distinguish since this is of limited
reliability ( e.g. source intensity as described below has the same
units as a surface brightness but is a very different thing)
and since it's a difference in CONCEPT not a difference in UNIT
that is key here.
Concept My new proposal Your proposal
Energy flux vs energy phot.flux.density;em.energy phot.flux.density;em.energy
Number flux vs energy phot.flux.density;em.energy,phys.photon phot.flux.density;em.energy
or phot.flux.number;em.energy
Flux density (vs wl) phot.flux.density;em.wl phot.flux.density;em.wl
Polarized flux (vs wl) phot.flux.density;em.wl,phys.polarization phot.flux.density;em.wl
Flux density (vs wl) phot.flux.density;em.wl phot.flux.density;em.wl
Source intensity phys.luminosity; em.wl,phys.angArea phot.flux.density;em.wl
Flux density phot.flux.density phot.flux.density
Brightness temperature phot.flux.density; phys.temperature phot.flux.density
Aperture area phys.area phys.area
Effective area phys.area;phys.transmission phys.area
Continuum-only vs wl phot.flux.density;em.wl,spect.continuum phot.flux.density,spect.continuum
Continuum-only vs freq phot.flux.density;em.freq,spect.continuum phot.flux.density,spect.continuum
Can you (1) explain what is wrong with my proposals (ok the first one uses
a so-far-nonexistent UCD I agree, the others seem legal to me) and
(2) provide alternatives that do not lose these distinctions?
I have a few other issues with your list vs mine:
Why is src.var.amplitude;arith.ratio not acceptable for
the variability as fraction of mean? To say that this
should have the same UCD as the actual amplitude is
losing valuable discriminatory power.
What is wrong with meta.ucd and meta.unit for e.g.
SpatialAxis.ucd, unit?
With e.g.
StatError you suggest
"stat.error;em" while I suggested "stat.error;em.wl" or
"stat.error;em.freq" etc. Isn't it better to specify here, so that
software can use the UCD to tell which is being given?
Why don't you like: em.wl;obs.atmos for Air Wavelength? I would really
like to be able to distinguish it from vacuum wavelength. How can I tell
the client (user) that air wavelengths are being used?
- thanks,
Jonathan
- GaryFuller - 19 Jun 2007
- Does the model allow for double sideband spectra? That is spectra for which each pixel can correspond to two different frequencies or wavelengths? This is a fairly common situation in the mm and submm.
- More generally what about the issue of describing how to convert between two different spectral axes, for example in the dsb case the upper and lower sideband frequency?
- Is there support for non-uniform spectral (and temporal) axes?
- In derived data fields signal-to-noise ratio seems a poorly defined quantity as it depends on what signal, spectral feature, is of interest to a particular user. A value for the rms noise level on the other hand would be very useful but doesn't seem to be in the model.
Gary:
We discussed the double sideband spectrum case but decided that
it complicated the model too much for version 1. We'll revisit
this in future, and for now you just have to pick one sideband as the
nominal frequency, maybe providing two Spectrum instances, one for
each sideband. We acknowledge this is a limitation, as is
the inability to represent two different spectra axes in a
single instance.
The S/N is a bit vague, but was requested by some data providers
for who a single target object is the norm. There is no rms noise, but
there is a 'typical (1 sigma) error' in
StatError which does much the
same thing. We may add rms in a later rev but not this one.
We do, however, support non-uniform axes, since each spectral coordinate
is listed explicitly.
Jonathan,
1) first of all, a general consideration: if we miss an ucd-word for expressing a given quantity, then let’s create the corresponding new ucd-word.
2) as an alternative, we can agree that e.g. phot.flux.density;em.wl (or freq or energy) indicates the flux density per unit wavelength (or frequency, or energy) : no syntax problem (valid syntax
), but cumbersome if you want to add other modifiers/qualifiers (em.opt, stat.max,...). What I do not like in the form phot.flux.density;em.wl is the use of a quantity (em.wl) as a modifier/qualifier of the first quantity (phot.flux.density).
In this case I prefer going to point (1) above, and re-present the proposal (see Victoria 2006) of adding:
phot.flux.density.perFreq,
phot.flux.density.perWave,
phot.flux.density.perEnergy, ..
phot.flux.photon.per*, ..
phot.flux.brTemperature, ..
phot.intensity.perWave ..
So, phot.flux.density;em.wl;spect.continuum will become phot.flux.density.perWave;spect.continuum
About Polarized Flux: I don’t understand the problem. I would have added an additional attribute for the degree of polarization or the fraction of polarized flux.
the same concern applies to phys.area;phys.transmission: let’s go to phys.area.effective
src.amplitude;arith.ratio : you are right
SpatialAxis.ucd, unit: I took out meta.ucd, meta.unit for uniformity with the other axes. You can replace them, adding them also to TimeAxis .
stat.error;em : yes, if a * is legal in your context, you can write stat.error;em* (but not meaning all the em-word branch! )
em.wl;obs.atmos: I prefer em.wl.air.
3) in order to avoid confusion in multi-word ucds, I suggest also to introduce
stat.error.lower instead of stat.error;stat.min
stat.error.upper instead of stat.error;stat.max
Ciao, Andrea
* JonathanMcDowell -23 Jun 2007
Andrea,
I guess I don't feel that
phot.flux.density;em.wl,stat.max
(and similar) are 'cumbersome' - I find them much easier
to understand than extra custom-crafted UCDs like ..perWave.
I feel that em.wl, em.freq, etc. are naturally adjectives as well as
nouns; since they are 'Q' and not 'P', what I have done seems perfectly legal, and
I guess I would like a reason based on the standard that it's not right,
rather than just 'Andrea doesn't like it'...
I accept the proposal to add stat.error.lower, stat.error.upper, em.wl.air
as new UCDs. But I'm a bit concerned that you give me all this input now
at/after the end of the RFC period, we were hoping to get approval for Spectrum
at the July 15 exec - if we need to add all these new UCDs, does this
mean we have to wait for a new update to the words list to be approved
before Spectrum can go to approval? (Plus, of course, the time required
for all the implementers to change their implementation, since we've published
essentially the current list of Spectrum UCDs over a year ago and everyone is using
them - this mostly is important for the Flux.Value cases).
For polarized flux: yes, a data provider could choose to separate
it out into (total flux vs wavelength) and (fraction of polarized flux vs
wavelength), but I believe it's fairly common (certainly
in AGN work) to show the product of these two things as the 'polarized flux',
and I want a UCD for that. Since phys.polarization is an 'E' word, I believe
that phot.flux.density;em.wl,phys.polarization is a reasonably unambiguous
UCD for this?
For em.*: in the context of the doc, this is a shorthand for
(EITHER em.wl OR em.freq OR em.energy).
- Jonathan
* FrancoisBonnarel - 25 Jun 2007
Jonathan,
Reviewing the document, I discover two little points which may
be a problem:
A) Page 8 the document claims that
"
In section 4 we elaborate these concepts in detail, including some complications that we
explicitly do not attempt to handle in this version. The data model fields and possible values arelisted. We distinguish between optional and required fields in the text, as well as via a ”Req”
column in the tables which has values of R (Required) and O (Optional). Where appropriate
we list those values of the physical units which interoperable implementations are required to
recognize."
But actually at page 13 and in the following table there is
no more Required and Optional but "MANDATORY" abreviated as MAN and
"OPTIONAL" abreviated as "OPT". In addition we have "REC" for recommanded. I think i remember the statement at page 8 was what
was planned at the beginning...
B ) Page 17 for the Spectrum.Char.*.Calibration utypes I
think the meaning should be "coord calibration status" instead of
Type of Coord Calibartion. Type can be interpreted as "Data Type"
for calibration, no ? (Or is that due to French being my mother
language?)
François Bonnarel
* FrancoisBonnarel - 25 Jun 2007
Page 56 you write:
"How can this be generalized to mapping an arbitrary data model schema to VOTABLE? The
only tricky parts are
• Spotting where the tabledata parts are. We could require any DM schema that maps
to VOTABLE to include elements called FIELDS and Data (perhaps ROWS would be a
better name), otherwise you would get a VOTABLE with no data section.
• Spotting where to start the main TABLE (i.e. the fact that SPECTRUM is
special). We could change the schema to have an explicit attribute, annotation or other
marker to tell us this.
These issues will require further discussion for future models."
We started a discussion on that in Moscow... (yes Moscow, not
Beijing !!!)
I think the formulation here is a little too restrictive.
We may imagine any elemnt with a simple content as a "FIELD"
in some context, as long as you have more than one representative
of this element. It is not the case here but what when we will
get several segments in a main SED or Echelle spectrum ?
Will we have each segment as a separate Spectrum serialisation and
all this concatenated ? or redefine the PARAMS with several representatives as FIELDS... An index FIELD can be defined to hook
information in the "metadata" TABLE to information in the data
TABLE. This is the way we try to implement data models in the
DAL Extension mechanism (see SSA 1.01 and references therin), and I
think we need to let that open.
So I will rephrase:
"map all elements with simple content to PARAM" as
"map all elements with simple content and one single representative to PARAM"
and add a third "spotting".
"Spotting if and where we need secondary TABLES with Metadata
FIELDS and how we hook the information therin to the main TABLE rows"
François Bonnarel
* FelixStoehr - 02 Jul 2007
Dear all,
Currently, the SpectrumDM states under section 5.3 Derived Data fields, Signal-to-noise:
"A suitable method, used by the STScI MAST group, is to define the noise by the median absolute value of the difference between adjacent independent flux values in the spectrum. (The MAST definition multiplies this noise value by a 1.048 correction factor for precise applications)."
The STScI(MAST)/ST-ECF/CADC Spectral Container Working Group has worked quite a bit on the algorithm and tested several possibilities. The final "best" algorithm is presented in the issue #42 of the ST-ECF Newsletter
http://www.spacetelescope.org/about/further_information/newsletters/html/newsletter_42.html
Implementations of the new algorithm can be obtained from:
http://stecf.org/software/ASTROsoft/DER_SNR/
As a suggestion, we propose to change the sentences cited above by:
"A suitable method, set forth by the STScI/ST-ECF/CADC Spectral Container Working Group, is to define the signal as the median of the flux values in the spectrum and the noise as the median absolute third-order difference of flux values spaced two pixels apart. This noise value is then multiplied by 1.482602 / sqrt(6). Padded zeros in the flux values are skipped. A detailed description and discussion of the algorithm can be found in the issue #42 of the ST-ECF newsletter at www.spacetelescope.org/about/further_information/newsletters/html/newsletter_42.html. Implementations of the algorithm can be obtained from www.stecf.org/software/ASTROsoft/DER_SNR. We recommend to use this algorithm where suitable to allow meaningful cross-instrument searches."
Felix Stoehr
Adding a reference to a recommended method for computing comparable SNRs is an excellent idea.
MarkusDolensky
Comments from the Working Group Chairs and Interest Group Chairs
Chairs should add their comments under their name.
Mark Allen (Applications WG)
Approved
The Spectrum Data Model appears to contain all the pieces necessary
to fully describe spectra as is needed in the VO. The examples are
especially useful as they will help guide usage in services and
applications.
Christophe Arviset (TCG vice Chair)
Matthew Graham (Grid & Web Services WG)
Bob Hanisch (Data Curation & Preservation IG)
Gerard Lemson (Theory IG)
Mireille Louys (Data Models WG)
Keith Noddle (Data Access Layer WG)
Francois Ochsenbein (VOTable WG)
Pedro Osuna (VOQL WG)
Ray Plante (Resource Registry WG)
Andrea Preite-Martinez (Semantics WG)
I approve the document for Recommendation. My comments are expressed above.
Roy Williams (VOEvent WG)
This is a well-written, well-thought out definition of a spectrum, and the authors are to be congratulated. I approve it for Recommendation.
The examples may be all that a reader sees. They may read NONE of the text! Therefore these are important. After looking carefully, I find only two things worthy of note:
-- In the Characterisation section of example 7.2, the flux units are difficult to see, it says
name="Flux density" ucd="phot.flux;em.wavelength" unit="erg cm**(-2) s**(-1) Angstrom**(-1)", and this is still not enough for me to see which kind of flux you mean. Perhaps a comment from Table 3 would help, it explains simply: "Flux density per unit wave", and now I am clear.
-- In the VOTable example 8.2, there are no schema locations for either VOTable or SpectrumModel. This means that simply copying the example does not allow XML validation.