Survey among Spectral Data Providers and Consumers

At the IVOA meeting in Cambridge in May, one of the highest priority projects identified for the DAL working group was to define a Simple Spectral Access (SSA) protocol for accessing 1D spectra and SEDs.

To help plan this effort a little survey was conducted among potential data providers and consumers. The results are summarized below. Of course, the summary is only as complete as the response we've received. Thanks a lot to all who took the time to answer to our questions!

More feedback is always welcome. Regards,
DougTody, MarkusDolensky

• ESA ISO Science Data Archive (Pedro Osuna Alcalaya)
• ESA XMM-Newton Science Data Archive (Pedro Osuna Alcalaya)
• ESO/ST-ECF archive (Pirenne, Wicenec, Dolensky)
• Fermilab (Stephen Kent)
• HEASARC (Tom McGlynn)
• INTA/LAEFF (Enrique Solano)
• NCSA (Ray Plante)
• NOAO (Mike Fitzpatrick)
• NOAO and CFLIB project (Frank Valdes)

1b. Data Consumer

• Specview (Ivo Busko, STScI)

About Data Providers

2. Spectral Data Collections

- Sloan Digital Sky Survey, Data Release 1

- NCSA Astronomy Digital Image Library (NCSA BIMA Data Archive)

- Variety of spectral data sources from X-ray and gamma-ray missions, including both relatively high and low resolution spectral data (HEASARC)

- Indo-U.S. Library of Coude Feed Stellar Spectra (NOAO & CFLIB)

- at NOAO:

• High Resolution Atlas of Arcturus Spectra - 0.92-5.36 microns
• High Resolution Atlas of Arcturus Spectra - 3727-9300 A
• Library of High Resolution Infrared Stellar Spectra in K-Band
• Library of Medium Resolution Infrared Stellar Spectra
• Coude Feed Spectral Library
• Library of Stellar Spectra
• Optical Spectrophotometric Atlas of SN 1987A
• Atlases of ThAR, FeAr, and HeNeAr comparison spectra

- at ESO/ST-ECF as of June 26, 03

- XMM-Newton 1D Scpectral Data

- ISO 1D Spectral Data

- INES (IUE Newly-Extracted Spectra)

3. Characteristics of Data

• number: ~100 K - 1 Mio. spectra/data provider; often only several 100s in very specific catalogs
• size: few KBs - few MBs/spectrum
• wavelength/energy ranges: gamma, X-ray, UV, optical, IR, radio
• resolution: 20 - 45.000
• linear and non-linear/irregularly sampling (for instance in energy domain)
• often with WCS
• stellar parameters: e.g.: Teff, log(g), [Fe/H]
• noise
• masks (coded aperture mask data, INTEGRAL)
• observation dependent transfer matrix to convert counts to phys. units
• spectral features (SLOAN): emission, absorption lines and parameters, emission redshift params. cross-correlation parameters
• time resolved spectral information (XTE, BATSE, Swift, HETE)

Columns/data items: wavelength (better fravergy: frequency, wavelength, energy) flux (magnitude, flux, flux density, counts) flux error quality flag (quality of what exactly?) variance arrays photon events antenna temperature vs. frequency (e.g. SWAS spectra)

4. Current Storage Format

• FITS images (often used for raw data (CCD images) or stacks thereof; radio spectral image cubes)
• FITS binary tables
• ASCII tables (catalogs, highly processed data)

5. Is the data available online?

Yes. Data are generally on-line, but proprietary periods and restricted access depending on nationality may apply.

About Data Consumers

6. Name of application, package, demo, etc.

• NOAO/IRAF-SPECTOOL
• MIDAS-MOS
• Specview
• aXe

Tools for SEDs:

• NED (on-line)
• AVO Prototype SED utility

7. Summarize capabilities of software

• reading various instrument dependent FITS and ASCII formats
• overplotting of multiple arrays
• various display options (labels, zoom, colors, log/linear scaling, ...)
• physical unit conversion
• model fitting capability
• rebinning
• quality filters

see, for instance http://www.stsci.edu/resources/software_hardware/specview/what_is_specview

8. Desired characteristics of input data

• FITS WCS, most useful for well sampled rasters
• As a minimum, data should consist of (x,y) pairs of wavelength-flux values expressed in wavelength/frequency/energy units, and spectral flux density units respectively. The units information should be present as well, in header keywords or table column descriptors or any other suitable form.
• Additional information that could be included with the above, (if available) is a third value associated with each (x,y) pair representing the measurement error in flux density. Also, it won't hurt if obvious pieces of information such as the object name, are provided as well.
• Uncalibrated data (such as in counts/s units) can also be ingested but not much else can be done with it besides plain plotting.

9. Desired input data format or formats (e.g. graphics, FITS, XML)

10. General Comments

An interesting service would be to generate 1-D spectra from cubes on the fly. In addition to the usual search parameters, users would have to provide a spatial resolution. This service would be analogous to an image cutout service in that the output is generated to user specs on-the-fly from the underlying data. (Ray Plante)

Even at this level of detail it may be inappropriate to discuss the HEASARC monolithically. The characteristics of individual missions/instruments varies enormously. (Tom McGlynn)

1D and 2D currently have more applications than 3D. FITS binary tables format and support under development. Will be evolved to support VO spectral standards. (Frank Valdes)

[...] the different storage formats and associated data tables would provide for some interesting use cases. Something like a cutout service might be appropriate for some of the data, the challenging thing for us however to put all these into an SSA is simply the variety of local data access methods required. (Mike Fitzpatrick)

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->

Survey among Spectral Data Providers and Consumers

At the IVOA meeting in Cambridge in May, one of the highest priority projects identified for the DAL working group was to define a Simple Spectral Access (SSA) protocol for accessing 1D spectra and SEDs.

To help plan this effort a little survey was conducted among potential data providers and consumers. The results are summarized below. Of course, the summary is only as complete as the response we've received. Thanks a lot to all who took the time to answer to our questions!

More feedback is always welcome. Regards,
DougTody, MarkusDolensky

1a. Data Provider (name of person responding to DAL list)

• ESA ISO Science Data Archive (Pedro Osuna Alcalaya)
• ESA XMM-Newton Science Data Archive (Pedro Osuna Alcalaya)
• ESO/ST-ECF archive (Pirenne, Wicenec, Dolensky)
• Fermilab (Stephen Kent)
• HEASARC (Tom McGlynn)
• INTA/LAEFF (Enrique Solano)
• NCSA (Ray Plante)
• NOAO (Mike Fitzpatrick)
• NOAO and CFLIB project (Frank Valdes)

1b. Data Consumer

• Specview (Ivo Busko, STScI)

About Data Providers

2. Spectral Data Collections

- Sloan Digital Sky Survey, Data Release 1

- NCSA Astronomy Digital Image Library (NCSA BIMA Data Archive)

- Variety of spectral data sources from X-ray and gamma-ray missions, including both relatively high and low resolution spectral data (HEASARC)

- Indo-U.S. Library of Coude Feed Stellar Spectra (NOAO & CFLIB)

- at NOAO:

• High Resolution Atlas of Arcturus Spectra - 0.92-5.36 microns
• High Resolution Atlas of Arcturus Spectra - 3727-9300 A
• Library of High Resolution Infrared Stellar Spectra in K-Band
• Library of Medium Resolution Infrared Stellar Spectra
• Coude Feed Spectral Library
• Library of Stellar Spectra
• Optical Spectrophotometric Atlas of SN 1987A
• Atlases of ThAR, FeAr, and HeNeAr comparison spectra

- at ESO/ST-ECF as of June 26, 03

- XMM-Newton 1D Scpectral Data

- ISO 1D Spectral Data

- INES (IUE Newly-Extracted Spectra)

3. Characteristics of Data

• number: ~100 K - 1 Mio. spectra/data provider; often only several 100s in very specific catalogs
• size: few KBs - few MBs/spectrum
• wavelength/energy ranges: gamma, X-ray, UV, optical, IR, radio
• resolution: 20 - 45.000
• linear and non-linear/irregularly sampling (for instance in energy domain)
• often with WCS
• stellar parameters: e.g.: Teff, log(g), [Fe/H]
• noise
• masks (coded aperture mask data, INTEGRAL)
• observation dependent transfer matrix to convert counts to phys. units
• spectral features (SLOAN): emission, absorption lines and parameters, emission redshift params. cross-correlation parameters
• time resolved spectral information (XTE, BATSE, Swift, HETE)

Columns/data items: wavelength (better fravergy: frequency, wavelength, energy) flux (magnitude, flux, flux density, counts) flux error quality flag (quality of what exactly?) variance arrays photon events antenna temperature vs. frequency (e.g. SWAS spectra)

4. Current Storage Format

• FITS images (often used for raw data (CCD images) or stacks thereof; radio spectral image cubes)
• FITS binary tables
• ASCII tables (catalogs, highly processed data)

5. Is the data available online?

Yes. Data are generally on-line, but proprietary periods and restricted access depending on nationality may apply.

About Data Consumers

6. Name of application, package, demo, etc.

• NOAO/IRAF-SPECTOOL
• MIDAS-MOS
• Specview
• aXe

Tools for SEDs:

• NED (on-line)
• AVO Prototype SED utility

7. Summarize capabilities of software

• reading various instrument dependent FITS and ASCII formats
• overplotting of multiple arrays
• various display options (labels, zoom, colors, log/linear scaling, ...)
• physical unit conversion
• model fitting capability
• rebinning
• quality filters

see, for instance http://www.stsci.edu/resources/software_hardware/specview/what_is_specview

8. Desired characteristics of input data

• FITS WCS, most useful for well sampled rasters
• As a minimum, data should consist of (x,y) pairs of wavelength-flux values expressed in wavelength/frequency/energy units, and spectral flux density units respectively. The units information should be present as well, in header keywords or table column descriptors or any other suitable form.
• Additional information that could be included with the above, (if available) is a third value associated with each (x,y) pair representing the measurement error in flux density. Also, it won't hurt if obvious pieces of information such as the object name, are provided as well.
• Uncalibrated data (such as in counts/s units) can also be ingested but not much else can be done with it besides plain plotting.

9. Desired input data format or formats (e.g. graphics, FITS, XML)

FITS (images, tables), XML, and plain ASCII are all supported at this point. It is fairly easy to write ingestor modules for almost any conceivable spectral format that uses one of those media.

10. General Comments

An interesting service would be to generate 1-D spectra from cubes on the fly. In addition to the usual search parameters, users would have to provide a spatial resolution. This service would be analogous to an image cutout service in that the output is generated to user specs on-the-fly from the underlying data. (Ray Plante)

Even at this level of detail it may be inappropriate to discuss the HEASARC monolithically. The characteristics of individual missions/instruments varies enormously. (Tom McGlynn)

1D and 2D currently have more applications than 3D. FITS binary tables format and support under development. Will be evolved to support VO spectral standards. (Frank Valdes)

[...] the different storage formats and associated data tables would provide for some interesting use cases. Something like a cutout service might be appropriate for some of the data, the challenging thing for us however to put all these into an SSA is simply the variety of local data access methods required. (Mike Fitzpatrick)

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->