Data Modeling Meeting

ESO, Garching, Jan. 23/24, 2003

Agenda

Day 1, Room 231, 1:30 pm - 5:15 pm

Day 2, Room 232, 9 am - 12 am

Meeting split for informal discussions within ESO.

Notes About Individual Agenda Items

1 [MD]: IVOA meeting conclusions

2 [JMD]: VO Data Models: status

report on boston meeting, Oct 2002

See above minutes for meaning of conforming data model and modelling exercise on spectral bandpass.

Wavelength/Frequency model

  <ENERGY>
   <MODE>Wavelength</MODE>
   <VALUE>512.02</VALUE>
   <UNIT>nm</UNIT>
  </ENERGY>

or

  <WAVELENGTH UNIT="nm">512.02</WAVELENGTH>
  (#PC: <FRAVERGY TYPE="wavelength" UNIT="nm">512.02</WAVELENGTH>?)

What is an image?

• sparse (with bad pixel map) is ok
• must have regular pixellated array
• must have pixels with scalar simple data types (int, real, complex, char?)
• can have n dimensions, n arbitrary

Aggregation is generally better/simpler than inheritance (HAS-A instead of IS-A)

What is data, hypermaps, images, tables, documents, code, plots, links, etc?

related documents

3 [MIL]: open discussion on ...

• reduction pipelines
• data quality
• hierarchy/granularity in metadata
• versioning

IDHA (Images Distribuées Hétérogènes pour l'Astronomie) team:

• CDS FB, A.Oberto, et al
• LSIIT ML, C.Collet
• IAP, PD, Y.Mellier, E.Bertin
• QUB: F.Murtagh

images in astronomy

• A. metadata definition for image sets in diverse arch.
• B. multicolor image analysis for obj. recognition and cross identification

Goal: provide a mode for astronomical image data http://alinda.u-strasbg.fr/IDHA/datamodel.html

a) reduction pipeline

• [JMD] 'region object' to describe FOV
• make more general object which has a common concept for raw and processed images

b) data quality

• measure for quality: dm richness (detailed), traceability (raw<=>publi.)

c) granularity

• [BV] How about applicability XSLTs to manipulate XML DMs?
• raw - cal. - mosaic - rgb ...

d) versioning

• feedback loop: obs. - reduc. - visual. - interpret. (and back to obs.)
• software versioning/data versioning

4 [FB]: How data model + votable was used in AVO GOODS demo

Bonnarel, Louys, Boch, Fernique, Schaaf, Derriere, Ochsenbein

from datamodel to xml

• visions of data model must be translated in documents for describing metadata (different scenarii for different users/applications)
• xml good for exchanging
• votable or some other dtd ? -> votable because already accepted
• how to code a Tree (partial view of a DM) into a table oriented DTD?
• chance: definition of VOTable RESOURCE is recursive (1->n links)
• each class of the model is defined as a RESOURCE (fixed name, no ID in order to allow multi-occurance), with a specific table part
• children classes as nested resources
• [JMD]: How to link a filter in the Observing_group table to the filter given in the filter resource?
• GOODS image server has more functionalities than public aladin; public aladin will inherit those, but not before summer
• SIAP is flat, no hierarchy, output = single table

5 [JMD]: Chandra pipeline

chandra -> telemetry -> fits files with mnemonics (Level 0)

Level 1: takes level 0 data -> calibrated scientific file

Events ASOL (Aspecs solution, geometry vs time ra,dec,roll(t)), spacecraft attitude, bending of telescope tube, (1 arcsec within an observation!) HK, temperature, radiation environment, etc -> ra, dec for all photons,

between level 1 and 2: observation can be split up -> level two combines them

typical exptime: 50ksec to 100ksec (some 1ksec, some 1Msec)

level 2 removes garbage: L2 data is just a starting point used by naive users, experts will go back to L1

L1->L2 pipeline is publicly available

L3 pipeline (not yet in place): to generate uniform source catalogs, etc

L2 event list (EVT) -> binned on the fly to provide an image of requested resolution/size

dmcopy evt2.fits[energy=2000,5500][time=from,to][bin x=3000:4000:4,y=...] produces the requested image.

IDHA covers some part of Chandra calibration system, but not the bookkeeping.

6 [AM]: HST pipeline

in contrast to Chandra ...

• pixel info and header info come from different sources (files + DBs) when assembling raw files
• science and engineering streams are separated for HST, but not for Chandra

[PB] reports that IAU interferometry WG exists to define data format for interferometry

7 [MD]: Forming IVOA Data Modeling Standardisation WG

• Forum: dm@ivoa.net
• Lead: Jonathan McDowell (CfA)
• Members
  • Alberto Micol,
  • Mireille Louys,
  • ?Dave Giarretta?,
  • others...

For events 2003 see Avo:AvoEvents

further ...

• NVO meeting in Nashville with AAS in June
• NVO team meeting, where?, March,

Meeting adjourned. Splinter discussions followed on:

• a possible model for a distributed Aladin, to access both data and metadata directly from the VO data centres, without the need of storing them in the Aladin server.
• UCD discussion
• a possible design for a Data Model suitable for an ASTROVIRTEL Exposure-Time-Calculator-based archive search engine. Version 0.2 will be drafted and sent to participants.

Action Items

• [JMD] announce creation of DM group to ivoa@ivoa.net
• [MD] register below members with dm@ivoa.net: Haase, Micol, Leoni, Didelon, Bonnarel, McDowell, Louys, Dolensky, Voisin

(minutes written by A. Micol; edited by M. Dolensky)

Data Modeling Meeting

ESO, Garching, Jan. 23/24, 2003

Agenda

Notes About Individual Agenda Items

from datamodel to xml

between level 1 and 2: observation can be split up -> level two combines them

L3 pipeline (not yet in place): to generate uniform source catalogs, etc

L2 event list (EVT) -> binned on the fly to provide an image of requested resolution/size

dmcopy evt2.fits[energy=2000,5500][time=from,to][bin x=3000:4000:4,y=...] produces the requested image.

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->

THIS PAGE IS UNDER CONSTRUCTION

Data Modeling Meeting

ESO, Garching, Jan. 23/24, 2003

Agenda

Day 1, Room 231, 1:30 pm - open ended (~5 pm)

M Dolensky: report on IVOA meeting conclusions J McDowell: status of data modeling efforts M Louys: start an open discussion of - reduction pipelines - data quality - hierarchy/granularity in the metadata - versioning

AOB

Day 2: room 232, 9am - max 12pm

- F. Bonnarel: how data model + VOTable was used in AVO GOODS first light demo

AOB (P.Didelon ?)

-M.Dolensky: events 2003, plans wrap up

Notes About Individual Agenda Items

MD: IVOA meeting conclusions

JMD: VO Data Models: status

boston meeting goals were: compare existing models ... ... compliant data model: first: to write the concept one wants to describe second: UML or equivalent based model third: convert UML to XMI or equiv. and verify model schema satisfies VO DM standards (see below) fourth(optional): provide a reference XSD schema and XML instance examples. IVOA InterOp. group can adopt model as a recommended VO model for the given concept.

Conforming IVOA Data Models unique URI unique name version description curation metadata url for white paper class descriptions: - name and description - attributes, name,type,quantity,cardinality,defaults, units? - properties - methods

bandpass methods and attributes effective lambda min and max lambda/freq/energy/wave numbers transmission curve (add or multiply?) theoretical transmission curves (also as function of time, temperature)

uncertainties on each number! quality object (non numerical description)

problems: red leaks, periodic filters, wedge and band stop (notch) filters multiple filters atmospheric transmission ?

bandpass attributes - name, generic name, waveband name - central and end wavelengths - measured ot theoretical transmission function what about zeropoints ?

bandpass classes NOT DISCUSSED

Wavelength/Frequency model Wavelength 512.02 nm

or

512.02

(#PC: 512.02?)

What is an image? - sparse (with bad pixel map) is ok - must have regular pixellated array - must have pixels with scalar simple data types (int, real, complex, char?) - can have n dimensions, n arbitrary

aggregation better than inheritance (HAS-A instead of IS-A)

what is data, hypermaps, images, tables, documents, code, plots, links, etc)

http://hea-www.harvard.edu/~jcm/vo/dmtm

ML: open discussion - reduction pipelines - data quality - hierarchy/granularity in metadata - versioning

CDS FB, A.Oberto, et al LSIIT ML, C.Collet IAP, PD, Y.Mellier, E.Bertin QUB: F.Murtagh

images in astronomy A. metadata definition for image sets in diverse arch. B. multicolor image analysis for obj. recognition and cross identification

Goal: provide a mode for astronomical image data http://alinda.u-strasbg.fr/IDHA/datamodel.html

Granularity: BV? better having XSLTs onto your XML DM ?

FB: How data model + votable was used in AVO GOODS demo Bonnarel, Louys, Boch, Fernique, Schaaf, Derriere, Ochsenbein

from datamodel to xml - visions of data model must be translated in documents for describing metadata (different scenarii for different users/applications) - xml good for exchanging

- votable or some other dtd ? -> votable because already accepted - how to code a Tree (partial view of a DM) into a table oriented DTD ? - chance: definition of VOTable RESOURCE is recursive (1->n links) - each class of the model is defined as a RESOURCE (fixed name, no ID in order to allow multi-occurance), with a specific table part

- children classes as nested resources ?JMD: how to link a filter in the Observing_group table to the filter given in the filter resource ? (GOODS image server has more functionalities than public aladin; public aladin will inherit those, but not before summer)

- SIAP is flat, no hierarchy, output = single table

JMD: Chandra pipeline

chandra -> telemetry -> fits files with mnemonics (Level 0) Level 1: takes level 0 data -> calibrated scientific file Events ASOL (Aspecs solution, geometry vs time ra,dec,roll(t)), spacecraft attitude, bending of telescope tube, (1 arcsec within an observation!) HK, temperature, radiation environment, etc -> ra, dec for all photons,

between level 1 and 2: observation can be split up -> level two combines them typical exptime: 50ksec to 100ksec (some 1ksec, some 1Msec) level 2 removes garbage: L2 data is just a starting point used by naive users, experts will go back to L1 L1->L2 pipeline is publicly available

L3 pipeline (not yet in place): to generate uniform source catalogs, etc

L2 event list (EVT) -> binned on the fly to provide an image of requested resolution/size

dmcopy evt2.fits[energy=2000,5500][time=from,to][bin x=3000:4000:4,y=...] produces the requested image.

AM: HST pipeline ...

MD: IVOA Data Modeling Standardisation WG

Forum: dm@ivoa.net

Lead: Jonathan McDowell (CfA) Members: Alberto Micol (ST-ECF), Mireille Louys (CDS), ?Dave Giarretta (ASTROGRID)?, ? someone else ?

Remarks (scope, goal, ...)

for events 2003 see Avo:AvoEvents

us-vo Nashville, June aas meeting (but sidney ...) NVO team meeting, ?, March, status reports

Meeting adjourned. Splinter discussions followed on: - a possible model for a distributed Aladin, to access both data and metadata directly from the VO data centres, without the need of storing them in the Aladin server. - UCD discussion - a possible design for a Data Model suitable for an ASTROVIRTEL Exposure-Time-Calculator-based archive search engine. Version 0.2 will be drafted and sent to participants.

(created by MarkusDolensky 14.03.2025@08:10:33)

<-- 

 

 

 Set ALLOWTOPICRENAME = TWikiAdminGroup
 
 
-->