Image Data Model Effort

The Image Data Model, currently under development, will describe multdimensional image and image cube datasets and instances. It will serve as the basis for the description of Image datasets in archives and VO queries, as well as the basis for Image data access protocols such as SIAV2 and related access methods such as siav2.accessData.

ImageDM-ObsCore Convergence

It was agreed in the fall 2013 interop to do whatever is necessary to bring the Image data model into convergence with ObsCore (e.g., ObsTAP), in particular making data models such as ImageDM and SpectralDM extensions of ObsCore, all sharing a common core data model. We are fast-tracking an updated "stable" version of both the ImageDM and SIAV2 drafts that will make a first cut at this convergence, sufficient to support ongoing development for the Cube initiative, while further development of the working drafts goes forward. During prototyping the stable version of ImageDM will be used for prototype development while the full standard evolves (i.e., there will be two versions of ImageDM until eventually the stable version is retired and replaced by the final standard). An initial analysis of what is required has been performed by a subset of the ImageDM author group (FB, ML, MCD, DT) and the results are summarized here:

The consensus at this point appears to be that we finalize a set of Characterisation V2 Utypes, use these for both ImageDM and SpectralDM, and eventually update the older ObsCore/ObsTAP document to comply, however what is eventually done will depend upon a more detailed analysis still to be performed. This will take a while, but all we need for ongoing Cube development is the updated ImageDM and SIAV2 drafts plus a summary of the revised Utypes planned for ObsCore.

Further updates will be posted here as this effort progresses. Discussion should be carried out on the DM mailing list.

Discussion Points:

This effort carries on from the above draft (Nov. 2013).

The following is a list of topics to be discussed regarding the draft and will guide the future evolution of the document. It is a dynamic list, topics will be added as needed. Discussion should be carried out on the DM mailing list. Here we will summarize the topics, provide links to the relevant threads and note the resolution. It would be greatly appreciated if we can keep discussion topics within the relevant thread.

1. Relation to ObsCore

Carries on from the Nov. draft, formalizing the relation to the ObsCore 1.0. This is a fairly broad topic, and will undoubtedly spawn other threads. The goal of this topic is to explicitely illustrate the Observation model used by the Image and Spectral models and how that relates to the existing ObsCore model. The result of this effort may lead to a recommendation to update that standard, but should not require it.

Threads:

Links: 2. Observation Relation to Dataset

This topic spawned from "Relation to ObsCore". This topic is very "ObsCore" centric, but since the Observation/Dataset relation is central to the Image and Spectral Model structure, it has significant ramifications on their design.

It is clear that there is a relationship between "Observation" and a more generic "Dataset". This "Dataset" would contain elements such as the dataProductType, and dataProductSubtype, presumably others. This object has not been formally defined.

In ObsCore, there is an implied relationship for Observation as an Extension of Dataset in the location of these attributes. So, I have always interpreted that Observation "is" a Dataset. This is reflected in my choice of the name "ObservationDataset" in the left hand package of my diagram. It implies that it is a Dataset extended for Observation purposes. Recent discussion brings this relationship into question, with assertions that an Observation can be associated with 0 or more Datasets. "

The goal of this thread is to gain concensus on what this relationship currently is, and how it should be represented in the Observation model used by the Image/Spectral models.

Threads:

Links: Resolution:

The Spectral/Image/Cube model work will show "Observation" as a type of Experiment which is associated with 0:* "Dataset"s. The Dataset will be generic, with ObsDataset extension to include metadata from the Observation model.

This includes elements such as Target, Proposal, ObsConfig.. etc.

3. Mapping

Major dopic for discussion of the Mapping object, its definition and purpose.

Threads:

Links: Resolution:

The Image/Cube model will be refactored to distribute the Mapping information to the corresponding STC based structure for coordinate systems and frames.

The current STC model does not accomodate the Linear + Projection model for the coordinate system transforms, so adjustments will be made in that area.

IF access protocols (SIAP2) requires a Mapping object to encapsulate the transform information in a query response, it should be defined there, with a 'mapping' to indicate how to populate that structure from the model components (as outlined in the PDF file above).

4 . Provenance

Hopefully a very minor thread. There is an ambiguity in the ObsCore document, where the Utypes imply a "Provenance" container class in the model which is not reflected in the UML diagrams.

The goal of this thread is merely to decide if this is a mistake in the diagram which should be 'fixed' or not.

Thread:

Resolution:

The concensus seems to be that there is no logical head for Provenance at this time, and the model is not defined enough at this point to presume structure, so we will not include a Provenance class at this time.

5. Provenance: new elements in ImageDM.

The image model adds new elements under the Provenance umbarella (Bandpass, DataSource, ObsData). There is question regarding where/how these should be included.

Threads:

Resolution:

This thread has had no discussion.. At present, these items are to be modeled under the Observation section, brought to the Dataset via the ObsConfig container.

6. Derived object.

Minor topic regarding the need and placement of the Derived class.

Threads:

Resolution:

Yes, Derived should be included.. attached to Dataset as 'information derived from other model elements.'

Unified Models:

The next stage involves folding the above discussion points into a new set of model diagrams which are consistent with each other, and correctly model the object relationships.

Models involved:

1. Models involved:

+ Observation/Dataset: High level metadata for generic Dataset, plus Observation (hypothetical) experiment model, from which certain metadata is referenced as 'provenance' in an ObservationDataset.

+ NDCube: Specification of SparseCube and NDImage Datasets.

+ STC: STC-1.33 model (my interpretation of schema) provides baseline for STC elements. STCMod diagrams illustrate recommended modifications to that model to support NDCube.

+ Characterisation: Char-1.13 model elements used in NDCube, expanded to enough detail to verify consistency in axis detail between CharAxis and DataAxis coordinates.

+ ivoa Types: For this project, each of these models are defined using a common set of basic datatypes. We use the ivoa types model from the VO-DML project in order to fill our need, and facilitate collaboration with efforts to port the models into that framework.

2. SVN Repository:

The model documents (Modelio save set and XMI exports), along with generated diagrams are stored in the Volute repository .

3. Ongoing Discussions:

Discussion threads on model diagrams:

Second Draft:

The above model diagrams were presented at the IVOA interop in Madrid, May-2014. A link to the presentation PDF is here.

The requirements for moving forward from that point included:

  • define CubeDM as extension of a common baseline Observation-Dataset model usable for Cube, ObsCore, Spectral, and future models.
  • define and incorporate a 'simpler' STC-2 model which meets requirements to support Cube model and compatible with official STC-2 development being done in parallel by Arnold.
  • Goal: RFC ready standard by October interop.
Revised CubeDM document(s) were delivered to the IVOA DM working group Sept. 30. (see announcement)

The documents and Modelio UML project have been updated in the Volute repository. As explained in the announcement, this delivery split the models into 2 documents to better separate the areas of concern for Dataset Metadata, and particular Dataset types (e.g. NDCube). Pointers to the document PDFs are:

Discussion Threads:

Current Drafts

Latest version of the draft:

Previous versions: Some related documents can be found here:

In particular, there is a new draft of the SIAV2 image access specification which is sync-ed up with the current version of the ImageDM.

The VAO Cube whitepaper relevant use cases and analysis addressing the problem of modeling and accessing multidimensional image data in the VO:

This includes a section on the ImageDM, mainly addressing issues pertaining to the image cube use cases.

=


Topic attachments
I Attachment History ActionSorted ascending Size Date Who Comment
PNGpng DMArch-ImageFull.png r1 manage 107.5 K 2013-11-11 - 04:10 DougTody ImageDM UML Class Diagram
Texttxt ImageDM-ObsCore.txt r1 manage 16.2 K 2013-10-28 - 13:36 DougTody Initial analysis of ImageDM-ObsCore Convergence
Microsoft Excel Spreadsheetxls ImageDMObscoreSTC-Utypes-ML8Nov.xls r1 manage 22.0 K 2013-11-16 - 06:02 DougTody Char and ObsCore Utypes updated for Char2 (ML, FB)
PDFpdf ImageDMObscoreSuggestionForDMElementsbasedOnSTC-19Nov.pdf r1 manage 128.1 K 2013-11-20 - 10:10 MireilleLouys  
Unknown file formatods ImageDMObscoreSuggestionForDMElementsbased_OnSTC-19Nov.ods r1 manage 25.2 K 2013-11-20 - 10:01 MireilleLouys column C and blue cells as suggestions for attribute name updates
PDFpdf WD-ImageDM-20130505.pdf r4 r3 r2 r1 manage 1365.2 K 2013-05-17 - 08:25 DougTody Image Data Model draft (still under development)
PDFpdf WD-ImageDM-20130812.pdf r1 manage 1224.0 K 2013-08-16 - 19:56 DougTody ImageDM draft as of 12Aug2013 (first fairly complete version)
PDFpdf WD-ImageDM-20131110.pdf r1 manage 1335.8 K 2013-11-11 - 04:12 DougTody ImageDM draft - architecture section updated to reflect Image/ObsCore integration
Unknown file formatdocx WD-ImageDM-20131115.docx r1 manage 334.3 K 2013-11-16 - 06:08 DougTody ImageDM-Stable (15Nov draft)
PDFpdf WD-ImageDM-20131115.pdf r1 manage 1278.8 K 2013-11-16 - 06:08 DougTody ImageDM-Stable (15Nov draft)
Unknown file formatdocx WD-ImageDM-20131117.docx r1 manage 451.2 K 2013-11-18 - 04:33 DougTody ImageDM-Stable (17Nov draft)
PDFpdf WD-ImageDM-20131117.pdf r1 manage 1349.9 K 2013-11-18 - 04:31 DougTody ImageDM-Stable (17Nov update)
PDFpdf imagedm-obscore.pdf r3 r2 r1 manage 89.2 K 2013-12-25 - 19:54 DougTody PDF version of imagedm-obscore spreadsheet
Microsoft Excel Spreadsheetxls imagedm-obscore.xls r3 r2 r1 manage 143.0 K 2013-12-25 - 19:49 DougTody ImageDM Utypes integrated with ObsCore, Char2 (25Dec)
Microsoft Excel Spreadsheetxls siav2-keywords.xls r1 manage 136.0 K 2013-08-16 - 19:58 DougTody Image (and SIAV2) data model spreadsheet
Microsoft Excel Spreadsheetxls siav2-obstap-FB.xls r1 manage 35.5 K 2013-10-28 - 13:42 DougTody ImageDM and ObsCore Utypes vs Char2 (FB)
Microsoft Excel Spreadsheetxls siav2-obstap.xls r1 manage 140.5 K 2013-10-28 - 13:38 DougTody Spreadsheet contrasting ImageDM, ObsCore diffs (DT)
Edit | Attach | Watch | Print version | History: r23 < r22 < r21 < r20 < r19 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r23 - 2015-06-01 - OmarLaurino
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback