Image Data Model Effort
The Image Data Model, currently under development, will describe multdimensional image and image cube datasets and instances. It will serve as the basis for the description of Image datasets in archives and VO queries, as well as the basis for Image data access protocols such as SIAV2 and related access methods such as siav2.accessData.
It was agreed in the fall 2013 interop to do whatever is necessary to bring the Image data model into convergence with ObsCore
), in particular making data models such as ImageDM
extensions of ObsCore
, all sharing a common core data model. We are fast-tracking an updated "stable" version of both the ImageDM
and SIAV2 drafts that will make a first cut at this convergence, sufficient to support ongoing development for the Cube initiative, while further development of the working drafts goes forward. During prototyping the stable version of ImageDM
will be used for prototype development while the full standard evolves (i.e., there will be two versions of ImageDM
until eventually the stable version is retired and replaced by the final standard). An initial analysis of what is required has been performed by a subset of the ImageDM
author group (FB, ML, MCD, DT) and the results are summarized here:
The consensus at this point appears to be that we finalize a set of Characterisation V2 Utypes, use these for both ImageDM
, and eventually update the older ObsCore
/ObsTAP document to comply, however what is eventually done will depend upon a more detailed analysis still to be performed. This will take a while, but all we need for ongoing Cube development is the updated ImageDM
and SIAV2 drafts plus a summary of the revised Utypes planned for ObsCore
Further updates will be posted here as this effort progresses. Discussion should be carried out on the DM mailing list.
This effort carries on from the above draft (Nov. 2013).
The following is a list of topics to be discussed regarding the draft and will guide the future evolution of the document. It is a dynamic list, topics will be added as needed. Discussion should be carried out on the DM mailing list. Here we will summarize the topics, provide links to the relevant threads and note the resolution. It would be greatly appreciated if we can keep discussion topics within the relevant thread.
1. Relation to ObsCore
Carries on from the Nov. draft, formalizing the relation to the ObsCore
1.0. This is a fairly broad topic, and will undoubtedly spawn other threads. The goal of this topic is to explicitely illustrate the Observation model used by the Image and Spectral models and how that relates to the existing ObsCore
model. The result of this effort may lead to a recommendation to update that standard, but should not require it.
2. Observation Relation to Dataset
This topic spawned from "Relation to ObsCore
". This topic is very "ObsCore" centric, but since the Observation/Dataset relation is central to the Image and Spectral Model structure, it has significant ramifications on their design.
It is clear that there is a relationship between "Observation" and a more generic "Dataset". This "Dataset" would contain elements such as the dataProductType, and dataProductSubtype, presumably others. This object has not been formally defined.
, there is an implied relationship for Observation as an Extension of Dataset in the location of these attributes. So, I have always interpreted that Observation "is" a Dataset. This is reflected in my choice of the name "ObservationDataset" in the left hand package of my diagram. It implies that it is a Dataset extended for Observation purposes. Recent discussion brings this relationship into question, with assertions that an Observation can be associated with 0 or more Datasets. "
The goal of this thread is to gain concensus on what this relationship currently is, and how it should be represented in the Observation model used by the Image/Spectral models.
The Spectral/Image/Cube model work will show "Observation" as a type of Experiment which is associated with 0:* "Dataset"s. The Dataset will be generic, with ObsDataset
extension to include metadata from the Observation model.
This includes elements such as Target, Proposal, ObsConfig
Major dopic for discussion of the Mapping object, its definition and purpose.
The Image/Cube model will be refactored to distribute the Mapping information to the corresponding STC
based structure for coordinate systems and frames.
The current STC
model does not accomodate the Linear + Projection model for the coordinate system transforms, so adjustments will be made in that area.
IF access protocols (SIAP2) requires a Mapping object to encapsulate the transform information in a query response, it should be defined there, with a 'mapping' to indicate how to populate that structure from the model components (as outlined in the PDF file above).
4 . Provenance
Hopefully a very minor thread. There is an ambiguity in the ObsCore
document, where the Utypes imply a "Provenance" container class in the model which is not reflected in the UML diagrams.
The goal of this thread is merely to decide if this is a mistake in the diagram which should be 'fixed' or not.
The concensus seems to be that there is no logical head for Provenance at this time, and the model is not defined enough at this point to presume structure, so we will not include a Provenance class at this time.
5. Provenance: new elements in ImageDM.
The image model adds new elements under the Provenance umbarella (Bandpass, DataSource
). There is question regarding where/how these should be included.
This thread has had no discussion.. At present, these items are to be modeled under the Observation section, brought to the Dataset via the ObsConfig
6. Derived object.
Minor topic regarding the need and placement of the Derived class.
Yes, Derived should be included.. attached to Dataset as 'information derived from other model elements.'
The next stage involves folding the above discussion points into a new set of model diagrams which are consistent with each other, and correctly model the object relationships.
1. Models involved:
+ Observation/Dataset: High level metadata for generic Dataset, plus Observation (hypothetical) experiment model, from which certain metadata is referenced as 'provenance' in an ObservationDataset
+ NDCube: Specification of SparseCube
and NDImage Datasets.
-1.33 model (my interpretation of schema) provides baseline for STC
elements. STCMod diagrams illustrate recommended modifications to that model to support NDCube.
+ Characterisation: Char-1.13 model elements used in NDCube, expanded to enough detail to verify consistency in axis detail between CharAxis
+ ivoa Types: For this project, each of these models are defined using a common set of basic datatypes. We use the ivoa types model from the VO-DML project in order to fill our need, and facilitate collaboration with efforts to port the models into that framework.
2. SVN Repository:
The model documents (Modelio save set and XMI exports), along with generated diagrams are stored in the Volute repository
3. Ongoing Discussions:
Discussion threads on model diagrams:
The above model diagrams were presented at the IVOA interop in Madrid, May-2014. A link to the presentation PDF is here
The requirements for moving forward from that point included:
- define CubeDM as extension of a common baseline Observation-Dataset model usable for Cube, ObsCore, Spectral, and future models.
- define and incorporate a 'simpler' STC-2 model which meets requirements to support Cube model and compatible with official STC-2 development being done in parallel by Arnold.
- Goal: RFC ready standard by October interop.
document(s) were delivered to the IVOA DM working group Sept. 30. (see announcement
The documents and Modelio UML project have been updated in the Volute repository. As explained in the announcement, this delivery split the models into 2 documents to better separate the areas of concern for Dataset Metadata, and particular Dataset types (e.g. NDCube). Pointers to the document PDFs are:
Latest version of the draft:
Some related documents can be found here:
In particular, there is a new draft of the SIAV2 image access specification
which is sync-ed up with the current version of the ImageDM
The VAO Cube whitepaper relevant use cases and analysis addressing the problem of modeling and accessing multidimensional image data in the VO:
This includes a section on the ImageDM
, mainly addressing issues pertaining to the image cube use cases.