From dtody@nrao.edu Sat Oct 19 11:45:07 2013 Date: Sat, 19 Oct 2013 11:44:18 -0600 (MDT) From: Douglas Tody To: Jesus Salgado , Mark Cresitello-Dittmar , Francois Bonnarel , Mireille Louys , Arnold Rots , Jose Enrique Ruiz Cc: Patrick Dowler Subject: ImageDM and ObsCore Hi All - I hope everyone has recovered from the interop and is ready for the next steps. I took about a week off myself (helped by the government shutdown here which conveniently came immediately after the interop!). We agreed in the interop that I would as a high priority prepare the next version of the ImageDM draft, reflecting the interop discussions as well as feedback already received prior to the interop on the initial draft. Mark will then edit the final version, providing consistency with the SpectralM in the process (SpectralDM will also be updated). Meanwhile Pat, Francois, and I are preparing an updated version of the SIAV2 draft. These next versions of SIAV2 and the ImageDM will be synchronized (consistent), and should be sufficient to support related development over the next few months while we iterate on the next versions of the drafts leading up to the May interop. In particular we have major development of cube data access and analysis facilities going forward here at NRAO this fall, involving other non-VO groups within NRAO, for which updated and consistent drafts are required ASAP this fall. The main change to ImageDM coming out of the interop discussions is to make the ImageDM and ObsCore consistent - ideally ImageDM extends ObsCore, so that if we already implement ObsCore/ObsTAP we can add more metadata to support the ImageDM, and ObsTAP and SIAV2 (and a future generic AccessData also being developed) can play well together. Our main focus at this point are the data model Utypes; the image data access model (AccessData etc.) will become the focus once we have the basic model defined. To get this started I have gone through the most recent ImageDM and compared this to ObsCore to see what differences remain. The attached ImageDM spreadsheet details the differences. In the spreadsheet, column B contains the ObsCore field IDs. Those in bold are the mandatory ObsCore fields. Utypes which differ between the two models are highlighted with a yellow background. The Utypes shown are the ImageDM/SpectralDM versions. Refer to Appendix B of the ObsCore doc (pg 35) to see the corresponding Utypes defined by ObsCore. A summary of the differences follows below. The edited ImageDM spreadsheet highlighting the differences with ObsCore is also attached. We need to decide what to do to bring ImageDM/SpectralDM in line with ObsCore - do we merely adopt ObsCore as the core model and change ImageDM/SpectralDM and possibly Char/Char2 etc., or do we produce an updated version of ObsCore to minimize changes to the other data models? Please think about it and suggest what you think is the best approach. We have an opportunity here to possibly make all the major data models consistent. But we will need to move fast for this next update. - Doug ImageDM vs ObsCore Summary of Differences ------------------------------------------ ObsTAP ID vs ImageDM shortname / field ID These serve much the same purpose, e.g., a column name in a table, or a shortname for get/put calls. The simplest solution is to just adopt the ObsCore IDs and extend the list to the full set of attributes in ImageDM. Dataset vs Observation ImageDM and SpectralDM use Dataset.X for attributes describing the overall dataset, e.g., Dataset.Type, Dataset.DataModel, etc. ObsCore uses "Obs" for the equivalent Utypes, and is oriented more toward arbitrary (VO and non-VO) data products, e.g., Obs.dataProductType. ObsCore has no equivalent to Dataset.DataModel. Convergence requires that we choose one or the other of Dataset or Obs. ObsCore Utypes Missing in ImageDM obs_id DataID.ObservationID facility_name Provenance.ObsConfig.facility.name s_ra, s_dec ImageDM uses the field Location.Coord instead s_resolution_min ImageDM currently defines only Resolution.RefVal s_resolution_max em_respower_min ImageDM currently defines only ResolPower.RefVal em_respower_max Char.FluxAxis ImageDM and SpectralDM use Char.FluxAxis, whereas ObsCore uses Char.ObservableAxis. ObsCore is more correct in this case, since the observable is not always flux. But we refer to a generic Flux quantity in many of our models. Utype Inconsistencies I identified about 20 of these (marked with a yellow background in the spreadsheet; we just need to decide on the final Utypes. Do we keep the ObsCore Utypes and change all our other DMs, or have a new version of ObsCore? [ Part 2: "Deleted Attachment" ] [ The following attachment was DELETED when this message was saved: ] [ A Application/VND.MS-EXCEL (Name="siav2-obstap.xls") segment of abou ] [ described as "siav2-obstap.xls" ] From mdittmar@cfa.harvard.edu Wed Oct 23 15:14:14 2013 Date: Wed, 23 Oct 2013 17:13:35 -0400 From: "CresitelloDittmar, Mark" To: Douglas Tody Cc: Jesus Salgado , Mark Cresitello-Dittmar , Francois Bonnarel , Mireille Louys , Arnold Rots , Jose Enrique Ruiz , Patrick Dowler Subject: Re: ImageDM and ObsCore All, I've been tied up with other work.. so haven't had a chance to review this well. I'll be giving it good look tomorrow.   some quick comments below.. If I recall, ObsCore makes use of Characterisation primarily through reference. It leaves the details to that model and only talks about it's usage of certain components.  The Reference listed is for Char v2.0, so I'd say that as far as Utypes go, ObsCore is definitive for the Observation level stuff, while Char 2.0 is definitive for the Characterisation stuff.  We need to agree which to put in Char 2.0, then ObsCore should adjust to that if needed.  On Sat, Oct 19, 2013 at 1:44 PM, Douglas Tody wrote: ImageDM vs ObsCore Summary of Differences ------------------------------------------     Dataset vs Observation         ImageDM and SpectralDM use Dataset.X for attributes describing         the overall dataset, e.g., Dataset.Type, Dataset.DataModel, etc.         ObsCore uses "Obs" for the equivalent Utypes, and is oriented         more toward arbitrary (VO and non-VO) data products, e.g.,         Obs.dataProductType.  ObsCore has no equivalent to         Dataset.DataModel.  Convergence requires that we choose one or         the other of Dataset or Obs. During the adass meetings, I began migrating my UML diagrams to reflect the ObsCore extention.  From that perspective, I recall there is some difficulty with the Obs vs DataSet objects.  They have similar info at different levels.  I was working toward getting "Spectral" to be an extension of "Obs" where the extension is the data portions, etc.     Char.FluxAxis         ImageDM and SpectralDM use Char.FluxAxis, whereas ObsCore uses         Char.ObservableAxis.  ObsCore is more correct in this case,         since the observable is not always flux.  But we refer to a         generic Flux quantity in many of our models. ImageDM probably shouldn't use FluxAxis. SpectralDM can include FluxAxis as an extension to Char.     Utype Inconsistencies         I identified about 20 of these (marked with a yellow background         in the spreadsheet; we just need to decide on the final Utypes.         Do we keep the ObsCore Utypes and change all our other DMs, or         have a new version of ObsCore? more on this tomorrow. Mark From mireille.louys@unistra.fr Thu Oct 24 13:13:29 2013 Date: Thu, 24 Oct 2013 21:12:30 +0200 From: Louys Mireille To: Douglas Tody , François Bonnarel Subject: Re: ImageDM and ObsCore Dear Doug , thanks for the convergence effort . I inserted my comments below . Le 19/10/2013 19:44, Douglas Tody a écrit : Hi All - .... The main change to ImageDM coming out of the interop discussions is to make the ImageDM and ObsCore consistent - ideally ImageDM extends ObsCore, so that if we already implement ObsCore/ObsTAP we can add more metadata to support the ImageDM, and ObsTAP and SIAV2 (and a future generic AccessData also being developed) can play well together.  Our main focus at this point are the data model Utypes; the image data access model (AccessData etc.) will become the focus once we have the basic model defined. To get this started I have gone through the most recent ImageDM and compared this to ObsCore to see what differences remain.  The attached ImageDM spreadsheet details the differences. In the spreadsheet, column B contains the ObsCore field IDs.  Those in bold are the mandatory ObsCore fields.  Utypes which differ between the two models are highlighted with a yellow background.  The Utypes shown are the ImageDM/SpectralDM versions.  Refer to Appendix B of the ObsCore doc (pg 35) to see the corresponding Utypes defined by ObsCore. A summary of the differences follows below.  The edited ImageDM spreadsheet highlighting the differences with ObsCore is also attached. We need to decide what to do to bring ImageDM/SpectralDM in line with ObsCore - do we merely adopt ObsCore as the core model and change ImageDM/SpectralDM and possibly Char/Char2 etc., or do we produce an updated version of ObsCore to minimize changes to the other data models? Please think about it and suggest what you think is the best approach. We have an opportunity here to possibly make all the major data models consistent.  But we will need to move fast for this next update.     - Doug I think most discrepencies can be treated by: -  changing Obscore Utypes when possible. - include LoLim, HiLim values specified in ObsCore into the ImageDM when it defines only a refval: this allows to anticipate extensions for new use-cases . I am ok to change Observation. or Obs. to Dataset ImageDM vs ObsCore Summary of Differences ------------------------------------------     ObsTAP ID vs ImageDM shortname / field ID         These serve much the same purpose, e.g., a column name in a         table, or a shortname for get/put calls.  The simplest solution         is to just adopt the ObsCore IDs and extend the list to the full         set of attributes in ImageDM.     Dataset vs Observation         ImageDM and SpectralDM use Dataset.X for attributes describing         the overall dataset, e.g., Dataset.Type, Dataset.DataModel, etc.         ObsCore uses "Obs" for the equivalent Utypes, and is oriented         more toward arbitrary (VO and non-VO) data products, e.g.,         Obs.dataProductType.  ObsCore has no equivalent to         Dataset.DataModel.  Convergence requires that we choose one or         the other of Dataset or Obs. so Ok to homogeneize to Dataset. Consequently we can add also Dataset.DataModel for completion. However, I think we worked out a precise definition of dataproductType in ObsCore , that we should re-use in ImageDM     ObsCore Utypes Missing in ImageDM         obs_id              DataID.ObservationID         facility_name       Provenance.ObsConfig.facility.name         s_ra, s_dec         ImageDM uses the field Location.Coord instead         s_resolution_min    ImageDM currently defines only Resolution.RefVal         s_resolution_max         em_respower_min     ImageDM currently defines only ResolPower.RefVal         em_respower_max ResolPower accessed through  Resolution in ObsCore. It belongs to this Property as in Charv2.0     Char.FluxAxis         ImageDM and SpectralDM use Char.FluxAxis, whereas ObsCore uses         Char.ObservableAxis.  ObsCore is more correct in this case,         since the observable is not always flux.  But we refer to a         generic Flux quantity in many of our models. What about adding ObservableAxis in ImageDM for the cases where Flux is not appropriate?     Utype Inconsistencies         I identified about 20 of these (marked with a yellow background         in the spreadsheet; we just need to decide on the final Utypes.         Do we keep the ObsCore Utypes and change all our other DMs, or         have a new version of ObsCore? More to come in details , namely by François. Best wishes , Mireille From francois.bonnarel@astro.unistra.fr Fri Oct 25 10:41:08 2013 Date: Fri, 25 Oct 2013 18:40:06 +0200 From: François Bonnarel To: Douglas Tody Cc: Louys Mireille , "CresitelloDittmar, Mark" , Jesus Salgado , Mark Cresitello-Dittmar , Francois Bonnarel , Arnold Rots , Jose Enrique Ruiz , Patrick Dowler Subject: Re: ImageDM and ObsCore Hi all, Here are my comments The basic thing to do to understand most of the discripencies between ObsCore, ImageDM (or SpecDM) is to go back to Char2 utype list. Char2 utype list is an evolution of Char1 utype list which has been reworked mainly by Igor, Mireille and me during the past years and which is not yet public. (it will be posted rather soon on IVOA now) I am currently documenting it. It's basically cleaning up Char1 and has inspired most of the ObscOre innovations with respect to Spectrum (Spectrum 1 at that time) The main thing to understand is that Characterisation is reusing STC class instances at the low level. For example Char.(AnyCharAxis).coverage.location.coord designates an AstroCoord instance. AstroCoord because this had to come with a reference to a coordinate system. AstroCoord (or subclasses of it) and AstroCoordArea (or subclasses of it) are the basic classes reused in coverage location, bounds and support. This was first intended to have fully complet STC xml elements in case of xml serialisation of char metadata. But there was another reason to do this : benefit of the semantic power of STC classes and attributes names. You may have a look to the list of char1 utypes if you want. This was used not only for coverage but also for accurate definitions of Errors, PixelSize and Resolution elements. For example STC allows various definition of error bins in 2-d or 3-d spaces: circular, rectangular or ellipsoidal, tilted or not, etc ... The joint file is an upgrade of a work I allready sent Doug and Mark at the end of June. Column 1 is the ImageDM utype list taken from Doug's file. Column 2 is the corresponding ObsCore utype. Column 3 is the specific path of char2 DM, before using terminal stc utypes . Column 4 gives the STC structure reused. Column 5 explains the discripencies and proposes various solutions. Cheers François