ObscCore1dot1RFC < IVOA

---++ ObsCore v1.1: Proposed Recommendation: Request for Comments

This page contains public discussion of the Obscore 1.1 Proposed Recommendation.

%ICON{updated}% Latest version: [[http://wiki.ivoa.net/internal/IVOA/ObscCore1dot1RFC/PR-ObsCore-v1.1-20160923.pdf][Proposed Recommendation ObsCore 1.1- new release ]]
---+++ Reference Interoperable Implementations

Two !ObsTAP services are currently serving !ObsTAP 1.1.
   * CADC: [[http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/][http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/]]
   * !XCatDB: [[http://xcatdb.unistra.fr/3xmmdr5/tap ][http://xcatdb.unistra.fr/3xmmdr5/tap ]] <br />Compare table updates by browsing ivoa.obscore compliant with V1.0 and ivoa.obcore1.1 compliant with V1.1. <p> </p>
The following links provide direct access through !TapHandle.
   * CADC: [[http://saada.unistra.fr/taphandle?url=http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/][http://saada.unistra.fr/taphandle?url=http%3A//www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/ ]]
   * !XCatDB: [[http://saada.unistra.fr/taphandle?url=http://xcatdb.unistra.fr/3xmmdr5/tap/][http://saada.unistra.fr/taphandle?url=http%3A//xcatdb.unistra.fr/3xmmdr5/tap/ ]]
Others applications can be used to connect to these services (e.g. !TopCat)

---+++ Validators

STILTS [[http://www.starlink.ac.uk/stilts/sun256/taplint.html][taplint]] can validate !ObsCore. A version that can validate !ObsCore 1.1 is available at [[ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/stilts.jar]]. Run java -jar stilts.jar taplint tapurl=http://... and look at the results of the OBS stage (you can minimally use stages='CAP TMS OBS' if you don't care about the rest of the validation).

*Note:* At time of writing (18 Apr 2016) the reference implementations listed above *do not pass* this validation.

-- IVOA.MarkTaylor - 2016-04-18

http://xcatdb.unistra.fr/3xmdr5/tap has been updated to pass the Taplint validation.

-- IVOA.LaurentMichel - 2016-04-27

The CADC implementation has been updated but still has two errors from taplint:

1. s_region has no listed unit; in my opinion this is correct since the VOTable field for this is currently datatype="char" arraysize="*" and units are not appropriate. I think this was a bug in !ObsCore-1.0 and !ObsCore-1.1 should be fixed to be more careful with s_region. It also has to be more carefully written to permit use of new ADQL xtypes (polygon) which would have a different datatype and units. As an involved party I will submit some new text to the editor.

2. dataproduct_type "catalog" is in use pending a decision on what new value should be added to the list.

There are also some warnings about other things the CADC TAP service does that are not !ObsCore specific.

-- IVOA.PatrickDowler - 2016-05-18

---++ Comments from the IVOA Community during RFC period: 01 April 2016 - 15 May 2016

In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your Wiki Name so that authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.

---+++ Comments from IVOA.MarkTaylor

There is a problem with the registration of !ObsCore 1.1 services. Section 5 of the PR says this:
<blockquote>Since !ObsCore-1.1 is a superset of 1.0, TAP services that support !ObsCore-1.1 also support !ObsCore-1.0 and should include both 'dataModel' elements, e.g.: <pre>   &lt;dataModel ivo-id="ivo://ivoa.net/std/ObsCore/v1.0"&gt;ObsCore-1.0&lt;/dataModel&gt;
   &lt;dataModel ivo-id="ivo://ivoa.net/std/ObsCore/v1.1"&gt;ObsCore-1.1&lt;/dataModel&gt;
</pre> </blockquote>

But since some of the field metadata has been changed between !ObsCore 1.0 and 1.1 (by my count: 8 mandatory field Utypes, 1 mandatory field UCD and 8 optional field Utypes), !ObsCore 1.1 is not a superset of !ObsCore 1.0. Any service attempting to implement both will have to choose which version of the UCDs and Utypes to use (e.g. is the Utype for dataproduct_type "obscore:Obs.dataProductType" or "obscore:ObsDataset.dataProductType"?), it can't comply with both simultaneously. This anomaly has to be resolved in some way in the text.

<span style="color: mediumblue;"> </span>

_-- Anser by editor_

_This is correct. The text has been changed accordingly and will not allow the two data models tags._ _---- IVOA.MireilleLouys - 2016-06-02_

Also some minor comments on the detail:
   * Sec 1: the paragraph "...an initiative of the IVOA Take Up Committee ... all archived astronomical observations" is included twice in the Introduction. <em><span style="color: mediumblue;" class="WYSIWYG_COLOR">changed</span></em>
   * Figure 1: there are a couple of standards not appearing in the architecture diagram that maybe should be there. &lt;nop&gt;DALI is mentioned in figure 1 caption, and &lt;nop&gt;DatasetDM is mentioned in section 3.3.3. <em><span style="color: mediumblue;">changed</span></em>
   * Sec 4.4: could usefully make an explicit reference to sec 3.3.3. <em><span style="color: mediumblue;">changed</span></em>
   * Sec 6: Some of the FIELDs have @unit="sec", which is not VOUnit compliant, it should be @unit="s". Plus some other incorrect units and UCDs you can discover by running the taplint validator mentioned above on the service. There are also one or two (cut'n'paste?) typos that make this text badly-formed XML ("&lt; ?xml", "/FIELD&gt;", incorrect double-quote character). <span style="color: mediumblue;"> _changed_ </span>
   * Sec A: I don't understand some of the notation in this section, e.g. "AND=SERVICE REQ", "(SERVICE REQ+NEEDS ANOTHER SERVICE (CATALOG)". Did I miss something? <span style="color: mediumblue;" class="WYSIWYG_COLOR"> _text changed to be more explicit_ </span>
   * Sec A.1: "em_max &gt;&gt; 2.48E10" should read "em_max &gt; 2.48E10". <em><span style="color: mediumblue;">changed</span></em>
   * Sec A.1.6: "AND | (t_max + t_min)/2 &ndash; user_date | &lt; 1 d" - is this valid &lt;nop&gt;ADQL? 
      * *PR-ObsCore-v1.1-20160709: still not addressed*
      * <strong><em><span style="color: mediumblue;" class="WYSIWYG_COLOR">Adql parser fails: currently tested on ref implementation for the correct syntax, <strong><em><span style="color: mediumblue;">-- IVOA.MireilleLouys - 2016-08-26</span></em></strong></span></em></strong>
      * <strong><em><span style="color: mediumblue;" class="WYSIWYG_COLOR"><strong><em><span style="color: mediumblue;">changed in the new release using abs() and the defined default units -- IVOA.MireilleLouys - 2016-09-12</span></em></strong></span></em></strong>
   * Table 5: There's a slight inconsistency in the Type column of this table: some enumeration columns are listed with types and others not, e.g. dataproduct_type has "enum string" while data_rights just says "enum". <em><span style="color: mediumblue;">changed</span></em>
   * Sec B.6.3.1: "A format like MJD is useful for easy calculations and preferred for the Observation Core components model.". This makes it sound like use of MJD is optional here, but from sec 4.14 I understand that MJD is mandatory for t_min and t_max. Reword? <span style="color: mediumblue;"> _changed ---- IVOA.MireilleLouys - 2016-06-02_ </span> 
      * *PR-ObsCore-v1.1-20160709: it has been reworded to "these should be stored in MJD format..." which still sounds advisory rather than compulsory. Shouldn't it read "must"?*
      * <strong><em><span style="color: mediumblue;" class="WYSIWYG_COLOR">yes changed, thanks -- IVOA.MireilleLouys - 2016-08-26</span></em></strong>
-- IVOA.MarkTaylor - 2016-04-15

The reference implementations currently do not pass validation (see above). This has to be fixed. -- IVOA.MarkTaylor - 2016-04-18

_Xcatdb reference implementation is fixed (see above) -- IVOA.LaurentMichel - 2016-05-03_

---+++ Comments from IVOA.PatrickDowler

The CADC implementation of !ObsCore-1.1 (referenced above) makes use of a non-standard dataproduct_type value: catalog. I would like to see this value added to the list of allowed values.

-- IVOA.PatrickDowler - 2016-04-15

See author response below

-- IVOA.LaurentMichel - 2016-04-27
---+++ Comments from IVOA.JamesDempsey

The CASDA implementation of !ObsCore v1.0 includes catalogue data products - the ASKAP processing pipeline will produce one or more catalog files for each image/cube. These currently have a blank dataproduct_type in accordance with the !ObsCore spec. We have had numerous questions about this omission in our workshops as there is an expectation that they should have a type value. Thus we would like to see a 'catalog' value added to the allowed types.

-- IVOA.JamesDempsey - 2016-04-15

_We replied to this issue on the mailing list. In conclusion, the catalogue in a broad sense can also be an agregation of tables or a compilation of multiple source lists. On the contrary a source list is well identified as a list of sources detected in one observation that fits better your use case. Thus we propose to add sourcelist as a new dataproduct_type._

-- IVOA.LaurentMichel - 2016-04-27

_A new dataproduct _type has been added for "analysis data product" and should cover this use-case . Updated in last PR document_

<span style="color: mediumblue;">-- IVOA.MireilleLouys - 2016-08-26</span>

<span style="color: mediumblue;"> _we finally opted for dataproduct_type='measurements' to cover the case of extraction sources related to another dataset described in Obscore. the general case for catalogs was not planned and is not covered for this specification._ </span>

<span style="color: mediumblue;"> _-- IVOA.MireilleLouys - 2016-09-23_ </span>
---+++ Comments from IVOA.TomMcGlynn

These comments are mostly on the understandability of the standard. So this isn't particularly associated with 1.1 versus 1.0 and I doubt that you will wish to address these comments in this version but hope it might have some effect on future versions. These are from my personal perspective of looking at the need to implement this standard in the future and trying to see if I can understand what it implies. If I were (and I will be) trying to implement the protocol, then it takes a while to understand where the pieces of this standard that matter are located.

Sections 1 and 2 are the usual filler and can be easily ignored. I think that what I really care about in 3.2, 3.3, 4 and appendix C and these are pretty clear. Part 5 is needed for entry into the registry and I find it confusing, but that's par for the course. I remain deeply baffled by section 3.1. I think it does a brilliant job of obfuscating a simple standard. I remain similarly unclear of what the role Appendix B is. If you want to note that there is something more complex on which this standard is based, do it by reference please rather than importing the complexity into this document.

%BLUE% _Section 3.1 is the UML description for the model, a requirement for the IVOA data models standards. Details on the way parts of a datamodel are re-used need to be explained carefully. Section 4 describes the TAP implementation of the mandatory data model fields only. Section B describes all data model items including optional ones and provides some usage context and examples. This B Appendix can be considered as a Glossary for all data model items. -- IVOA.MireilleLouys - 2016-07-09_ %ENDCOLOR%

I would suggest that section 6 be retained at most as an appendix.<span style="color: blue;"><em> A</em></span>%BLUE%<em>greed, changed </em><span style="color: blue;"> _-- IVOA.MireilleLouys - 2016-07-09_ </span>%ENDCOLOR%

Appendix C should be included in the normative text. Appendix A seems fine (though I don't know why it sometimes gives ADQL translations and sometimes not).

On a slightly more substantive note, I am not clear what role UCDs play? Since we specify column names and utypes for all columns are UCDs given simply for completeness or do they play some actual role in the way users interact with this table? I.e., what breaks if we were to omit or change a UCD for one of these columns?

-- IVOA.TomMcGlynn - 2016-05-10
---+++ Late Comments from IVOA.FrancoisBonnarel

Here are some comments and suggestions for typos correction and the like ...
   * Page 7, list of acronyms. "!ObsTAP TAP interface to Observation Data Model" ----&gt; "!ObsTAP TAP interface to Observation Core components Data Model" <em><span style="color: mediumblue;">changed</span></em>
   * Page 8 "Such information will be prototyped as new features in DAL protocols...." ---&gt; "Such information will be tackled as new features in future versions of DAL protocols...."<em> <span style="color: mediumblue;" class="WYSIWYG_COLOR">changed</span></em>
   * Page 10 "The described datasets can be directly downloaded, or IVOA Data Access Layer (DAL) protocols such as for accessing images (SIA).." ---&gt; "The described datasets can be directly downloaded, or IVOA Data Access Layer (DAL) protocols such as for accessing images (SODA).." <em><span style="color: mediumblue;">changed</span></em>
   * Page 11 Introduction of section 3: there is a conceptual problem here; Observation model referred here (basically = result) is a little different of the concept described in 3.3.3. Maybe we should say there the concept has evolved ? <em><span style="color: mediumblue;">The general Observation discussion is in the scope of the more general Dataset Metadata DM. Sorting out precisely the concept of an Observation as experiment or result is context dependant and will be illustrated later as in the CTA DM for instance.</span></em>
   * Page 12 New efforts on the provenance model may be inconsistent with the Provenance package drawn on Figure 2. If possible this should be made consistent. <em><span style="color: mediumblue;">This reflects the simple ideas for Provenance considered in Obscore and not the full Provenance DM currently under definition <br /></span></em>
   * Page 16 SSA protocol : the author list should give "& al". Tody and Dolensky alone are the editors. <em><span style="color: mediumblue;">changed</span></em>
   * Page 18 ASDM (archive science data model) should be added to the list of achronyms of page 7. maybe with a reference <em><span style="color: mediumblue;">added in acronyms with Fits</span></em>
   * <em><span style="color: mediumblue;"> </span></em>Page 23 "called !DataLinking, currently in development in the DM and DAL Working group" ---&gt; to be replaced by "called !DataLinking (reference to 2015-06-17 spec) <em><span style="color: mediumblue;">changed</span></em>
   * Page 26 4.19 Observable axis description (o_ucd) I think event lists are datasets where "observable" is not applicable so the corresponding field should be NULL. Can we add this ? <em><span style="color: mediumblue;">changed</span></em>
   * Page 30+31 Example : is it really necessary to let the "free" fields ? And the service descriptor for the datalink service after the main resource ? <em><span style="color: mediumblue;" class="WYSIWYG_COLOR">discussion to happen for a better illustration of a serialisation </span></em>
   * Page 33 : Example service : I am not really understanding what is on the main page and how I should use this apart from reading what is in "Protocol & Languages" page extensively. A revision of the introductory page is probably needed (I am ready to help). A possibility to get responses from real services would be well appreciated. <em><span style="color: mediumblue;">Current improvement on the way<br /></span></em>
   * Page 33 to 40 : use cases; AS already stated by Mark TAYLOR. Anything related to service functionalities ora capabilities (LIST = SERVICE REQ) should be removed. When available the SELECT should be given ONLY. <em><span style="color: mediumblue;" class="WYSIWYG_COLOR">changed </span></em>
   * Maybe some use cases expressed with abreviation such as FREQ or STOKES should be rephrased. <em><span style="color: mediumblue;">These are just illustrations of the way a user would express Use-cases, not a predefined language</span><span style="color: mediumblue;"> </span></em>
   * page 44 : data product subtypes. If it is a field with free syntax contrary to dataproduct_type its should be said explicitly. <em><span style="color: mediumblue;">changed </span></em>
   * page 46 : Creation date . Remove the reference to SIAV2.0 This protocol is no more using time stamps explicitly. <em><span style="color: mediumblue;">Done </span></em>
   * page 46 : Publisher Dataset ID . I think "It may differ" has to be replaced by "It is generally different". And the parenthesis has to be removed. The main reason why it is different is that creator and publisher will differ in general. <em><span style="color: mediumblue;">changed </span></em>
   * page 48 : B.6.1.1 Utypes in char are at the moment "Char.SpatialAxis.numBins2.I1" and "Char.SpatialAxis.numBins2.I2" Do you want to change this in Char2 ?
   * page 50 : B;6.2.2 add "corresponding to utype !Char.SpatialAxis.calibrationStatus" after "This attribute of the spectral axis"
   * page 51 : B6.2.4 a) ---&gt; fix "Full Width at half maximum(FWHM)" <span style="color: mediumblue;" class="WYSIWYG_COLOR"> _changed_ </span>
   * page 52 : B.6.3.4 ---&gt; this parameter corresponding to utype "Char.TimeAxis.calibrationStatus"
   * page 52 : B.6.4.1 complete the first paragraph this way: "This field corresponds to utype "Char.ObservableAxis.ucd".
   * page 53 : B.6.4.2 add : "The corresponding utype is !Char.ObservableAxis.calibrationStatus".
<span style="color: mediumblue;"><em>A general comment : </em></span><em><span style="color: mediumblue;">All Utypes are presented in the main data model table, Table 5 beginning of Appendix B. These are eluded in the description of data model elements when the building logic for these strings are easy to derive from the "ClassName.attributeName" pattern. </span></em>

-- IVOA.FrancoisBonnarel 2016-06-01

Two more things from validation discussion with IVOA.PierreLesidaner:

   * In Obscore Table 1 we have "AstroCoordArea" type for the s_region. This is valid because we wanted to have STC-S there. This type is actually not a VOTABLE datattype but an xtype. In further tables we have datatype in tha TAP_SCHEMA to be ADQL:REGION. How does all this fit together. I think we want this field to be reused for selection in UPLOAD context ? <em><span style="color: dodgerblue;">Changed s_region type to 'string' and left the implementation details to the protocol version shipping the data model serialisation , TAP 1.0 or TAP1.1. <br /></span></em>
   * s_resolution has type "float" in table 1 but has datatype ADQL:DOUBLE in other tables.The VOtable datatype is also double in the implementations. <em><span style="color: dodgerblue;">Changed homogeneized all definitions to double -- IVOA.MireilleLouys - 2016-06-10</span></em>
-- IVOA.FrancoisBonnarel 2016-06-03

---+++ Comments from TCG member during the TCG Review Period: 01 April 2016 - 15 May 2016

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or not the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.
---+++ TCG Chair & Vice Chair ( _Matthew Graham_, _Pat Dowler_)
---+++ Applications Working Group ( _Pierre Fernique, Tom Donaldson_ )
---+++ Data Access Layer Working Group ( _François Bonnarel, Marco Molinaro_ )

Valid and useful for DAL WG. I definitely think that the elements number along each axis and the tackling of velocity cubes is a real progress compared with 1.0 from the DAL perspective (ObsTAP and SIAV2.0 dealing with data cubes), as well as some cleaning in the Relationship to DatasetMetadata and Cube data model.

-- IVOA.FrancoisBonnarel 2016-06-01

---+++ Data Model Working Group ( _Mark Cresitello Dittmar, Laurent Michel_ )
---+++ Grid & Web Services Working Group ( _Brian Major, Giuliano Taffoni_ )
---+++ Registry Working Group ( _Markus Demleitner, Theresa Dower_ )
---+++ Semantics Working Group ( _Mireille Louys, Alberto Accomazzi_ )
---+++ Education Interest Group ( _Massimo Ramella, Sudhanshu Barway_ )
---+++ Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick_ )
---+++ Data Curation & Preservation Interest Group ( _Françoise Genova_ )
---+++ Knowledge Discovery in Databases Interest Group ( _Kai Lars Polsterer_ )
---+++ Theory Interest Group ( _Franck Le Petit_ )

&lt;!--<br /> * Set ALLOWTOPICRENAME = IVOA.TWikiAdminGroup<br />--&gt;
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
docx	PR-ObsCore-v1.1-20160923.docx	r2 r1	manage	469.8 K	2016-09-23 - 17:31	MireilleLouys	ObsCore1.1 New release for PR
pdf	PR-ObsCore-v1.1-20160923.pdf	r2 r1	manage	1727.6 K	2016-09-23 - 17:31	MireilleLouys	ObsCore1.1 New release for PR / pdf
pdf	PR-ObsCore-v1.1-201630032016.pdf	r1	manage	839.9 K	2016-04-01 - 11:13	MireilleLouys	proposed rec for Obscore 1.1
This topic: IVOA > WebHome > IvoaDataModel > ObscCore1dot1RFC
Topic revision: r27 - 2016-09-23 - MireilleLouys