MOC11RFC < IVOA

IVOA Web>IvoaApplications>MocInfo>MOC11RFC (2019-10-09, AdaNebot) (raw view)
---+ MOC 1.1 Proposed Recommendation: Request for Comments

<span style="background-color: transparent;">%TOC{depth="2"}%</span>

---

---++ Summary

Latest Draft: <span style="text-decoration: underline;">[[http://www.ivoa.net/documents/MOC/][MOC 1.1]]</span> (Proposed Recommendation)

<span style="background-color: transparent;">The substantive differences between version 1.1 of MOC and the preceding version 1.0 are:</span>

<div title="Page 30">

   * <span style="background-color: transparent;">The String MOC serialization was moved from an informative section </span><span style="background-color: transparent;">(suggested syntax) to the normative section.</span>
   * <span style="background-color: transparent;">A MOCORDER convention for String MOC and JSON MOC was added.</span>
</div>
---++ 
---++ Reference Interoperable Implementations

(Indicate here the links to at least two Reference Interoperable Implementations)

   * [[https://github.com/cds-astro/mocpy][MOCPy &gt;=0.5.6]]
   * [[https://aladin.u-strasbg.fr/java/nph-aladin.pl?frame=downloading#Proto][Aladin (&gt;v10.126)]]: read/write ASCII MOC (select ASCII format when exporting MOC - see snapshot below) + new script command dedicated to ASCII MOC (ex: draw MOC 3/1,2,3 4/56-67)
   * The relational registry at http://dc.g-vo.org/tap parses the ASCII MOCs in Registry records into the rr.stc_spatial table (consumer)
   * DaCHS since version 1.1 produces ASCII MOCs when serialising MOC-values database columns.
   * The [[http://starlink.eao.hawaii.edu/devdocs/sun211.htx/sun211ss141.html][Moc]] class within the [[http://starlink.eao.hawaii.edu/devdocs/sun211.htx/sun211.html][AST]]/[[http://timj.github.io/starlink-pyast/pyast.html][PyAST]] WCS library now supports reading and writing ASCII and JSON MOCs in addition to FITS.
To see non-trivial examples for ASCII MOCs coming out of a database, run something like

<pre>select top 100 * from rr.stc_spatial
where 1=contains(point(10, 10), coverage)
and 0=contains(point(20, 20), coverage)
</pre>

on http://dc.g-vo.org/tap.

---++ 
---++ Implementations Validators

   * This validator is written in Python. It check for valid ASCII MOC syntax and that pixel numbers are in the legal range for thier order: https://volute.g-vo.org/svn/trunk/projects/apps/MOC/
---++ 
---++ Comments Prior to Official RFC Period

<span style="background-color: transparent;"> </span>

As I was considering semantic validation cases, I noticed this sentence in section 2.3.2:
<blockquote>"Warning: In this basic simple ASCII string format the values may be not sorted, and the MOC may be not well-formed."</blockquote>

which seems to contradict section 2.2.3:
<blockquote>" An encoded MOC must also be &ldquo;well-formed&rdquo;. To keep the canonical property of MOC, the redundant cells are not allowed and the hierarchical encoding principle must be respected. Thus it is not allowed to encode 4 sibling cells instead of their parent (only at level 0 for the 12 first diamonds). Using this simple constraint, there is one and only one way to describe a specific region and this makes the interoperability principle easy to implement."</blockquote>

Maybe the former was just a legacy from MOC 1.0 where ASCII MOC was less formal, perhaps intended for hand-crafted MOCs not intended for serious use. With ASCII MOC now being formalized, can we remove that sentence and require that ASCII MOCs must be just as well-formed as their FITS counterparts?

-- IVOA.TomDonaldson - 2019-06-14

_As an ASCII MOC is often written by hand (in a Web form for instance, or as an Aladin script command), it was difficult to insure that it is always well formed at this level. Is it an ASCII MOC, yes, but potentially not well formed (ex: 3/1-210) . So the tools which have to manipulate such ASCII MOCs must check before used. At the opposite, FITS MOC is always generated by a code. So there is a real interest to force it to be stored well-formed for accelerating the future reloads.<br />That said, you right that it is a little bit inhomogeneous, and the ASCII MOCs will be used in other contexts that just for forms and scripts. So we can considere that an ASCII MOC written by hand is not - strickly speaking an ASCII MOC (not serialized in a file) - and in this consideration - extend the well-formed constraint to all serialized MOC formats (FITS & ASCII) by just removing the sentence._

-- IVOA.PierreFernique - 2019-06-18<br /><br />Tom answer 2019-06-21<br />That would be great. Thank you for doing that.
---
---
---

---++ Comments from the IVOA Community during RFC/TCG review period: 2019-06-14 - 2019-07-29

The comments from the TCG members during the RFC/TCG review should be included in the next section.

In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your Wiki Name so that authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.

Additional discussion about any of the comments or responses can be conducted on the WG mailing list. However, please be sure to enter your initial comments here for full consideration in any future revisions of this document

   * Comment from Carlo Zwölf (CarloMariaZwoelf) : It is not clear for me in this proposed recommendation what comes natively from the Healpix definition and what is the IVOA-specific layer. Is there a way for highlighting this difference in the text? <br /><br /><em>This new document is a very minor modification of the MOC 1.0 REC to standardize the ASCII MOC so that it can be used as quickly as possible in other IVOA standards, including VOregistry. The original text was not modified voluntarily in order to quickly reach a consensus. As your suggestion may imply a significant refactoring of the document, I propose to keep this point in mind for the next major release. It is clear that the introduction of the temporal MOC and the space-time MOC will involve a deep change in the document, especially in the direction you have suggested.<br />-- IVOA.PierreFernique - 2019-09-03</em>

   * Sample comment by TWiki.WikiName 
      * Response (by TWiki.WikiName)

---
---

---++ Comments from TCG member during the RFC/TCG Review Period: 2019-06-14 - 2019-07-29

WG chairs or vice chairs must read the Document, provide comments if any (including on topics not directly linked to the Group matters) or indicate that they have no comment.

IG chairs or vice chairs are also encouraged to do the same, althought their inputs are not compulsory.

---+++ TCG Chair & Vice Chair

---+++ [[IvoaApplications][Applications Working Group]]

---+++ [[IvoaDAL][Data Access Layer Working Group]]

MOC specification changes to bring String serialization to the normative part looks a good addition to this standard. The document itself is fine in general. I have two small things I'd like to point out.
   * letting [space,CR,LF,comma], multiples/mixes of them, as the elements separator has a human readability effect (see [[http://mail.ivoa.net/pipermail/apps/2019-June/001408.html][email]]), not so alerting, but also requires some code in MOC usage. Isn't it really possible to force a bit this serialization to be more smooth with, at least, single and homogeneous separators throughout a single ASCII MOC? Could it be at least warmly suggested? <br /><br /><em>The flexibility of the ASCII MOC has been driven by three main pragmatic reasons: <br /> 1- this syntax was exactly the suggested syntax in MOC1.0. <br /> =&gt; consequently there are already existing ASCII MOCs following this syntax,<br /> =&gt; It was easier to keep the syntax as it was suggested in MOC 1.0<br /> 2- an ASCII MOC is often written by hand (in a Web form for instance, or as an Aladin script command), <br /> =&gt; a certain level of tolerance may be useful to avoid rejecting MOC with double space separators...<br /> 3- an ASCII MOC can be a very very long string line<br /> =&gt; As for BASE64 ASCII streaming, it may be really problematic to read/edit an ASCII MOC in a regular ASCII editor without the CR/LF separator and multiple space for possible indentation. Also, we will have serious potential problems for reading and writting a so long line due to the classical flush mecanism of ASCII libs.<br /> <br />The two first points are managable even if we really want to reduce the flexibility of the ASCII MOC syntax. Concretely the MOC clients/libs will have to be smart enough to support proto ASCII MOCs and possible separator typos (same argument that Tom's point). Quite reasonnable in fact. But I'm strongly in favor to take into account the 3nd point and keep the possibility to split an ASCII MOC if required<br /><br />My conclusion would be to remove the "comma", and let space,CR,LF possibly multiple<br /> -- IVOA.PierreFernique - 2019-09-03<br /><br /></em>
   * MOCORDER is a MUST and clearly described in the FITS serialization, while in the String (and JSON) one(s) it is less clear how to grab that piece of information. This because it is only said to be a must "If the best HEALPix resolution of the MOC (MOCORDER) is greater than the greatest stored order" (in which case it would be the last token of the String, followed by a slash and no npix). I agree that if it's not so it's implicit the MOCORDER is the last/highest order available, but I'd prefer it to be explicit. <em><br /><br />You are right. It should be a "MUST" without ambiguity.<br />The current sentence for ASCII MOC (section 2.3.2) is : "If the best HEALPix resolution of the MOC (MOCORDER) is greater than the greatest stored order, the MOCORDER *must* be provided, followed by a slash ("/") without any associated npix value."</em><em><br />I suggest this new sentence: "The best HEALPix resolution of the MOC (MOCORDER) MUST be always provided, even if there is no explicit npix value at this best resolution (due to the MOC hierarchization process). In the latter case, the MOCORDER is specified followed by a slash ("/") without any associated npix value."<br />Concerning JSON serialization (suggested syntax in a non normative section). Here the proposed new sentence : "In the same way as for ASCII MOC serialization, if the best HEALPix resolution of the MOC (MOCORDER) is greater than the greatest stored order, the MOCORDER should be provided with an empty npix list."<br />-- IVOA.PierreFernique - 2019-09-03</em>
-- IVOA.MarcoMolinaro - 2019-07-11

---+++ [[IvoaDataModel][Data Model Working Group]]
   * I support the Marco's comment about MOCORDER in string. <br /><em><br />See below -- IVOA.PierreFernique - 2019-09-03</em>

   * It would be nice if figures had a caption and a number used to reference them. <br /><br /><em>Good point. Concretely the few figures in this document are already fully described in the main text and the insertion of captions introduces not very well come repetitions (see section 1.3 for instance) without modifying deeply the text. As presented, this document would like to be as light as possible modification of the MOC 1.0 standard. So, I would propose to keep in mind this suggestion for the next major release of MOC. -- IVOA.PierreFernique - 2019-09-03</em>
-- IVOA.LaurentMichel - 2019-07-25
---+++ [[IvoaGridAndWebServices][Grid & Web Services Working Group]]

I support Marco suggestions and at this stage I have no further comments. Please add caption to figures and tables and reference them in the text.

-- IVOA.GiulianoTaffoni - 2019-09-11
---+++ [[IvoaResReg][Registry Working Group]]

I would like to echo Marco's comments above, regarding simplifying the separators in serialization, and perhaps go a bit further. At best, there should be only one way to do this. I am willing to settle for consistency within a MOC using any one of those, no mix and match, with a preferred character separator suggested. -- IVOA.TheresaDower - 2019-07-24 <br /><br /><em>See below -- IVOA.PierreFernique - 2019-09-03</em>
---+++ [[IvoaSemantics][Semantics Working Group]]

The Semantics WG doesn't see anything concerning us in the proposed changes (that otherwise sound reasonable and useful to us). The spurious indent of the rule for moc in the EBNF grammar could be removed, though.

-- IVOA.MarkusDemleitner - 2019-07-16

---+++ [[IvoaDCP][Data Curation & Preservation Interest Group]]

---+++ [[IvoaTheory][Education Interest Group]]

---+++ [[IvoaKDD][Knowledge Discovery Interest Group]]

---+++ [[IvoaSS][Solar System Interest Group]]

---+++ [[IvoaTheory][Theory Interest Group]]

---+++ [[IvoaVOEvent][Time Domain Interest Group]]

---+++ [[IvoaOps][Operations]]

Mostly OK, but some comments:
   * Table of contents: the page numbers are mostly wrong. <br /><br /><em>Right -&gt; and fixed -- IVOA.PierreFernique - 2019-09-03</em>

   * Sec 1.4: The existence of a FITS HEALPix serialization format on which the MOC serialization format is based is claimed, but no reference is given. Since this text (written by Eric Hivon) does now just about exist in a public form [[https://healpix.sourceforge.io/data/examples/healpix_fits_specs.pdf]], can it be referenced here?
   * Sec 2.3.1: The HEALPix FITS encoding standard is mentioned here as well: _"As the FITS MOC is derived from a more generic standard dedicated to HEALPix maps, the FITS header must contain the following keywords/value."_ That's a bit misleading, since the MOC serialization is not a special case of the HEALPix encoding (it doesn't contain some of the keywords required by that standard, such as INDEXSCHM and NSIDE) and some of the keywords have values which are not allowed in the HEALPix encoding (ORDERING='NUNIQ'). Given the difference between the two encodings, I'm not sure there's much point making the comparison here, but if so I'd suggest something more like "The following keywords are mandatory and derive from the HEALPix MOC encoding [with a document reference]." <br /><br /><em>I agree. The FITS HEALPix map serialization was and is unfortunately not a standard in the sense of IVOA REC. HEALPix authors have deployed libs and tools which follow an evolving convention. The recent Eric Hivon document has clarified this situation. Some FITS keywords which was required in previous conventions have not been kept, and some other have been introduced. So I totally agree Mark's point and I will take the suggestion "The following keywords are mandatory and derive from the HEALPix MOC encoding [with a document reference]."<br />-- IVOA.PierreFernique - 2019-09-03<br /><br /></em>
   * Sec 2.3.1: The MOCID and ORIGIN keys both _"suggest using IVOA identification like ..."_ where the examples are of the form =ivo://...= . If these are intended to be IVOA Identifiers, it should say something more like _"suggest using an IVOA Identifier ..."_ with a reference to the [[http://www.ivoa.net/documents/IVOAIdentifiers/index.html][IVOA Identifiers document]], and the examples given should be valid IVOIDs (i.e. should resolve in the registry - I don't think that "CADC" is an authority in the IVOID sense). If not, the =ivo://= prefix seems a bit misleading. Reg WG may want to comment on this. <br /><em>Right -&gt; Modified following the suggestion =&gt; MOCID: Specify an identifier for the original data used to build the MOC. If the original data set has been declared in the VO registry [11], we strongly encourage to use the IVOA identifier (ex: &ldquo;ivo://wfau.roe.ac.uk/twomass-dsa&rdquo;')" and "ORIGIN: This keyword could be use to store the name of the MOC origin. If the authority of the origin has been declared in the VO registry, we strongly encourage to use the IVOA identifier (ex: &ldquo;ivo://cadc.nrc.ca&rdquo;)." -- IVOA.PierreFernique - 2019-09-03<br /><br /></em>
   * Sec 2.3.2: _"hightnpix"_ -&gt; _"highnpix"_ ? <br /><em>ok -&gt; IVOA.PierreFernique - 2019-09-03<br /><br /></em>
   * Sec 2.3.2: I'm inclined to agree with Tom's and Marco's comments about the form of the ASCII MOC serialization - it's not clear to me why FITS MOCs have to be well-formed and ASCII ones don't, and why so many separator choices are allowed. Also, the BNF doesn't constrain where the optional MOCORDER marker has to go, or how many of these appear. Consider constraining this format a bit further, e.g. require the MOCORDER marker to be present and/or at the end (or start?) of the string? <br /><br /><em>See above -- IVOA.PierreFernique - 2019-09-03<br /><br /></em>
   * Appendix B: most of the decimal separators are "." characters, but one is a "," ("1,17").<em><br />Ok -&gt; IVOA.PierreFernique - 2019-09-03</em>
-- IVOA.MarkTaylor - 2019-07-13

---+++ [[IvoaStdsDocsProc][Standards and Processes Committee]]

---

---++ TCG Vote: 2019-06-14 - 2019-07-29

If you have minor comments (typos) on the last version of the document please indicate it in the Comments column of the table and post them in the TCG comments section above with the date.

| Group | Yes | No | Abstain | Comments |
| TCG | * | | | |
| Apps | * | | | |
| DAL |  *  | | | -- IVOA.MarcoMolinaro / IVOA.JamesDempsey 2019-07-17 |
| DM | * | | | -- IVOA.LaurentMichel - 2019-09-26 |
| GWS | * | | | |
| Registry | * | | | |
| Semantics | * | | | |
| DCP | | | | |
| KDIG | | | | |
| SSIG | * | | | |
| Theory | | | | |
| TDIG | * | | | -- IVOA.AdaNebot - 2019-10-09 |
| Ops | * | | | |
| <nop>StdProc | | | | |

---

<br /> <!--</p> <ul> <li> <ul> <li> Set ALLOWTOPICRENAME =<span class="WYSIWYG_PROTECTED"> IVOA.TWikiAdminGroup</span> </li> </ul></li> </ul>-->
   * MOC_ASCII_in_Aladin.png: <br /> <img width="546" alt="MOC_ASCII_in_Aladin.png" src="%ATTACHURLPATH%/MOC_ASCII_in_Aladin.png" height="158" />
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
png	MOC_ASCII_in_Aladin.png	r1	manage	9.3 K	2019-05-17 - 08:32	PierreFernique
Topic revision: r25 - 2019-10-09 - AdaNebot
IVOA
Log in or Register
IVOA.net
Wiki Home
WebChanges
WebTopicList
WebStatistics
Twiki Meta & Help
IVOA
Know
Main
Sandbox
TWiki
TWiki intro
TWiki tutorial
User registration
Notify me
Working Groups
Interest Groups
Time Domain
Committees
Stds&Procs
www.ivoa.net
Documents
Events
Members
XML Schema