HiPS 1.0 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA HiPS 1.0 Proposed Recommendation.

The latest version of the HiPS specification can be found at:

• http://www.ivoa.net/documents/HiPS/index.html

Reference Interoperable Implementations

• HiPS clients:
  • Aladin Desktop / CDS / java
  • Aladin Lite / CDS / java script (+derived tools ESAsky/ESAC, Judo2/JAXA, Skymap Viewer/Ligo-Virgo, etc)
  • MIZAR / CNES / WebGL
  • Mast prototype portal / STScI / java script
• HiPS servers => http://aladin.unistra.fr/hips/list
  • 12 servers distributing 350 HiPS
• HiPS generators => doc
  • Aladin-Hipsgen / CDS / java => Image HiPS and Cube HiPS generator
  • Hipsgen-cat / CDS / java => catalog HiPS generator

Validators

• Hipslint / CDS / java => Hips server validator & Hips validator

Comments from the IVOA Community and TCG members during RFC period: 2016-12-15 - 2017-01-26

Comments from Markus Demleitner

p. 16, "Each line is ended by CR..." Are you sure you want what this is saying? The only platform I'm aware of that ever used single CRs as line separators was the classic Mac, and AFAIK with OS X they migrated to having just LF as a line separator as $DEITY had originally intended. So, essentially what you're specifying here will make everyone equally unhappy. Note that the IETF somewhere requires anything with a text/ major media type to use CRLF as line separator. So, much as I'd like to see single LFs here, I guess you should just require CRLF as line separator throughout and be done with it. (Incidentally, on p.22, there's mention of "property1\nkey1"; \n is the conventional escape for LF rather than CR, so whatever you do, this would need reconciliation).

=> It was a mistake. The document has been fixed following the usual ASCII convention end of line. -- PierreFernique - 2017-02-07

p. 17, point 3. First, there's "image His" rather than "image HiPS". Then -- Obscore 1.1 now has "measurements" as dataproduct_type instead of "catalog". Everyone will be grateful if we keep terminology uniform between standards, and given that ObsCore was there first, I guess HiPS would have to bite the bullet and change.

=> The recent discussion/evolution in the ObsCore vocabulary for replacing the "catalog" keyword for "measurements" keyword was ignored by the HiPS editor. The adjustement of this keyword in HiPS would imply a big impact on already existing catalog HiPS distribution and compatible clients => JAXA, Leiden and CDS are already distributing catalog HiPS and the dataproduct_type=catalog is notably used by the clients for discriminating the type of the HiPS and not only for providing user information. Thus, if we modify this keyword, we will have to manage a transition period... always problematic. Additionnally, the term "catalog" in the HiPS context is really what it is. And the replacement with "measurements" appears quite "artificial" - again in the HiPS context. These arguments has been transmitted to the Obscore authors/contributors. It is still in discussion in the Obscore team. -- PierreFernique - 2017-02-07

p. 18, point 6. What should catalog HiPSes put into hips_file_format?

=> "tsv" possibiliity has been added. -- PierreFernique - 2017-02-07

p. 18, point 8: hips_frame. For equatorial, you should mention in the specification that you're assuming the ICRS reference frame (you do, don't you?).

=> ICRS has been explicitely specified. -- PierreFernique - 2017-02-07

p. 18, table header: I suppose "Rec or Should" should be "Req or Should", no? Linguistically more consistent would be "Must or Should", but since you'd have to change all the letters in the table then I'd say it's not worth the effort.

=> Fixed (Rec -> Req) -- PierreFernique - 2017-02-07

p. 18, "obs_description". You're saying this contains "one paragraph", which people would probably take to mean "a couple of (perhaps physical) lines", which of course is outlawed by the basic metadata format. To reduce the risk of confusion, I'd suggest writing something like "longer free text description of the dataset"

=> The proposal has been incorporated in the document. -- PierreFernique - 2017-02-07

p. 20, "Catalogue HiPS metadata": Are you sure we can't just require metadata.xml? If I were to write a client and all I had to work with are the column labels from the TSV, I'd be cross with you.

=> The goal of the HiPS authors/contributors was to keep the generation of catalog HiPS as simple as possible. In this perscpective, the tile TSV header is trivial to implement. The metadata.xml is optional, and can be used/added only when/if it is required. Complementary, keeping a TSV header in each tile helps a lot for debugging/controlling individual tile content if required. -- PierreFernique - 2017-02-07

Also, two typos:

p. 12 "The mean method is appropriated": -> appropriate

=> Fixed. -- PierreFernique - 2017-02-07

p. 19, hips_order: Deeper -> Deepest (or perhaps Highest?)

=> Fixed. -- PierreFernique - 2017-02-07

-- MarkusDemleitner - 2016-12-15

Comments from TCG members during the TCG Review Period: 2016-12-15 - 2017-01-26

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Approved -- PierreFernique - 2017-04-07

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

- We approve this document which is describing and standardizing in detail a very innovative and already successful data management system. Some recommendations to the authors

- A specification source file (doc or latex) will be appreciated

=> Doc. -- PierreFernique - 2017-04-03

- Text should be justified for better reading quality.

=> Done -- PierreFernique - 2017-04-03

- From a DAL point of view it's now time to make classical DAL protocol and HiPS discovery and access functionality reach some convergence. See DAL future page for that, for future versions of HiPS.

- Detailed list of typos or editing remarks:

page 3 : "of the original data into account,and" --> introduce space

page 5 : "Each cell at a given level is covers...." ---> suppress "is".

page 6 : "a collection HiPS tiles stored in regular files inside...." ---> a collection of HiPS tiles.

page 6 : Paragraph with "These three aspects ..." and figure 1 : "Hips Server" which contains "Hips servers", "HipS data", "Hips" list.

=> Fixed -- PierreFernique - 2017-04-03

The sentence and figure will be more consistent and the whole clearer with the "HipS server" rectangle text replaced by "HipS distribution" and the "Registry" rectangle text replaced by "registration"

=> Good suggestion. Done -- PierreFernique - 2017-04-03

page 10 : In the paragraph "To avoid tiles containing ...." we don't understand "packaging the 2sx2s HEALPIX cell values of the S sub-orders". Isn't that "k+S" order with k being the order of the considered tile ?

Same remark in the example.

=> After discussion, we change a little bit the sentence to be clearer. -- PierreFernique - 2017-04-03

page 13 : "A HiPS may be delivered" ----> an image HiPS

page 15 : "two enablers may be implemented for the 4 lower orders [0..3]" --> this is not totally right according to what follows (order omission stops at 2). And "These are" can be suppressed this way

"two enablers may be implemented for the lower orders : order omission and Allsky preview."

=> Ok. Done -- PierreFernique - 2017-04-03

page 15 : "The method to generate the Allsky file depends to the nature" ---> depends on the nature.

=> Fixed -- PierreFernique - 2017-04-03

page 16 : (typically 64x64 pixels rather than 512x512) ---> what are these pixels ?

=> The description is provided in the document "The tiles at low orders (0 to 3) may be packaged together into a unique file called Allsky... To avoid having a too large Allsky file, the resolution of each tile may be reduced but must stay a power of two (typically 64x64 pixels rather than 512x512) -- PierreFernique - 2017-04-03

page 16 "the same rule that regular cube frame tiles" ---> the same rule than the regular cube frame tiles one

=> Ok. Done -- PierreFernique - 2017-04-03

page 18 : "The list below is not exhaustive and new keywords may be added if required". How will this be done ? List managed by "semantics" ? List stored on the ivoa page ?

=> This is a good question. Our goal is to let very open the possibility to create new keywords if required - more or less as "a la FITS". With the good and the bad of such a principle. If required we could imagine to provide a list of new introduced keywords on the IOA site for avoiding duplications. -- PierreFernique - 2017-04-03

page 19 : "dataproduct-type" . From the text We don't understans what 'meta" is. Same thing for hips_progenitor_url.

=> meta is used for HiPS prototype progenitor extension not described at all in this document. There is effectively no reason to mention it here. Removed. -- PierreFernique - 2017-04-03

page 20 : "t_min, t_max" - > MJD (like JD, ISO, etc...) is not only a unit. It's a "representation" of time.

=> Right. Done. -- PierreFernique - 2017-04-03

page 29 : "a Allsky.xxxfile" ---> "A Allsky.xxx file"

=> Fixed -- PierreFernique - 2017-04-03

page 29 and 30 : instead of (intermediate step between 2 and 3 in the previous algorithm) in parenthesis. Add this to the subtitle this way

"Moc.fits usage improvment by adding an intermediate step 2 and 3 in the previous algorithm"

=> Good suggestion. Done -- PierreFernique - 2017-04-03

-- FrancoisBonnarel - 2017-03-28

-- MarcoMolinaro - 2017-03-28

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

(1) Sentence "The more you ..." starts in italic and ends in plain text

=> Fixed. -- PierreFernique - 2017-02-07

(2) "Going beyong...": This sentence suggests that HIPs can be used even for non spatial data. Must be clarified

=> Right. New formulation: "Going beyond its emphasis on visualisation, HiPS can also be described as a generalised spatial mapping of data into a common architecture..." -- PierreFernique - 2017-02-07

(3) P5 "... given level is covers an .."
- The HEALPix tile defintion vs HEALPix cell is not clear (to me at least). After reading back and forward, I understand that one tile contains data of one HEALPix cell at an order ranging between 0 and a deepest order which must be lower that the HEALPix order of the resampled data. This key concept, very important to understand the following, could be a bit more developed

=> Difficult to introduce immediately in this section the specificity of the Images HiPS. Here we described the global mechanism of HiPS for any type: catalog, image, cube .... The fact that a HiPS tile for images and cubes contains several sub HEALPix cells is described in the dedicated section 4.2.1 -- PierreFernique - 2017-02-07

(4.2.1) It is first said that HIPs tiles must contain "orginial pixels" and after that "original pixels are resampled": not consistant. Better to say that " HIPs tiles must contain orginal pixels resamples inthe the HEALPix geometry".

=> Good point. we just removed the "original" word in the first sentence. The fact that these pixels are original (for instance in the case of native HEALPix data such as PLANCK or WMAP), or resampled are discussed in the end of the paragraph. -- PierreFernique - 2017-02-07

- "... tile hierarchy is S orders" must or should?
- The subsection "To avoid tiles ...." actually introduces the concept of deepest order. Would be nice to mention it before to enter 4.2.1.1
- 4.2.1 said that tiles contain resampled original data and 4.2.1.1 said the pixels are directly computed from the orginal data: I do not see the distinction between both
- 4.2.1.2: "Each pixel..." better to talk about cells nistead of pixels are in fig5

=> See the previous remark about the image HiPS specificities. -- PierreFernique - 2017-02-07

(4.2.2.1) Same remark as Markus: what about string with characters not encoded in 7Bits?

=> Good suggestion. As UTF-8 is a superset of 7-bit ASCII, there is no good reason to restrict HiPS to 7bits. The document has been updated in this direction. -- PierreFernique - 2017-02-07

(4.2.3) If you put 10,000 cube tiles with 1024 frames each within a directory, you will have ~ 10,000,000 files in that directory. This breaks the reason for setting that 10,000 limit. In a future version the limit on files per diresctory could be adapted to each HIPs.

=> After some mails between HiPS authors and contributors, we agreed with Laurent Michel to wait a future version of HiPS standard for managing this specifical potential cube HiPS limitation (on idea would be to define a dedicated property keyword modifying the default 10,000 tiles packaging). -- PierreFernique - 2017-02-07

(4.3.2) The type of the allsky file may be precised.
- The way image tiles are arranged in allSkys files is not clear. Is that a mosaic, a multi extension FITS file?

=> After discussion with Laurent Michel, the current paragraph is ok. The figure illustrating this section has been enlarge a little bit to better show the mosaic packaging. -- PierreFernique - 2017-02-07

(4.4.3) What is the "FITs header used for global meta data information"?

=> Most of surveys distributed as FITS image collection uses the same FITS header keywords for describing globally the survey (mission, instruments, observatory localisation...). So providing one header is a very simple method to provide this generic information. -- PierreFernique - 2017-02-07

(5.1) "...only the view as directory..." already specified in 4.1

=> Right. But a repetition in this section, dedicated to the distribution, seems appropriate. -- PierreFernique - 2017-02-07

(5.2) "... any HIPs server via a dedicated URL": How to discover that URL, through the registry? must be specified since the global HIPs design tends to be self consistent: just take the root URL and go.

=> That's a good point. Any HiPS list aggregator is free to be declared or not in the VO registry. Typically the MOCserver (CDS) which are doing this task will be declare in the VO registry. -- PierreFernique - 2017-02-07

General remarks

+ Both implementers and consumers will appreciate (among other things) how the registration is detailed (5) and the cookbook section (6)
- The concept of tiles/pixels/cells/order/nside could be a bit better explained before the normative section. (lexicon in a table?)

-- LaurentMichel (and MarkCresitelloDittmar) - 2016-12-16

Approved

-- LaurentMichel - 2017-04-05

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

In general, I found this to be a very readable and clear document. I have a couple of comments from the GWS perspective:

5.1 HiPS Server - Since the HiPS server is essentially an IVOA web service, I think it should adopt the mechanisms laid out in the VOSI (VO Support Interfaces) specification. In particular, the /availability and /capabilities endpoints would be useful.

The /availability endpoint would help clients decide whether the server is ready for use. If not, it could switch to another implementation from the registry.

The /capabilities would provide a means for clients to dynamically discover the location of the HiPS list after finding it in the registry. The HiPS list should be identified by an IVOA standard ID. This might have some implications to the wording in the section describing the registration of HiPS implementations.

=> After discussion with Brian Major, we decided to let the HiPS document at is it for HiPS 1.0, and to study the VOSI usage probably in a larger context than just HiPS for future version (considering HiPS access as one possible "interface" for accessing a dataset collection, one amongs others. For instance, we can imagine to have a HiPS capability and a CS capability for the same catalog. Or a SIA interface and a HiPS interface for an image collection - also to manage possible multi HiPS interfaces) -- PierreFernique - 2017-02-09

It would be nice to see an example HTTP URL to a tile, perhaps repeating the example earlier in the document.

=> Good idea. The generic example provided in section 4.1 has been re-used in section 6.1 and 6.2 for illustrating a local access to a HiPS tile and an HTTP remote access to the same HiPS tile. -- PierreFernique - 2017-02-09

Should the HiPS server set HTTP headers for tile requests to help the client? For example, content-type: image/png

=> As a HiPS tile is (usually) a regular file, the content-type is automatically set by the apache server. image/png for PNG tiles, image/fits for FITS tiles... -- PierreFernique - 2017-02-09

Proprietary Data

This may only be a concern for future HiPS implementations, but I think some forethought regarding access to proprietary astronomical surveys should be given. Namely, there should be a way for clients to authenticate. If the /capabilities metadata of a HiPS server is available then there would be a standard mechanism for determining the supported authentication methods when it comes to proprietary HiPS lists.

There may be cases where only certain tiles at a certain order or sky location are proprietary. I wonder if there should be space reserved in the tile metadata for authorized read permission information. This field would likely specify the standardID of a group that is allowed to read the tile.

=> Any possible authentification or encryption mechanism can be applied just by using the HTTP or HTTPS support. Concretly, today there are already some HiPS with restricted access (basic HTTP authentification, or more complex system). And we can already limit the access partially quite easily (for instance, we can ask authentification for tiles located in NorderNNN directory for NNN greater than... : all users can see the HiPS at low resolutions, but only authorized people can access to the high resolutions.). We mentioned this point in the 6.2 section. -- PierreFernique - 2017-02-09

-- BrianMajor - 2017-02-04

I approve this PR.

-- BrianMajor - 2017-02-27

Registry Working Group ( _Markus Demleitner, Theresa Dower )

(1) 4.4.2.1 Is there a terribly strong reason to require 7-bit ASCII in the TSVs? This would make Registry's life quite a bit harder when we produce a HiPS of the Registry (once we have enough coverages), as our authors and titles can potentially contain all kinds of odd characters. My take is: Just allow UTF-8. There's few platforms where that's a problem these days, and UTF-8 doesn't clobber Tabs, so there's probably little that can go severly wrong.

=> We moved to UTF-8 (cf Laurent Michel comment and response below) -- PierreFernique - 2017-02-09

(2) While I like the term obs_regime in the table on p. 18, I have to mention that the same concept, with essentially the same terms (but different spelling -- ugh) is called waveband within VODataService's vs:Coverage element. Now, the different concept names probably are not really dramatic, but the differences in the terms contained are fairly painful, and that pain will only increase as we may extend the list with neutrinos, gravitational waves, or whatever. So, here's what the VODataService creators came up with about ten years ago:

"Radio" | "Millimeter" | "Infrared" | "Optical" | "UV" | "EUV" | "X-ray" | "Gamma-ray"

Couldn't you just adopt this list or even better, defer to VODataService's vs:Waveband type so future additions will just pull through? I realise it's a pain given the installed base, but perhaps clients can contain a mapping for a while to ease transition?

=> Good point. We just differed by "Millimeter" and "EUV" additions, and uppercase first letter. The document has been fixed to follow the VODataService list. -- PierreFernique - 2017-02-09

(3) Thanks for making the implemented global HiPS list something of an epiphenomenon of a proper registration -- much appreciated. Since, on the other had, I suspect that all actual clients go through that global list exclusively, I think either sect. 6 should say somewhere that that service exists and how to use it (probably in a non-normative section) or there should be an appendix discussing that. Except, of course, if you actually have plans to phase out that service. But if, as I suspect, it is what clients are expected to do for the forseeable future, it should IMHO be documented in this standard.

=> Difficult to provide a potentially non permanent URL in such a document. But as you suggested, we added as a page note the URL to the CDS hips list aggregator -- PierreFernique - 2017-02-09

-- MarkusDemleitner - 2016-12-15

Given the changes during PR, I'm happy to approve.

-- MarkusDemleitner - 2017-04-05

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

The document is well detailed and describes precisely the features needed to understand the Hierarchical Progressive Survey data structure for distribution and client access.

Here is a short list of minor typos noticed:

p11 table has caption but no number. This would be easier for reference.

=> Done -- PierreFernique - 2017-04-03

p15 To ease the client drawing process, two following two “enablers” may be implemented for the first 4 lower orders [0..3]. These are order omission, and
Allsky preview. what does it mean?

Is the following more appropriate?

"two strategies are proposed to ease the client drawing process: first low order ommission , and second building up the "Allsky preview" file. ", etc...

=> Fixed. see DAL comments -- PierreFernique - 2017-04-03

p18-19
The "keywords" table has no number/ no caption . That would be useful for reference in other parts of the text too.

=> Done -- PierreFernique - 2017-04-03

hips_order / hips_order_min
this is not symetric: should not we have hips_order_max / hips_order_min ?

in the definition : replace "deepest" HiPS order by "highest" which is really opposite to "lowest"

example definition in the table could be :
hips_order_max R Highest HiPS order (corresponding to the finest scale) – Format: positive integer

hips-order_min then corresponds to the coarsest scale, and the deepest you browse with the viewer, the highest order maps appear on the screen.

=> totally true, but really very impacting in case of modification on all already existing HiPS collections and tools (hips_order is a key parameter). hips_order_min has been introduced in the study of HiPS progenitors not described in this document. It is preferable to simply remove hips_order_min. With this modification the "deepest" term already used in the document make sense. -- PierreFernique - 2017-04-03

data_cube_crpix3 Coef for computing physical canal value. Should not it read "channel" instead of "canal"?

=> Fixed -- PierreFernique - 2017-04-03

p 31, last line before References
"metadata" is used in the VO instead of "meta data".

=> Fixed -- PierreFernique - 2017-04-03

Appendix p32

"and {.ext} is depending of the /nature/ of surveys (.fits, .jpg, .png …)"

I would prefer /file format/ of the surveys (...)

=> replaced by "... the file format of the tiles..." -- PierreFernique - 2017-04-03

The functions mentionned are part of a library I suppose: where are they defined?.
A simple reference to this library could be added here.

=> Done -- PierreFernique - 2017-04-03

-- MireilleLouys - 2017-03-29

I approve the document in the last version produced by April 6, 2017 and hope for wide adoption of this specification.

-- MireilleLouys - 2017-04-07

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Dave Morris )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

The document looks basically OK, but I have some points that should be clarified. These comments concern the version PR-HiPS-20170207.

• Sec 4.2.2.1: "Columns providing the location on the sky of each source MUST be present." That's reasonable, but it doesn't prescribe how to find such columns. Sec 4.4.3 says that the catalogue metadata file is optional. If there is no such metadata file, then presumably one has to guess from the column names in one of the catalogue files which columns are sky positions. In that case there should be some rules provided for how to do that. If on the other hand the metadata.xml file really is mandatory in this case, that should be stated, and there should be rules about how to identify sky positions: e.g. are columns with UCD=pos.eq.{ra,dec};meta.main mandatory?

• Sec 4.4.1:
  • ".. or spaces (blank, TSV, ...)" - should "TSV" here read "TAB"?
  • "possible spaces between the keyword and the character "=" MUST be ignored" - presumably the same applies to spaces between the equals sign and the value text too.

• • dataproduct_subtype description: I don't understand the explanation of this keyword's content. Does "color" just mean that it's using PNG/JPEG tiles (in which case you can work it out from the hips_tile_format keyword), or does it mean something else, if so what?

• • "some keywords are not exhaustive and may be completed if required" - would "extended" be better than "completed"?

• Keyword table in sec 4.4.1:
  • hips_status keyword is marked R[equired], but not mentioned as one of the 8+2 Mandatory keywords in the preceding text

• • hips_reg_blue should read hips_rgb_blue

• • hips_rgb_* keywords format " creator_did [cutLow cutMiddle cutHight TransferFunction] " seems a bit underexplained. What's the creator_did doing here? What are the options for the transfer function? what do you do with the low/middle/high values?

• • hips_data_range - I don't understand what is a "default data range"?

• • hips_pixel_scale - if I understand correctly this is redundant with hips_order - is there any point in defining this value?

• • addendum_did - what does "(must be multiple)" mean? If it means that it has to be specified multiple times, shouldn't it be marked with "(*)"?

• Sec 5.2: "a HiPS list MAY be easily generated as the concatenation of the properties files of all distributed HiPS, with blank line separators" This at least needs to be done with care, since sec 4.4.1 says that blank lines MUST be ignored, and if the properties files have embedded blank lines they would mess up such a concatentation.

• Sec 5.4: "... may mirror a HiPS survey ... with the explicit property hips_status = ... clonable ..." That sounds like mirroring is not permissible if hips_status has cloneableOnce , or if it doesn't say anything about clonability. But the hips_status description in sec 4.4.1 says that the default is clonableOnce . Is explicit inclusion of the clonable term really essential, or does it just need to lack unclonable ?

There are a few typos/language issues too:

• a Table des matieres heading appears alongside the Table of Contents .
• Sec 1: "The aim of HiPS system" -> "The aim of the HiPS system"
• Sec 1: "the term 'astronomical survey' term" -> "the term 'astronomical survey'"
• Sec 3: "Each cell at a given level is covers" -> "Each cell at a given level covers"
• Sec 3: "... a collection HiPS tiles stored in regular files..." -> "... a collection of HiPS tiles stored in regular files..."
• Sec 4.2.1.1: "choosen" -> "chosen"
• Sec 5.1: "Additionnally" -> "Additionally"
• In many places "depends of" is written where it should be "depends on"

-- MarkTaylor - 2017-02-21

Knowledge Discovery Interest Group ( Kaï Polsterer )

Theory Interest Group ( _Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova)