TWiki> IVOA Web>WebPreferences>HiPSRFC (revision 15)EditAttach

HiPS 1.0 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA HiPS 1.0 Proposed Recommendation.

The latest version of the HiPS specification can be found at:

Reference Interoperable Implementations

Comments from the IVOA Community and TCG members during RFC period: 2016-12-15 - 2017-01-26

Comments from Markus Demleitner

p. 16, "Each line is ended by CR..." Are you sure you want what this is saying? The only platform I'm aware of that ever used single CRs as line separators was the classic Mac, and AFAIK with OS X they migrated to having just LF as a line separator as $DEITY had originally intended. So, essentially what you're specifying here will make everyone equally unhappy. Note that the IETF somewhere requires anything with a text/ major media type to use CRLF as line separator. So, much as I'd like to see single LFs here, I guess you should just require CRLF as line separator throughout and be done with it. (Incidentally, on p.22, there's mention of "property1\nkey1"; \n is the conventional escape for LF rather than CR, so whatever you do, this would need reconciliation).

=> It was a mistake. The document has been fixed following the usual ASCII convention end of line. -- PierreFernique - 2017-02-07

p. 17, point 3. First, there's "image His" rather than "image HiPS". Then -- Obscore 1.1 now has "measurements" as dataproduct_type instead of "catalog". Everyone will be grateful if we keep terminology uniform between standards, and given that ObsCore was there first, I guess HiPS would have to bite the bullet and change.

=> The recent discussion/evolution in the ObsCore vocabulary for replacing the "catalog" keyword for "measurements" keyword was ignored by the HiPS editor. The adjustement of this keyword in HiPS would imply a big impact on already existing catalog HiPS distribution and compatible clients => JAXA, Leiden and CDS are already distributing catalog HiPS and the dataproduct_type=catalog is notably used by the clients for discriminating the type of the HiPS and not only for providing user information. Thus, if we modify this keyword, we will have to manage a transition period... always problematic. Additionnally, the term "catalog" in the HiPS context is really what it is. And the replacement with "measurements" appears quite "artificial" - again in the HiPS context. These arguments has been transmitted to the Obscore authors/contributors. It is still in discussion in the Obscore team. -- PierreFernique - 2017-02-07

p. 18, point 6. What should catalog HiPSes put into hips_file_format?

=> "tsv" possibiliity has been added. -- PierreFernique - 2017-02-07

p. 18, point 8: hips_frame. For equatorial, you should mention in the specification that you're assuming the ICRS reference frame (you do, don't you?).

=> ICRS has been explicitely specified. -- PierreFernique - 2017-02-07

p. 18, table header: I suppose "Rec or Should" should be "Req or Should", no? Linguistically more consistent would be "Must or Should", but since you'd have to change all the letters in the table then I'd say it's not worth the effort.

=> Fixed (Rec -> Req) -- PierreFernique - 2017-02-07

p. 18, "obs_description". You're saying this contains "one paragraph", which people would probably take to mean "a couple of (perhaps physical) lines", which of course is outlawed by the basic metadata format. To reduce the risk of confusion, I'd suggest writing something like "longer free text description of the dataset"

=> The proposal has been incorporated in the document. -- PierreFernique - 2017-02-07

p. 20, "Catalogue HiPS metadata": Are you sure we can't just require metadata.xml? If I were to write a client and all I had to work with are the column labels from the TSV, I'd be cross with you.

=> The goal of the HiPS authors/contributors was to keep the generation of catalog HiPS as simple as possible. In this perscpective, the tile TSV header is trivial to implement. The metadata.xml is optional, and can be used/added only when/if it is required. Complementary, keeping a TSV header in each tile helps a lot for debugging/controlling individual tile content if required. -- PierreFernique - 2017-02-07

Also, two typos:

p. 12 "The mean method is appropriated": -> appropriate

=> Fixed. -- PierreFernique - 2017-02-07

p. 19, hips_order: Deeper -> Deepest (or perhaps Highest?)

=> Fixed. -- PierreFernique - 2017-02-07

-- MarkusDemleitner - 2016-12-15

Comments from TCG members during the TCG Review Period: 2016-12-15 - 2017-01-26

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

(1) Sentence "The more you ..." starts in italic and ends in plain text

=> Fixed. -- PierreFernique - 2017-02-07

(2) "Going beyong...": This sentence suggests that HIPs can be used even for non spatial data. Must be clarified

=> Right. New formulation: "Going beyond its emphasis on visualisation, HiPS can also be described as a generalised spatial mapping of data into a common architecture..." -- PierreFernique - 2017-02-07

(3) P5 "... given level is covers an .."
- The HEALPix tile defintion vs HEALPix cell is not clear (to me at least). After reading back and forward, I understand that one tile contains data of one HEALPix cell at an order ranging between 0 and a deepest order which must be lower that the HEALPix order of the resampled data. This key concept, very important to understand the following, could be a bit more developed

=> Difficult to introduce immediately in this section the specificity of the Images HiPS. Here we described the global mechanism of HiPS for any type: catalog, image, cube .... The fact that a HiPS tile for images and cubes contains several sub HEALPix cells is described in the dedicated section 4.2.1 -- PierreFernique - 2017-02-07

(4.2.1) It is first said that HIPs tiles must contain "orginial pixels" and after that "original pixels are resampled": not consistant. Better to say that " HIPs tiles must contain orginal pixels resamples inthe the HEALPix geometry".

=> Good point. we just removed the "original" word in the first sentence. The fact that these pixels are original (for instance in the case of native HEALPix data such as PLANCK or WMAP), or resampled are discussed in the end of the paragraph. -- PierreFernique - 2017-02-07


- "... tile hierarchy is S orders" must or should?
- The subsection "To avoid tiles ...." actually introduces the concept of deepest order. Would be nice to mention it before to enter 4.2.1.1
- 4.2.1 said that tiles contain resampled original data and 4.2.1.1 said the pixels are directly computed from the orginal data: I do not see the distinction between both
- 4.2.1.2: "Each pixel..." better to talk about cells nistead of pixels are in fig5

=> See the previous remark about the image HiPS specificities. -- PierreFernique - 2017-02-07

(4.2.2.1) Same remark as Markus: what about string with characters not encoded in 7Bits?

=> Good suggestion. As UTF-8 is a superset of 7-bit ASCII, there is no good reason to restrict HiPS to 7bits. The document has been updated in this direction. -- PierreFernique - 2017-02-07

(4.2.3) If you put 10,000 cube tiles with 1024 frames each within a directory, you will have ~ 10,000,000 files in that directory. This breaks the reason for setting that 10,000 limit. In a future version the limit on files per diresctory could be adapted to each HIPs.

=> After some mails between HiPS authors and contributors, we agreed with Laurent Michel to wait a future version of HiPS standard for managing this specifical potential cube HiPS limitation (on idea would be to define a dedicated property keyword modifying the default 10,000 tiles packaging). -- PierreFernique - 2017-02-07

(4.3.2) The type of the allsky file may be precised.
- The way image tiles are arranged in allSkys files is not clear. Is that a mosaic, a multi extension FITS file?

=> After discussion with Laurent Michel, the current paragraph is ok. The figure illustrating this section has been enlarge a little bit to better show the mosaic packaging. -- PierreFernique - 2017-02-07

(4.4.3) What is the "FITs header used for global meta data information"?

=> Most of surveys distributed as FITS image collection uses the same FITS header keywords for describing globally the survey (mission, instruments, observatory localisation...). So providing one header is a very simple method to provide this generic information. -- PierreFernique - 2017-02-07

(5.1) "...only the view as directory..." already specified in 4.1

=> Right. But a repetition in this section, dedicated to the distribution, seems appropriate. -- PierreFernique - 2017-02-07

(5.2) "... any HIPs server via a dedicated URL": How to discover that URL, through the registry? must be specified since the global HIPs design tends to be self consistent: just take the root URL and go.

=> That's a good point. Any HiPS list aggregator is free to be declared or not in the VO registry. Typically the MOCserver (CDS) which are doing this task will be declare in the VO registry. -- PierreFernique - 2017-02-07

General remarks

+ Both implementers and consumers will appreciate (among other things) how the registration is detailed (5) and the cookbook section (6)
- The concept of tiles/pixels/cells/order/nside could be a bit better explained before the normative section. (lexicon in a table?)

-- LaurentMichel (and MarkCresitelloDittmar) - 2016-12-16

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

In general, I found this to be a very readable and clear document. I have a couple of comments from the GWS perspective:

5.1 HiPS Server - Since the HiPS server is essentially an IVOA web service, I think it should adopt the mechanisms laid out in the VOSI (VO Support Interfaces) specification. In particular, the /availability and /capabilities endpoints would be useful.

The /availability endpoint would help clients decide whether the server is ready for use. If not, it could switch to another implementation from the registry.

The /capabilities would provide a means for clients to dynamically discover the location of the HiPS list after finding it in the registry. The HiPS list should be identified by an IVOA standard ID. This might have some implications to the wording in the section describing the registration of HiPS implementations.

=> After discussion with Brian Major, we decided to let the HiPS document at is it for HiPS 1.0, and to study the VOSI usage probably in a larger context than just HiPS for future version (considering HiPS access as one possible "interface" for accessing a dataset collection, one amongs others. For instance, we can imagine to have a HiPS capability and a CS capability for the same catalog. Or a SIA interface and a HiPS interface for an image collection - also to manage possible multi HiPS interfaces) -- PierreFernique - 2017-02-09

It would be nice to see an example HTTP URL to a tile, perhaps repeating the example earlier in the document.

=> Good idea. The generic example provided in section 4.1 has been re-used in section 6.1 and 6.2 for illustrating a local access to a HiPS tile and an HTTP remote access to the same HiPS tile. -- PierreFernique - 2017-02-09

Should the HiPS server set HTTP headers for tile requests to help the client? For example, content-type: image/png

=> As a HiPS tile is (usually) a regular file, the content-type is automatically set by the apache server. image/png for PNG tiles, image/fits for FITS tiles... -- PierreFernique - 2017-02-09

Proprietary Data

This may only be a concern for future HiPS implementations, but I think some forethought regarding access to proprietary astronomical surveys should be given. Namely, there should be a way for clients to authenticate. If the /capabilities metadata of a HiPS server is available then there would be a standard mechanism for determining the supported authentication methods when it comes to proprietary HiPS lists.

There may be cases where only certain tiles at a certain order or sky location are proprietary. I wonder if there should be space reserved in the tile metadata for authorized read permission information. This field would likely specify the standardID of a group that is allowed to read the tile.

=> Any possible authentification or encryption mechanism can be applied just by using the HTTP or HTTPS support. Concretly, today there are already some HiPS with restricted access (basic HTTP authentification, or more complex system). And we can already limit the access partially quite easily (for instance, we can ask authentification for tiles located in NorderNNN directory for NNN greater than... : all users can see the HiPS at low resolutions, but only authorized people can access to the high resolutions.). We mentioned this point in the 6.2 section. -- PierreFernique - 2017-02-09

-- BrianMajor - 2017-02-04

I approve this PR.

-- BrianMajor - 2017-02-27

Registry Working Group ( _Markus Demleitner, Theresa Dower )

(1) 4.4.2.1 Is there a terribly strong reason to require 7-bit ASCII in the TSVs? This would make Registry's life quite a bit harder when we produce a HiPS of the Registry (once we have enough coverages), as our authors and titles can potentially contain all kinds of odd characters. My take is: Just allow UTF-8. There's few platforms where that's a problem these days, and UTF-8 doesn't clobber Tabs, so there's probably little that can go severly wrong.

=> We moved to UTF-8 (cf Laurent Michel comment and response below) -- PierreFernique - 2017-02-09

(2) While I like the term obs_regime in the table on p. 18, I have to mention that the same concept, with essentially the same terms (but different spelling -- ugh) is called waveband within VODataService's vs:Coverage element. Now, the different concept names probably are not really dramatic, but the differences in the terms contained are fairly painful, and that pain will only increase as we may extend the list with neutrinos, gravitational waves, or whatever. So, here's what the VODataService creators came up with about ten years ago:

"Radio" | "Millimeter" | "Infrared" | "Optical" | "UV" | "EUV" | "X-ray" | "Gamma-ray"

Couldn't you just adopt this list or even better, defer to VODataService's vs:Waveband type so future additions will just pull through? I realise it's a pain given the installed base, but perhaps clients can contain a mapping for a while to ease transition?

=> Good point. We just differed by "Millimeter" and "EUV" additions, and uppercase first letter. The document has been fixed to follow the VODataService list. -- PierreFernique - 2017-02-09

(3) Thanks for making the implemented global HiPS list something of an epiphenomenon of a proper registration -- much appreciated. Since, on the other had, I suspect that all actual clients go through that global list exclusively, I think either sect. 6 should say somewhere that that service exists and how to use it (probably in a non-normative section) or there should be an appendix discussing that. Except, of course, if you actually have plans to phase out that service. But if, as I suspect, it is what clients are expected to do for the forseeable future, it should IMHO be documented in this standard.

=> Difficult to provide a potentially non permanent URL in such a document. But as you suggested, we added as a page note the URL to the CDS hips list aggregator -- PierreFernique - 2017-02-09

-- MarkusDemleitner - 2016-12-15

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Dave Morris )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

The document looks basically OK, but I have some points that should be clarified. These comments concern the version PR-HiPS-20170207.

  • Sec 4.2.2.1: "Columns providing the location on the sky of each source MUST be present." That's reasonable, but it doesn't prescribe how to find such columns. Sec 4.4.3 says that the catalogue metadata file is optional. If there is no such metadata file, then presumably one has to guess from the column names in one of the catalogue files which columns are sky positions. In that case there should be some rules provided for how to do that. If on the other hand the metadata.xml file really is mandatory in this case, that should be stated, and there should be rules about how to identify sky positions: e.g. are columns with UCD=pos.eq.{ra,dec};meta.main mandatory?
=> After discussion with Mark and with the agreement of all HiPS authors, we decided that the metada.xml VOTable file will be mandatory (SHOULD) for catalog HiPS. By this basic modification, any HiPS client will be able to follow the usual VOTable rules and methods to retrieve coordinate columns -- PierreFernique - 2017-03-14

  • Sec 4.4.1:
    • ".. or spaces (blank, TSV, ...)" - should "TSV" here read "TAB"?
    • "possible spaces between the keyword and the character "=" MUST be ignored" - presumably the same applies to spaces between the equals sign and the value text too.
=> Right, fixed! -- PierreFernique - 2017-03-14
    • dataproduct_subtype description: I don't understand the explanation of this keyword's content. Does "color" just mean that it's using PNG/JPEG tiles (in which case you can work it out from the hips_tile_format keyword), or does it mean something else, if so what?
=> JPEG or PNG tile usage do not imply that the HiPS is colored. Most of grey levels HiPS provide grey level tiles in JPEG or PNG. The dataproduct_subtype = color means that the tiles contains colored pixel. I precised this point in the document, thanks. -- PierreFernique - 2017-03-14
    • "some keywords are not exhaustive and may be completed if required" - would "extended" be better than "completed"?
=> Done! -- PierreFernique - 2017-03-14

  • Keyword table in sec 4.4.1:
    • hips_status keyword is marked R[equired], but not mentioned as one of the 8+2 Mandatory keywords in the preceding text
=> Right. Added in the mandatory keywords section -- PierreFernique - 2017-03-14
    • hips_reg_blue should read hips_rgb_blue
=> Fixed. -- PierreFernique - 2017-03-14
    • hips_rgb_* keywords format " creator_did [cutLow cutMiddle cutHight TransferFunction] " seems a bit underexplained. What's the creator_did doing here? What are the options for the transfer function? what do you do with the low/middle/high values?
=> These parameters are too specific to the Aladin/Hipsgen colored HiPS generation. Removed -- PierreFernique - 2017-03-14
    • hips_data_range - I don't understand what is a "default data range"?

=> This is the pixel range taken into account during the HiPS generation, notably if the BITPIX is modified. I modified the explanation in this sens. -- PierreFernique - 2017-03-14

    • hips_pixel_scale - if I understand correctly this is redundant with hips_order - is there any point in defining this value?
=> You are right! we can compute this value from the hips_order, but it is really useful to have it directly expressed in angular unit, which can be compared to the original pixel scale (s_pixel_scale) -- PierreFernique - 2017-03-14
    • addendum_did - what does "(must be multiple)" mean? If it means that it has to be specified multiple times, shouldn't it be marked with "(*)"?

=> Right, fixed ! -- PierreFernique - 2017-03-14

  • Sec 5.2: "a HiPS list MAY be easily generated as the concatenation of the properties files of all distributed HiPS, with blank line separators" This at least needs to be done with care, since sec 4.4.1 says that blank lines MUST be ignored, and if the properties files have embedded blank lines they would mess up such a concatentation.
=> You are right. A note has been added to specify this point -- PierreFernique - 2017-03-14
  • Sec 5.4: "... may mirror a HiPS survey ... with the explicit property hips_status = ... clonable ..." That sounds like mirroring is not permissible if hips_status has cloneableOnce , or if it doesn't say anything about clonability. But the hips_status description in sec 4.4.1 says that the default is clonableOnce . Is explicit inclusion of the clonable term really essential, or does it just need to lack unclonable ?
=> Better explained in the first citation of clonableOnce (the copy is allowed but only from the master site). -- PierreFernique - 2017-03-14

There are a few typos/language issues too:

  • a Table des matieres heading appears alongside the Table of Contents .
  • Sec 1: "The aim of HiPS system" -> "The aim of the HiPS system"
  • Sec 1: "the term 'astronomical survey' term" -> "the term 'astronomical survey'"
  • Sec 3: "Each cell at a given level is covers" -> "Each cell at a given level covers"
  • Sec 3: "... a collection HiPS tiles stored in regular files..." -> "... a collection of HiPS tiles stored in regular files..."
  • Sec 4.2.1.1: "choosen" -> "chosen"
  • Sec 5.1: "Additionnally" -> "Additionally"
  • In many places "depends of" is written where it should be "depends on"
=> Fixed. -- PierreFernique - 2017-03-14

-- MarkTaylor - 2017-02-21

Knowledge Discovery Interest Group ( Kaï Polsterer )

Theory Interest Group ( _Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova)


Edit | Attach | Watch | Print version | History: r25 | r17 < r16 < r15 < r14 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r15 - 2017-03-14 - PierreFernique
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback