Discovering Data Collections Within Services: Endorsed Note - Request for TCG Comments and Approvals

Document Link: http://ivoa.net/documents/Notes/discovercollections/20161111/

Temporary location of document build updated according to the RFC comments below: http://docs.g-vo.org/discovercollections.pdf

Source Repository: https://volute.g-vo.org/svn/trunk/projects/registry/discovercollections

RFC period: 2017-03-01 through 2017-04-12

Implementations

Several publishing registries (at least GAVO's, HEASARC's, and CADC's) already produce records as defined in this specification. Also, the future Registry Interfaces 1.1 (also in RFC) intends to employ the method proposed here. Note: see below for caveat on CADC's use of this.

On the client side, WIRR implements discovery of data collections in this way -- sample query.

Also, TOPCAT has experimented with this scheme since several releases. Since v4.5, you'll find it in the TAP window (TAP -> Service Discovery -> Reg Prototype). If your TAP service isn't shown with the right number of tables there, please consider fixing your service's registry interface.

Comments from the interested public

Comments from TCG members

WG chairs or vice chairs must read the Note, provide comments if any and formally indicate if they approve or do not approve of the Note. IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

In trying to configure the ivo://cadc.nrc.ca/IRIS data collection with related service relationships and aux capabilities I ran into a problem: the capability child element is not allowed (schema validation exception) and when I looked in the relevant xsd I see that it is only allowed in a resource of type Service. The example A.1 is actually a resource of type CatalogService (extends DataService extends Service) whereas ivo://cadc.nrc.ca/IRIS is a resource of type DataCollection (extends Resource). The end of page 6 covers this issue and promises that VOResource-1.1 will allow capabilities in DataCollection... so at this point I cannot implement aux capablities since we used DataCollection.

The recommendation to use CatalogService in place of DataCollection as a stop gap seems to force clients back into service-oriented discovery. I have a feeling we would regret making that recommendation. Is it sensible to put this note on hold until we have VOResource-1.1 and can describe DataCollection resources properly?

• In practice, the resource type is rarely used for data discovery (it's relevant in Authorities, Standards, etc). So, the actual type of something containing data isn't that big a deal. In particular, once we have added capability to DataCollection, its content model will essentially be identical to DataService. So, using CatalogService instead of DataCollection is an aesthetic defect at worst. Also note that DataCollection is defined in VODataService rather than VOResource, and it will not be updated before 2019. Hence, making this an EN would be delayed fairly significantly if we waited for it. -- MarkusDemleitner - 2018-02-20

On a more general note, since DataCollection can have zero or more relationship(s) it looks like a suitably complex join in RegTAP could find DataCollection resources that are servedBy TAP services without the aux capabilities, so it looks like this is a denormalisation of the VOResource model to make it possibly to do this (efficiently) without a join? Is that the essence?

• More or less. Sect. 1.2.2 of the PEN discusses why the originally envisioned plan of having "unsullied" DataCollections that just declare relationships to actual DataServices yields very clumsy query patterns in common cases. So, it's not a denormalisation in the sense of relational algebra, but certainly an adaptation of the underlying model to facilitate all of usability, migration of legacy structures, and efficient queries. -- MarkusDemleitner - 2018-02-20

-- PatrickDowler - 2018-02-14

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

I have to thanks the authors of this note to bring a solution to a serious problem in the orginal design of VO registry. I have to say that elements presented in this note would have help a lot in the implementation of Aladin v10 which had to do, in some cases fragile assumption (based on prefix url...), and in some other cases, has no solution at all. If this note is followed by the providers, my client developper life will be definitively easier. Also, I appreciate a lot that the solution proposed has concretely a very light impact on the existing content of the registry. I will clearly encourage all providers to follow this note as fast as possible in order to improve the quality and the usability of the VO registry content.

I approve this note

Some minor suggestions:

• Page 6: The detal of the rational for standardID suffix syntax is probably not really required in this note. or ?
  • Hm -- frankly, I'd like to keep the explanation in, as it's a bit subtle -- and, I think, helps understanding what problem this is intended to solve. If you really feel it's out of place here, it could be separated typographically (perhaps with a little header in between) or put in an appendix. Would you strongly prefer that? -- MarkusDemleitner
• Page 9 : the specific citation of VizieR current state for illustrating the impact of the addition of aux. will be obsoleted when VizieR will applied the note recommendation. I would suggest to remove VizieR citation, and just evocate "a service"...
  • Removed VizieR reference hier in Volute rev. 4571 -- MarkusDemleitner
• page 11: The reference http://gavo... for WIRR is a little bit obscured for me.
  • You mean, as in "smear on the rendered document"? What user agent is that? (Sounds like an ivoatex issue, but I'll take these, too) -- MarkusDemleitner
• Page 15: typo: Removed secion on => section
  • Fixed, thanks. -- MarkusDemleitner

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

The problem tackled by this note is real and the proposed solution looks smart and working. We strongly support this proposal for recommendation.

However, a few remarks could be taken into account by the authors:

page 3 : introduction

• It should be nice to describe the "reverse case" where a collection is served by several services (eg : non Standard interface, ConeSearch, TAP and HiPs for a catalogue)
  • That's simply mulitple capabilities on the same record; I'm now mentioning this in volute rev. 4572 -- MarkusDemleitner 2017-11-13
• It's better to use the word ObsTAP instead of ObsCore: Obscore is the model but the ObsCore spec explicity designates TAP services serving ObsCore tables as "ObsTAP".
  • Fair enough -- changed this (except where we actually reference the data model) in volute rev. 4572 -- MarkusDemleitner 2017-11-13
• page 5 / 2.1 The statement "either one of main or auxiliary capabilities" : does that mean inclusive or ?
  • That is inclusive – I don't see a scenario where a service could have both main and auxiliary capaiblities of the same standard, though, so perhaps that's academic. I could be wrong, though -- MarkusDemleitner 2017-11-13
• page 6 = the sentence "This note proposes to mark the auxiliary ...." expresses the main modification proposed by the note. It could be usefully repeated in a new short section just after 2.1 called "Implication for standardID expression" to clarify the actual modification (all the other stuff being "good practice actually"
  • Hm – I'm not wild about this; it seems uglification by repetition to me, without giving an additional utility proportional to the uglification. Perhaps one could just put "(normative)" into the respective section titles? Does anyone else feel something like this would be a useful thing to have? -- MarkusDemleitner 2017-11-13
• page 9 : sentence "In order to offer an immediate solution for the outlined problem... ". We propose "In order to offer a solution for legacy identifiers" we propose to form auxiliary identifiers for them..." because it seems to be something which may last a while...
  • True -- put in "In order to offer a solution for legacy standards and standard identifiers already in use in the VO" in volute rev. 4572; improvements to the wording are welcome directly in Volute -- MarkusDemleitner 2017-11-13

-- FrancoisBonnarel - 2017-11-09

-- MarcoMolinaro - 2017-11-09

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

a bit out of my area of expertise, but the content seems consistent and logical. This does seem like content where we should be giving a 'preferred' method and guidance to implement.

I had one question in reading the text, which does not effect the assessment, merely a curiosity. Section 2.1.2 specifically states that an ID of form "ivo://ivoa.net/std/sia#query-aux-2.0" generates queries for all minor versions as well (tag-n.%). My question is what happens for ID with a specified minor version "ivo://ivoa.net/std/sia#query-aux-2.2"? If this returns exact match only, how does one get exact match for '2.0'? It seems that the minor version is irrelevant in this thread, given the backward compatibility requirements.

• No, the query-aux-2.0 is not the query pattern ignoring the minor version, it is the full identifier that lets people match for minor versions if they have to for some reason. The actual query patterns are discussed in sect. 2.2. Any hints on how to make that clearer are appreciated -- MarkusDemleitner - 2017-09-13

typos in section 2.1.4: 2 instances of 'notherntel' rather than 'northerntel', and 1 instance of 'sourtherntel' rather than 'southerntel'

• Fixed, thanks -- MarkusDemleitner - 2017-09-13

I have no objection to the content and approve the Note. -- MarkCresitelloDittmar - 2017-09-11

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

Knowledge Discovery in Databases Interest Group ( Kai Polsterer )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Solar System Interest Group ( _Baptiste Cecconi)

Although the EPNcore-v2 data model is not an IVOA standard (yet), it would be nice to have example implementation of such a service. I can try to work with Markus Demleitner and Pierre Le Sidaner to propose something.

-- BaptisteCecconi - 2018-02-22

Standards and Processes Committee ( Françoise Genova)