RegTAP 1.1 Proposed Recommendation: Request for Comments

Summary

Registries provide a mechanism with which VO applications can discover and select resources - first and foremost data and services - that are relevant for a particular scientific problem. The RegTAP specification defines an interface for searching this resource metadata based on the IVOA's TAP protocol. It specifies a set of tables that comprise a useful subset of the information contained in the registry records, as well as the table's data content in terms of the XML VOResource data model. The general design of the system is geared towards allowing easy authoring of queries.

RegTAP 1.1, as a minor version increment over RegTAP 1.0, is backward compatible. The main differences between 1.1 and 1.0 are as follows:

• new allowed res_detail values for testQueryStrings
• more generic type information in schema tables
• mapped terms in fields for dates and resource relationships to a vocabulary for DataCite compatibility
• case-insensitive query support from ADQL 2.1
• new columns for service mirrors, rights, and authentication and authorization of protected data and services
• new table for alternate identifiers, supporting DOIs, ORCIDs, bibcodes, and future identification schemes

• http://www.ivoa.net/documents/RegTAP/20190529/index.html

Reference Interoperable Implementations

Two separate reference implementations of server-side architecture exist at GAVO (and other archives using GAVO's codebase) and STScI

• GAVO: The machines serving http://reg.g-vo.org/tap implement all proposed changes (on top of DaCHS and Postgres). The Implementation Validators section has a list of queries exercising the new features.
• STScI: http://vao.stsci.edu/regtap/tapservice.aspx implements all proposed changes minus optional detail insertions for test queries. Runs MSSQL server, C# ASP.NET web service code plus the ADQL translator from the VOLLT project (https://github.com/gmantele/vollt)

Implementations Validators

The RegTAP validator (currently at http://docs.g-vo.org/regtap-val; should this move into the VCS for the standard?) has been updated to cover the main new features.

For reviewers, here's a set of RegTAP queries exercising the main user-visible new features (TAP access URLs above):

alt_identifiers – find VO resources and their titles that have DOIs:

select ivoid, res_title, alt_identifier
from rr.resource
natural join rr.alt_identifier
where alt_identifier like 'doi:%'

rights_uri – find VO resources that have a CC license declared:

select ivoid, res_title, rights_uri
from rr.resource
where rights_uri like 'http://creativecommons.org/%'

or find what license URIs are already in use:

select distinct rights_uri from rr.resource

mirror_url – find mirrors available for a known access url (in this case, indicating that the service is available through https, too):

select ivoid, mirrors.mirror_url
from rr.interface as intfs
join rr.interface as mirrors
using (ivoid,intf_index, cap_index)
where intfs.access_url='http://dc.zah.uni-heidelberg.de/antares/q/cone/form'

authenticated_only – find resources unavailable without authentication (note that we do not claim that's enough to actually operate them; the use case at this point is filtering them out with a view to a VO that has more of them):

select distinct ivoid from rr.interface where authenticated_only=1

vocabulary mapping – use just a single term to find out services of data collections:

select res_title
from rr.resource as res
natural join rr.relationship as rel
where relationship_type='isservedby'
  and rel.related_id='ivo://nasa.heasarc/services/xamin'

Comments from the IVOA Community during RFC/TCG review period: 2019-06-15 - 2019-07-31

The comments from the TCG members during the RFC/TCG review should be included in the next section.

In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your Wiki Name so that authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.

Additional discussion about any of the comments or responses can be conducted on the WG mailing list. However, please be sure to enter your initial comments here for full consideration in any future revisions of this document

• Sample comment by WikiName
  • Response (by WikiName)

Comments from TCG member during the RFC/TCG Review Period: TCG_start_date - TCG_end_date

WG chairs or vice chairs must read the Document, provide comments if any (including on topics not directly linked to the Group matters) or indicate that they have no comment.

IG chairs or vice chairs are also encouraged to do the same, althought their inputs are not compulsory.

TCG Chair & Vice Chair

Applications Working Group

Data Access Layer Working Group

Data Model Working Group

Grid & Web Services Working Group

Registry Working Group

Semantics Working Group

Remarks from Carlo Maria Zwölf

• Page 4
  • "Even if it were, data discovery would at least be fairly time consuming if
    each client had to query dozens or, potentially,hundreds of publishing
    registries". --> If a client performs queries in parallel, the number of
    registries is not a matter. In some cases a central service gathering all the
    information could be a bottleneck of the IVOA infrastructure. A discussion
    would be welcome on these aspects. It it also important to mention sync-issue
    between the publishing registries and the RegistryOfRegistries.
  • "this first attempt which was quickly" --> check if 'which' should be removed
  • "The simplification yields 14 tables" --> I would suggest to replace by "The simplification
    yields to a schema composed of 14 tables"
  • TAP_SCHEMA is not defined. What does this stand for? Why is it in red? What is the implied
    color convention?
• Page 5
  • "The largest table,table_column, has about a million rowsat the time of
    writing". --> A standard document is neutral about the practical and specific
    implementation. For highlighting this neutrality I would suggest to write
    something like 'If we use this standard for describing all the resources available in the publishing
    registries, then the table table_column would contain about a million row'
  • "table_column" --> why in green. What is the color convention?
• Page 6
  • Caption of figure 1 : "Relational Registry" tagging does not appear in the
    figure, as it is said in the caption. Please highlight the current described
    standard in red on the figure.
  • "...table using xpaths into the registry documents.This document should not
    in general" --> Does 'this document' refer to the registry document or the
    current document? It is ambiguous.
• Page 15
  • In both TAP_SCHEMA and the VODataService tableset, this schema MUST be
    associated with a utype matching the data model identifier given in sect. 7"
    --> It would be more clear to write "In both TAP_SCHEMA and the VODataService
    tableset, the rr schema MUST be
    associated with a utype matching the data model identifier given in sect. 7
  • "On the values in the utype columns within TAP_SCHEMA except for the schema
    utype, see section 6." --> Please check this sentence. It is not clear for me.

Data Curation & Preservation Interest Group

Education Interest Group

Knowledge Discovery Interest Group

Solar System Interest Group

Theory Interest Group

Time Domain Interest Group

Operations

• As of 2019-06-13, the "latest" version pointed at from this RFC page is http://ivoa.net/documents/RegTAP/20190503/index.html, but the current latest version is actually http://www.ivoa.net/documents/RegTAP/20190529/index.html. It's 20190529 that I'm commenting on here.
• Fig 1: the caption says '(tagged with "Relational Registry")' . I can't work out what this note means - should it say something like '(tagged as "RegTAP")' ?
• Appendix D: In this version (20190529) all the changes since REC-1.0 have been condensed into a single section (D.1) to avoid extra content that only serves historical interest of VO archaeologists; I think this is good practice. By the same token, does it makes sense to delete all the sections (D.2-D.9 in 20190529) that detail how it got to REC-1.0? As an issue affecting preparation of all REC-track documents, this is probably something which also ought to be discussed at TCG or SDP level.
• Appendix D.1: One change in the text which I don't see noted here is changes to the standard_id constraints in the example queries from section 10, e.g. 10.3 in REC-1.0 has WHERE standard_id='ivo://ivoa.net/std/sia' , and in PR-1.1-20190529 has WHERE standard_id LIKE 'ivo://ivoa.net/std/sia%' . I appreciate this may be a result of changes elsewhere in the standards landscape rather than in RegTAP itself, but it would be useful for people making use of these example queries if the change log summarised what's changed in recommended practice since REC-RegTAP-1.0. (There's also a correction to the argument order in the ivo_hashlist_has invocation in section 10.3 which should probably be logged for completeness).
• This RFC page says "The TOPCAT client is aware of RegTAP 1.1 features and is interoperable with both reference implementations" . Yes it is interoperable, because of the 1.1->1.0 backward compatibility. However, can you refresh my memory about what 1.1 features it's aware of? I'm not denying it! Just can't remember what I may have claimed.
• Plus a few typos:
  • Sec 4.1: "whereever" -> "wherever"
  • Sec 6: "specifiation" -> "specification"
  • Sec 8.1, vr:organisation: "is to be references by IVOID" -> "is to be referenced by IVOID" ?
  • Sec 8.10: "slight denormalization of the vr:Relationship type: Whereas..." -> "slight denormalization of the vr:Relationship type: whereas..." ?

Mark, I updated the link. Thanks and apologies.

As for the TOPCAT support, I'm unaware of specific 1.1 features and will leave that with other comments for Markus to respond to as author; possibly the intention was noting the schema additions didn't break anything, with altIdentifier queries working and no broken baked-in example queries? Those are the main points that came to my mind when testing my reference implementation with TOPCAT. -- TheresaDower - 2019-07-24

Standards and Processes Committee

TCG Vote : Vote_start_date - Vote_end_date

If you have minor comments (typos) on the last version of the document please indicate it in the Comments column of the table and post them in the TCG comments section above with the date.