UserAgentUsage

NOTE: this content has been included in the SoftID draft Note, currently under review.

-- MarkTaylor - 2021-03-05

Identifying operations-related service requests

Introduction

Client software may make requests to VO services for various reasons. The most straightforward case is when a science user makes a query in order to obtain data for scientific purposes. However clients may also wish to connect for operational purposes, for instance in order to perform service validation or monitoring. If service providers are gathering statistics on service usage, they may wish to distinguish these different classes of request.

User-Agent header standard usage

The HTTP User-Agent header may be used by clients to identify their nature or origin. The definiition and usage of this header is described in RFC2616 section 14.43, with additional text on syntax in sections 3.8 and 2.2. The basic rule is that the content of this field should consist of a sequence of tokens, where each token is either a product name (with an optional version indicator), or a free-text comment enclosed in parentheses. Formally (BNF from RFC2616):

   User-Agent      = "User-Agent" ":" 1*( product | comment )
   product         = token ["/" product-version]
   product-version = token
   comment         = "(" *( ctext | quoted-pair | comment ) ")"
   ctext           = <any TEXT excluding "(" and ")">

(the quoted-pair business here is a character escaping mechanism).

Additional rules/conventions are that more-significant tokens should appear earlier in the sequence, and that the content should be "short and to the point".

User-Agent field IVOA recommendations

The Operations IG endorses and encourages use of these rules concerning the User-Agent header, and adds a further convention, which does not conflict with the above rules: clients whose primary purpose relates to operations within the VO should indicate that purpose by including a comment token of the form "(IVOA-<op-purpose> <optional-extra-text>)".

Suggested op-purpose values are currently:

• validate : client is performing service validation, e.g. to assess service conformance to standards
• monitor : client is performing service monitoring, e.g. to assess service availability or performance
• harvest : client is harvesting records e.g. to populate a registry or registry-like database

This list may evolve in the future (suggestions should be discussed on the ops@ivoa.net mailing list). Custom op-purpose values are permitted. Case is not significant in op-purpose values or in its "IVOA-" prefix.

The <optional-extra-text> may (should?) be used to indicate a URL at which more information about the client, or perhaps about the results it is gathering from the current request, can be found.

Formally:

   ivoa-comment  = "(IVOA-" op-purpose *( ctext | quoted-pair | comment ) ")"
   op-purpose   = "validate" | "monitor" | "harvest" | token

Tokens of the form ivoa-comment should not appear in the user-agent field if the request is a "normal" user science query. There are obviously grey areas between operational and science requests; this convention does not attempt to provide a rigid definition of these categories.

This arrangement allows service operators to test in their logs for User-agent values whose content matches the sequence "(IVOA-", or perhaps "(IVOA-validate", and adjust their usage statistics appropriately. Note however that it is not feasible to force operational clients to follow this convention, so service operators will still need to be careful in analysing their usage statistics.

Examples

A science query from the STILTS tapquery TAP client might contain the HTTP header

   User-Agent: STILTS/3.1-4 Java/1.8.0_181

while a query from the STILTS taplint TAP service validator might contain the header

   User-Agent: STILTS/3.1-4 (IVOA-Validate) Java/1.8.0_181

or maybe

   User-Agent: STILTS/3.1-4 (ivoa-validate http://validators.org/results) Java/1.8.0_181

Implementations

Here is a list of operational clients that declare themselves using this convention. Please update this list if you know of such clients not listed here:

• STILTS taplint, as "(IVOA-Validate)" since version 3.3-3 (05/2019)

History

This topic was originally raised by Alberto Micol, and discussed in the Operations IG session at the November 2018 Interop meeting in College Park.

This page was written up by Mark Taylor following that discussion to facilitate further consideration by the community. If there is agreement, it may make sense to write it up as an IVOA Note at some point.

-- MarkTaylor - 2018-11-22