UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

• http://ivoa.net/documents/UWS/20160505/index.html (May 5, 2016)

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

• Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

• The new optional attributes "mime-type" and "size" for the XML nodes <result>

IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

-- GregoryMantelet - 2015-10-30

Response to the above question:

According to the recently published XML schema versioning document http://ivoa.net/documents/Notes/XMLVers/20151029/NOTE-schemaVersioning-1.0-20151029.html#tth_sEc3, the document located at

http://www.ivoa.net/xml/UWS/v1.0

should remain as it is (UWS 1.0 stuff only), but a modified 'filename' URL will exist at:

http://www.ivoa.net/xml/UWS/UWS-v1.0.xsd

that has the "version=1.1" attribute within and the additions brought above by the 1.1 specification.

-- BrianMajor - 2016-03-11

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

CADC has implemented all new features introduced UWS 1.1 for all their UWS-based services including TAP:

• http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/sync (auth-sync for userid/password)
• http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/async (auth-async for userid/password)

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We implemented the new UWS1.1 features, i.e. support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase, WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added as well.

Example usage (using gaia.aip.de as example UWS1.1 service):

• uws --user uwstest --password gavo --host https://gaia.aip.de/uws/query list --archived
• uws --user uwstest --password gavo --host https://gaia.aip.de/uws/query list --error --completed --last 10
• uws --user uwstest --password gavo --host https://gaia.aip.de/uws/query list --after 2015-10-26T20:00+01:00
• uws --user uwstest --password gavo --host https://gaia.aip.de/uws/query job show 1445885741809796726 --wait=10 --phase=PENDING

The newest updates (sort by creationTime, optional runId, ownerId and creationTime in shortjob-description of the job list) are supported in the version of branch uws11_update.

-- KristinRiebe - 2016-03-31

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

• http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
• http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
• http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
  This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
• http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
• http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
  This creates a new pending job. Use its jobid in the next two steps.
• http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
• http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN

Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sorted by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value, so they were just ignored.

This is now improved since sorting is done by a newly introduced 'creationTime'. -- KristinRiebe - 2016-03-31

The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, redirect with LAST still needs to be implemented.

Our datetimes are not returned in UTC, but with timezone; if UWS will require UTC datetimes in the future, I will adjust this. -- KristinRiebe - 2016-03-31

TOPCAT/STILTS

The asynchronous TAP clients in topcat v4.3-3 and stilts v3.0-7 now use the blocking behaviour ( ?WAIT=-1&PHASE...=) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document. This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07 (updated 2016-07-11)

Implementations Validators

Validator's for UWS 1.1:

• GAVO uws-validator: It's rather a definition of tests for checking the new functionality of UWS 1.1, i.e. mainly job list filtering and blocking behaviour. It is using behave (Python pacakge) and a definition of features and steps for testing the behaviour of UWS services. Some tests may still fail if I defined them too strict. Especially blocking behaviour may not behave as expected: my tests would fail but the service would still be valid. Tested with AIP's Daiquiri service (e.g. gaia.aip.de) and Gregory's TAP service at mintaka. Here are configuration files for these services: userconfig-gaia-aip.json (AIP Daiquiri service), userconfig-gaia-mintaka.json (CDS/GAVO TAP service).
  They can be used like this (needs additional installation of behave first):
  • For UWS 1.0 services only:
    behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-uws1_1
  • For the new UWS 1.1 functionality, excluding slow tests (waiting a long time):
    behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-slow --tags=-neverending
• Any questions and comments are welcome! (Kristin Riebe, GAVO)

CADC (and others?) have UWS 1.0 validators that still pass:

• CADC UWS 1.0 Validator: OpenCADC cadcTestUWS

• taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07 I'm removing this action on me :-), since UWS blocking is hard to validate, as it's defined to be backward-compatible and can always behave as if it was non-blocking -- MarkTaylor - 2016-02-17

Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

• • there is no significance in the different types of braces - I have made them all curly.
• Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
  • I think you are right - I have removed it
• Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
  • you are right - and quote should be 0..1
• Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
  • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
• Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
  • highly unlikely - I have made the statement less precise about which binding may be added.
• Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
  • done
• Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
  • done
• Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
  • done
• Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
  • corrected
• Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
  • changed wording to make it clear that AstroGrid is no longer active
• Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
  • done
• Sec 4.3: "the emerging XForms technology" - is it still emerging?
  • not really - have updated wording.
• Schema: version attribute to be made optional? (see schema evolution discussion)
  • done in latest version of document and schema

-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

• /{jobs}?QUERY=PHASE=EXECUTING
• /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
• /{jobs}?QUERY=TOP+100

"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL

• /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
2. There is a greater burden on the implementors to do the query parsing.

It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

---

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Approved.

-- PatrickDowler - 2016-05-12

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

I approve this document.

-- PierreFernique - 2016-04-29

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

I approve the document.

A few minor remarks for edition of the rec maybe :

The diagram with the objects as "phase" while the text has "Execution phase"

2.2 reads " In this first version of the UWS pattern only a REST (Representational State Transfer) style binding is presented". Is it really now the first version ?

As staed by Marco the text reads a 1.0 xsd ? is there any 1.1 associated to UWS 1.1 ? or is the xsd unchanged since first version ?

-- FrancoisBonnarel - 2016-05-09

There is a new XSD, but it has retained the namespace of the 1.0 version - the rationale behind this is explained in this note http://www.ivoa.net/documents/Notes/XMLVers/

-- PaulHarrison - 2016-05-10

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

I have given the document a light review and have no comments.. I approve this document.

-- MarkCresitelloDittmar - 2016-04-19

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

I approve this document.

-- BrianMajor - 2016-03-11

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

• agreed - changed in latest version

p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

• have used xsi:nil in both places to be more precise about what is meant.

2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

• actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.

On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

• I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago

Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

• I think that you may be one of the few people who has the tools available to do this!

Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

*

Given that clients have now demonstrated blocking mode, I approve this document with the caveat that a standards RegExt for UWS needs to be created separately.

-- TheresaDower - 2016-03-17

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

1- The last line of the document reads:

$Revision: 1.36 $ $Date:
        2015-09-02 17:41:42 +0100 (Wed, 02 Sep 2015) $ $HeadURL:
        [[https://volute.g-vo.org/svn/trunk/][https://volute.g-vo.org/svn/trunk/]]projects/grid/uws/doc/UWS.html$

is it needed in the standard document ?

• It is not a requirement, but I think that it is useful for a future author to know where the source file is managed - I think that you will see these sorts of references in future in most of the documents that are authored using the IVOATeX preparation system

2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .

• I will update the diagram

-- MireilleLouys - 2015-10-31 replies in bullets by PaulHarrison

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

I have tested a working implementation of UWS1.1 blocking, against Gregory Mantelet's mintaka TAP service on 2015-12-07. Good! The list of implementations now looks much more healthy too. Ops is now happy to see this progress to REC. -- MarkTaylor - 2016-02-17

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova)