(r12) UWS11RFC < IVOA

IVOA Web>IvoaGridAndWebServices>UWS11RFC (revision 12) (raw view)~~EditAttach~~
---++ UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:
   * [[http://ivoa.net/documents/UWS/20150907/index.html][http://ivoa.net/documents/UWS/20150907/index.html]]

---+++ Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

---++++ CDS / GAVO Implementation (Gregory Mantelet)
---++++ VO Paris Implementation (Mathieu Servillat)
---++++ CADC Implementation (Pat Dowler)

---++++ GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at <a target="_blank" href="https://github.com/aipescience/uws-client">https://github.com/aipescience/uws-client</a> is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords is being added in branch 21_job_list_filter.
---+++ Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

   * CADC UWS 1.0 Validator: [[https://code.google.com/p/opencadc/source/browse/#git%2Fprojects%2FcadcTestUWS][OpenCADC cadcTestUWS]]

   * [[http://www.star.bris.ac.uk/~mbt/stilts/sun256/taplint.html][taplint]] TAP validator within [[http://www.starlink.ac.uk/stilts][stilts]] does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See [[https://github.com/Starlink/starjava/blob/master/ttools/src/main/uk/ac/starlink/ttools/taplint/JobStage.java][UWS Stage]]. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled =version='1.1'= ) I'll try to update the validation for it. -- IVOA.MarkTaylor - 2015-07-03
Please add your validator (even if 1.0) if applicable.

---+++ RFC Review Period: 2015-06-30 - 2015-08-18

---+++ TCG Review Period: 2015-09-09 - 2015-10-07

---++ Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

---+++ Comments from IVOA.MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?
   * 
      * there is no significance in the different types of braces - I have made them all curly.
   * Sec 1.3: "an XML document (e.g. <nop>ADQL, CEA [harrison05])" - is it anachronistic to refer to <nop>ADQL in this context since post-ADQL-2.0 there is no <nop>ADQL/X representation? 
      * I think you are right - I have removed it
   * Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram 
      * you are right - and quote should be 0..1
   * Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram. 
      * I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
   * Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really? 
      * highly unlikely - I have made the statement less precise about which binding may be added.
   * Sec 2.2: "an non-existant job id" -&gt; "a non-existent job-id" 
      * done
   * Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"? 
      * done
   * Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=&lt;value&gt;" not just "WAIT"? 
      * <span style="background-color: transparent;"> done</span>
   * Sec 4.2: "<nop>ADQL [1] can serve as a JDL." - bibliography referencing is wrong. 
      * corrected
   * Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by <nop>AstroGrid." - it's a bit anachronistic. 
      * changed wording to make it clear that AstroGrid is no longer active
   * Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop 
      * done
   * Sec 4.3: "the emerging XForms technology" - is it still emerging? 
      * not really - have updated wording.
   * Schema: =version= attribute to be made optional? (see [[http://mail.ivoa.net/pipermail/grid/2015-July/002768.html][schema evolution discussion]]) 
      * done in latest version of document and schema
-- IVOA.MarkTaylor - 2015-07-02 -- Replies IVOA.PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- IVOA.MarkTaylor - 2015-09-11

---+++ Comments from IVOA.WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like<br />we are creating yet another query syntax with new keywords. To be<br />concrete, I would change the examples to
   * /{jobs}?QUERY=PHASE=EXECUTING
   * /{jobs}?QUERY=STARTTIME&gt;2014-09-10T10:01:02.000
   * /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS<br />SCHEMA. For now, the spec will only specify simple queries. So there<br />will be only one parameter in QUERY. But this allows future<br />expansion, so a more sophisticated service could allow full SQL
   * /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME&gt;2014-09-10+ORDER+BY+ENDTIME

---++++ Reply by IVOA.PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages
   1 <span style="background-color: transparent;">More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES</span>
   1 <span style="background-color: transparent;">There is a greater burden on the implementors to do the query parsing.</span>
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs
---

---++ Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

---+++ TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler_ )

---+++ Applications Working Group ( _Pierre Fernique, Tom Donaldson_ )

---+++ Data Access Layer Working Group ( _François Bonnarel, Marco Molinaro_ )

---+++ Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel_ )

---+++ Grid & Web Services Working Group ( _Brian Major, Giuliano Taffoni_ )

---+++ Registry Working Group ( _Markus Demleitner, Theresa Dower_ )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job *have been* retained" (or I don't get what this is intended to say).

p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- IVOA.MarkusDemleitner - 2015-09-25

---+++ Semantics Working Group ( _Mireille Louys, Alberto Accomazzi_ )

---+++ Education Interest Group ( _Massimo Ramella, Sudhanshu Barway_ )

---+++ Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick_ )

---+++ Data Curation & Preservation Interest Group ( _Françoise Genova_ )

---+++ Operations Interest Group ( _Tom !McGlynn, Mark Taylor_ )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- IVOA.MarkTaylor - 2015-09-11

---+++ Knowledge Discovery in Databases Interest Group ( _George Djorgovski_ )

---+++ Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo_ )

---+++ Standards and Processes Committee ( _Françoise Genova_)

<br /> <!--
      * Set ALLOWTOPICRENAME = IVOA.TWikiAdminGroup
-->
Topic revision: r12 - 2015-10-09 - KristinRiebe
IVOA
Log in or Register
IVOA.net
Wiki Home
WebChanges
WebTopicList
WebStatistics
Twiki Meta & Help
IVOA
Know
Main
Sandbox
TWiki
TWiki intro
TWiki tutorial
User registration
Notify me
Working Groups
Interest Groups
Time Domain
Committees
Stds&Procs
www.ivoa.net
Documents
Events
Members
XML Schema