Difference: UWS11RFC (1 vs. 36)

Revision 362016-07-11 - MarkTaylor

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.

The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version is available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site has also been upgraded to UWS 1.1.

Please, report any bug and/or missing feature you may found while testing it!

This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)

  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important

  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

Response to the above question:

According to the recently published XML schema versioning document http://ivoa.net/documents/Notes/XMLVers/20151029/NOTE-schemaVersioning-1.0-20151029.html#tth_sEc3, the document located at

http://www.ivoa.net/xml/UWS/v1.0

should remain as it is (UWS 1.0 stuff only), but a modified 'filename' URL will exist at:

http://www.ivoa.net/xml/UWS/UWS-v1.0.xsd

that has the "version=1.1" attribute within and the additions brought above by the 1.1 specification.

-- BrianMajor - 2016-03-11

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

CADC has implemented all new features introduced UWS 1.1 for all their UWS-based services including TAP:

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We implemented the new UWS1.1 features, i.e. support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase, WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added as well.

Example usage (using gaia.aip.de as example UWS1.1 service):

The newest updates (sort by creationTime, optional runId, ownerId and creationTime in shortjob-description of the job list) are supported in the version of branch uws11_update.

-- KristinRiebe - 2016-03-31

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use its jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sorted by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value, so they were just ignored.

This is now improved since sorting is done by a newly introduced 'creationTime'. -- KristinRiebe - 2016-03-31

The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, redirect with LAST still needs to be implemented.

Our datetimes are not returned in UTC, but with timezone; if UWS will require UTC datetimes in the future, I will adjust this. -- KristinRiebe - 2016-03-31

TOPCAT/STILTS

Changed:
<
<		The asynchronous TAP clients in topcat and stilts (prerelease available) now use the blocking behaviour ( ?WAIT=-1&PHASE...=) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document. This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07
>
>		The asynchronous TAP clients in topcat v4.3-3 and stilts v3.0-7 now use the blocking behaviour ( ?WAIT=-1&PHASE...=) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document. This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07 (updated 2016-07-11)
 

Implementations Validators

Validator's for UWS 1.1:

  • GAVO uws-validator: It's rather a definition of tests for checking the new functionality of UWS 1.1, i.e. mainly job list filtering and blocking behaviour. It is using behave (Python pacakge) and a definition of features and steps for testing the behaviour of UWS services. Some tests may still fail if I defined them too strict. Especially blocking behaviour may not behave as expected: my tests would fail but the service would still be valid. Tested with AIP's Daiquiri service (e.g. gaia.aip.de) and Gregory's TAP service at mintaka. Here are configuration files for these services: userconfig-gaia-aip.json (AIP Daiquiri service), userconfig-gaia-mintaka.json (CDS/GAVO TAP service).
    They can be used like this (needs additional installation of behave first):
    • For UWS 1.0 services only:
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-uws1_1
    • For the new UWS 1.1 functionality, excluding slow tests (waiting a long time):
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-slow --tags=-neverending
  • Any questions and comments are welcome! (Kristin Riebe, GAVO)
CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07 I'm removing this action on me :-), since UWS blocking is hard to validate, as it's defined to be backward-compatible and can always behave as if it was non-blocking -- MarkTaylor - 2016-02-17
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Approved.

-- PatrickDowler - 2016-05-12

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

I approve this document.

-- PierreFernique - 2016-04-29

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

I approve the document.

A few minor remarks for edition of the rec maybe :

The diagram with the objects as "phase" while the text has "Execution phase"

2.2 reads " In this first version of the UWS pattern only a REST (Representational State Transfer) style binding is presented". Is it really now the first version ?

As staed by Marco the text reads a 1.0 xsd ? is there any 1.1 associated to UWS 1.1 ? or is the xsd unchanged since first version ?

-- FrancoisBonnarel - 2016-05-09

There is a new XSD, but it has retained the namespace of the 1.0 version - the rationale behind this is explained in this note http://www.ivoa.net/documents/Notes/XMLVers/

-- PaulHarrison - 2016-05-10

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

I have given the document a light review and have no comments.. I approve this document.

-- MarkCresitelloDittmar - 2016-04-19

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

I approve this document.

-- BrianMajor - 2016-03-11

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

*

Given that clients have now demonstrated blocking mode, I approve this document with the caveat that a standards RegExt for UWS needs to be created separately.

-- TheresaDower - 2016-03-17

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

1- The last line of the document reads:

Changed:
<
<
$Revision: 1.34 $ $Date:
>
>
$Revision: 1.35 $ $Date:
  2015-09-02 17:41:42 +0100 (Wed, 02 Sep 2015) $ $HeadURL: https://volute.g-vo.org/svn/trunk/projects/grid/uws/doc/UWS.html$

is it needed in the standard document ?

  • It is not a requirement, but I think that it is useful for a future author to know where the source file is managed - I think that you will see these sorts of references in future in most of the documents that are authored using the IVOATeX preparation system
2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .

  • I will update the diagram
-- MireilleLouys - 2015-10-31 replies in bullets by PaulHarrison

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

I have tested a working implementation of UWS1.1 blocking, against Gregory Mantelet's mintaka TAP service on 2015-12-07. Good! The list of implementations now looks much more healthy too. Ops is now happy to see this progress to REC. -- MarkTaylor - 2016-02-17

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

META FILEATTACHMENT attachment="userconfig-gaia-mintaka.json" attr="" comment="" date="1456302914" name="userconfig-gaia-mintaka.json" path="userconfig-gaia-mintaka.json" size="743" user="KristinRiebe" version="1"
META FILEATTACHMENT attachment="userconfig-gaia-aip.json" attr="" comment="" date="1456303205" name="userconfig-gaia-aip.json" path="userconfig-gaia-aip.json" size="720" user="KristinRiebe" version="1"

Revision 352016-05-12 - PatrickDowler

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.

The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version is available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site has also been upgraded to UWS 1.1.

Please, report any bug and/or missing feature you may found while testing it!

This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)

  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important

  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

Response to the above question:

According to the recently published XML schema versioning document http://ivoa.net/documents/Notes/XMLVers/20151029/NOTE-schemaVersioning-1.0-20151029.html#tth_sEc3, the document located at

http://www.ivoa.net/xml/UWS/v1.0

should remain as it is (UWS 1.0 stuff only), but a modified 'filename' URL will exist at:

http://www.ivoa.net/xml/UWS/UWS-v1.0.xsd

that has the "version=1.1" attribute within and the additions brought above by the 1.1 specification.

-- BrianMajor - 2016-03-11

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

CADC has implemented all new features introduced UWS 1.1 for all their UWS-based services including TAP:

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We implemented the new UWS1.1 features, i.e. support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase, WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added as well.

Example usage (using gaia.aip.de as example UWS1.1 service):

The newest updates (sort by creationTime, optional runId, ownerId and creationTime in shortjob-description of the job list) are supported in the version of branch uws11_update.

-- KristinRiebe - 2016-03-31

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use its jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sorted by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value, so they were just ignored.

This is now improved since sorting is done by a newly introduced 'creationTime'. -- KristinRiebe - 2016-03-31

The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, redirect with LAST still needs to be implemented.

Our datetimes are not returned in UTC, but with timezone; if UWS will require UTC datetimes in the future, I will adjust this. -- KristinRiebe - 2016-03-31

TOPCAT/STILTS

The asynchronous TAP clients in topcat and stilts (prerelease available) now use the blocking behaviour ( ?WAIT=-1&PHASE...=) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document. This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07

Implementations Validators

Validator's for UWS 1.1:

  • GAVO uws-validator: It's rather a definition of tests for checking the new functionality of UWS 1.1, i.e. mainly job list filtering and blocking behaviour. It is using behave (Python pacakge) and a definition of features and steps for testing the behaviour of UWS services. Some tests may still fail if I defined them too strict. Especially blocking behaviour may not behave as expected: my tests would fail but the service would still be valid. Tested with AIP's Daiquiri service (e.g. gaia.aip.de) and Gregory's TAP service at mintaka. Here are configuration files for these services: userconfig-gaia-aip.json (AIP Daiquiri service), userconfig-gaia-mintaka.json (CDS/GAVO TAP service).
    They can be used like this (needs additional installation of behave first):
    • For UWS 1.0 services only:
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-uws1_1
    • For the new UWS 1.1 functionality, excluding slow tests (waiting a long time):
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-slow --tags=-neverending
  • Any questions and comments are welcome! (Kristin Riebe, GAVO)
CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07 I'm removing this action on me :-), since UWS blocking is hard to validate, as it's defined to be backward-compatible and can always behave as if it was non-blocking -- MarkTaylor - 2016-02-17
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Added:
>
>		Approved.

-- PatrickDowler - 2016-05-12

 

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

I approve this document.

-- PierreFernique - 2016-04-29

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

I approve the document.

A few minor remarks for edition of the rec maybe :

The diagram with the objects as "phase" while the text has "Execution phase"

2.2 reads " In this first version of the UWS pattern only a REST (Representational State Transfer) style binding is presented". Is it really now the first version ?

As staed by Marco the text reads a 1.0 xsd ? is there any 1.1 associated to UWS 1.1 ? or is the xsd unchanged since first version ?

-- FrancoisBonnarel - 2016-05-09

There is a new XSD, but it has retained the namespace of the 1.0 version - the rationale behind this is explained in this note http://www.ivoa.net/documents/Notes/XMLVers/

-- PaulHarrison - 2016-05-10

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

I have given the document a light review and have no comments.. I approve this document.

-- MarkCresitelloDittmar - 2016-04-19

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

I approve this document.

-- BrianMajor - 2016-03-11

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

*

Given that clients have now demonstrated blocking mode, I approve this document with the caveat that a standards RegExt for UWS needs to be created separately.

-- TheresaDower - 2016-03-17

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

1- The last line of the document reads:

Changed:
<
<
$Revision: 1.33 $ $Date:
>
>
$Revision: 1.34 $ $Date:
  2015-09-02 17:41:42 +0100 (Wed, 02 Sep 2015) $ $HeadURL: https://volute.g-vo.org/svn/trunk/projects/grid/uws/doc/UWS.html$

is it needed in the standard document ?

  • It is not a requirement, but I think that it is useful for a future author to know where the source file is managed - I think that you will see these sorts of references in future in most of the documents that are authored using the IVOATeX preparation system
2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .

  • I will update the diagram
-- MireilleLouys - 2015-10-31 replies in bullets by PaulHarrison

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

I have tested a working implementation of UWS1.1 blocking, against Gregory Mantelet's mintaka TAP service on 2015-12-07. Good! The list of implementations now looks much more healthy too. Ops is now happy to see this progress to REC. -- MarkTaylor - 2016-02-17

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

META FILEATTACHMENT attachment="userconfig-gaia-mintaka.json" attr="" comment="" date="1456302914" name="userconfig-gaia-mintaka.json" path="userconfig-gaia-mintaka.json" size="743" user="KristinRiebe" version="1"
META FILEATTACHMENT attachment="userconfig-gaia-aip.json" attr="" comment="" date="1456303205" name="userconfig-gaia-aip.json" path="userconfig-gaia-aip.json" size="720" user="KristinRiebe" version="1"

Revision 342016-05-11 - BrianMajor

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Changed:
<
<
>
>
 

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.

The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version is available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site has also been upgraded to UWS 1.1.

Please, report any bug and/or missing feature you may found while testing it!

This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)

  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important

  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

Response to the above question:

According to the recently published XML schema versioning document http://ivoa.net/documents/Notes/XMLVers/20151029/NOTE-schemaVersioning-1.0-20151029.html#tth_sEc3, the document located at

http://www.ivoa.net/xml/UWS/v1.0

should remain as it is (UWS 1.0 stuff only), but a modified 'filename' URL will exist at:

http://www.ivoa.net/xml/UWS/UWS-v1.0.xsd

that has the "version=1.1" attribute within and the additions brought above by the 1.1 specification.

-- BrianMajor - 2016-03-11

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

CADC has implemented all new features introduced UWS 1.1 for all their UWS-based services including TAP:

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We implemented the new UWS1.1 features, i.e. support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase, WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added as well.

Example usage (using gaia.aip.de as example UWS1.1 service):

The newest updates (sort by creationTime, optional runId, ownerId and creationTime in shortjob-description of the job list) are supported in the version of branch uws11_update.

-- KristinRiebe - 2016-03-31

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use its jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sorted by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value, so they were just ignored.

This is now improved since sorting is done by a newly introduced 'creationTime'. -- KristinRiebe - 2016-03-31

The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, redirect with LAST still needs to be implemented.

Our datetimes are not returned in UTC, but with timezone; if UWS will require UTC datetimes in the future, I will adjust this. -- KristinRiebe - 2016-03-31

TOPCAT/STILTS

The asynchronous TAP clients in topcat and stilts (prerelease available) now use the blocking behaviour ( ?WAIT=-1&PHASE...=) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document. This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07

Implementations Validators

Validator's for UWS 1.1:

  • GAVO uws-validator: It's rather a definition of tests for checking the new functionality of UWS 1.1, i.e. mainly job list filtering and blocking behaviour. It is using behave (Python pacakge) and a definition of features and steps for testing the behaviour of UWS services. Some tests may still fail if I defined them too strict. Especially blocking behaviour may not behave as expected: my tests would fail but the service would still be valid. Tested with AIP's Daiquiri service (e.g. gaia.aip.de) and Gregory's TAP service at mintaka. Here are configuration files for these services: userconfig-gaia-aip.json (AIP Daiquiri service), userconfig-gaia-mintaka.json (CDS/GAVO TAP service).
    They can be used like this (needs additional installation of behave first):
    • For UWS 1.0 services only:
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-uws1_1
    • For the new UWS 1.1 functionality, excluding slow tests (waiting a long time):
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-slow --tags=-neverending
  • Any questions and comments are welcome! (Kristin Riebe, GAVO)
CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07 I'm removing this action on me :-), since UWS blocking is hard to validate, as it's defined to be backward-compatible and can always behave as if it was non-blocking -- MarkTaylor - 2016-02-17
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

I approve this document.

-- PierreFernique - 2016-04-29

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

I approve the document.

A few minor remarks for edition of the rec maybe :

The diagram with the objects as "phase" while the text has "Execution phase"

2.2 reads " In this first version of the UWS pattern only a REST (Representational State Transfer) style binding is presented". Is it really now the first version ?

As staed by Marco the text reads a 1.0 xsd ? is there any 1.1 associated to UWS 1.1 ? or is the xsd unchanged since first version ?

-- FrancoisBonnarel - 2016-05-09

There is a new XSD, but it has retained the namespace of the 1.0 version - the rationale behind this is explained in this note http://www.ivoa.net/documents/Notes/XMLVers/

-- PaulHarrison - 2016-05-10

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

I have given the document a light review and have no comments.. I approve this document.

-- MarkCresitelloDittmar - 2016-04-19

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

I approve this document.

-- BrianMajor - 2016-03-11

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

*

Given that clients have now demonstrated blocking mode, I approve this document with the caveat that a standards RegExt for UWS needs to be created separately.

-- TheresaDower - 2016-03-17

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

1- The last line of the document reads:

Changed:
<
<
$Revision: 1.32 $ $Date:
>
>
$Revision: 1.33 $ $Date:
  2015-09-02 17:41:42 +0100 (Wed, 02 Sep 2015) $ $HeadURL: https://volute.g-vo.org/svn/trunk/projects/grid/uws/doc/UWS.html$

is it needed in the standard document ?

  • It is not a requirement, but I think that it is useful for a future author to know where the source file is managed - I think that you will see these sorts of references in future in most of the documents that are authored using the IVOATeX preparation system
2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .

  • I will update the diagram
-- MireilleLouys - 2015-10-31 replies in bullets by PaulHarrison

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

I have tested a working implementation of UWS1.1 blocking, against Gregory Mantelet's mintaka TAP service on 2015-12-07. Good! The list of implementations now looks much more healthy too. Ops is now happy to see this progress to REC. -- MarkTaylor - 2016-02-17

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

META FILEATTACHMENT attachment="userconfig-gaia-mintaka.json" attr="" comment="" date="1456302914" name="userconfig-gaia-mintaka.json" path="userconfig-gaia-mintaka.json" size="743" user="KristinRiebe" version="1"
META FILEATTACHMENT attachment="userconfig-gaia-aip.json" attr="" comment="" date="1456303205" name="userconfig-gaia-aip.json" path="userconfig-gaia-aip.json" size="720" user="KristinRiebe" version="1"

Revision 332016-05-10 - PaulHarrison

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.

The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version is available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site has also been upgraded to UWS 1.1.

Please, report any bug and/or missing feature you may found while testing it!

This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)

  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important

  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

Response to the above question:

According to the recently published XML schema versioning document http://ivoa.net/documents/Notes/XMLVers/20151029/NOTE-schemaVersioning-1.0-20151029.html#tth_sEc3, the document located at

http://www.ivoa.net/xml/UWS/v1.0

should remain as it is (UWS 1.0 stuff only), but a modified 'filename' URL will exist at:

http://www.ivoa.net/xml/UWS/UWS-v1.0.xsd

that has the "version=1.1" attribute within and the additions brought above by the 1.1 specification.

-- BrianMajor - 2016-03-11

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

CADC has implemented all new features introduced UWS 1.1 for all their UWS-based services including TAP:

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We implemented the new UWS1.1 features, i.e. support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase, WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added as well.

Example usage (using gaia.aip.de as example UWS1.1 service):

The newest updates (sort by creationTime, optional runId, ownerId and creationTime in shortjob-description of the job list) are supported in the version of branch uws11_update.

-- KristinRiebe - 2016-03-31

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use its jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sorted by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value, so they were just ignored.

This is now improved since sorting is done by a newly introduced 'creationTime'. -- KristinRiebe - 2016-03-31

The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, redirect with LAST still needs to be implemented.

Our datetimes are not returned in UTC, but with timezone; if UWS will require UTC datetimes in the future, I will adjust this. -- KristinRiebe - 2016-03-31

TOPCAT/STILTS

The asynchronous TAP clients in topcat and stilts (prerelease available) now use the blocking behaviour ( ?WAIT=-1&PHASE...=) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document. This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07

Implementations Validators

Validator's for UWS 1.1:

  • GAVO uws-validator: It's rather a definition of tests for checking the new functionality of UWS 1.1, i.e. mainly job list filtering and blocking behaviour. It is using behave (Python pacakge) and a definition of features and steps for testing the behaviour of UWS services. Some tests may still fail if I defined them too strict. Especially blocking behaviour may not behave as expected: my tests would fail but the service would still be valid. Tested with AIP's Daiquiri service (e.g. gaia.aip.de) and Gregory's TAP service at mintaka. Here are configuration files for these services: userconfig-gaia-aip.json (AIP Daiquiri service), userconfig-gaia-mintaka.json (CDS/GAVO TAP service).
    They can be used like this (needs additional installation of behave first):
    • For UWS 1.0 services only:
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-uws1_1
    • For the new UWS 1.1 functionality, excluding slow tests (waiting a long time):
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-slow --tags=-neverending
  • Any questions and comments are welcome! (Kristin Riebe, GAVO)
CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07 I'm removing this action on me :-), since UWS blocking is hard to validate, as it's defined to be backward-compatible and can always behave as if it was non-blocking -- MarkTaylor - 2016-02-17
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

I approve this document.

-- PierreFernique - 2016-04-29

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

I approve the document.

A few minor remarks for edition of the rec maybe :

The diagram with the objects as "phase" while the text has "Execution phase"

Changed:
<
<		2.2 reads " In this first version of the UWS pattern only a REST (Representational
>
>		2.2 reads " In this first version of the UWS pattern only a REST (Representational State Transfer) style binding is presented". Is it really now the first version ?
Deleted:
<
<		State Transfer) style binding is presented". Is it really now the first version ?
  As staed by Marco the text reads a 1.0 xsd ? is there any 1.1 associated to UWS 1.1 ? or is the xsd unchanged since first version ?

-- FrancoisBonnarel - 2016-05-09

Added:
>
>		There is a new XSD, but it has retained the namespace of the 1.0 version - the rationale behind this is explained in this note http://www.ivoa.net/documents/Notes/XMLVers/

-- PaulHarrison - 2016-05-10

 

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

I have given the document a light review and have no comments.. I approve this document.

-- MarkCresitelloDittmar - 2016-04-19

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

I approve this document.

-- BrianMajor - 2016-03-11

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

*

Given that clients have now demonstrated blocking mode, I approve this document with the caveat that a standards RegExt for UWS needs to be created separately.

-- TheresaDower - 2016-03-17

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

1- The last line of the document reads:

Changed:
<
<
$Revision: 1.31 $ $Date:
>
>
$Revision: 1.32 $ $Date:
  2015-09-02 17:41:42 +0100 (Wed, 02 Sep 2015) $ $HeadURL: https://volute.g-vo.org/svn/trunk/projects/grid/uws/doc/UWS.html$

is it needed in the standard document ?

  • It is not a requirement, but I think that it is useful for a future author to know where the source file is managed - I think that you will see these sorts of references in future in most of the documents that are authored using the IVOATeX preparation system
2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .

  • I will update the diagram
-- MireilleLouys - 2015-10-31 replies in bullets by PaulHarrison

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

I have tested a working implementation of UWS1.1 blocking, against Gregory Mantelet's mintaka TAP service on 2015-12-07. Good! The list of implementations now looks much more healthy too. Ops is now happy to see this progress to REC. -- MarkTaylor - 2016-02-17

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

META FILEATTACHMENT attachment="userconfig-gaia-mintaka.json" attr="" comment="" date="1456302914" name="userconfig-gaia-mintaka.json" path="userconfig-gaia-mintaka.json" size="743" user="KristinRiebe" version="1"
META FILEATTACHMENT attachment="userconfig-gaia-aip.json" attr="" comment="" date="1456303205" name="userconfig-gaia-aip.json" path="userconfig-gaia-aip.json" size="720" user="KristinRiebe" version="1"

Revision 322016-05-09 - FrancoisBonnarel

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.

The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version is available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site has also been upgraded to UWS 1.1.

Please, report any bug and/or missing feature you may found while testing it!

This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)

  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important

  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

Response to the above question:

According to the recently published XML schema versioning document http://ivoa.net/documents/Notes/XMLVers/20151029/NOTE-schemaVersioning-1.0-20151029.html#tth_sEc3, the document located at

http://www.ivoa.net/xml/UWS/v1.0

should remain as it is (UWS 1.0 stuff only), but a modified 'filename' URL will exist at:

http://www.ivoa.net/xml/UWS/UWS-v1.0.xsd

that has the "version=1.1" attribute within and the additions brought above by the 1.1 specification.

-- BrianMajor - 2016-03-11

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

CADC has implemented all new features introduced UWS 1.1 for all their UWS-based services including TAP:

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We implemented the new UWS1.1 features, i.e. support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase, WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added as well.

Example usage (using gaia.aip.de as example UWS1.1 service):

The newest updates (sort by creationTime, optional runId, ownerId and creationTime in shortjob-description of the job list) are supported in the version of branch uws11_update.

-- KristinRiebe - 2016-03-31

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use its jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sorted by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value, so they were just ignored.

This is now improved since sorting is done by a newly introduced 'creationTime'. -- KristinRiebe - 2016-03-31

The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, redirect with LAST still needs to be implemented.

Our datetimes are not returned in UTC, but with timezone; if UWS will require UTC datetimes in the future, I will adjust this. -- KristinRiebe - 2016-03-31

TOPCAT/STILTS

The asynchronous TAP clients in topcat and stilts (prerelease available) now use the blocking behaviour ( ?WAIT=-1&PHASE...=) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document. This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07

Implementations Validators

Validator's for UWS 1.1:

  • GAVO uws-validator: It's rather a definition of tests for checking the new functionality of UWS 1.1, i.e. mainly job list filtering and blocking behaviour. It is using behave (Python pacakge) and a definition of features and steps for testing the behaviour of UWS services. Some tests may still fail if I defined them too strict. Especially blocking behaviour may not behave as expected: my tests would fail but the service would still be valid. Tested with AIP's Daiquiri service (e.g. gaia.aip.de) and Gregory's TAP service at mintaka. Here are configuration files for these services: userconfig-gaia-aip.json (AIP Daiquiri service), userconfig-gaia-mintaka.json (CDS/GAVO TAP service).
    They can be used like this (needs additional installation of behave first):
    • For UWS 1.0 services only:
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-uws1_1
    • For the new UWS 1.1 functionality, excluding slow tests (waiting a long time):
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-slow --tags=-neverending
  • Any questions and comments are welcome! (Kristin Riebe, GAVO)
CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07 I'm removing this action on me :-), since UWS blocking is hard to validate, as it's defined to be backward-compatible and can always behave as if it was non-blocking -- MarkTaylor - 2016-02-17
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

I approve this document.

-- PierreFernique - 2016-04-29

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Added:
>
>		I approve the document.

A few minor remarks for edition of the rec maybe :

The diagram with the objects as "phase" while the text has "Execution phase"

2.2 reads " In this first version of the UWS pattern only a REST (Representational State Transfer) style binding is presented". Is it really now the first version ?

As staed by Marco the text reads a 1.0 xsd ? is there any 1.1 associated to UWS 1.1 ? or is the xsd unchanged since first version ?

-- FrancoisBonnarel - 2016-05-09

 

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

I have given the document a light review and have no comments.. I approve this document.

-- MarkCresitelloDittmar - 2016-04-19

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

I approve this document.

-- BrianMajor - 2016-03-11

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

*

Given that clients have now demonstrated blocking mode, I approve this document with the caveat that a standards RegExt for UWS needs to be created separately.

-- TheresaDower - 2016-03-17

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

1- The last line of the document reads:

Changed:
<
<
$Revision: 1.30 $ $Date:
>
>
$Revision: 1.31 $ $Date:
  2015-09-02 17:41:42 +0100 (Wed, 02 Sep 2015) $ $HeadURL: https://volute.g-vo.org/svn/trunk/projects/grid/uws/doc/UWS.html$

is it needed in the standard document ?

  • It is not a requirement, but I think that it is useful for a future author to know where the source file is managed - I think that you will see these sorts of references in future in most of the documents that are authored using the IVOATeX preparation system
2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .

  • I will update the diagram
-- MireilleLouys - 2015-10-31 replies in bullets by PaulHarrison

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

I have tested a working implementation of UWS1.1 blocking, against Gregory Mantelet's mintaka TAP service on 2015-12-07. Good! The list of implementations now looks much more healthy too. Ops is now happy to see this progress to REC. -- MarkTaylor - 2016-02-17

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

META FILEATTACHMENT attachment="userconfig-gaia-mintaka.json" attr="" comment="" date="1456302914" name="userconfig-gaia-mintaka.json" path="userconfig-gaia-mintaka.json" size="743" user="KristinRiebe" version="1"
META FILEATTACHMENT attachment="userconfig-gaia-aip.json" attr="" comment="" date="1456303205" name="userconfig-gaia-aip.json" path="userconfig-gaia-aip.json" size="720" user="KristinRiebe" version="1"

Revision 312016-04-29 - PierreFernique

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.

The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version is available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site has also been upgraded to UWS 1.1.

Please, report any bug and/or missing feature you may found while testing it!

This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)

  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important

  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

Response to the above question:

According to the recently published XML schema versioning document http://ivoa.net/documents/Notes/XMLVers/20151029/NOTE-schemaVersioning-1.0-20151029.html#tth_sEc3, the document located at

http://www.ivoa.net/xml/UWS/v1.0

should remain as it is (UWS 1.0 stuff only), but a modified 'filename' URL will exist at:

http://www.ivoa.net/xml/UWS/UWS-v1.0.xsd

that has the "version=1.1" attribute within and the additions brought above by the 1.1 specification.

-- BrianMajor - 2016-03-11

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

CADC has implemented all new features introduced UWS 1.1 for all their UWS-based services including TAP:

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We implemented the new UWS1.1 features, i.e. support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase, WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added as well.

Example usage (using gaia.aip.de as example UWS1.1 service):

The newest updates (sort by creationTime, optional runId, ownerId and creationTime in shortjob-description of the job list) are supported in the version of branch uws11_update.

-- KristinRiebe - 2016-03-31

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use its jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sorted by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value, so they were just ignored.

This is now improved since sorting is done by a newly introduced 'creationTime'. -- KristinRiebe - 2016-03-31

The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, redirect with LAST still needs to be implemented.

Our datetimes are not returned in UTC, but with timezone; if UWS will require UTC datetimes in the future, I will adjust this. -- KristinRiebe - 2016-03-31

TOPCAT/STILTS

The asynchronous TAP clients in topcat and stilts (prerelease available) now use the blocking behaviour ( ?WAIT=-1&PHASE...=) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document. This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07

Implementations Validators

Validator's for UWS 1.1:

  • GAVO uws-validator: It's rather a definition of tests for checking the new functionality of UWS 1.1, i.e. mainly job list filtering and blocking behaviour. It is using behave (Python pacakge) and a definition of features and steps for testing the behaviour of UWS services. Some tests may still fail if I defined them too strict. Especially blocking behaviour may not behave as expected: my tests would fail but the service would still be valid. Tested with AIP's Daiquiri service (e.g. gaia.aip.de) and Gregory's TAP service at mintaka. Here are configuration files for these services: userconfig-gaia-aip.json (AIP Daiquiri service), userconfig-gaia-mintaka.json (CDS/GAVO TAP service).
    They can be used like this (needs additional installation of behave first):
    • For UWS 1.0 services only:
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-uws1_1
    • For the new UWS 1.1 functionality, excluding slow tests (waiting a long time):
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-slow --tags=-neverending
  • Any questions and comments are welcome! (Kristin Riebe, GAVO)
CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07 I'm removing this action on me :-), since UWS blocking is hard to validate, as it's defined to be backward-compatible and can always behave as if it was non-blocking -- MarkTaylor - 2016-02-17
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Added:
>
>		I approve this document.

-- PierreFernique - 2016-04-29

 

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

I have given the document a light review and have no comments.. I approve this document.

-- MarkCresitelloDittmar - 2016-04-19

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

I approve this document.

-- BrianMajor - 2016-03-11

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

*

Given that clients have now demonstrated blocking mode, I approve this document with the caveat that a standards RegExt for UWS needs to be created separately.

-- TheresaDower - 2016-03-17

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

1- The last line of the document reads:

Changed:
<
<
$Revision: 1.29 $ $Date:
>
>
$Revision: 1.30 $ $Date:
  2015-09-02 17:41:42 +0100 (Wed, 02 Sep 2015) $ $HeadURL: https://volute.g-vo.org/svn/trunk/projects/grid/uws/doc/UWS.html$

is it needed in the standard document ?

  • It is not a requirement, but I think that it is useful for a future author to know where the source file is managed - I think that you will see these sorts of references in future in most of the documents that are authored using the IVOATeX preparation system
2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .

  • I will update the diagram
-- MireilleLouys - 2015-10-31 replies in bullets by PaulHarrison

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

I have tested a working implementation of UWS1.1 blocking, against Gregory Mantelet's mintaka TAP service on 2015-12-07. Good! The list of implementations now looks much more healthy too. Ops is now happy to see this progress to REC. -- MarkTaylor - 2016-02-17

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

META FILEATTACHMENT attachment="userconfig-gaia-mintaka.json" attr="" comment="" date="1456302914" name="userconfig-gaia-mintaka.json" path="userconfig-gaia-mintaka.json" size="743" user="KristinRiebe" version="1"
META FILEATTACHMENT attachment="userconfig-gaia-aip.json" attr="" comment="" date="1456303205" name="userconfig-gaia-aip.json" path="userconfig-gaia-aip.json" size="720" user="KristinRiebe" version="1"

Revision 302016-04-19 - MarkCresitelloDittmar

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.

The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version is available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site has also been upgraded to UWS 1.1.

Please, report any bug and/or missing feature you may found while testing it!

This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)

  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important

  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

Response to the above question:

According to the recently published XML schema versioning document http://ivoa.net/documents/Notes/XMLVers/20151029/NOTE-schemaVersioning-1.0-20151029.html#tth_sEc3, the document located at

http://www.ivoa.net/xml/UWS/v1.0

should remain as it is (UWS 1.0 stuff only), but a modified 'filename' URL will exist at:

http://www.ivoa.net/xml/UWS/UWS-v1.0.xsd

that has the "version=1.1" attribute within and the additions brought above by the 1.1 specification.

-- BrianMajor - 2016-03-11

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

CADC has implemented all new features introduced UWS 1.1 for all their UWS-based services including TAP:

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We implemented the new UWS1.1 features, i.e. support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase, WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added as well.

Example usage (using gaia.aip.de as example UWS1.1 service):

The newest updates (sort by creationTime, optional runId, ownerId and creationTime in shortjob-description of the job list) are supported in the version of branch uws11_update.

-- KristinRiebe - 2016-03-31

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use its jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Changed:
<
<		Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sorted by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value, so they were just ignored.
>
>		Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sorted by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value, so they were just ignored.
  This is now improved since sorting is done by a newly introduced 'creationTime'. -- KristinRiebe - 2016-03-31

The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, redirect with LAST still needs to be implemented.

Our datetimes are not returned in UTC, but with timezone; if UWS will require UTC datetimes in the future, I will adjust this. -- KristinRiebe - 2016-03-31

TOPCAT/STILTS

The asynchronous TAP clients in topcat and stilts (prerelease available) now use the blocking behaviour ( ?WAIT=-1&PHASE...=) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document. This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07

Implementations Validators

Validator's for UWS 1.1:

  • GAVO uws-validator: It's rather a definition of tests for checking the new functionality of UWS 1.1, i.e. mainly job list filtering and blocking behaviour. It is using behave (Python pacakge) and a definition of features and steps for testing the behaviour of UWS services. Some tests may still fail if I defined them too strict. Especially blocking behaviour may not behave as expected: my tests would fail but the service would still be valid. Tested with AIP's Daiquiri service (e.g. gaia.aip.de) and Gregory's TAP service at mintaka. Here are configuration files for these services: userconfig-gaia-aip.json (AIP Daiquiri service), userconfig-gaia-mintaka.json (CDS/GAVO TAP service).
    They can be used like this (needs additional installation of behave first):
    • For UWS 1.0 services only:
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-uws1_1
    • For the new UWS 1.1 functionality, excluding slow tests (waiting a long time):
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-slow --tags=-neverending
  • Any questions and comments are welcome! (Kristin Riebe, GAVO)
CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07 I'm removing this action on me :-), since UWS blocking is hard to validate, as it's defined to be backward-compatible and can always behave as if it was non-blocking -- MarkTaylor - 2016-02-17
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Added:
>
>		I have given the document a light review and have no comments.. I approve this document.

-- MarkCresitelloDittmar - 2016-04-19

 

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

I approve this document.

-- BrianMajor - 2016-03-11

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

*

Given that clients have now demonstrated blocking mode, I approve this document with the caveat that a standards RegExt for UWS needs to be created separately.

-- TheresaDower - 2016-03-17

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

1- The last line of the document reads:

Changed:
<
<
$Revision: 1.28 $ $Date:
>
>
$Revision: 1.29 $ $Date:
  2015-09-02 17:41:42 +0100 (Wed, 02 Sep 2015) $ $HeadURL: https://volute.g-vo.org/svn/trunk/projects/grid/uws/doc/UWS.html$

is it needed in the standard document ?

  • It is not a requirement, but I think that it is useful for a future author to know where the source file is managed - I think that you will see these sorts of references in future in most of the documents that are authored using the IVOATeX preparation system
2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .

  • I will update the diagram
-- MireilleLouys - 2015-10-31 replies in bullets by PaulHarrison

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

I have tested a working implementation of UWS1.1 blocking, against Gregory Mantelet's mintaka TAP service on 2015-12-07. Good! The list of implementations now looks much more healthy too. Ops is now happy to see this progress to REC. -- MarkTaylor - 2016-02-17

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

META FILEATTACHMENT attachment="userconfig-gaia-mintaka.json" attr="" comment="" date="1456302914" name="userconfig-gaia-mintaka.json" path="userconfig-gaia-mintaka.json" size="743" user="KristinRiebe" version="1"
META FILEATTACHMENT attachment="userconfig-gaia-aip.json" attr="" comment="" date="1456303205" name="userconfig-gaia-aip.json" path="userconfig-gaia-aip.json" size="720" user="KristinRiebe" version="1"

Revision 292016-03-31 - KristinRiebe

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.

The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version is available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site has also been upgraded to UWS 1.1.

Please, report any bug and/or missing feature you may found while testing it!

This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)

  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important

  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

Response to the above question:

According to the recently published XML schema versioning document http://ivoa.net/documents/Notes/XMLVers/20151029/NOTE-schemaVersioning-1.0-20151029.html#tth_sEc3, the document located at

http://www.ivoa.net/xml/UWS/v1.0

should remain as it is (UWS 1.0 stuff only), but a modified 'filename' URL will exist at:

http://www.ivoa.net/xml/UWS/UWS-v1.0.xsd

that has the "version=1.1" attribute within and the additions brought above by the 1.1 specification.

-- BrianMajor - 2016-03-11

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

CADC has implemented all new features introduced UWS 1.1 for all their UWS-based services including TAP:

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

Changed:
<
<		The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added in branch 21_job_list_filter.
>
>		The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We implemented the new UWS1.1 features, i.e. support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase, WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added as well.
 
Changed:
<
<		Example usage (using gaia.aip.de as example UWS1.1 service):
>
>		Example usage (using gaia.aip.de as example UWS1.1 service):
Deleted:
<
<
 
Added:
>
>		The newest updates (sort by creationTime, optional runId, ownerId and creationTime in shortjob-description of the job list) are supported in the version of branch uws11_update.
 
Added:
>
>		-- KristinRiebe - 2016-03-31
 

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

Changed:
<
<		I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:
>
>		I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:
Deleted:
<
<
 
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
Changed:
<
<
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use it's jobid in the next two steps.
>
>
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use its jobid in the next two steps.
 
Changed:
<
<		Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sort by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value. Now they are just ignored. I'd like to have a sort by 'creationTime' in the future for these cases. The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, but does not do a LAST-redirect since it would then lose all PENDING and QUEUED jobs in the job list. See grid mailing list for more discussions on AFTER/LAST.
>
>		Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sorted by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value, so they were just ignored.
 
Added:
>
>		This is now improved since sorting is done by a newly introduced 'creationTime'. -- KristinRiebe - 2016-03-31

The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, redirect with LAST still needs to be implemented.

Our datetimes are not returned in UTC, but with timezone; if UWS will require UTC datetimes in the future, I will adjust this. -- KristinRiebe - 2016-03-31

 

TOPCAT/STILTS

The asynchronous TAP clients in topcat and stilts (prerelease available) now use the blocking behaviour ( ?WAIT=-1&PHASE...=) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document. This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07

Implementations Validators

Validator's for UWS 1.1:

Changed:
<
<
  • GAVO uws-validator: It's rather a definition of tests for checking the new functionality of UWS 1.1, i.e. mainly job list filtering and blocking behaviour. It is using behave (Python pacakge) and a definition of features and steps for testing the behaviour of UWS services. Some tests may still fail if I defined them too strict. Especially blocking behaviour may not behave as expected, my tests would fail but the service would still be valid. Tested with AIP's Daiquiri service (e.g. gaia.aip.de) and Gregory's TAP service at mintaka. Here are configuration files for these services: userconfig-gaia-aip.json (AIP Daiquiri service), userconfig-gaia-mintaka.json (CDS/GAVO TAP service).
    They can be used like this (needs additional installation of behave first):
>
>
  • GAVO uws-validator: It's rather a definition of tests for checking the new functionality of UWS 1.1, i.e. mainly job list filtering and blocking behaviour. It is using behave (Python pacakge) and a definition of features and steps for testing the behaviour of UWS services. Some tests may still fail if I defined them too strict. Especially blocking behaviour may not behave as expected: my tests would fail but the service would still be valid. Tested with AIP's Daiquiri service (e.g. gaia.aip.de) and Gregory's TAP service at mintaka. Here are configuration files for these services: userconfig-gaia-aip.json (AIP Daiquiri service), userconfig-gaia-mintaka.json (CDS/GAVO TAP service).
    They can be used like this (needs additional installation of behave first):
 
    • For UWS 1.0 services only:
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-uws1_1
    • For the new UWS 1.1 functionality, excluding slow tests (waiting a long time):
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-slow --tags=-neverending
  • Any questions and comments are welcome! (Kristin Riebe, GAVO)
CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07 I'm removing this action on me :-), since UWS blocking is hard to validate, as it's defined to be backward-compatible and can always behave as if it was non-blocking -- MarkTaylor - 2016-02-17
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

I approve this document.

-- BrianMajor - 2016-03-11

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

*

Given that clients have now demonstrated blocking mode, I approve this document with the caveat that a standards RegExt for UWS needs to be created separately.

-- TheresaDower - 2016-03-17

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

1- The last line of the document reads:

Changed:
<
<
$Revision: 1.27 $ $Date:
>
>
$Revision: 1.28 $ $Date:
  2015-09-02 17:41:42 +0100 (Wed, 02 Sep 2015) $ $HeadURL: https://volute.g-vo.org/svn/trunk/projects/grid/uws/doc/UWS.html$

is it needed in the standard document ?

  • It is not a requirement, but I think that it is useful for a future author to know where the source file is managed - I think that you will see these sorts of references in future in most of the documents that are authored using the IVOATeX preparation system
2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .

  • I will update the diagram
-- MireilleLouys - 2015-10-31 replies in bullets by PaulHarrison

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

I have tested a working implementation of UWS1.1 blocking, against Gregory Mantelet's mintaka TAP service on 2015-12-07. Good! The list of implementations now looks much more healthy too. Ops is now happy to see this progress to REC. -- MarkTaylor - 2016-02-17

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

META FILEATTACHMENT attachment="userconfig-gaia-mintaka.json" attr="" comment="" date="1456302914" name="userconfig-gaia-mintaka.json" path="userconfig-gaia-mintaka.json" size="743" user="KristinRiebe" version="1"
META FILEATTACHMENT attachment="userconfig-gaia-aip.json" attr="" comment="" date="1456303205" name="userconfig-gaia-aip.json" path="userconfig-gaia-aip.json" size="720" user="KristinRiebe" version="1"

Revision 282016-03-17 - TheresaDower

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.

The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version is available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site has also been upgraded to UWS 1.1.

Please, report any bug and/or missing feature you may found while testing it!

This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)

  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important

  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

Response to the above question:

According to the recently published XML schema versioning document http://ivoa.net/documents/Notes/XMLVers/20151029/NOTE-schemaVersioning-1.0-20151029.html#tth_sEc3, the document located at

http://www.ivoa.net/xml/UWS/v1.0

should remain as it is (UWS 1.0 stuff only), but a modified 'filename' URL will exist at:

http://www.ivoa.net/xml/UWS/UWS-v1.0.xsd

that has the "version=1.1" attribute within and the additions brought above by the 1.1 specification.

-- BrianMajor - 2016-03-11

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

CADC has implemented all new features introduced UWS 1.1 for all their UWS-based services including TAP:

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added in branch 21_job_list_filter.

Example usage (using gaia.aip.de as example UWS1.1 service):

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use it's jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sort by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value. Now they are just ignored. I'd like to have a sort by 'creationTime' in the future for these cases. The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, but does not do a LAST-redirect since it would then lose all PENDING and QUEUED jobs in the job list. See grid mailing list for more discussions on AFTER/LAST.

TOPCAT/STILTS

The asynchronous TAP clients in topcat and stilts (prerelease available) now use the blocking behaviour ( ?WAIT=-1&PHASE...=) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document. This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07

Implementations Validators

Validator's for UWS 1.1:

  • GAVO uws-validator: It's rather a definition of tests for checking the new functionality of UWS 1.1, i.e. mainly job list filtering and blocking behaviour. It is using behave (Python pacakge) and a definition of features and steps for testing the behaviour of UWS services. Some tests may still fail if I defined them too strict. Especially blocking behaviour may not behave as expected, my tests would fail but the service would still be valid. Tested with AIP's Daiquiri service (e.g. gaia.aip.de) and Gregory's TAP service at mintaka. Here are configuration files for these services: userconfig-gaia-aip.json (AIP Daiquiri service), userconfig-gaia-mintaka.json (CDS/GAVO TAP service).
    They can be used like this (needs additional installation of behave first):
    • For UWS 1.0 services only:
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-uws1_1
    • For the new UWS 1.1 functionality, excluding slow tests (waiting a long time):
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-slow --tags=-neverending
  • Any questions and comments are welcome! (Kristin Riebe, GAVO)
CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07 I'm removing this action on me :-), since UWS blocking is hard to validate, as it's defined to be backward-compatible and can always behave as if it was non-blocking -- MarkTaylor - 2016-02-17
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

I approve this document.

-- BrianMajor - 2016-03-11

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

Added:
>
>		*

Given that clients have now demonstrated blocking mode, I approve this document with the caveat that a standards RegExt for UWS needs to be created separately.

-- TheresaDower - 2016-03-17

 

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

1- The last line of the document reads:

Changed:
<
<
$Revision: 1.26 $ $Date:
>
>
$Revision: 1.27 $ $Date:
  2015-09-02 17:41:42 +0100 (Wed, 02 Sep 2015) $ $HeadURL: https://volute.g-vo.org/svn/trunk/projects/grid/uws/doc/UWS.html$

is it needed in the standard document ?

  • It is not a requirement, but I think that it is useful for a future author to know where the source file is managed - I think that you will see these sorts of references in future in most of the documents that are authored using the IVOATeX preparation system
Added:
>
>		2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .
 
Deleted:
<
<
2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .
 
  • I will update the diagram
Deleted:
<
<
 -- MireilleLouys - 2015-10-31 replies in bullets by PaulHarrison

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

I have tested a working implementation of UWS1.1 blocking, against Gregory Mantelet's mintaka TAP service on 2015-12-07. Good! The list of implementations now looks much more healthy too. Ops is now happy to see this progress to REC. -- MarkTaylor - 2016-02-17

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

META FILEATTACHMENT attachment="userconfig-gaia-mintaka.json" attr="" comment="" date="1456302914" name="userconfig-gaia-mintaka.json" path="userconfig-gaia-mintaka.json" size="743" user="KristinRiebe" version="1"
META FILEATTACHMENT attachment="userconfig-gaia-aip.json" attr="" comment="" date="1456303205" name="userconfig-gaia-aip.json" path="userconfig-gaia-aip.json" size="720" user="KristinRiebe" version="1"

Revision 272016-03-15 - PaulHarrison

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.

The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version is available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site has also been upgraded to UWS 1.1.

Please, report any bug and/or missing feature you may found while testing it!

This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)

  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important

  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

Response to the above question:

According to the recently published XML schema versioning document http://ivoa.net/documents/Notes/XMLVers/20151029/NOTE-schemaVersioning-1.0-20151029.html#tth_sEc3, the document located at

http://www.ivoa.net/xml/UWS/v1.0

should remain as it is (UWS 1.0 stuff only), but a modified 'filename' URL will exist at:

http://www.ivoa.net/xml/UWS/UWS-v1.0.xsd

that has the "version=1.1" attribute within and the additions brought above by the 1.1 specification.

-- BrianMajor - 2016-03-11

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

CADC has implemented all new features introduced UWS 1.1 for all their UWS-based services including TAP:

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added in branch 21_job_list_filter.

Example usage (using gaia.aip.de as example UWS1.1 service):

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use it's jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sort by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value. Now they are just ignored. I'd like to have a sort by 'creationTime' in the future for these cases. The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, but does not do a LAST-redirect since it would then lose all PENDING and QUEUED jobs in the job list. See grid mailing list for more discussions on AFTER/LAST.

TOPCAT/STILTS

The asynchronous TAP clients in topcat and stilts (prerelease available) now use the blocking behaviour ( ?WAIT=-1&PHASE...=) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document. This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07

Implementations Validators

Validator's for UWS 1.1:

  • GAVO uws-validator: It's rather a definition of tests for checking the new functionality of UWS 1.1, i.e. mainly job list filtering and blocking behaviour. It is using behave (Python pacakge) and a definition of features and steps for testing the behaviour of UWS services. Some tests may still fail if I defined them too strict. Especially blocking behaviour may not behave as expected, my tests would fail but the service would still be valid. Tested with AIP's Daiquiri service (e.g. gaia.aip.de) and Gregory's TAP service at mintaka. Here are configuration files for these services: userconfig-gaia-aip.json (AIP Daiquiri service), userconfig-gaia-mintaka.json (CDS/GAVO TAP service).
    They can be used like this (needs additional installation of behave first):
    • For UWS 1.0 services only:
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-uws1_1
    • For the new UWS 1.1 functionality, excluding slow tests (waiting a long time):
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-slow --tags=-neverending
  • Any questions and comments are welcome! (Kristin Riebe, GAVO)
CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07 I'm removing this action on me :-), since UWS blocking is hard to validate, as it's defined to be backward-compatible and can always behave as if it was non-blocking -- MarkTaylor - 2016-02-17
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

I approve this document.

-- BrianMajor - 2016-03-11

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

1- The last line of the document reads: 

$Revision: 1.26 $ $Date:
        2015-09-02 17:41:42 +0100 (Wed, 02 Sep 2015) $ $HeadURL:
        [[https://volute.g-vo.org/svn/trunk/][https://volute.g-vo.org/svn/trunk/]]projects/grid/uws/doc/UWS.html$

is it needed in the standard document ?

Added:
>
>
  • It is not a requirement, but I think that it is useful for a future author to know where the source file is managed - I think that you will see these sorts of references in future in most of the documents that are authored using the IVOATeX preparation system
 
2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .
Changed:
<
<		-- MireilleLouys - 2015-10-31
>
>
  • I will update the diagram
Added:
>
>		 -- MireilleLouys - 2015-10-31 replies in bullets by PaulHarrison
 

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

I have tested a working implementation of UWS1.1 blocking, against Gregory Mantelet's mintaka TAP service on 2015-12-07. Good! The list of implementations now looks much more healthy too. Ops is now happy to see this progress to REC. -- MarkTaylor - 2016-02-17

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

META FILEATTACHMENT attachment="userconfig-gaia-mintaka.json" attr="" comment="" date="1456302914" name="userconfig-gaia-mintaka.json" path="userconfig-gaia-mintaka.json" size="743" user="KristinRiebe" version="1"
META FILEATTACHMENT attachment="userconfig-gaia-aip.json" attr="" comment="" date="1456303205" name="userconfig-gaia-aip.json" path="userconfig-gaia-aip.json" size="720" user="KristinRiebe" version="1"

Revision 262016-03-12 - BrianMajor

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.

The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version is available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site has also been upgraded to UWS 1.1.

Please, report any bug and/or missing feature you may found while testing it!

This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)

  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important

  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

Added:
>
>		Response to the above question:

According to the recently published XML schema versioning document http://ivoa.net/documents/Notes/XMLVers/20151029/NOTE-schemaVersioning-1.0-20151029.html#tth_sEc3, the document located at

http://www.ivoa.net/xml/UWS/v1.0

should remain as it is (UWS 1.0 stuff only), but a modified 'filename' URL will exist at:

http://www.ivoa.net/xml/UWS/UWS-v1.0.xsd

that has the "version=1.1" attribute within and the additions brought above by the 1.1 specification.

-- BrianMajor - 2016-03-11

 

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

CADC has implemented all new features introduced UWS 1.1 for all their UWS-based services including TAP:

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added in branch 21_job_list_filter.

Example usage (using gaia.aip.de as example UWS1.1 service):

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use it's jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sort by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value. Now they are just ignored. I'd like to have a sort by 'creationTime' in the future for these cases. The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, but does not do a LAST-redirect since it would then lose all PENDING and QUEUED jobs in the job list. See grid mailing list for more discussions on AFTER/LAST.

TOPCAT/STILTS

The asynchronous TAP clients in topcat and stilts (prerelease available) now use the blocking behaviour ( ?WAIT=-1&PHASE...=) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document. This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07

Implementations Validators

Validator's for UWS 1.1:

  • GAVO uws-validator: It's rather a definition of tests for checking the new functionality of UWS 1.1, i.e. mainly job list filtering and blocking behaviour. It is using behave (Python pacakge) and a definition of features and steps for testing the behaviour of UWS services. Some tests may still fail if I defined them too strict. Especially blocking behaviour may not behave as expected, my tests would fail but the service would still be valid. Tested with AIP's Daiquiri service (e.g. gaia.aip.de) and Gregory's TAP service at mintaka. Here are configuration files for these services: userconfig-gaia-aip.json (AIP Daiquiri service), userconfig-gaia-mintaka.json (CDS/GAVO TAP service).
    They can be used like this (needs additional installation of behave first):
    • For UWS 1.0 services only:
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-uws1_1
    • For the new UWS 1.1 functionality, excluding slow tests (waiting a long time):
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-slow --tags=-neverending
  • Any questions and comments are welcome! (Kristin Riebe, GAVO)
CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07 I'm removing this action on me :-), since UWS blocking is hard to validate, as it's defined to be backward-compatible and can always behave as if it was non-blocking -- MarkTaylor - 2016-02-17
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Added:
>
>		I approve this document.

-- BrianMajor - 2016-03-11

 

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Deleted:
<
<

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

  1- The last line of the document reads:
Changed:
<
<
$Revision: 1.25 $ $Date:
>
>
$Revision: 1.26 $ $Date:
  2015-09-02 17:41:42 +0100 (Wed, 02 Sep 2015) $ $HeadURL: https://volute.g-vo.org/svn/trunk/projects/grid/uws/doc/UWS.html$

is it needed in the standard document ?


2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .

-- MireilleLouys - 2015-10-31

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

I have tested a working implementation of UWS1.1 blocking, against Gregory Mantelet's mintaka TAP service on 2015-12-07. Good! The list of implementations now looks much more healthy too. Ops is now happy to see this progress to REC. -- MarkTaylor - 2016-02-17

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

META FILEATTACHMENT attachment="userconfig-gaia-mintaka.json" attr="" comment="" date="1456302914" name="userconfig-gaia-mintaka.json" path="userconfig-gaia-mintaka.json" size="743" user="KristinRiebe" version="1"
META FILEATTACHMENT attachment="userconfig-gaia-aip.json" attr="" comment="" date="1456303205" name="userconfig-gaia-aip.json" path="userconfig-gaia-aip.json" size="720" user="KristinRiebe" version="1"

Revision 252016-02-24 - KristinRiebe

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.

The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version is available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site has also been upgraded to UWS 1.1.

Please, report any bug and/or missing feature you may found while testing it!
Deleted:
<
<
 
This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)

  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important

  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

CADC has implemented all new features introduced UWS 1.1 for all their UWS-based services including TAP:

Changed:
<
<
>
>
 

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added in branch 21_job_list_filter.

Example usage (using gaia.aip.de as example UWS1.1 service):

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use it's jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sort by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value. Now they are just ignored. I'd like to have a sort by 'creationTime' in the future for these cases. The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, but does not do a LAST-redirect since it would then lose all PENDING and QUEUED jobs in the job list. See grid mailing list for more discussions on AFTER/LAST.

TOPCAT/STILTS

The asynchronous TAP clients in topcat and stilts (prerelease available) now use the blocking behaviour ( ?WAIT=-1&PHASE...=) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document. This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07

Implementations Validators

Changed:
<
<		No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:
>
>		Validator's for UWS 1.1:
 
Added:
>
>
  • GAVO uws-validator: It's rather a definition of tests for checking the new functionality of UWS 1.1, i.e. mainly job list filtering and blocking behaviour. It is using behave (Python pacakge) and a definition of features and steps for testing the behaviour of UWS services. Some tests may still fail if I defined them too strict. Especially blocking behaviour may not behave as expected, my tests would fail but the service would still be valid. Tested with AIP's Daiquiri service (e.g. gaia.aip.de) and Gregory's TAP service at mintaka. Here are configuration files for these services: userconfig-gaia-aip.json (AIP Daiquiri service), userconfig-gaia-mintaka.json (CDS/GAVO TAP service).
    They can be used like this (needs additional installation of behave first):
    • For UWS 1.0 services only:
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-uws1_1
    • For the new UWS 1.1 functionality, excluding slow tests (waiting a long time):
      behave -D configfile="userconfig-gaia-mintaka.json" --no-skipped --tags=-slow --tags=-neverending
  • Any questions and comments are welcome! (Kristin Riebe, GAVO)
CADC (and others?) have UWS 1.0 validators that still pass:
 
Changed:
<
<
  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07 I'm removing this action on me :-), since UWS blocking is hard to validate, as it's defined to be backward-compatible and can always behave as if it was non-blocking -- MarkTaylor - 2016-02-17
  • uws-validator: using behave and a definition of features and steps for testing the behaviour of UWS services. Some tests may still fail if I defined them too strict. Tested with AIP's Daiquiri service (e.g. gaia.aip.de) and Gregory's TAP service at mintaka.
    You can find it here: uws-validator, questions and comments welcome. (Kristin Riebe, GAVO)
>
>
  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07 I'm removing this action on me :-), since UWS blocking is hard to validate, as it's defined to be backward-compatible and can always behave as if it was non-blocking -- MarkTaylor - 2016-02-17
Please add your validator (even if 1.0) if applicable.
Deleted:
<
<		Please add your validator (even if 1.0) if applicable.
 

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

1- The last line of the document reads:

Changed:
<
<
$Revision: 1.24 $ $Date:
>
>
$Revision: 1.25 $ $Date:
  2015-09-02 17:41:42 +0100 (Wed, 02 Sep 2015) $ $HeadURL: https://volute.g-vo.org/svn/trunk/projects/grid/uws/doc/UWS.html$

is it needed in the standard document ?


2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .

-- MireilleLouys - 2015-10-31

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

Changed:
<
<		I have tested a working implementation of UWS1.1 blocking, against Gregory Mantelet's mintaka TAP service on 2015-12-07. Good!
>
>		I have tested a working implementation of UWS1.1 blocking, against Gregory Mantelet's mintaka TAP service on 2015-12-07. Good! The list of implementations now looks much more healthy too. Ops is now happy to see this progress to REC. -- MarkTaylor - 2016-02-17
Deleted:
<
<		The list of implementations now looks much more healthy too. Ops is now happy to see this progress to REC. -- MarkTaylor - 2016-02-17
 

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->
Added:
>
>
META FILEATTACHMENT attachment="userconfig-gaia-mintaka.json" attr="" comment="" date="1456302914" name="userconfig-gaia-mintaka.json" path="userconfig-gaia-mintaka.json" size="743" user="KristinRiebe" version="1"
META FILEATTACHMENT attachment="userconfig-gaia-aip.json" attr="" comment="" date="1456303205" name="userconfig-gaia-aip.json" path="userconfig-gaia-aip.json" size="720" user="KristinRiebe" version="1"
 

Revision 242016-02-17 - MarkTaylor

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.

The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version is available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site has also been upgraded to UWS 1.1.

Please, report any bug and/or missing feature you may found while testing it!


This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)

  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important

  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

CADC has implemented all new features introduced UWS 1.1 for all their UWS-based services including TAP:

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added in branch 21_job_list_filter.

Example usage (using gaia.aip.de as example UWS1.1 service):

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use it's jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sort by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value. Now they are just ignored. I'd like to have a sort by 'creationTime' in the future for these cases. The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, but does not do a LAST-redirect since it would then lose all PENDING and QUEUED jobs in the job list. See grid mailing list for more discussions on AFTER/LAST.

TOPCAT/STILTS

The asynchronous TAP clients in topcat and stilts (prerelease available) now use the blocking behaviour ( ?WAIT=-1&PHASE...=) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document. This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

Changed:
<
<
  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07
>
>
  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07 I'm removing this action on me :-), since UWS blocking is hard to validate, as it's defined to be backward-compatible and can always behave as if it was non-blocking -- MarkTaylor - 2016-02-17
 
  • uws-validator: using behave and a definition of features and steps for testing the behaviour of UWS services. Some tests may still fail if I defined them too strict. Tested with AIP's Daiquiri service (e.g. gaia.aip.de) and Gregory's TAP service at mintaka.
    You can find it here: uws-validator, questions and comments welcome. (Kristin Riebe, GAVO)
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

1- The last line of the document reads:

Changed:
<
<
$Revision: 1.22 $ $Date:
>
>
$Revision: 1.24 $ $Date:
  2015-09-02 17:41:42 +0100 (Wed, 02 Sep 2015) $ $HeadURL: https://volute.g-vo.org/svn/trunk/projects/grid/uws/doc/UWS.html$

is it needed in the standard document ?


2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .

-- MireilleLouys - 2015-10-31

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

Added:
>
>		 I have tested a working implementation of UWS1.1 blocking, against Gregory Mantelet's mintaka TAP service on 2015-12-07. Good! The list of implementations now looks much more healthy too. Ops is now happy to see this progress to REC. -- MarkTaylor - 2016-02-17
 

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

Revision 232016-02-04 - BrianMajor

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.

The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version is available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site has also been upgraded to UWS 1.1.

Please, report any bug and/or missing feature you may found while testing it!


This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)

  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important

  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

Added:
>
>		CADC has implemented all new features introduced UWS 1.1 for all their UWS-based services including TAP:

 

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added in branch 21_job_list_filter.

Example usage (using gaia.aip.de as example UWS1.1 service):

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use it's jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sort by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value. Now they are just ignored. I'd like to have a sort by 'creationTime' in the future for these cases. The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, but does not do a LAST-redirect since it would then lose all PENDING and QUEUED jobs in the job list. See grid mailing list for more discussions on AFTER/LAST.

TOPCAT/STILTS

The asynchronous TAP clients in topcat and stilts (prerelease available) now use the blocking behaviour ( ?WAIT=-1&PHASE...=) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document. This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

Changed:
<
<
  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07
>
>
  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07
 
  • uws-validator: using behave and a definition of features and steps for testing the behaviour of UWS services. Some tests may still fail if I defined them too strict. Tested with AIP's Daiquiri service (e.g. gaia.aip.de) and Gregory's TAP service at mintaka.
    You can find it here: uws-validator, questions and comments welcome. (Kristin Riebe, GAVO)
Deleted:
<
<
 Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

1- The last line of the document reads:

Changed:
<
<
$Revision: 1.21 $ $Date:
>
>
$Revision: 1.22 $ $Date:
  2015-09-02 17:41:42 +0100 (Wed, 02 Sep 2015) $ $HeadURL: https://volute.g-vo.org/svn/trunk/projects/grid/uws/doc/UWS.html$

is it needed in the standard document ?


2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .

-- MireilleLouys - 2015-10-31

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

Revision 222015-12-16 - KristinRiebe

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Changed:
<
<
>
>
 

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.

The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version is available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site has also been upgraded to UWS 1.1.

Please, report any bug and/or missing feature you may found while testing it!


This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)

  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important

  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added in branch 21_job_list_filter.

Example usage (using gaia.aip.de as example UWS1.1 service):

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use it's jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sort by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value. Now they are just ignored. I'd like to have a sort by 'creationTime' in the future for these cases. The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, but does not do a LAST-redirect since it would then lose all PENDING and QUEUED jobs in the job list. See grid mailing list for more discussions on AFTER/LAST.

TOPCAT/STILTS

Changed:
<
<		The asynchronous TAP clients in topcat and stilts (prerelease available) now use the blocking behaviour (?WAIT=-1&PHASE=...) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document.
>
>		The asynchronous TAP clients in topcat and stilts (prerelease available) now use the blocking behaviour ( ?WAIT=-1&PHASE...=) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document. This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07
Deleted:
<
<		This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07
 

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

Changed:
<
<
  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07
>
>
  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07
Added:
>
>
  • uws-validator: using behave and a definition of features and steps for testing the behaviour of UWS services. Some tests may still fail if I defined them too strict. Tested with AIP's Daiquiri service (e.g. gaia.aip.de) and Gregory's TAP service at mintaka.
    You can find it here: uws-validator, questions and comments welcome. (Kristin Riebe, GAVO)
  Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Changed:
<
<		1- The last line of the document reads:
>
>		1- The last line of the document reads:
  
$Revision: 1.21 $ $Date:
        2015-09-02 17:41:42 +0100 (Wed, 02 Sep 2015) $ $HeadURL:
        [[https://volute.g-vo.org/svn/trunk/][https://volute.g-vo.org/svn/trunk/]]projects/grid/uws/doc/UWS.html$

is it needed in the standard document ?


2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .

-- MireilleLouys - 2015-10-31

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

Revision 212015-12-07 - MarkTaylor

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.

The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version is available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site has also been upgraded to UWS 1.1.

Please, report any bug and/or missing feature you may found while testing it!


This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)

  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important

  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added in branch 21_job_list_filter.

Example usage (using gaia.aip.de as example UWS1.1 service):

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use it's jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sort by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value. Now they are just ignored. I'd like to have a sort by 'creationTime' in the future for these cases. The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, but does not do a LAST-redirect since it would then lose all PENDING and QUEUED jobs in the job list. See grid mailing list for more discussions on AFTER/LAST.
Added:
>
>

TOPCAT/STILTS

The asynchronous TAP clients in topcat and stilts (prerelease available) now use the blocking behaviour (?WAIT=-1&PHASE=...) rather than repeated polls, for TAP services which identify themselves as UWS1.1 using the version attribute in the <job> document. This has been tested against Gregory Mantelet's TAP service implementation, and seems to be working fine. -- MarkTaylor - 2015-12-07

 

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

Changed:
<
<
  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03
>
>
  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03 ... OK, there is one now, action on me to update the validator -- MarkTaylor - 2015-12-07
Added:
>
>
 Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

1- The last line of the document reads:

Changed:
<
<
$Revision: 3052 $ $Date:
>
>
$Revision: 1.21 $ $Date:
  2015-09-02 17:41:42 +0100 (Wed, 02 Sep 2015) $ $HeadURL: https://volute.g-vo.org/svn/trunk/projects/grid/uws/doc/UWS.html$

is it needed in the standard document ?


2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .

-- MireilleLouys - 2015-10-31

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

Revision 202015-10-31 - MireilleLouys

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.

The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version is available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site has also been upgraded to UWS 1.1.

Please, report any bug and/or missing feature you may found while testing it!


This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)

  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important

  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added in branch 21_job_list_filter.

Example usage (using gaia.aip.de as example UWS1.1 service):

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use it's jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sort by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value. Now they are just ignored. I'd like to have a sort by 'creationTime' in the future for these cases. The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, but does not do a LAST-redirect since it would then lose all PENDING and QUEUED jobs in the job list. See grid mailing list for more discussions on AFTER/LAST.

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Added:
>
>

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

 
Added:
>
>		1- The last line of the document reads: 

$Revision: 3052 $ $Date:
        2015-09-02 17:41:42 +0100 (Wed, 02 Sep 2015) $ $HeadURL:
        [[https://volute.g-vo.org/svn/trunk/][https://volute.g-vo.org/svn/trunk/]]projects/grid/uws/doc/UWS.html$

is it needed in the standard document ?


2- Section 2.2.1.1.Determining the Protocol Version
refers to the VOSI specification , but the VOSI label not appear
on the architecture diagram .

-- MireilleLouys - 2015-10-31

 

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

Revision 192015-10-30 - GregoryMantelet

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Grégory Mantelet)

The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.
Changed:
<
<
The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version will be quite soon available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site will also be upgraded to UWS 1.1.
>
>
The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version is available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site has also been upgraded to UWS 1.1.
 
Please, report any bug and/or missing feature you may found while testing it!


This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)
Deleted:
<
<
 
  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important
Deleted:
<
<
 
  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added in branch 21_job_list_filter.

Example usage (using gaia.aip.de as example UWS1.1 service):

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use it's jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sort by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value. Now they are just ignored. I'd like to have a sort by 'creationTime' in the future for these cases. The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, but does not do a LAST-redirect since it would then lose all PENDING and QUEUED jobs in the job list. See grid mailing list for more discussions on AFTER/LAST.

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

Revision 182015-10-30 - GregoryMantelet

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

Changed:
<
<

CDS / GAVO Implementation (Gregory Mantelet)

>
>

CDS / GAVO Implementation (Grégory Mantelet)

Added:
>
>
The CDS/GAVO UWSLibrary is a Java framework designed to create and customize as easily as possible a UWS service. The last stable version of this library - targeting the UWS 1.0 standard - is available at http://cdsportal.u-strasbg.fr/uwstuto and the sources on Github at https://github.com/gmantele/taplib.

The implementation of the UWS 1.1 standard in this library seems now to be complete. Sources are available on the branch uws1.1 of Github. Since UWS 1.1 is still a Proposed Recommendation, no stable version of the library is for the moment released. However, a Beta version will be quite soon available at http://cdsportal.u-strasbg.fr/uwstuto/uws1.1Note. In addition, the demo provided on this web-site will also be upgraded to UWS 1.1.

Please, report any bug and/or missing feature you may found while testing it!


This beta version supports the following UWS1.1 features:

  • The new execution phase: ARCHIVED.
    • since no standard way is described in the UWS1.1 document to set a job into this phase, the library proposes 3 policies:
      • never (at destruction the job is just destroyed completely),
      • when the job reaches the destructionTime
      • and whenever the job is asked to be destroyed for the first time.
    • an archived job can be destroyed as any other job
    • an HTTP-GET request on /{jobs} does not list by default archived jobs ; it is therefore possible using job list filter (see below)
    • no result is retained in this phase. But input files and error summary are kept.
    • the former execution phase (i.e. the phase in which the job was before being ARCHIVED) is kept as additional information in the jobInfo XML node under the name "oldPhase" (this is hardcoded in the library, but it may be changed at any moment before the stable release if anyone notifies me on time)

  • Job list filters: PHASE, LAST and AFTER.
    • several PHASE filter may be specified ; if more than one is provided, all jobs in one of the given phase are returned (i.e. UNION of all PHASE filters).
    • using the PHASE filter with the value ARCHIVED is the only way to get the archived jobs
    • the LAST and AFTER filters work on the starting time (and not on the creation time as it could be intuitively expected)
    • only one AFTER filter is expected ; if several are provided, only the one with the most recent date is kept (which actually is the same as taking into account all AFTER filters)
    • only one LAST filter is expected ; if several are provided, only the one with the smallest value is kept (here again, it has actually the same behaviour as considering all LAST filters)
    • LAST is the only filter which performs a sorting of jobs ; sort by ascending starting time (from the first to the last started job ; from the oldest to the newest)
    • these 3 filters may be provided in the same time so that cumulating their effect
    • the character case (upper/lower/mixed case) is not important

  • Blocking behaviour for /{jobs}/{job-id} with the parameter WAIT and PHASE.
    • WAIT is mandatory and is taken into account by the library only if an integer value is provided
    • special values for WAIT are: 0=no blocking, <0=unlimited blocking time and >0=waiting time in seconds
    • if several WAIT parameters are provided, only the one equivalent to the smallest waiting time is kept
    • PHASE is optional, but if provided it represents the phase in which the job MUST be when the blocking is asked ; if not in the specified phase, no blocking is performed
    • if several PHASE parameters are provided, only the last legal occurrence is taken into account

  • Version attribute in the XML nodes <jobs> and <job> when they are the root document: version="1.1"

  • The new optional attributes "mime-type" and "size" for the XML nodes <result>
IMPORTANT NOTE: the XML schema for the XML representation of all UWS objects MUST be updated when the document goes into the "Recommended" state.

QUESTION: Will the XML schema still be available at http://www.ivoa.net/xml/UWS/v1.0? Maybe should it be at http://www.ivoa.net/xml/UWS/v1.1, no?

-- GregoryMantelet - 2015-10-30

 

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added in branch 21_job_list_filter.

Example usage (using gaia.aip.de as example UWS1.1 service):

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:

  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use it's jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sort by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value. Now they are just ignored. I'd like to have a sort by 'creationTime' in the future for these cases. The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, but does not do a LAST-redirect since it would then lose all PENDING and QUEUED jobs in the job list. See grid mailing list for more discussions on AFTER/LAST.

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

Revision 172015-10-27 - KristinRiebe

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Gregory Mantelet)

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added in branch 21_job_list_filter.

Example usage (using gaia.aip.de as example UWS1.1 service):

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

Changed:
<
<		I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQl server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:
>
>		I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQL server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:
 
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use it's jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Changed:
<
<		Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sort by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value and thus are ignored. I'd like to have a sort by 'creationTime' in the future for these cases. The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. See grid mailing list for more discussions on AFTER/LAST.
>
>		Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sort by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value. Now they are just ignored. I'd like to have a sort by 'creationTime' in the future for these cases. The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. The service only returns 1000 jobs by default, but does not do a LAST-redirect since it would then lose all PENDING and QUEUED jobs in the job list. See grid mailing list for more discussions on AFTER/LAST.
 

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

Revision 162015-10-26 - KristinRiebe

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Gregory Mantelet)

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

Changed:
<
<		The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords is being added in branch 21_job_list_filter.
>
>		The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords was added in branch 21_job_list_filter.
Added:
>
>		 Example usage (using gaia.aip.de as example UWS1.1 service):

 

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

Changed:
<
<		I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQl server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. We will make it available for one of our webservices early next week, so you can test it yourself.
>
>		I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQl server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. You can test it live at https://gaia.aip.de/uws/query, using the test user I just created with username: uwstest, password: gavo, or register for your own account. This allows you to test it e.g. with httpie like this:
Deleted:
<
<
<-- 
You can use the test version at http://dorado.aip.de:8097 for now (we plan to make it live on one of our websites at the beginning of next week.) I've created a test user with following credentials: username: uwstestuser, password: gavo. You can query the job list e.g. with httpie like this:
 
     
  •  http auth uwstestuser:gavo "http://dorado.aip.de:8097/uws/query?LAST=10" 
    •  
  •  http auth uwstestuser:gavo "http://dorado.aip.de:8097/uws/query?PHASE=ERROR&PHASE=PENDING" 
    •  
This version has no queuing system in the background, but you can check how WAIT works by requesting a WAIT=30 for a pending job and in the meantime starting it with PHASE=RUN:
 
     
  •  http auth uwstestuser:gavo form follow POST http://dorado.aip.de:8097/uws/query query="SELECT * FROM MDR1.FOF LIMIT 10"
     This creates a new pending job. Use it's jobid in the next two steps. 
    •  
  •  http auth uwstestuser:gavo "http://dorado.aip.de:8097/uws/query/{jobid}?WAIT=30&PHASE=PENDING 
    •  
  •  http auth uwstestuser:gavo form follow POST http://dorado.aip.de:8097/uws/query/{jobid}/phase PHASE=RUN. 
    •  
-->
 
Changed:
<
<		Notes on implementation: AFTER and LAST were a bit unclear because they sort by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value and thus are ignored. I'd like to have a sort by 'creationTime' in the future for these cases. The repeated PHASES keyword caused trouble with standard PHP query parsing (it requests PHASE[]=...&PHASE[]=...), but I solved this by parsing the request parameters manually. See grid mailing list for more discussions.
>
>
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?LAST=10"
Added:
>
>
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING"
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?PHASE=ERROR&PHASE=PENDING&LAST=5"
    This will remove all pending jobs from the list, since they do not have a startTime-value (LAST returns the most recently started jobs!)
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query?AFTER=2015-10-26T20:00"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query query="SELECT * FROM GUMS10.Galaxies LIMIT 10"
    This creates a new pending job. Use it's jobid in the next two steps.
  • http --auth uwstest:gavo "https://gaia.aip.de/uws/query/{jobid}?WAIT=30&PHASE=PENDING"
  • http --auth uwstest:gavo --form --follow POST https://gaia.aip.de/uws/query/{jobid}/phase PHASE=RUN
Notes on implementation: I like the new filter mechanisms and think they are really useful. AFTER and LAST were a bit unclear because they sort by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value and thus are ignored. I'd like to have a sort by 'creationTime' in the future for these cases. The repeated PHASES keyword caused trouble with standard PHP query parsing (it usually expects PHASE[]=...&PHASE[]=... if values shall be appended), but I solved this by parsing the request parameters manually. See grid mailing list for more discussions on AFTER/LAST.
 

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

Revision 152015-10-23 - KristinRiebe

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Gregory Mantelet)

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords is being added in branch 21_job_list_filter.

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

Changed:
<
<		I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQl server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. You can use the test version at http://dorado.aip.de:8097 for now (we plan to make it live on one of our websites at the beginning of next week.) I've created a test user with following credentials: username: uwstestuser, password: gavo. You can query the job list e.g. with httpie like this:
  • http --auth uwstestuser:gavo "http://localhost:8097/uws/query?LAST=10"
  • http --auth uwstestuser:gavo "http://localhost:8097/uws/query?PHASE=ERROR&PHASE=PENDING"
>
>		I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQl server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. We will make it available for one of our webservices early next week, so you can test it yourself. 
<-- 
You can use the test version at http://dorado.aip.de:8097 for now (we plan to make it live on one of our websites at the beginning of next week.) I've created a test user with following credentials: username: uwstestuser, password: gavo. You can query the job list e.g. with httpie like this:
 
     
  •  http auth uwstestuser:gavo "http://dorado.aip.de:8097/uws/query?LAST=10" 
    •  
  •  http auth uwstestuser:gavo "http://dorado.aip.de:8097/uws/query?PHASE=ERROR&PHASE=PENDING" 
    •  
This version has no queuing system in the background, but you can check how WAIT works by requesting a WAIT=30 for a pending job and in the meantime starting it with PHASE=RUN:
 
     
  •  http auth uwstestuser:gavo form follow POST http://dorado.aip.de:8097/uws/query query="SELECT * FROM MDR1.FOF LIMIT 10"
     This creates a new pending job. Use it's jobid in the next two steps. 
    •  
  •  http auth uwstestuser:gavo "http://dorado.aip.de:8097/uws/query/{jobid}?WAIT=30&PHASE=PENDING 
    •  
  •  http auth uwstestuser:gavo form follow POST http://dorado.aip.de:8097/uws/query/{jobid}/phase PHASE=RUN. 
    •  
-->
Deleted:
<
<		This version has no queuing system in the background, but you can check how WAIT works by requesting a WAIT=30 for a pending job and in the meantime starting it with PHASE=RUN:
  • http --auth uwstestuser:gavo --form --follow POST http://localhost:8097/uws/query query="SELECT * FROM MDR1.FOF LIMIT 10"
    This creates a new pending job. Use it's jobid in the next two steps.
  • http --auth uwstestuser:gavo "http://localhost:8097/uws/query/{jobid}?WAIT=30&PHASE=PENDING
  • http --auth uwstestuser:gavo --form --follow POST http://localhost:8097/uws/query/{jobid}/phase PHASE=RUN.
 Notes on implementation: AFTER and LAST were a bit unclear because they sort by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value and thus are ignored. I'd like to have a sort by 'creationTime' in the future for these cases. The repeated PHASES keyword caused trouble with standard PHP query parsing (it requests PHASE[]=...&PHASE[]=...), but I solved this by parsing the request parameters manually. See grid mailing list for more discussions.

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

Revision 142015-10-23 - KristinRiebe

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Gregory Mantelet)

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords is being added in branch 21_job_list_filter.

Added:
>
>

AIP Daiquiri - update to UWS 1.1 (Kristin Riebe)

I've updated our web application Daiquiri to include all (mandatory) UWS 1.1 extensions. It's not a full TAP server, it only supports UWS with a MySQl server in the background. The source code is available at https://github.com/aipescience/daiquiri/tree/uws_1_1. You can use the test version at http://dorado.aip.de:8097 for now (we plan to make it live on one of our websites at the beginning of next week.) I've created a test user with following credentials: username: uwstestuser, password: gavo. You can query the job list e.g. with httpie like this:

  • http --auth uwstestuser:gavo "http://localhost:8097/uws/query?LAST=10"
  • http --auth uwstestuser:gavo "http://localhost:8097/uws/query?PHASE=ERROR&PHASE=PENDING"
This version has no queuing system in the background, but you can check how WAIT works by requesting a WAIT=30 for a pending job and in the meantime starting it with PHASE=RUN:
  • http --auth uwstestuser:gavo --form --follow POST http://localhost:8097/uws/query query="SELECT * FROM MDR1.FOF LIMIT 10"
    This creates a new pending job. Use it's jobid in the next two steps.
  • http --auth uwstestuser:gavo "http://localhost:8097/uws/query/{jobid}?WAIT=30&PHASE=PENDING
  • http --auth uwstestuser:gavo --form --follow POST http://localhost:8097/uws/query/{jobid}/phase PHASE=RUN.
Notes on implementation: AFTER and LAST were a bit unclear because they sort by startTime; PENDING and QUEUED jobs and those aborted at these phases do not have a startTime value and thus are ignored. I'd like to have a sort by 'creationTime' in the future for these cases. The repeated PHASES keyword caused trouble with standard PHP query parsing (it requests PHASE[]=...&PHASE[]=...), but I solved this by parsing the request parameters manually. See grid mailing list for more discussions.
 

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

  • agreed - changed in latest version
Deleted:
<
<
 p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

  • have used xsi:nil in both places to be more precise about what is meant.
Deleted:
<
<
 2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

Changed:
<
<
  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
>
>
  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
Deleted:
<
<
 On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
Deleted:
<
<
 Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

  • I think that you may be one of the few people who has the tools available to do this!
Deleted:
<
<
 Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

Revision 132015-10-15 - PaulHarrison

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Gregory Mantelet)

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords is being added in branch 21_job_list_filter.

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

Added:
>
>
  • agreed - changed in latest version
 p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)
Added:
>
>
  • have used xsi:nil in both places to be more precise about what is meant.
 2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

Added:
>
>
  • actually the description of quote was the biggest inconsistency in REC1.0 - it was an error to descibe it in the table as being integer seconds as in the schema it was xs:dateTime. The text description mentions comparing with the Destruction Time (implying that quote is a dateTime) but in a contadiction also talks of returning a negative value to indicate "don't know"(I will remove this too!). The decision was to clarify in 1.1 and follow the schema - The quote always was intended to be a "wall-clock" time for the server to try to indicate how busy it though it was, however the usual caveat applies that it is almost impossible for a server to provide a reasonable estimate.
 On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

Added:
>
>
  • I am ideologically opposed to having the UWS capability derived from TAPRegExt as TAP derives partly from UWS and I do not like the circular relationships such a derivation would imply. In the original "grand vision" UWS can be used as a generalised pattern for any type of parameterised service beyond simply those that are part of DAL - as such the main piece of metadata for the service would be a reference to the PDL for the service - there was a start made to a specific UWSRegExt quite a time ago
 Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.
Added:
>
>
  • I think that you may be one of the few people who has the tools available to do this!
 Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.
Changed:
<
<		-- MarkusDemleitner - 2015-09-25
>
>		-- MarkusDemleitner - 2015-09-25 replies in bullets -- PaulHarrison - 2015-10-15
 

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

Revision 122015-10-09 - KristinRiebe

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Gregory Mantelet)

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

Changed:
<
<		The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included. The new version attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords is being added in branch 21_job_list_filter, the WAIT-keyword for blocking behaviour is being included with branch wait_blocking.
>
>		The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included as well as the WAIT keyword (with optional PHASE added) for the new blocking behaviour. The new "version" attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords is being added in branch 21_job_list_filter.
 

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

Revision 112015-10-07 - KristinRiebe

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Gregory Mantelet)

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

Changed:
<
<		The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included. Filtering using AFTER and LAST keywords is being added in branch 21_job_list_filter. The new version attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1.
>
>		The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included. The new version attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1. Filtering using AFTER and LAST keywords is being added in branch 21_job_list_filter, the WAIT-keyword for blocking behaviour is being included with branch wait_blocking.
 

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

Revision 102015-10-06 - KristinRiebe

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Gregory Mantelet)

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

Added:
>
>

GAVO uws-client Implementation (Kristin Riebe, Adrian Partl)

The uws-client at https://github.com/aipescience/uws-client is a Python tool to interact with UWS-services from the command line. We are implementing the UWS1.1 features; support for server-side PHASE-filtering (UWS standard section 2.2.2.1), including ARCHIVED-phase is already included. Filtering using AFTER and LAST keywords is being added in branch 21_job_list_filter. The new version attribute is parsed, the client switches from the new server-side filtering to default client-side phase-filtering, if the version attribute is missing or not set to 1.1.

 

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

Changed:
<
<		p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated
>
>		p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).
Deleted:
<
<		with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).
 
Changed:
<
<		p.9, "Indicated in the job XML as a Nill element": I believe this should
>
>		p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)
Deleted:
<
<		be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)
 
Changed:
<
<		2.1.11 I thought that during RFC I had ranted somewhere to the effect
>
>		2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.
Deleted:
<
<		that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.
 
Changed:
<
<		2.2.1 .../quote now returns an ISO timestamp. This differs from
>
>		2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.
Deleted:
<
<		REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.
 
Changed:
<
<		On p. 11 it says "This URI should be given as the access URL in the UWS
>
>		On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.
Deleted:
<
<		registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.
 
Changed:
<
<		Relatedly, I should have intervened a bit earlier because there's
>
>		Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.
Deleted:
<
<		actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.
 
Changed:
<
<		Has anyone already written a StandardsRegExt record for this standard?
>
>		Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.
Deleted:
<
<		If not and no one else steps forward, I volunteer to go ahead and draft one.
 
Changed:
<
<		Like Mark, I'd like to see a bit more details on the implementations and where
>
>		Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.
Deleted:
<
<		they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.
  -- MarkusDemleitner - 2015-09-25

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

Revision 92015-09-25 - MarkusDemleitner

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Gregory Mantelet)

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

Added:
>
>		p.7, paragraph "ARCHIVED: At...": I believe "but the metadata associated with the job must be retained" was actually intended to be "but the metadata associated with the job have been retained" (or I don't get what this is intended to say).

p.9, "Indicated in the job XML as a Nill element": I believe this should be "Nil" and that the double l in "nillable" is just morphological reduplication. (Also, Quote one paragraph down has "nil" with one l and in lower case; whatever you choose, it should be consistent)

2.1.11 I thought that during RFC I had ranted somewhere to the effect that one single way to change parameters should be made "canoical" and the other deprecated. This would greatly help the implementation of UWS generic clients. I still believe this would have been a good thing, though I admit introducing such a change during TCG review might be inapproriate.

2.2.1 .../quote now returns an ISO timestamp. This differs from REC-1.0, where it returns an integer number of seconds. Since I'm not aware of any client making good use of quote and few servers actually returning something meaningful there, I'd not consider this serious, 2.0-justifying breaking the standard, but it should certainly be noted in the change log. Incidentally, I'd rather reference DALI here, since the ISO standard actually allows an almost infinite number of serialisations, so standards trolls might be tempted to play games with the current formulation.

On p. 11 it says "This URI should be given as the access URL in the UWS registration," but then no further comment is made on how such a registration might actually look like. To foster registration of UWS services, I would volunteer to contribute an informative appendix with a sample registry record for a UWS service.

Relatedly, I should have intervened a bit earlier because there's actually stuff a special UWS capability should note (support for ARCHIVED, perhaps limits on blocking times, plus there's limits on execution duration and retention time as in TAPRegExt...). Actually, I wonder if we shouldn't extend TAPRegExt for that purpose. This would mean adding a couple of (optional) elements and making language and outputFormat optional. If you agree that'd be a good idea unless bad things result, all that it would take was "UWS services SHOULD use TAPRegExt 1.1 capabilities as those become available" somewhere near the existing registration language.

Has anyone already written a StandardsRegExt record for this standard? If not and no one else steps forward, I volunteer to go ahead and draft one.

Like Mark, I'd like to see a bit more details on the implementations and where they actually run. I'd also feel much better if the blocking mode were demonstrated with an actual client^W^W^WTOPCAT.

-- MarkusDemleitner - 2015-09-25

 

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Changed:
<
<		Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.
>
>		Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.
  There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).
Changed:
<
<		Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.
>
>		Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.
  -- MarkTaylor - 2015-09-11

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova)

Deleted:
<
<
  
<--  
-->

Revision 82015-09-11 - MarkTaylor

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.

CDS / GAVO Implementation (Gregory Mantelet)

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

TCG Review Period: 2015-09-09 - 2015-10-07

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03
Added:
>
>		Thanks Paul - fixes all look good. -- MarkTaylor - 2015-09-11
 

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs

Comments from TCG members during the TCG Review Period: 2015-09-09 - 2015-10-07

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel )

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

Registry Working Group ( _Markus Demleitner, Theresa Dower )

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Changed:
<
<

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

>
>

Operations Interest Group ( _Tom McGlynn, Mark Taylor )

Added:
>
>		 As I noted above, this is basically a good document.

However, the Implementations/Validators part of the RFC page still looks to me rather underpopulated for a document in TCG review.

Three implementations are listed, but the implementation authors have not as requested filled in details, so it's not clear what the status of these implementations is, or where to go to see the code or running UWS 1.1 services. In particular, it's important that the implementations conform to the version determination arrangements described in sec 2.2.1.1, which have been revised during this RFC period.

There are no validators listed that test UWS 1.1-specific functionality, partly (in taplint's case at least) because there are no services to validate (if service providers want to blame validators for this chicken-and-egg problem - OK then let me know and I'll write the validation code first).

Since TAP is currently the most important(?) UWS client standard, before recommending approval I'd really like to see a TAP service whose async endpoint implements the new UWS features (I'm especially personally interested in the WAIT blocking in accordance with sections 2.2.1.1 and 2.2.1.2). Possibly there is one out there, but (despite my request in the Implementations Validators section) it's not clear from the information on this page where it can be found. If no TAP service can be persuaded to do that, I'd at least like to see a running UWS service implementing the new features.

-- MarkTaylor - 2015-09-11

 

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )

Standards and Processes Committee ( Françoise Genova) 


<--  
-->

Revision 72015-09-09 - BrianMajor

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Changed:
<
<
>
>
 

Reference Interoperable Implementations

Changed:
<
<		Would the implementors of the UWS 1.1 specification please add details. Or, please remove your implementation from this list if not correct.
>
>		Would the implementors of the UWS 1.1 specification please add details; or, please remove your implementation from this list if not correct.
 

CDS / GAVO Implementation (Gregory Mantelet)

Deleted:
<
<
 

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18

Changed:
<
<
 

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Deleted:
<
<		Note that there is a new version of the document edited as a result of the comments by the end of the RFC period at https://volute.g-vo.org/svn/trunk/projects/grid/uws/doc/UWS.html, which has not yet been formally republished to the IVOA document store.

Example Comment (2015 June 30th) by BrianMajor

I comment on these three points:

  • point 1
  • point 2
  • point 3
 

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?

    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs
Changed:
<
<
  
<--  
-->

Revision 62015-09-03 - PaulHarrison

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details. Or, please remove your implementation from this list if not correct.

Deleted:
<
<

Astrogrid Implementation (Paul Harrison)

 

CDS / GAVO Implementation (Gregory Mantelet)

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03
Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18 

<-- REMOVE comment-out for TCG Review Period

 TCG Review Period: TCG_start_date - TCG_end_date 

-->

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Added:
>
>		Note that there is a new version of the document edited as a result of the comments by the end of the RFC period at https://volute.g-vo.org/svn/trunk/projects/grid/uws/doc/UWS.html, which has not yet been formally republished to the IVOA document store.
 Example Comment (2015 June 30th) by BrianMajor

I comment on these three points:

  • point 1
  • point 2
  • point 3

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

Changed:
<
<
  • Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
  • Schema: version attribute to be made optional? (see schema evolution discussion)
-- MarkTaylor - 2015-07-02
>
>		* Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?
    • there is no significance in the different types of braces - I have made them all curly.
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
    • I think you are right - I have removed it
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
    • you are right - and quote should be 0..1
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
    • I had not added these in the past as I thought it would complicate the diagram, as the conditions for transition into those states are more complex - you are probably right that they should be included though, and I have produced a version with them all in.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
    • highly unlikely - I have made the statement less precise about which binding may be added.
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
    • done
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
Added:
>
>
    • done
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
    • done
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
    • corrected
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
    • changed wording to make it clear that AstroGrid is no longer active
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
    • done
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
    • not really - have updated wording.
  • Schema: version attribute to be made optional? (see schema evolution discussion)
    • done in latest version of document and schema
-- MarkTaylor - 2015-07-02 -- Replies PaulHarrison - 2015-09-03
 

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME
Added:
>
>

Reply by PaulHarrison

Whilst I would normally be a strong advocate of trying to reuse existing syntax and not do re-invention, in this case the filtering capability is not intended to be a full query language for jobs. I think that your suggestions start to introduce the full query language approach which has the following disadvantages

  1. More effort has to be made in defining the query language than the current standard does - e.g. TOP in ADQL does not imply any ordering - by using the "keyword" style filtering that the current document specifies exactly what LAST means - so to be consistent with ADQL the "query language" should be something like /{jobs}?QUERY=TOP+100+ORDER+BY+STARTTIME+DES
  2. There is a greater burden on the implementors to do the query parsing.
It is for these reasons that I think that the "parameterized filter" style is more appropriate to a 1.1 update - if it turns out that there is demand for sophisticated querying then we can include a full ADQL query and associated schema in 2.0 - however in truth , I am still skeptical, as a UWS client should most of the time be keeping a local list of JOBIDs for the jobs that it submits and do all "queries" directly using the JOBID. The filtering mechanism in 1.1 is there just to make it easier for generic admin style web clients to not get overwhelmed with a long list of JOBs
 

<-- REMOVE comment-out for TCG Review Period

 Comments from TCG member during the TCG Review Period: 2014-10-01 - 2014-10-29 



WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.


IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.



 TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler ) 




 Applications Working Group ( _Pierre Fernique, Tom Donaldson ) 




 Data Access Layer Working Group ( François Bonnarel, Marco Molinaro ) 




 Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel ) 




 Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni ) 




 Registry Working Group ( _Markus Demleitner, Theresa Dower ) 




 Semantics Working Group ( _Mireille Louys, Alberto Accomazzi ) 




 Education Interest Group ( _Massimo Ramella, Sudhanshu Barway ) 




 Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick ) 




 Data Curation & Preservation Interest Group ( Françoise Genova ) 




 Operations Interest Group ( _Tom McGlynn, Mark Taylor ) 




 Knowledge Discovery in Databases Interest Group ( George Djorgovski ) 




 Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo ) 




 Standards and Processes Committee ( Françoise Genova) 



-->


<--  
-->

Revision 52015-08-14 - WalterLandry

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details. Or, please remove your implementation from this list if not correct.

Astrogrid Implementation (Paul Harrison)

CDS / GAVO Implementation (Gregory Mantelet)

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03
Deleted:
<
<
 Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18 

<-- REMOVE comment-out for TCG Review Period

 TCG Review Period: TCG_start_date - TCG_end_date 

-->

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Example Comment (2015 June 30th) by BrianMajor

I comment on these three points:

  • point 1
  • point 2
  • point 3

Comments from MarkTaylor

Changed:
<
<		(Still) a good clearly written document. One or two minor comments/suggestions:
>
>		(Still) a good clearly written document. One or two minor comments/suggestions:
 
Changed:
<
<
  • Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?
>
>
  • Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?
 
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
Changed:
<
<
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
>
>
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
 
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
Changed:
<
<
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
>
>
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
 
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
  • Schema: version attribute to be made optional? (see schema evolution discussion)
Deleted:
<
<
 -- MarkTaylor - 2015-07-02
Added:
>
>

Comments from WalterLandry

Regarding the new job query capability in Sec 2.2.2.1, it feels like
we are creating yet another query syntax with new keywords. To be
concrete, I would change the examples to

  • /{jobs}?QUERY=PHASE=EXECUTING
  • /{jobs}?QUERY=STARTTIME>2014-09-10T10:01:02.000
  • /{jobs}?QUERY=TOP+100
"QUERY" and "TOP" come from TAP, "STARTTIME" comes from the UWS
SCHEMA. For now, the spec will only specify simple queries. So there
will be only one parameter in QUERY. But this allows future
expansion, so a more sophisticated service could allow full SQL
  • /{jobs}?QUERY=(PHASE=EXECUTING+OR+PHASE=COMPLETED+OR+PHASE=SUSPENDED)+AND+STARTTIME>2014-09-10+ORDER+BY+ENDTIME
 

<-- REMOVE comment-out for TCG Review Period

 Comments from TCG member during the TCG Review Period: 2014-10-01 - 2014-10-29 



WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.


IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.



 TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler ) 




 Applications Working Group ( _Pierre Fernique, Tom Donaldson ) 




 Data Access Layer Working Group ( François Bonnarel, Marco Molinaro ) 




 Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel ) 




 Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni ) 




 Registry Working Group ( _Markus Demleitner, Theresa Dower ) 




 Semantics Working Group ( _Mireille Louys, Alberto Accomazzi ) 




 Education Interest Group ( _Massimo Ramella, Sudhanshu Barway ) 




 Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick ) 




 Data Curation & Preservation Interest Group ( Françoise Genova ) 




 Operations Interest Group ( _Tom McGlynn, Mark Taylor ) 




 Knowledge Discovery in Databases Interest Group ( George Djorgovski ) 




 Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo ) 




 Standards and Processes Committee ( Françoise Genova) 



-->


<--  
-->

Revision 42015-07-03 - MarkTaylor

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details. Or, please remove your implementation from this list if not correct.

Astrogrid Implementation (Paul Harrison)

CDS / GAVO Implementation (Gregory Mantelet)

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

Changed:
<
<		CADC UWS 1.0 Validator: OpenCADC cadcTestUWS
>
>
 
Changed:
<
<		taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage.
>
>
  • taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage. If someone can point me at a running UWS 1.1-compliant TAP service (minimum: WAIT-blocking and job documents labelled version='1.1' ) I'll try to update the validation for it. -- MarkTaylor - 2015-07-03
  Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18 

<-- REMOVE comment-out for TCG Review Period

 TCG Review Period: TCG_start_date - TCG_end_date 

-->

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Example Comment (2015 June 30th) by BrianMajor

I comment on these three points:

  • point 1
  • point 2
  • point 3

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

  • Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
  • Schema: version attribute to be made optional? (see schema evolution discussion)

-- MarkTaylor - 2015-07-02 

<-- REMOVE comment-out for TCG Review Period

 Comments from TCG member during the TCG Review Period: 2014-10-01 - 2014-10-29 



WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.


IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.



 TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler ) 




 Applications Working Group ( _Pierre Fernique, Tom Donaldson ) 




 Data Access Layer Working Group ( François Bonnarel, Marco Molinaro ) 




 Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel ) 




 Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni ) 




 Registry Working Group ( _Markus Demleitner, Theresa Dower ) 




 Semantics Working Group ( _Mireille Louys, Alberto Accomazzi ) 




 Education Interest Group ( _Massimo Ramella, Sudhanshu Barway ) 




 Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick ) 




 Data Curation & Preservation Interest Group ( Françoise Genova ) 




 Operations Interest Group ( _Tom McGlynn, Mark Taylor ) 




 Knowledge Discovery in Databases Interest Group ( George Djorgovski ) 




 Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo ) 




 Standards and Processes Committee ( Françoise Genova) 



-->


<--  
-->

Revision 32015-07-03 - MarkTaylor

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details. Or, please remove your implementation from this list if not correct.

Astrogrid Implementation (Paul Harrison)

CDS / GAVO Implementation (Gregory Mantelet)

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

CADC UWS 1.0 Validator: OpenCADC cadcTestUWS

taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage.

Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18 

<-- REMOVE comment-out for TCG Review Period

 TCG Review Period: TCG_start_date - TCG_end_date 

-->

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Example Comment (2015 June 30th) by BrianMajor

I comment on these three points:

  • point 1
  • point 2
  • point 3

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

  • Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?
Added:
>
>
  -- MarkTaylor - 2015-07-02 

<-- REMOVE comment-out for TCG Review Period

 Comments from TCG member during the TCG Review Period: 2014-10-01 - 2014-10-29 



WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.


IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.



 TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler ) 




 Applications Working Group ( _Pierre Fernique, Tom Donaldson ) 




 Data Access Layer Working Group ( François Bonnarel, Marco Molinaro ) 




 Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel ) 




 Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni ) 




 Registry Working Group ( _Markus Demleitner, Theresa Dower ) 




 Semantics Working Group ( _Mireille Louys, Alberto Accomazzi ) 




 Education Interest Group ( _Massimo Ramella, Sudhanshu Barway ) 




 Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick ) 




 Data Curation & Preservation Interest Group ( Françoise Genova ) 




 Operations Interest Group ( _Tom McGlynn, Mark Taylor ) 




 Knowledge Discovery in Databases Interest Group ( George Djorgovski ) 




 Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo ) 




 Standards and Processes Committee ( Françoise Genova) 



-->


<--  
-->

Revision 22015-07-02 - MarkTaylor

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details. Or, please remove your implementation from this list if not correct.

Astrogrid Implementation (Paul Harrison)

CDS / GAVO Implementation (Gregory Mantelet)

VO Paris Implementation (Mathieu Servillat)

Deleted:
<
<
 

CADC Implementation (Pat Dowler)

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

CADC UWS 1.0 Validator: OpenCADC cadcTestUWS

Added:
>
>		taplint TAP validator within stilts does non-exhaustive validation of the UWS v1.0 functionality of a TAP service: job submission, lifecycle tracking, deletion, job document parsing etc. See UWS Stage.
 Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18 

<-- REMOVE comment-out for TCG Review Period

 TCG Review Period: TCG_start_date - TCG_end_date 

-->

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Example Comment (2015 June 30th) by BrianMajor

I comment on these three points:

  • point 1
  • point 2
  • point 3
Added:
>
>

Comments from MarkTaylor

(Still) a good clearly written document. One or two minor comments/suggestions:

  • Notation: I don't understand the usage of curly and round brackets in referencing URL path elements. For instance in sec 2.1.11 I see: "/{jobs}/{job-id}/parameters/(parameter-name)", and in sec 2.2: "/(jobs)/(jobid)", and in sec 2.2.1.1: "/{jobs}/(job-id)". Are () and {} doing different, ahem, jobs here? If not, can you just pick one or the other?
  • Sec 1.3: "an XML document (e.g. ADQL, CEA [harrison05])" - is it anachronistic to refer to ADQL in this context since post-ADQL-2.0 there is no ADQL/X representation?
  • Sec 2.1: A count indicator ("1") is missing from the Job-Owner link in the UML diagram
  • Sec 2.1.3: The phases "HELD", "SUSPENDED" and "ARCHIVED" do not appear in the diagram.
  • Sec 2.2: "future versions of this document will add other bindings such as SOAP." Really?
  • Sec 2.2: "an non-existant job id" -> "a non-existent job-id"
  • Sec 2.2.1.1: "it should not return a response (i.e. block)" is a bit ambiguous. Prefer "it should not return a response (i.e. it should block)"?
  • Sec 2.2.1.1: "/{jobs}/(job-id)?WAIT&PHASE=QUEUED": I think it has to be "WAIT=<value>" not just "WAIT"?
  • Sec 4.2: "ADQL [1] can serve as a JDL." - bibliography referencing is wrong.
  • Sec 4.3: "Call it CEA v2 to distinguish it from CEA v1 as currently maintained by AstroGrid." - it's a bit anachronistic.
  • Sec 4.3: "The JDL in CEA v2 is similar to that in CEA v1 It is a formal ..." - missing full stop
  • Sec 4.3: "the emerging XForms technology" - is it still emerging?

-- MarkTaylor - 2015-07-02

 

<-- REMOVE comment-out for TCG Review Period

 Comments from TCG member during the TCG Review Period: 2014-10-01 - 2014-10-29 



WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.


IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.



 TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler ) 




 Applications Working Group ( _Pierre Fernique, Tom Donaldson ) 




 Data Access Layer Working Group ( François Bonnarel, Marco Molinaro ) 




 Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel ) 




 Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni ) 




 Registry Working Group ( _Markus Demleitner, Theresa Dower ) 




 Semantics Working Group ( _Mireille Louys, Alberto Accomazzi ) 




 Education Interest Group ( _Massimo Ramella, Sudhanshu Barway ) 




 Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick ) 




 Data Curation & Preservation Interest Group ( Françoise Genova ) 




 Operations Interest Group ( _Tom McGlynn, Mark Taylor ) 




 Knowledge Discovery in Databases Interest Group ( George Djorgovski ) 




 Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo ) 




 Standards and Processes Committee ( Françoise Genova) 



-->


<--  
-->

Revision 12015-06-30 - BrianMajor

 
META TOPICPARENT name="IvoaGridAndWebServices"

UWS v1.1 Proposed Recommendation: Request for Comments

Public discussion page for the IVOA UWS 1.1 Proposed Recommendation.

The latest version of the UWS Specification can be found at:

Reference Interoperable Implementations

Would the implementors of the UWS 1.1 specification please add details. Or, please remove your implementation from this list if not correct.

Astrogrid Implementation (Paul Harrison)

CDS / GAVO Implementation (Gregory Mantelet)

VO Paris Implementation (Mathieu Servillat)

CADC Implementation (Pat Dowler)

Implementations Validators

No implementation validators for UWS 1.1, though CADC (and others?) have UWS 1.0 validators that still pass:

CADC UWS 1.0 Validator: OpenCADC cadcTestUWS

Please add your validator (even if 1.0) if applicable.

RFC Review Period: 2015-06-30 - 2015-08-18 

<-- REMOVE comment-out for TCG Review Period

 TCG Review Period: TCG_start_date - TCG_end_date 

-->

Comments from the IVOA Community during RFC period: 2015-06-30 - 2015-08-18

Example Comment (2015 June 30th) by BrianMajor

I comment on these three points:

  • point 1
  • point 2
  • point 3 

<-- REMOVE comment-out for TCG Review Period

 Comments from TCG member during the TCG Review Period: 2014-10-01 - 2014-10-29 



WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or do not approve of the Standard.


IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.



 TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler ) 




 Applications Working Group ( _Pierre Fernique, Tom Donaldson ) 




 Data Access Layer Working Group ( François Bonnarel, Marco Molinaro ) 




 Data Model Working Group ( _Mark Cresitello-Dittmar, Laurent Michel ) 




 Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni ) 




 Registry Working Group ( _Markus Demleitner, Theresa Dower ) 




 Semantics Working Group ( _Mireille Louys, Alberto Accomazzi ) 




 Education Interest Group ( _Massimo Ramella, Sudhanshu Barway ) 




 Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick ) 




 Data Curation & Preservation Interest Group ( Françoise Genova ) 




 Operations Interest Group ( _Tom McGlynn, Mark Taylor ) 




 Knowledge Discovery in Databases Interest Group ( George Djorgovski ) 




 Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo ) 




 Standards and Processes Committee ( Françoise Genova) 



-->


<--  
-->
View topic | History: r36 < r35 < r34 < r33 | More topic actions...
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2025 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback