Difference: VOSpace20140918 (1 vs. 6)

From the mailing list announcement :

• Summary of recent changes
• Line by line diff

General

HTML anchors

The HTML anchors use name rather than id, is this intentional ?

    <a name="0.1_sec1">1. Introduction</a>

rather than

    <a id="0.1_sec1">1. Introduction</a>

-- BrianMajor - 05 Oct 2014 - No, this is not intentional. id is a better choice for the anchors. I will address this on conversion to a more easily editable document format (xhtml / ivotex).

Section 1.1

Create node path

In the create node operation described in section 1.1, the path encoded in the URL of the HTTP PUT request

    http://nvo.caltech.edu/vospace/myData/table123

does not match the path in the message content

    vos://nvo.caltech!vospace/mytable1

In addition, the URL of the HTTP PUT request does not point to a location within the {service}/nodes tree defined in the specification

If the endpoint for the VOSpace service is

    http://nvo.caltech.edu/vospace/

then the target URL for the HTTP PUT request should be

    http://nvo.caltech.edu/vospace/nodes/myData/table123

and the uri in the node XML should either be blank, or it should match the path in the HTTP PUT target URL

    vos://nvo.caltech!vospace/myData/table123

Create node text

There are a number of places where the description of the create operation could be clarified.

In the original VOSpace-1.x SOAP service, the create message contained the target URI as part of the message body.

The VOSpace-2.x REST service create message is essentially the same XML message content, but the request is sent to a location inside the {service}/nodes tree with target node path embeded in the URL.

    {service}/nodes/{path}

This means the node path is encoded in two different places within the same message.

• The current specification does not clearly state how the intended node path should be included in the target URL of the HTTP PUT request.
• The current specification does not clearly state whether or not the node uri should be included in the content of the create node message.
• The current specification does not clearly state whether or not the service should return an error if the path in the uri in the message content does not match the path in the target URL of the HTTP PUT request

Push transfer endpoint

In the description of the data transfer in section 1.1, the endpoint URL for the HTTP POST appears to point to a location relative to the target node

    http://nvo.caltech.edu/vospace/myData/table123/transfers

In VOSpace-2.x transfer operations should be initiated by a HTTP POST to a separate UWS {service}/transfers endpoint, outside the {service}/nodes tree.

If the endpoint for the VOSpace service is

    http://nvo.caltech.edu/vospace/

then the endpoint URL for initiating a transfer should be

    http://nvo.caltech.edu/vospace/transfers

Push transfer target

The path of the target in the body of the HTTP POST message should match the path of the node created in the first step of the example

    <transfer xmlns="http://www.ivoa.net/xml/VOSpace/v2.1">
        ....
        <target>vos://nvo.caltech!vospace/myData/table123</target>
        ....
    </transfer>

Push transfer protocol

The content for the transfer request given in the example uses a non-standard protocol URI

    <transfer ...>
        ....
        <protocol uri="ivo://ivoa.net/vospace/core#http-put"/>  
        ....
    </transfer>

The standard protocol URI for HTTP PUT transfers listed in section 3.5.3 of the same document is

    ivo://ivoa.net/vospace/core#httpput

Push transfer redirect

The description of the data transfer in section 1.1, misses out a step in the sequence of requests.

Following the initial HTTP POST to the {service}/transfers endpoint, the current text implies that the server responds with the HTTP PUT destination directly in the response.

"The service will reply with the URL that the user will HTTP PUT the data file"

This omits the details of the 303 redirect response

    HTTP/1.1 303 See Other
    ....
    Location: http://nvo.caltech.edu/vospace/transfers/581

Which directs the client to a representation of the UWS job, which contains the HTTP PUT destination that the client should send the data to

    <uws:job ....>
        <uws:jobId>581</uws:jobId>
        ....
        <uws:jobInfo>
            <vos:transfer>
                ....
                <vos:protocol uri="ivo://ivoa.net/vospace/core#http-put">
                    <vos:endpoint>http://d21.caltech.edu/temp/d558</vos:endpoint>
                </vos:protocol>
                ....
            </vos:transfer>
        <uws:jobInfo>
        ....
    </uws:job>

In addition, the example URL for the HTTP PUT destination,

    http://nvo.caltech.edu/bvospace/myData/table123/transfers/147516ab

appears to point to a location relative to the target node.

It would be clearer if the example used a URL that did not overlap with the VOSpace service endpoint in any way

    http://d21.caltech.edu/temp/d558

Pull transfer endpoint

The endpoint URL for the HTTP POST should point to the global {service}/transfers endpoint, not to what appears to be a location relative to the target node within the node tree.

If the endpoint for the VOSpace service is

    http://nvo.caltech.edu/vospace/

then the target URL to initate a transfer should be

    http://nvo.caltech.edu/vospace/transfers

Pull transfer target

The path of the target in the body of the HTTP POST message should match the path of the node created in the first step of the example.

    <transfer xmlns="http://www.ivoa.net/xml/VOSpace/v2.1">
        ....
        <target>vos://nvo.caltech!vospace/myData/table123</target>
        ....
    </transfer>

Pull transfer redirect

The current text implies that the server responds directly with the HTTP GET source URL that the client should use to fetch the data from.

This omits the details of the 303 redirect response

    HTTP/1.1 303 See Other
    ....
    Location: http://nvo.caltech.edu/vospace/transfers/581

which directs the client to a representation of the UWS job, which contains the HTTP GET location that client can fetch the data from.

In addition, the example URL for the HTTP GET location to transfer the data

    http://nvo.caltech.edu/bvospace/myData/table123/transfers/147516ab

appears to point to a location relative to the target node within the {service}/node tree.

It would be clearer if the example used a URL that did not overlap with the VOSpace service endpoint in any way.

    http://d22.caltech.edu/temp/e612

Push to VOSpace sequence

In section 5.4.1.4 Example: pushToVoSpace (missing section header) The text in the example suggests that the client should submit the transfer job, then poll the job details, waiting for the job to COMPLETE and then using the URL in 'transferDetails' result to fetch the transfer details, which contain the endpoint URL to send the data to.

I don't think that is the correct sequence.

Based on the way the original SOAP service worked, I think the correct pushToVoSpace sequence should be :

1. The client submits the job, and follows the redirect to the job details, but there is no need to poll the job status, because the server is now waiting for the client to send the data.
2. The job status SHOULD should be PENDING to indicate that the server is waiting for some data.
3. The job status MUST contain the full details of the vos:transfer in the uws:jobInfo section, including target URL(s) for each of the vos:protocol elements that the server is offering.
4. It is up to the client to pick one of the target URL(s) and transfer the data (pushing the data to the VOSpace).
5. During the transfer, the server SHOULD update the job status to RUNNING.
6. The server MAY decide to reject additional transfers once a transfer has succeed.
7. The server MAY get update the job status to FAILED if no data has been sent after a time limit has expired.
8. If a transfers fails, the client MAY try again if the server offered more than one vos:protocol target URL.
9. The client SHOULD set the job status to FAILED if it is unable to transfer the data.
10. The server MUST update the job status to FAILED if the data that arrives is not valid (see http://mail.ivoa.net/pipermail/grid/2014-September/002649.html).
11. The server MUST update the job status to COMPLETED once a transfer has succeed.
12. After the data has been sent the client SHOULD check that the final job status is set COMPLETED to verify that the server has accepted the data as valid.

Optional .auto

The list of changes state that .auto is being made optional. However many sections of the document still contain the original text describing how .auto applies, without describing how it can be made optional. If it is optional, then the specification needs to describe a mechanism for a client to discover whether a particular service supports it or not, and how this effects the behaviour in each case where it is applicable.

However, I think making it optional causes too many side effects and inconsistent behaviours between different spaces. As the original sponsor of the .auto functionality I'm happy for it to be removed completely.

And .null

If we are going to remove .auto, then .null should probably go too. There are better ways of specifying a null value for a direction.

Neutral URL examples

All our documents should probably use the reserved 'example' domain names for URLs. https://tools.ietf.org/html/rfc2606

From the mailing list announcement :

• Summary of recent changes
• Line by line diff

General

HTML anchors

The HTML anchors use name rather than id, is this intentional ?

    <a name="0.1_sec1">1. Introduction</a>

rather than

    <a id="0.1_sec1">1. Introduction</a>

Section 1.1

Create node path

In the create node operation described in section 1.1, the path encoded in the URL of the HTTP PUT request

    http://nvo.caltech.edu/vospace/myData/table123

does not match the path in the message content

    vos://nvo.caltech!vospace/mytable1

In addition, the URL of the HTTP PUT request does not point to a location within the {service}/nodes tree defined in the specification

If the endpoint for the VOSpace service is

    http://nvo.caltech.edu/vospace/

then the target URL for the HTTP PUT request should be

    http://nvo.caltech.edu/vospace/nodes/myData/table123

and the uri in the node XML should either be blank, or it should match the path in the HTTP PUT target URL

    vos://nvo.caltech!vospace/myData/table123

Create node text

There are a number of places where the description of the create operation could be clarified.

In the original VOSpace-1.x SOAP service, the create message contained the target URI as part of the message body.

The VOSpace-2.x REST service create message is essentially the same XML message content, but the request is sent to a location inside the {service}/nodes tree with target node path embeded in the URL.

    {service}/nodes/{path}

This means the node path is encoded in two different places within the same message.

• The current specification does not clearly state how the intended node path should be included in the target URL of the HTTP PUT request.
• The current specification does not clearly state whether or not the node uri should be included in the content of the create node message.
• The current specification does not clearly state whether or not the service should return an error if the path in the uri in the message content does not match the path in the target URL of the HTTP PUT request

Push transfer endpoint

In the description of the data transfer in section 1.1, the endpoint URL for the HTTP POST appears to point to a location relative to the target node

    http://nvo.caltech.edu/vospace/myData/table123/transfers

In VOSpace-2.x transfer operations should be initiated by a HTTP POST to a separate UWS {service}/transfers endpoint, outside the {service}/nodes tree.

If the endpoint for the VOSpace service is

    http://nvo.caltech.edu/vospace/

then the endpoint URL for initiating a transfer should be

    http://nvo.caltech.edu/vospace/transfers

Push transfer target

The path of the target in the body of the HTTP POST message should match the path of the node created in the first step of the example

    <transfer xmlns="http://www.ivoa.net/xml/VOSpace/v2.1">
        ....
        <target>vos://nvo.caltech!vospace/myData/table123</target>
        ....
    </transfer>

Push transfer protocol

The content for the transfer request given in the example uses a non-standard protocol URI

    <transfer ...>
        ....
        <protocol uri="ivo://ivoa.net/vospace/core#http-put"/>  
        ....
    </transfer>

The standard protocol URI for HTTP PUT transfers listed in section 3.5.3 of the same document is

    ivo://ivoa.net/vospace/core#httpput

Push transfer redirect

The description of the data transfer in section 1.1, misses out a step in the sequence of requests.

Following the initial HTTP POST to the {service}/transfers endpoint, the current text implies that the server responds with the HTTP PUT destination directly in the response.

"The service will reply with the URL that the user will HTTP PUT the data file"

This omits the details of the 303 redirect response

    HTTP/1.1 303 See Other
    ....
    Location: http://nvo.caltech.edu/vospace/transfers/581

Which directs the client to a representation of the UWS job, which contains the HTTP PUT destination that the client should send the data to

    <uws:job ....>
        <uws:jobId>581</uws:jobId>
        ....
        <uws:jobInfo>
            <vos:transfer>
                ....
                <vos:protocol uri="ivo://ivoa.net/vospace/core#http-put">
                    <vos:endpoint>http://d21.caltech.edu/temp/d558</vos:endpoint>
                </vos:protocol>
                ....
            </vos:transfer>
        <uws:jobInfo>
        ....
    </uws:job>

In addition, the example URL for the HTTP PUT destination,

    http://nvo.caltech.edu/bvospace/myData/table123/transfers/147516ab

appears to point to a location relative to the target node.

It would be clearer if the example used a URL that did not overlap with the VOSpace service endpoint in any way

    http://d21.caltech.edu/temp/d558

Pull transfer endpoint

The endpoint URL for the HTTP POST should point to the global {service}/transfers endpoint, not to what appears to be a location relative to the target node within the node tree.

If the endpoint for the VOSpace service is

    http://nvo.caltech.edu/vospace/

then the target URL to initate a transfer should be

    http://nvo.caltech.edu/vospace/transfers

Pull transfer target

The path of the target in the body of the HTTP POST message should match the path of the node created in the first step of the example.

    <transfer xmlns="http://www.ivoa.net/xml/VOSpace/v2.1">
        ....
        <target>vos://nvo.caltech!vospace/myData/table123</target>
        ....
    </transfer>

Pull transfer redirect

The current text implies that the server responds directly with the HTTP GET source URL that the client should use to fetch the data from.

This omits the details of the 303 redirect response

    HTTP/1.1 303 See Other
    ....
    Location: http://nvo.caltech.edu/vospace/transfers/581

which directs the client to a representation of the UWS job, which contains the HTTP GET location that client can fetch the data from.

In addition, the example URL for the HTTP GET location to transfer the data

    http://nvo.caltech.edu/bvospace/myData/table123/transfers/147516ab

appears to point to a location relative to the target node within the {service}/node tree.

It would be clearer if the example used a URL that did not overlap with the VOSpace service endpoint in any way.

    http://d22.caltech.edu/temp/e612

Push to VOSpace sequence

In section 5.4.1.4 Example: pushToVoSpace (missing section header) The text in the example suggests that the client should submit the transfer job, then poll the job details, waiting for the job to COMPLETE and then using the URL in 'transferDetails' result to fetch the transfer details, which contain the endpoint URL to send the data to.

I don't think that is the correct sequence.

Based on the way the original SOAP service worked, I think the correct pushToVoSpace sequence should be :

1. The client submits the job, and follows the redirect to the job details, but there is no need to poll the job status, because the server is now waiting for the client to send the data.
2. The job status SHOULD should be PENDING to indicate that the server is waiting for some data.
3. The job status MUST contain the full details of the vos:transfer in the uws:jobInfo section, including target URL(s) for each of the vos:protocol elements that the server is offering.
4. It is up to the client to pick one of the target URL(s) and transfer the data (pushing the data to the VOSpace).
5. During the transfer, the server SHOULD update the job status to RUNNING.
6. The server MAY decide to reject additional transfers once a transfer has succeed.
7. The server MAY get update the job status to FAILED if no data has been sent after a time limit has expired.
8. If a transfers fails, the client MAY try again if the server offered more than one vos:protocol target URL.
9. The client SHOULD set the job status to FAILED if it is unable to transfer the data.
10. The server MUST update the job status to FAILED if the data that arrives is not valid (see http://mail.ivoa.net/pipermail/grid/2014-September/002649.html).
11. The server MUST update the job status to COMPLETED once a transfer has succeed.
12. After the data has been sent the client SHOULD check that the final job status is set COMPLETED to verify that the server has accepted the data as valid.

Optional .auto

The list of changes state that .auto is being made optional. However many sections of the document still contain the original text describing how .auto applies, without describing how it can be made optional. If it is optional, then the specification needs to describe a mechanism for a client to discover whether a particular service supports it or not, and how this effects the behaviour in each case where it is applicable.

However, I think making it optional causes too many side effects and inconsistent behaviours between different spaces. As the original sponsor of the .auto functionality I'm happy for it to be removed completely.

And .null

If we are going to remove .auto, then .null should probably go too. There are better ways of specifying a null value for a direction.

Neutral URL examples

All our documents should probably use the reserved 'example' domain names for URLs. https://tools.ietf.org/html/rfc2606

http://server.example.com/vospace http://server.example.com/vospace/nodes http://server.example.com/vospace/nodes/container http://server.example.com/vospace/nodes/container/file http://server.example.com/vospace/nodes/container/table ....

http://server.example.com/vospace/transfers/810 http://server.example.com/vospace/transfers/810/status ....

and perhaps use separate server names for uploads and downloads

http://other.example.com/upload/312 ....

http://download.example.com/public/598 ....

From the mailing list announcement :

• Summary of recent changes
• Line by line diff

General

HTML anchors

    <a name="0.1_sec1">1. Introduction</a>

    <a id="0.1_sec1">1. Introduction</a>

Section 1.1

Create node path

    http://nvo.caltech.edu/vospace/myData/table123

does not match the path in the message content

    vos://nvo.caltech!vospace/mytable1

    http://nvo.caltech.edu/vospace/

then the target URL for the HTTP PUT request should be

    http://nvo.caltech.edu/vospace/nodes/myData/table123

    vos://nvo.caltech!vospace/myData/table123

Create node text

    {service}/nodes/{path}

This means the node path is encoded in two different places within the same message.

• The current specification does not clearly state how the intended node path should be included in the target URL of the HTTP PUT request.
• The current specification does not clearly state whether or not the node uri should be included in the content of the create node message.
• The current specification does not clearly state whether or not the service should return an error if the path in the uri in the message content does not match the path in the target URL of the HTTP PUT request

Push transfer endpoint

    http://nvo.caltech.edu/vospace/myData/table123/transfers

    http://nvo.caltech.edu/vospace/

then the endpoint URL for initiating a transfer should be

    http://nvo.caltech.edu/vospace/transfers

Push transfer target

    <transfer xmlns="http://www.ivoa.net/xml/VOSpace/v2.1">
        ....
        <target>vos://nvo.caltech!vospace/myData/table123</target>
        ....
    </transfer>

Push transfer protocol

    <transfer ...>
        ....
        <protocol uri="ivo://ivoa.net/vospace/core#http-put"/>  
        ....
    </transfer>

    ivo://ivoa.net/vospace/core#httpput

Push transfer redirect

This omits the details of the 303 redirect response

    HTTP/1.1 303 See Other
    ....
    Location: http://nvo.caltech.edu/vospace/transfers/581

    <uws:job ....>
        <uws:jobId>581</uws:jobId>
        ....
        <uws:jobInfo>
            <vos:transfer>
                ....
                <vos:protocol uri="ivo://ivoa.net/vospace/core#http-put">
                    <vos:endpoint>http://d21.caltech.edu/temp/d558</vos:endpoint>
                </vos:protocol>
                ....
            </vos:transfer>
        <uws:jobInfo>
        ....
    </uws:job>

In addition, the example URL for the HTTP PUT destination,

    http://nvo.caltech.edu/bvospace/myData/table123/transfers/147516ab

    http://d21.caltech.edu/temp/d558

Pull transfer endpoint

    http://nvo.caltech.edu/vospace/

then the target URL to initate a transfer should be

    http://nvo.caltech.edu/vospace/transfers

Pull transfer target

    <transfer xmlns="http://www.ivoa.net/xml/VOSpace/v2.1">
        ....
        <target>vos://nvo.caltech!vospace/myData/table123</target>
        ....
    </transfer>

Pull transfer redirect

    HTTP/1.1 303 See Other
    ....
    Location: http://nvo.caltech.edu/vospace/transfers/581

    http://nvo.caltech.edu/bvospace/myData/table123/transfers/147516ab

    http://d22.caltech.edu/temp/e612

From the mailing list announcement :

• Summary of recent changes
• Line by line diff

HTML anchors

    <a name="0.1_sec1">1. Introduction</a>

    <a id="0.1_sec1">1. Introduction</a>

Create node path

From the mailing list announcement :

• Summary of recent changes
• Line by line diff

HTML anchors

    <a name="0.1_sec1">1. Introduction</a>

    <a id="0.1_sec1">1. Introduction</a>

From the mailing list announcement :

• Summary of recent changes
• Line by line diff

HTML anchors

    <a name="0.1_sec1">1. Introduction</a>

    <a id="0.1_sec1">1. Introduction</a>

Example node path

In the text description :

"Let's consider the first sequence: the user provides a XML description of the data file which they HTTP PUT to the appropriate VOSpace URI - this will be the HTTP identifier for the data file in the VOSpace, e.g. http://nvo.caltech.edu/vospace/myData/table123."

the URI target is given as http://nvo.caltech.edu/vospace/myData/table123

In the example node XML :

    <node xmlns="http://www.ivoa.net/xml/VOSpaceTypes-v2.1"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
        uri="vos://nvo.caltech!vospace/mytable1"
        xsi:type="vost:UnstructuredDataNode">  
        <properties> 
            <property uri="ivo://ivoa.net/vospace/core#mimetype">text/xml</property>     
        </properties> 
    </node> 

the URI target is given as vos://nvo.caltech!vospace/mytable1