Difference: Siav2RFC (42 vs. 43)
Revision 432015-10-31 - MireilleLouys
<--
SIA v2.0 Proposed Recommendation: Request for CommentsPublic discussion page for the IVOA Simple Image Access (version 2.0) Proposed Recommendation. Latest version of the IVOA SIAV2.0 can be found at:Reference Interoperable Implementations(Indicate here the links to at least two Reference Interoperable Implementations)AMIGA SIAv2 Archive PrototypeThe AMIGA SIAv2 Archive Prototype provides discovery and access interfaces for two collections of radiointerferometric single-line emission datacubes of galaxies. The B0DEGA collection is composed of 30 velocity datacubes of southern nearby galaxies obtained using the Submillimeter Array (SMA) for CO(2-1) emission of the circumnuclear regions (1 arcmin). The WHISP collection provides 33 velocity datacubes of the more extended public WHISP survey observed for HI 21cm emission line. RESTful interfaces may be tested with HTML forms that were initially developed for debugging purposes. Interfaces provided by AMIGA SIAv2 Archive Prototype:
CADC implementation of the SIA-2.0The CADC implementation has been updated to support all the query parameters in the PR document. Resources: http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/sia/availability http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/sia/capabilities http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/sia/v2query The landing page (http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/sia/) provides some example queries. The access_url in the output is currently a link to the CADC DataLink service (as described by the access_format value); this may change in the future to be a direct link to science data. We also include a DataLink service descriptor in the response (VOTable) document that can be used to generate calls to the DataLinks service using ID values from the table. Implementation note: This implementation is really just a front-end for the CADC TA service. It generates an ADQL query on the ivoa.ObsCore table and submits that as a query job to TAP sync endpoint; the caller gets the outpu directly from the CADC TAP service. All of the code to implement such a front-end is available in the cadcSIA module at https://code.google.com/p/opencadc/ -- the code here can be used to parse query params, generate ADQL, and provide a "TAP-frontend resource" using the sync support in the cadcUWS library.Client implementationThe last beta version of Aladin is able to query the test implementation servers and to ingest the query reponse. -- FrancoisBonnarel - 2015-06-08DALServer implementation of the SIA-2.0An implementation of the current RFC SIAV2 interface, including an accessData prototype, is in progress within the DALServer framework. Details will be posted here once this becomes available. The DALServer framework was developed with NVO and VAO resources within the US. Earlier (2014) prototype SIAV2 and accessData services, including multidimensional image data from a half dozen or so projects, can be viewed and exercised at http://vaosa-vm1.aoc.nrao.edu/ivoa-dal/siav2query.html -- DougTody - 2015-02-25CASDA implementation of SIA-2.0The CSIRO ASKAP Science Data Archive project has implemented a "minimal viable product" level SIA2, Datalink and SODA 'caravan' for retrieval of multi-dimensional FITS images and CASA visibilities. See my presentation on the InteropOct2015DAL page for more detail. This will be available on 7 Nov 2015 with support for full set of SIA2 parameters to be released in Feb 2016. -- JamesDempsey - 2015-10-31Validators(If any, indicate here the links to Implementations Validators)RFC Review Period: 2014-07-31 - 2014-09-09<-- REMOVE comment-out for TCG Review Period Comments from the IVOA Community during RFC period: 2014-07-14 - 2014-09-09Comments from José Enrique Ruiz
Comments from FrancoisBonnarel
I have other personal comments. One (2.2) is a real question
Answer by DougTody -Since the full metadata (WCS etc.) for a single dataset can be very large, and a discovery query may find many datasets, I think getMetadata should be a separate operation, restricted to a single image. Also, for reasons of complexity, only queryData should do discovery, and getMetadata and accessData should be limited to single datasets. However, if we want basic WCS info in the discovery query response then that is possible as we probably do not need full image metadata. The minimal stuff we have proposed for ObsTAP 1.1, plus s_region, already comes close. If we were to add just a bit more (e.g. reference point and projection type) then it would suffice for the typical nsubarrays=1 use case. -- DougTody - 2014-07-31Comments from WalterLandry -2014-09-26* Reminder: the parameters you are critiquing are from the SIAv2 query capability. This capability is designed to query for datasets that match the specified conditions, so the parameters are inherently those that express the conditions and not (necessarily) those that describe some ideal (construct-able) data. -- Answer by PatrickDowler - 2014-09-29 * Comment/response inline -- Answer by PatrickDowler - 2014-09-29 Back in July, I sent a note to this list about some issues I had with SIA 2.0. Since then, we have implemented a synthetic image generation service for the Planck satellite. We tried to implement this is in a way consistent with SIA 2.0, but we had some difficulties. 1) BAND The Planck satellite detector bands are all specified in GHz: 30, 44, 70, etc. These are nice, integer numbers. Mapping to wavelength leaves me with numbers that are not exactly representable in floating point. This means that every search has to give a range. It would be nice if I could specify the frequency instead of the wavelength. We ended up using the keyword FREQ in MHz, since I do not know of any astronomical observations of EM radiation that go lower than that. * We've had this argument so many times and it always comes own to "just pick something". In ObsCore-1.0 the em_min and em_max fields are wavelength in meters, so that is what SIA-2.0 query uses to query those fields. No matter what you pick for the standard, someone has to transform nice looking values into something with scientific notation... Implementing FREQ in addition is fine for you; requiring it in the standard is more work for all services; implementing FREQ instead means services are not compm>atible or (at best) clients have to get/grok service capabilities before being able to call them. -- Answer by PatrickDowler - 2014-09-29 2) RANGE As I mentioned in July, RANGE is prohibitively expensive for this data set. So we do not support it, will never support it, and I still think it should not be part of the spec. * I don't disagree with this; at best it is a convenience for making some common (large) polygons. -- Answer by PatrickDowler - 2014-09-29 3) POS Since this is a synthetic image generation service, it would be nice to make rectangular images. The current SIA 2.0 spec has no great circle rectangles. The client has to construct the polygons themselves, which is non-trivial. Given the negative reaction I got to Box's last time, we ended up ditching SIA 2.0 for this entirely and using SIA 1 syntax: POS, SIZE, CFRAME, CDELT (though SPATRES would have been fine). I would really prefer a better mechanism than this. * Well, I basically grok what you re trying to do and it is just using a small bit of FTS WCS to describe what you want to get back. That seems to me to fit much better in an AccessData-ish service and not in a pure data discovery service. Since it seems to be driven from real data, there is obviously some part of the usage that would involve discovery... let's make sure to discuss this kind of usage in Banff next week. -- Answer by PatrickDowler - 2014-09-29 4) TARGET vs OBJECT Why does SIA 2.0 use the TARGET parameter? OBJECT is an existing standard FITS convention. * The query result is ObsCore; in there the name of the field being constrained is target_name, hence TARGET. -- Answer by PatrickDowler - 2014-09-29 5) Syntax Consider these issues: a) In July, I highlighted a problem with the syntax of POS parameters. It requires spaces, which must be URL encoded or things silently break. Silent breakage is the worst kind of breakage. * Syntax requires encoding, yes. Even without syntax, parameter values must be encoded to be safe or strange things happen. How many times have I cursed the IAU naming convention that includes + sign? Lost count Must encode. -- Answer by PatrickDowler - 2014-09-29 b) Polygon searches use a straight list of numbers. It would be better to have a list of pairs to make typos more obvious. * It is possible to make syntax errors. More syntax solves it? -- Answer by PatrickDowler - 2014-09-29 c) We need to be able to select multiple detectors at once, so we would like to have an array of strings. * Not sure I follow... you said synthetic but now are talking about multiple detectors. If the underlying data is some kind of mosaic camera then you have several choices on what constitutes a single ObsCore entity (been there, we can discuss off-line), but describing the complexity of 1 observation -> N subarrays is not in the scope of ObsCore-1.0 so not in the scope of SIA-2.0 query. There is ObsCore-1.1 work underway, plus the ImageDM and consequent SIAv2 "metadata" capability for exposing it. -- Answer by PatrickDowler - 2014-09-29 d) There is no way to add arbitrary parameters. COORD was the way to do that in old versions of SIA 2.0. Now I have to use up a keyword and hope it does not accidentally conflict with new versions of the standard. This is not going to scale. * SELECT and COORD were never part of SIA-2.0 query; they were part of WD-AccessData-1.0 to show a way that SimDAL could be supported within that spec. -- Answer by PatrickDowler - 2014-09-29 e) We have a smart client doing searches on behalf of the user. In general, we would like to set arbitrary metadata that are not necessary for the search but convenient for the user. * You can always add custom fields to your ObsCore output. If they aren't really custom, but just in the optional fields of the appendix, then chosing standard names would be a good thing to do. -- Answer by PatrickDowler - 2014-09-29 This prompted me to use a more general syntax to express queries. Specifically, I used json5 https://github.com/aseemk/json5 It is an extension of JSON to make it friendlier to write. It is a strict superset of JSON and a strict subset of Javascript. So every valid JSON file is valid json5, and eval() will still work for those of you foolish enough to run it on unverified user input To be specific, a sample query would be http://irsa.ipac.caltech.edu/cgi-bin/Planck_TOI/nph-planck_toi_sia?POS=[0.053,-0.062]&CFRAME='GAL'&ROTANG=90&SIZE=1&CDELT=0.05&FREQ=44000&ITERATIONS=20&INSTRUMENT=['24m','24s']&TIME=[[0,55300],[55500,Infinity]]&USER_METADATA={CLIENT:'IRSA Smart Client'} Note that the service is not public yet, so this URL will not work for you yet. Internally, every parameter is converted into a json5 element. So this would turn into the json5 document { POS:[0.053,-0.062], CFRAME:'GAL', ROTANG:90, SIZE:1, CDELT:0.05, FREQ:44000, ITERATIONS:20, INSTRUMENT:['24m','24s'], TIME:[[0,55300],[55500,Infinity]], USER_METADATA:{CLIENT:'IRSA Smart Client'} } Modulo whitespace, this is just replacing '&' with ',' and '=' with ':'. We also support submitting a json5 document directly http://irsa.ipac.caltech.edu/cgi-bin/Planck_TOI/nph-planck_toi_sia?{POS:[0.053,-0.062],CFRAME:'GAL',ROTANG:90,SIZE:1,CDELT:0.05,FREQ:44000,ITERATIONS:20,INSTRUMENT:['24m','24s'],TIME:[[0,55300],[55500,Infinity]],USER_METADATA:{CLIENT:'IRSA Smart Client'}} * This is interesting and I remember talking about json5 at the last interop. Hopefully we can see/discuss this further in Banff. -- Answer by PatrickDowler - 2014-09-29 On a side note, I have used JSON (not json5) as input for simulations in geology [1]. The SAMRAI Adaptive Mesh Refinement framework for massively parallel simulations [2] also uses a format that is almost indistinguishable [3] from json5 for input files. So I would claim that json5 would be able to cover any needs that SIMDAL would need in specifying model parameters. Given all of the issues that I ran into, it is not clear to me that it would be a good idea to ratify SIA 2 as it is now. Whether or not you like the changes I made, there seem to be some major deficiencies that need to be addressed before the standard can be accepted. [1] http://geodynamics.org/cig/software/gale [2] https://computation-rnd.llnl.gov/SAMRAI/index.php [3] You can separate elements in an array or object with newlines instead of commas. -- WalterLandry 2014-09-26Comments from José Enrique Ruiz 2014-10-28Comments for SIAv2 PR 2014-10-24
Answer by FrancoisBonnarelI go back to this discussion between Walter and Pat (see above). I think the point has been clarified at Banff and in the new version. -- Walter said 3) POS Since this is a synthetic image generation service, it would be nice to make rectangular images. The current SIA 2.0 spec has no great circle rectangles. The client has to construct the polygons themselves, which is non-trivial. Given the negative reaction I got to Box's last time, we ended up ditching SIA 2.0 for this entirely and using SIA 1 syntax: POS, SIZE, CFRAME, CDELT (though SPATRES would have been fine). I would really prefer a better mechanism than this. * Well, I basically grok what you re trying to do and it is just using a small bit of FTS WCS to describe what you want to get back. That seems to me to fit much better in an AccessData-ish service and not in a pure data discovery service. Since it seems to be driven from real data, there is obviously some part of the usage that would involve discovery... let's make sure to discuss this kind of usage in Banff next week. -- Answer by PatrickDowler - 2014-09-29 ** This has been clarified in the introduction (differences SIAV1 and SIAV2) " The capabilities for dynamic access to image datasets are expanded in scope, but are separated from data discovery and download of whole image datasets. A separate "AccessData" specification currently under development will define the more advanced dynamic data access functionality. Automated virtual data generation and discovery (as in SIA- 1.0) is not currently supported but is being considered for a future version of SIA." Actually, AccessData 1.0 allows to extract cutouts. Regridding, changes of WCS will be covered by AccessData 1.1. Discovery of virtual data in the query phase will be covered by SIAV2.1Comments from TCG member during the TCG Review Period: 2014-11-20- 2015-09-13WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or not the Standard. IG chairs or vice chairs are also encouraged to do the same, althought their inputs are not compulsory.TCG Chair & Vice Chair ( _Séverin Gaudet, Matthew Graham )
Applications Working Group ( _Pierre Fernique, Tom Donaldson )
Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )Data Model Working Group ( _Jesus Salgado, Omar Laurino )Overall I found that this document goes a long way in meeting its goal, though there are some areas that need some improvement. I hope my feedback is helpful in meeting that goal. Reference Implementations. It looks like the reference implementations are both server implementations. In what sense are they interoperable? It would be good to have a client implementation, and probably there is one already, although it's not listed on the RFC page. This would show that a) the services are interoperable (with each other and with an independent client), and b) that the use cases are fulfilled. * There is now a client implementation within Aladin (see above) -- FrancoisBonnarel 2015-06-08 Examples The document does not present any examples that show possible fulfillments of the use cases (i.e. the triple: use case, example query, example response). * This is done now (see above) -- FrancoisBonnarel 2015-06-08 Syntax I agree with Walter that it is discouraging to see yet another syntax defined by the spec. This may limit uptake, as it increases the burden on both server and client implementors, and makes the spec less future-proof. I don't know if there is still room for reviewing this decision. Adopting an existing syntax would also make more advanced uses of the specification easier (including the ones declared as "future" features, in particular related to ImageDM, inputs based on complex metadata, or request-based data manipulation). The concrete risk is that the standard will need to define more than one syntax for simple and advanced cases falling into its scope. * - two alternatives answers (to be discussed in Sesto) : The syntax is not fairly new. It is an adaptation for images to the SSA Syntax defined several years ago. having SSA and SIA syntax close together is a strong driver. -- FrancoisBonnarel Changing to the VOTable-consistent syntax would mostly resolve this issue, with cost of changing the parameter syntax and inroducing new spatial params _ IVOA.Patrick.Dowler 2015-06-08 Mandatory features There has been strong feedback about some mandatory features that came from implementors during the RFC period (I am thinking about RANGE in particular). Would it be possible to make the problematic feature(s) optional, and maybe add some clarifications to the problematic keywords in the spec? As the original comment is actually deeper than that, I am also concerned that not taking the comment into account will result in more issues down the road. * It would be much easier to make spatial coordinate range query optional if it was a separate parameter rather than a separate keyword in the value of POS. In the RFC review my response was that RANGE could be dropped since it adds burden for services and only convenience for clients... but we never revisisted that in Banff since there were bigger issues... it isn't too late to drop spatial coordinate range -- IVOA.Patrick.Dowler 2015-06-08 Response serialization (and relationship with Data Models). It seems that the document does not directly specify the serialization format for service responses, leaving this to referenced specs. This may be a major issue for implementors. * It is actually similar to ObsTAp response. There are now examples available. -- FrancoisBonnarel - 2015-06-09 The abstract claims that SIA is based upon the Image DM. However, later in the text, the relationship with ImageDM seems to be part of "future" updates while the spec explicitly states that SIA is based on ObsTAP. * Hopefully this has been clarified now (see below) -- FrancoisBonnarel - 2015-06-09 Notice that in general I would suggest to limit references to "future" work as much as possible, as such references can possibly be confusing. (see "what's in 2.0 and what is not" below). * We have taken this remark into account. -- FrancoisBonnarel - 2015-06-09 More specifically, when SIAv2 states (3.1 Successful Query): "The response [...] is a table with instances of the ObsCore data model [that] specifies all the (VOTable) field names, utypes, UCDs, and units to use in the response" I believe it is referring to the specific "4 Implementation of ObsCore in a TAP service" section in the ObsTAP document, plus it assumes a VOTable serialization (why the parentheses btw?), as in section 2.9 of the TAP spec, although the TAP spec is not referenced. It takes too much work, in my opinion, to figure out what exactly the response should be, and the lack of examples does not help either. Obviously this is not an issue for those who have followed the discussions or implemented ObsTAP already, but we cannot assume the regular reader knows what the spec implies. * The paragraph has been modified for clarification. -- FrancoisBonnarel - 2015-06-09 The aforementioned statement in section 3.1 (assuming I correctly interpreted it) clashes with the different semantics of similar sentences in other sections, e.g. 1.1 The Role in the IVOA Architecture: "The metadata returned [...] is defined by the ObsCore data model; this may be extended with additional metadata (columns) from ImageDM in the future" (more on the "in the future" part later). ImageDM is abstract and serialization-agnostic, thus it does not, by itself, specify the VOTable field names, UCDs, etc. (This seems better suited for a DAL specification or a standardized serialization specification). For instance, the "ObsCore in TAP" specification limits the semantics of the DM (e.g. central coordinates must be in ICRS), as it is expected by a DAL spec, but again this does not apply to ImageDM. ImageDM is agnostic on how a DAL protocol response specification (or any other serializations) may restrict its semantics and make a "column" out of one of the leaves in its hierarchy. * section 1.1 has been modified to clarify this point. -- FrancoisBonnarel - 2015-06-09 On the other hand, section 2 Resources is more clear by itself (except for the "future" part, again more on this later), but again the context is ambiguous: "an optional {metadata} resource for obtaining [...] metadata conforming to the ImageDM." How is this statement to be interpreted? As in what applies to "ObsCoreDM in ObsTAP", i.e. a flattened serialization in a specific format with specific semantics, or as in what applies to ImageDM , i.e. any valid serialization of the abstract concepts defined by the DM? In this case the answer is probably "we don't know yet, as this is unspecified", which segue perfectly to the next comment. * The subsection describing metadata resource has been suppressed and the content partially moved to the introduction. -- FrancoisBonnarel - 2015-06-09 In conclusion, modulo the contents of the next comment, I would suggest that the document replaces the references to ObsCoreDM with less ambiguous phrasing, as proposed below. In the abstract, I would suggest to remove the reference to ImageDM (unless I am missing something ImageDM will be used to support future versions). * This has been changed and hopefully clarified but not removed. -- FrancoisBonnarel - 2015-06-09 Assuming that a SIA v2.0 response is identical to an ObsTAP response (is this true?), in section 1.1 I would suggest to remove the reference to ImageDM and I would rephrase the reference to ObsCoreDM as follows: "The metadata returned [...] is defined by the ObsCore data model components [7] and serialized according to the ObsTAP specification [ibid., section 4], and [missing reference to TAP, section 2.9]. * Done -- FrancoisBonnarel - 2015-06-09 In section 3.1, instead of: "The response [...] is a table with instances of the ObsCore data model [that] specifies all the (VOTable) field names, utypes, UCDs, and units to use in the response" I would suggest: "The response is a table consistent with ObsTAP responses as described in [7, section 4] and [missing reference to TAP, section 2.9]. Examples are provided in Appendix <X>." * Changed along these lines -- FrancoisBonnarel - 2015-06-09 What is in 2.0 and what is not. It is not trivial for the reader to keep track of what is defined in 2.0 and what is left to later releases. It is indeed true that the text tries to specify it, but the references to next releases are dispersed in the document, which will also make a future review hard to implement. I would suggest to add a section with a title like "Notable Absences", or "Future support", to explicitly notify the reader of what was left out of this version and will be done in the future. This should also make future reviews easier to accomplish. * This has been integrated to the introduction. -- FrancoisBonnarel - 2015-06-09 Similarly, it is unclear which Use Cases are addressed by v2.0 and which are not. In some cases it is clearly stated that a use case is not even supposed to be covered by the spec. One may probably work that out by cross-matching the specified features with the use cases section, but that seems too much work for implementors just so they can figure out whether what they want to achieve is supported or not by the standard. As the list of use cases is supposed to be a list of the "motivating" drivers, it's probably fine to keep all of them in the same section, but I would suggest to clearly mention in the title whether they are covered in the 2.0 specs or not. If it is true that some use cases are not going to be covered eventually, I suggest those use cases to be removed. * This has been clarified in the "Scope and Related documents section" -- FrancoisBonnarel - 2015-06-09 Other, possibly minor, issues (I am marking the minor ones with (M)): (M) 1. SIA-2.0 specification will make use of features... Why the future tense? Does it or does it not make use of those features? * _Does. Fixed _ -- FrancoisBonnarel - 2015-06-09 (M) 1.1 SIA is a data discovery and metadata service that works with other DAL services... I would have been less surprised to read that SIA is a protocol with relationships with other protocols and standards. Applications need to implement the SIA standard from the client side as much as services need to implement it from the server side. * Fixed -- FrancoisBonnarel - 2015-06-09 (M) 2.1 Repetition: "units for numeric values are are specified" * Fixed -- FrancoisBonnarel - 2015-06-09 2.1 "multi-valued" parameters. I guess this refers to multiple values of the same GET/POST parameter. However, it is unclear what the syntax is here, and as far as I can tell the HTTP does not define a standard one. DALI seems to define "multiple values" as multiple occurrences of the "parameter=value" pair, e.g. in 3.1.3 (Multiple Values) and 3.2.5 (UPLOAD), while the "," syntax is left to separate values of the single occurrence of a parameter (e.g. UPLOAD in 3.2.5, where the table, uri pair is separated by commas). This is consistent with what is specified in section 2.1.4 (POL). SIAv2 should probably clarify the syntax for multiple values and maybe refer to DALI here. * Hopefully clarified -- FrancoisBonnarel - 2015-06-09 (M) 2.1.4 POL. The specification here is redundant, as the syntax for multiple values was already defined in the 2.1 preamble (or it should have been), and does not apply to POL only. The sentence "possible values for the POL parameter are defined in [...]" does seems incomplete. * Reworded to make this clear -- PatrickDowler - 2015-06-09 2.2 {metadata}. I am not sure I understand why this is useful. As the resource is not specified and left to future revisions this section seems out of place and I would suggest to remove it. * _Information has been moved somewhere else. _ -- FrancoisBonnarel - 2015-06-09 (M) As most of the query parameters description is boilerplate text, I would suggest to insert more information in the section titles. DataType and Multiplicity are probably enough, but maybe the corresponding ObsCore field name would be useful as well. In my experience this eases implementations a good deal, as most of the relevant information becomes readily available in the TOC. (M) When referencing other specs (e.g. DALI) it would help a lot to have a specific section referenced, especially in the normative part. * I am somewhat reluctant to do that because I would prefer to reference things like DALI and ObsCore in a loose DALI-1.0 or later sort of way and putting section numbers in just breaks at some point. If someone can't find the appropriate bit in DALI the the DALI doc needs improvement as a base reference doc -- PatrickDowler - 2015-06-09 3.1 Successful Query. The spec should probably refer to the RESPONSEFORMAT parameter defined in DALI (Not necessarily in this section, but this section mentions the response format, so I am bringing this up here). This may also be true for VERSION and possibly other standard parameters from DALI. The spec does redefine standard pieces from DALI, sometimes unchanged (e.g. 2.3 Availability), so it seems appropriate to name all the parts of DALI that should be implemented faithfully with DALI. This is probably just boilerplate, but the kind of boilerplate that helps implementors by guiding them in tracking the relevant sections in different documents. * _RESPONSEFORMAT sentence has been added with reference to DALI _ -- FrancoisBonnarel - 2015-06-09 3.1.1 Related Service Metadata. Again, this section seems to specify what the document claimed is left for future versions (e.g. {metadata}). * _{metadata} information has been heavily modified elsewhere. should be sufficient _ -- FrancoisBonnarel - 2015-06-09 -- OmarLaurino - 2014-12-31Grid & Web Services Working Group ( André Schaaff, Brian Major )It would be nice to know, briefly and upfront, what the differences are between ImageDM and ObsCore. There is a suggestion that ObsCore is a more abstract data model in section 1.4, but apart from that it is difficult to understand why there are the two models and what purpose each serves. This, perhaps, could be clarified in a terminology section after the abstract? * _Modifications made along the lines stressed by the DM WG should help. _ -- FrancoisBonnarel - 2015-06-09 An overall easy to read document except for (as Pierre notes above) for the required knowledge of other documents. * The example should allow to avoid that. -- FrancoisBonnarel - 2015-06-09 -- BrianMajor - 2014-12-23Registry Working Group ( _Markus Demleitner, Pierre Le Sidaner )In the following comments, what we've marked with (*) are things we feel must be addressed one way or the other before the document can pass. General: We miss language on having custom parameters on SIAv2 services, something that was possible in SIAv1 -- and frequently necessary, too. Relatedly, the DALI metadata response (e.g., MAXREC=0) needs to be specified. (*) We suggest that even for the standard parameters, means for clients to discover what, if any, values of them can be expected to match. * a MAXREC section has been added -- PatrickDowler - 2015-06-09 We note the absence of a validator and a client implementation. For something as use-case driven SIAv2, a client implementation really is indispensable. Also, as we understand SIAv2 was positioned as "something easier to implement for non-IVOA folks than TAP+Obscore", a solid validator for these external implementors should really be provided. (*) * The answer to this can be found above (Basically the answer is yes) -- FrancoisBonnarel - 2015-06-09 For readability, we would prefer if parameter descriptions were grouped in a physically meaningful way (band and specrp together, time, timeres, exposure_time together, etc). * Not done yet. -- FrancoisBonnarel - 2015-06-09 1 Introduction: "While the data model is general..." -- We suppose the "data model" here means whatever data model is implied by the protocol response. If that is true, that should be made clear here so as avoid confusion with a future Image DM. * I think relationship with ObsCore and Image DM have been clarified now. -- FrancoisBonnarel - 2015-06-09 1.2 Motivating Use Cases: Stylistically, it would be preferable if there were less "probably", "may", etc in that section. Also, as 1.2.4 isn't actually part of SIA2, wouldn't it be better to not have them as two of four motivating use cases, regardless of what follows in 1.3? * Ok for "may". For the second part the answer is no because the documents and protocol are separated for convenience but the use cases may actually be inseparable. -- FrancoisBonnarel - 2015-06-09 2.1. {query} resource. As in Datalink, I'd like some guaranteed way to get from the URL of a {query} resource to the capabilities endpoint, e.g., by saying slashes are forbidden in {query}. (*) * _ It means the {query} resource has to be a sibling of the VOSI resources. I have added this restriction on naming to section 2. _ -- PatrickDowler - 2015-06-09 2.1. UNKNOWN frames in BAND and TIME. I believe what's meant here is that the reference positions of the incoming data should be used as-is. It should be put like this in the spec. The prose about UNKNOWN reference position is not helpful -- how to I interpret data as being computed for "UNKNOWN"?. Either way, the document would profit from an analysis on why leaving the reference position open in a discovery protocol is acceptable and doesn't conflict with any requirements (in particular for BAND, that's not obvious to us) * Yes it means that it should be used as-is and is assumed to be the same as the internal one -- PatrickDowler 2015-06-09 2.1.1 POS. It might be wise to stress even here that all values are decimal. * Done -- FrancoisBonnarel - 2015-06-09 2.1.3 TIME. Is it really necessary to allow both MJD and ISO time stamps? What additional use cases are enabled by this? It seems that allowing both is a bad compromise on the backs of service implementors, needlessly requiring a certain amount of guessing in parameter interpretation. As SIAv2 is design-- FrancoisBonnarel - 2015-06-09ed as a relatively thin layer on top of obscore, we suggest going for MJD exclusively. * still in discussion among authors : ISO, is for compatibility with SSA I Guess. MJD is natural with OBsCore. So I would say no for this demand -- FrancoisBonnarel - 2015-06-09 * I actually agree with this ; the backwards compatibility comes at a price because allowing both MJD and ISO means that the param datatype can't be double -- PatrickDowler - 2015-06-09 * I actually agree with this ; the backwards compatibility comes at a price because allowing both MJD and ISO means that the param datatype can't be double FrancoisBonnarel - 2015-06-09 2.1.8 ID. The text should say: "Note that IVORNs MUST be compared case-insensitively. As PubDIDs in the VO generally are IVORNs, implementations will usually have to use case-insensitive comparisons here." * Done FrancoisBonnarel - 2015-06-09 2.1.9....2.1.12, 2.1.14, 2.1.17: Do the string-valued parameters make any sense at all without a means to discover what values might be valid on a given service? Also, experience shows that people will want to use some sort of regular expression/wildcard scheme on these, and that we'll have implementations accepting them, leading to interoperability problems when no sane standard exists. It would be preferable if some provision were made for both discovery and patterns on enumerated/string-valued parameters. (*) * Since I had to add a little MAXREC subsection, I also included there the usefulness of providing a self-describing service descriptor (example 4.6 in Datalink) where once could list values for the string params. This provides a mechanism to do this and indirectly to describe all supported params including custom ones (exactly why we added that example to datalink). I can try to implement this and then be able to provide a good MAXREC=0 output example, or at worst work up an example by hand next week -- PatrickDowler - 2015-06-09 2.1.9 COLLECTION. Obscore has here: "In practice this is not a very precisely defined field" -- this, of course, makes discoverability even more important. Or SIAv2 should grasp the chance to define best practices: maybe IVORNs for the collections after all since there's a Resource type for them now? Or maybe propose some list of common short names of the data collections, perhaps maintained on the IVOA wiki? * I don't think we can really do something valuable right now. may be reconsidered with version 2.1 -- FrancoisBonnarel - 2015-06-09 2.1.10 FACILITY. Obscore has here "For combined observations stemming from multiple facilities the name may contain a list of comma separated strings..." SIAv2 really should comment about what that means for interpreting FACILITY queries. Also, having some sort of vocabulary or at least a means of discovering values supported by the service appear preconditions for having this work. * same answer than for COLLECTION -- FrancoisBonnarel - 2015-06-09 2.1.12 DTYPE. Is there any strong reason not to include other, fairly image-like data like visibilities (ok, their edges in space may be a bit fuzzy, but strictly speaking you have stray like on CCDs, too) or event lists? * Visibilities and event list can be discovered using ObsTap. Or via DataLink if they are related to the cube. Do we really want to discover with SIAV2 ? AccessData will be difficult for such Datasets. I would say no for this version -- FrancoisBonnarel - 2015-06-09 2.1.17 FORMAT. Previous incarnations of FORMAT included lots of funky magic terms for ("compliant", "native", "image"...). Just to be sure: There's no intention of have them any more? * No there isn't -- FrancoisBonnarel - 2015-06-09 2.1.18 UPLOAD. The text appears a bit too vague for us. Couldn't this just be "The DALI UPLOAD parameter is not used by this version of SIA. The use case of uploading lists of coordinates is covered by the multiple-valued parameter values." or so? * Done -- FrancoisBonnarel - 2015-06-09 2.2. {metadata} resource. If we've learned one thing from VO standards development it is that predictions are tricky, in particular when the concern the future. Non-normative or not, this section may have to do fairly little with what will actually happen, and so more likely than not will only confuse future readers. And we don't think it helps a lot right now. * This has been moved (see above) -- FrancoisBonnarel - 2015-06-09 2.4 Capabilities, example. The Registry-approved canonical namespace prefix for VODataService is vs, not vod. The example document also needs the prefix declaration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" in addition to what's already there. (*) * Done -- FrancoisBonnarel - 2015-06-09 2.4 Capabilities. Do we really intend to drop the special SIA capability? If not, we'd be happy to assist in drawing up a Registry extension for SIAv2. It seems very clear to us that maintaining these in a separate document (as for SimpleDALRegExt) is not a sustainable thing, and so we propose that if extra capability metadata is intended for SIAv2, it should go into this document. If it is dropped, we would strongly suggest some other means (examples endpoint?) is defined that allows validators to discover a set of parameters that should return data. * I know that one thing that is missing with SimpleDALregExt is the ability to say which version of ObsCore the query interface uses. If ObsCore-1.1 is accepted at the interop and we can tweak SIA_2.0 to depend on it (or later) – and I think we should - then I think this won't be an problem for quite some time -- PatrickDowler - 2015-06-09 3.1. Successful Query. We had the thing with the "recommended HTTP Response Headers" in Datalink already, and we think the table and associated text should go here, too, for identical reasons. 3.1. Successful Query. We suggest that the reference to Obscore might be a bit too terse for implementors. Anyway, there should be an example response in the document. This could be together with a sample query string, which might also be helpful. (*) * Done -- FrancoisBonnarel - 2015-06-09 3.2. Error "If the requested format is VOTable, the error document should be VOTable..." -- this "should" is bad from a validation perspective. This must be a MUST of some sort, or our validators cannot flag services that don't do that. It's clear that exceptional errors can result in other error document formats, but such services are then just broken, and we should flag them. Please have text here that gives us the legal basis to do so. (*) * An attempt was made -- FrancoisBonnarel - 2015-06-09Semantics Working Group ( _Norman Gray, Mireille Louys )I think the document offers a clear and precise list of features to be present to implement the SIAv2 protocol. Units and UCD semantic tags are defined via the other re-used specifications, namely ObsCore, VOTable, so there are no conflicting issues concerning the Semantic WG approval. An implementation section could be added directly as appendix of the specification , in order to keep their examples content close to the description of the specifiactions. * We added the examples section -- FrancoisBonnarel - 2015-06-09 A few minor comments and suggestions :
Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )Data Curation & Preservation Interest Group ( Françoise Genova )Knowledge Discovery in Databases Interest Group ( George Djorgovski )Theory Interest Group ( _Franck Le Petit, Rick Wagner )Standards and Processes Committee ( Françoise Genova)--><--
Comments from TCG member during the TCG Review Period extension: 2015-09-14- 2015-09-30WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or not the Standard. IG chairs or vice chairs are also encouraged to do the same, althought their inputs are not compulsory.TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler )Applications Working Group ( _Pierre Fernique, Tom Donaldson )We approve this document. Just to fix a problem p19: there is a empty section 2.1.18 * Done --IVOA.FrancoisBonnarel - 2015-10-28 -- PierreFernique - 2015-10-26Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )Approved as a version 2.0 -- FrancoisBonnarel 2015-09-16Data Model Working Group ( _Mark Cresitello, Laurent Michel )I believe this version addresses the previous comments by the DM working group of 2014-12-31. A couple general notes of my own:
Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )I approve this document. -- BrianMajor - 2015-10-26Registry Working Group ( _Markus Demleitner, Theresa Dower )Two minor new points:
Finally, points from our original review we believe are not entirely satisfactorily addressed: (1) François said in his response to our questions on COLLECTION and FACILITY that there's probably nothing valuable that can be done with the two parameters at this point. Given the potential confusion (e.g., commas in obscore's facility_name), I think the logical thing to do is remove them ("Underspecified is worse than unspecified"). * There was a confusion in undersatnding my answer. Probably my fault. I thought it would be difficult to get a significant valuable VO List from scratch before implementation. As soon as you know (by chance or whatever other ad hoc reason) some COLLECTION and FACILITY values in a service these parameters are allready usefull for discovering datasets in a service. However It would be very intersting to get these list and standradize at the IVOA level after some experience and feddback and I added a slot in the "current and future support" section -- FrancoisBonnarel 2015-10-29 (2) I'm still not happy with the language in DPTYPE. If we really want to specify that SIAv2 services only return image and cube-typed things, that needs to be said somewhere else, not en passant in the description of a parameter. * the restriction was allready stated several times in the introduction. I made it explicitly more clear in the "current and future support" section -- FrancoisBonnarel 2015-10-29 Markus is all for dropping this restriction entirely. Let operators decide what they want to return. On François' objection that they might be trouble in server side data processing services ("AccessData"): From Markus' experience in writing such services, you'll have to be very flexible in their parameters anyway, even with your run-of-the-mill array-typed data sets. Markus happens to believe the proposed three-factor semantics is powerful enough. And either way for SIAv2 itself that's a secondary consideration. * I think it will be interesting to have visibilities and event list in a future version. A cutout on visibility is really not trivial at then moment. How to provide a POS cutout ? Do we want to put this in the AccessData thing today ? and a cutout on u,v coordinates values (spatial frequencies) should be done. How ? For event list the recent discussion in Obscore show it is not easy to define relationship between the event list concept and the sparse cube concept. -- FrancoisBonnarel 2015-10-29 (3) We're still missing the facility to let validators find out parameters that should return values (the old testQuery thing). It's too late to address this properly now, but we in Registry would be happy to lift the standardId locks in StandardsRegExt if DAL asked for that. In that case, you could say here something like: For SIAv2, it is recommended to use sia:SimpleImageAccess-typed capabilities as soon as their standardIDs are no longer fixed to the SIAv1 value. We'd promise to fix SimpleDALRegExt as quickly as possible. Without a testQuery, it's really hard to build automated validators. We believe we really want those. * I think I agree with that. Where should I put this ? In capabilities section ? -- FrancoisBonnarel 2015-10-29Semantics Working Group ( _Mireille Louys, Alberto Accomazi ) | ||||||||
| Added: | ||||||||
| > > | A few typos need to be fixed as far as Semantics is concerned :
| |||||||
Operations Working Group ( _Tom McGlynn, Mark Taylor)While we recognize that it is important to make progress on this standard, the lack of a validator is troubling. We strongly urge that validation of the new SIA capabilities be treated as a very high priority. DAL standards are particularly appropriate for validation since they have well defined inputs and outputs. Validation helps assure robust services and provision of a validator makes it easier for standard implementers to develop their services by making it easy to identify subtle bugs and possible misinterpretations of the standard. * This question has been adressed during the september TCG teleconf. The "VO Paris data center" group was working on that again since a couple of weeks --IVOA.FrancoisBonnarel 2015-10-29Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )Data Curation & Preservation Interest Group ( Françoise Genova )Knowledge Discovery in Databases Interest Group ( George Djorgovski )Theory Interest Group ( _Franck Le Petit, Carlos Rodrigo )Standards and Processes Committee ( Françoise Genova)--><--
| ||||||||
View topic | History: r48 < r47 < r46 < r45 | More topic actions...
Copyright © 2008-2025 by the contributing authors. All material on this collaboration platform is the property of the contributing authors. Ideas, requests, problems regarding TWiki? Send feedback