VOEvent Proposed Recommendation: Request for Comments

This document will act as RFC center for the VOEvent V1.1 Proposed Recommendation.
Specification at http://www.ivoa.net/Documents/PR/VOE/VOEvent-20060629.html
XML Schema at http://www.ivoa.net/xml/VOEvent/VOEvent-v1.1.xsd

In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your WikiName so authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.

Discussion about any of the comments or responses should be conducted on the VOEvent mailing list, voevent@ivoa.net.

Comments from the Community

• (1) This first comment is a sample comment

-- RoyWilliams - 2006-07-11

• (2) I have a comment about the content of the Description element. I agree that it will sometimes be desirable to use plain text and sometimes to use HTML, but for software which encounters the content the current proposal presents the problem of how to present it - does it assume it's HTML and attempt to format it thus (which will look wrong if it's plain) or assume it's plain text and display it raw (which will look wrong if it's HTML) or attempt to guess by looking which it is (which is hard and error-prone). Can I suggest an additional attribute format on Description which may take (at least) the values plain (the default) and html. I'd suggest that it is not limited to these values so that specialised applications can use other formatting methods if they wish to in the future (on the understanding that not all software may know what to do with exotic formatting methods).

-- MarkTaylor - 12 Jul 2006

• Thank you Mark, good call. We suggest that "Description" have a "format" attribute, and that a parser be considered compliant if it recognizes two values: text/plain and text/html, the default being text/plain.

-- RoyWilliams - 2006-07-12

• (3) I have a comment about Param tag. Did you ever try to write XML parser? Do you know how big pain is it when you have attributes and you need to search for value of particular one?

You are making in my point-of-view classical XML-designers mistake. XML was designed to transport well organized data, so they can be machine readable. Param fields with UCDs are not well organized - because you cannot validate document for UCD validity, and you cannot reasonably parse it and store in target language (you can save it to collection holding parameters and then search in parameters..think how much overhead you will create by such decision, and how many tests will you need to perform - and XML was created to remove from you burden of such tests, and you just put them the different way inside).

To solve that, I recomend putting UCDs in tags. So instead of the example:

<Param name="TRIGGER_NUM" value="114299" ucd="meta.id" />
<Param name="RATE_SIGNIF" value="20.49"  ucd="stat.snr" />
<Param name="GRB_INTEN  " value="73288"  ucd="phot.count" />

Let's have something like:

<GRB>
  <trigger_num>114299</trigger_num>
  <rate_signif>20.49</rate_signif>
  <grb_inten>73288</grb_intent>
</GRB>

etc. for other events (transits,..).

In schema that will maternize as choice for different event types.

By letting free UCDs parameters you actually say: we don't care about researching which ever possible events can be transported by VOEvent, so we left submiters freedom to create mess of different parameters, so as results no machine will be able to use VOEvent entry and it will be again human, who have to read it and understood what was author intention. Mayby that's right for scientics, but for computer programmer it's a nightmare. And for robotic telescope operation it is nearly useless.

You really need to define possible content, most probably starting from what we know and use currently (e.g. GCN format is well established) and going to more difficult areas, where GCN example is not available.

-- PetrKubanek - 2006-08-07

While your suggestion certainly has advantages, the best choice is not obvious.

The difference here is between what I call a generic container and a semantic container. The latter is helpful when the scope of meaning is well contained. As you allude, a generic container is necessary when you can't constrain the semantics at the time the schema is defined, perhaps because it may change either depending on the application or just over time. In my experience, sometimes the advantage of Schema-based validation works against you in the face of this change.

If we are to have interoperable semantics that allow machines to interpret these terms, then the terms must be well documented and standardized--whether they are part of the XML Schema or not.

(BTW, just to avoid confusion, UCD refers to the values of the ucd attribute.)

-- RayPlante - 2006-08-07

The difference here is between what I call a generic container and a semantic container. The latter is helpful when the scope of meaning is well contained.

The later is useful if you know the contents the message will carry, the "Param" element was designed the way it was precisely because we have absolutely no idea the semantic content that the message will have to carry. If you look at, for instance, OGLE and GCN messages, the "Param" elements carry totally different things. That's why it has a ucd="" attribute to provide the semantic meaning that you seem to want.

-- AlasdairAllan - 2006-08-07

This is not a mistake, but on the contrary, there was a great deal of thought (and argument) went into it. This is a compromise between structure (special elements) and flexibility (generic Param elements).

The structured approach is certainly easier for the computers to validate, and makes nicer code when the code-binding tools are used. But the problem here is that we don't know in advance what kind of measurements will be made. Perhaps one group measures something called "trigger_num", other VOEvent providers have other names for parameters. It would be very difficult for those providers to use a structured approach. There would be "extra" schemas for each dialect of VOEvent, the authors would need to build and document such a sub-schema, put the schema into a permanent web-accessible repository, and be careful with versioning every time it changes. The subscriber to such events would also have trouble bringing in the correct schema and validating it.

And there's the rub: the more effort we demand from authors, the less likely they are to become part of VOEvent.

By letting free UCDs parameters you actually say: we don't care about researching which ever possible events can be transported by VOEvent,

No. We have said that in one part of the VOEvent there will be data whose structure we cannot know in advance, and that the structure there will change faster than our versioning process.

Perhaps you could help us in a constructive way by defining what kind of parameters the INTEGRAL instruments will be producing when they report an event. We would be happy to help you to add structure in the form of Group elements and UCDs.

When evaluating a VOEvent for possible follow-up, we see different approaches by different groups. Some may look at the name of the author organization and the "importance" score of the event, and know the meaning of that scale of importance. Others may look at the "Why" section where the author has a hypothesis ("80% chance it is a supernova"). Others may know the meanings of the Group names and Param names in the "What" section and be able to decide based on these values.

-- RoyWilliams - 2006-08-07

You are making in my point-of-view classical XML-designers mistake.

I disagree - we are making avant garde XML choices, not classical XML mistakes. An explicit architectural trade-off was made between supporting general purpose params versus elements custom tailored for each alert type. VOEvent happens to be realized using XML, but it isn't an XML technology per se. Just as XML itself has a broader range of applicability than just as a delivery mechanism for the zen of schema.

In an ideal world, your suggestion is superior. We have known this from the beginning. Note that nothing about VOEvent should be taken as forbidding such usage in a future version, say v2.0. We would need to be clear on the VOEvent ground rules (not just the XML rules) for including additional alert-specific schema. The success or failure of these hand-tailored schema would be driven by market pressures, just like VOEvent itself.

The current state of the VOEvent community is different. We are attempting to bootstrap VOEvent from today's complex world of many non-interoperable, non-XML, astronomical alert systems. The choices made under such circumstances may be quite distinct from some idealized vision. We should all be gratified at the success that VOEvent has demonstrated to date. Success speaks for itself.

Mayby that's right for scientics, but for computer programmer it's a nightmare. And for robotic telescope operation it is nearly useless.

Well, actually, it is the robotic telescope community who have adopted VOEvent most enthusiastically. VOEvent compliant code is already driving nightly robotic operations.

It has certainly been entertaining to watch the recurring debates between the astronomy side and the computer science side of the Virtual Observatory efforts. I've yet to see a single issue where one side is entirely right or entirely wrong. The more genuine disagreements tend to come between the scientific programmers and the computer science programmers - the folks who need to make it work. Whatever else is true of all of the many fascinating VO projects - at the end of the day, a pragmatic real world solution must exist and must be proven to be sustainably operable.

You really need to define possible content, most probably starting from what we know and use currently (e.g. GCN format is well established) and going to more difficult areas, where GCN example is not available.

VOEvent is the same type of effort as GCN was originally. GCN-2 will be VOEvent compliant. We are precisely in the business of defining formats, just as GCN has been before us. To require a full formal data-modeling effort in advance of permitting new alert types to be issued, would be to cut the legs out from under the entire concept of issuing alerts for transient astronomical phenomena.

It is precisely the poorly characterized phenomena whose alerts are the most valuable. We could do a wonderful job of crafting a fully specified, semantically rich, content-complete alert for eclipsing binary stars. Few would bother to either author or subscribe to such alerts.

-- RobSeaman - 2006-08-07

Note: this thread of comments continues in a philosophical vein for several more messages, see the mail archive for full details.

Comments from the Technical Working Group

TWG members should add their comments under their name.

Marc Allen (Applications IG)

If IG chairs count, then - I approve.

Bob Hanisch (Standards & Documentation WG)

please edit

Gerard Lemson (Theory IG)

In my reply to Roy (as an IG chair, this morning) I pointed out a problem I have with the dependency of the VOEvent specification (in particular the XML schema) on the STC specification, which as far as I am aware has not been approved as a recommendation yet. If I have missed that momentuous event I applogise, approve of this spec and ask you to ignore the rest of this email.

In my opinion this dependency on a non-approved spec should prevent recommendation of the current specification for technical reasons (until STC has been approved and its possible changes can be incorporated into the current spec, or until the dependency has been removed). But this issue seems to have a formal, procedural side as well (do we want to make incomplete recommendations, do we want to constrain STC development not to endanger VOEvent) and therefore should maybe be discussed by the exec.

[the following is a response from Bob Hanisch] We have other cases in which one IVOA standard relies upon another, and we do not require that all such dependencies proceed in order. I do not think this is a problem as long as the dependencies are understood and the interfaces between standards are clear. Typically the external dependency comes as a package as is simply adopted in total. There is undoubtedly some risk of introducing incompatibilities, but I think it is small and probably has to be tolerated if we are to make progress. We must all remember, too, that our standards evolve and problems can be fixed.

Tony Linde (Resource Registry WG)

The reason that VOEvent and VOTable has stuff like "param" tags and the like (and VOTable has a lot worse) is because they are dead-end technologies. They cannot be used for anything except being looked at by humans (possibly mediated by pretty graphics). If they were used in downstream applications (by people other than those creating the standards) they would cause errors and failures and would return false data and so would not have come into existence or would have been replaced before getting ratified.

The problem will come when people do want to use them to do something useful (Rob's V2) because the argument will be that everyone now knows the format (except, like VOTable, everyone will interpret it differently) and so it cannot be changed so stuff will be bolted on the side to try and make it work - like the utype in VOTable to take account of data models (which is the most daft fudge I've seen in over 30 years of development) . And, of course, that will simply cause even more problems and errors downstream.

This is nothing to do with pragmatism - the schema approach is the more pragmatic - nor to do with ease of development - as Petr has said, the schema approach is the easier to develop, but only to do with what looks easy.

Yes, I know I've said all this before but at least this reiteration might save Petr wasting his breath and time trying to convince people otherwise.

I will say that at least VOEvent does have its core data in properly tagged elements: the param data is useless for downstream processing but at least the core information can be used - unfortunately the same cannot be said of VOTable where none of the data can be used downstream with any certainty. (But I fully expect sometime in the future for someone to want to add the utype attribute to the param tag making VOEvent just as bad as VOTable.)

I think the inclusion of the faux-xml element, Param, in this spec will lead to technical problems in the future and so I cannot give my approval.

Jonathan MacDowell (Data Models WG)

I approve the VOEvent Proposed Recommendation. I don't think it's perfect, but I think you have done a great job in reconciling different agendas and interests - congratulations. I particularly like the elegant "param" container Seriously, there has to be a balance in our use of XML for strictly semantically defined information and our use of generic containers in the traditional and largely successful astronomical tradition. VOEvent enforces the bits that robotic telescopes have to act on, and leaves looser the capability to convey extra data that programs can format for the astronomer, but do not need always to understand (although specialized programs may). So I respectfully disagree with Tony.

Reagan Moore and Francoise Genova (Data Curation & Preservation IG)

I approve the proposed recommendation.

Having written programs to parse semi-structured data, either params or specific attributes requires new coding effort to know what to do with the added data. No matter which way is used to add more information content, the programs parsing the data will have to be revised to take advantage of the increased information content.

Francois Ochsenbein (VOTable WG)

Your document is very well structured, clear -- without the verbosity and redundancy that is so common in the this kind of document -- bravo !

I won't enter in the "Param" controversy -- a pragmatic approach in which the fundamental parameters have to be present, plus a facility to add computer-interpretable additional parameters look really adequate.

Just a few details:

-- a table of contents would be useful -- section 3.2.3 (date) -- if time is UTC, shouldn't the ending Z be removed ? -- section 3.3.1 (param): - the blanks within the quotes: name="GRB_INTEN " would better be removed - adding the units attribute in one of the examples (of sections 3.3.1 and 3.3.2) could be more illustrative

which are not fundamental...

Pedro Osuna (VOQL WG)

After having gone through the document, checked that there seem to be reference implementations on VOEvent (Caltech, Los Alamos, Exeteer) and that the whole procedure for Standards and Processes has been followed properly, I give my approval for the Proposed Recommendation to be upgraded.

I understand nevertheless that and eventual upgrade does not mean that the document is frozen forever, and should problems or new needs arise, new versions could be discussed and eventually generated.

I would also like to ask for RFC periods to try to avoid, whenever feasible, big holidays periods so that all parties can have time to react properly.

Andrea Priete-Martinez (Semantics WG)

I found the document well written, so well that I could understand almost everything in it! So, I approve the VOEvent PR document

Guy Rixon (Grid & Web Services WG)

A few exec meetings ago (Kyoto?) it was confirmed that a standard in PR review should be supported by two or more interoperating implementations. Further, it was decided that the interoperation trials should be written up as an IVOA note. I don't see a note for VOEvent, and, not having been involved with the development of the standard, I have no insight into the success or failure of the trials.

I have no objection to the content of the standard. I understand Tony's concerns about the use of XML schema, but I feel that the proposed group/param system is the lesser evil in this case. If the schema were made open, then it would have to be made completely open: to allow any elements, those elements being defined by the authors and publishers in external schemata. This makes practical difficulties for the handling of VOEvents (it rules out static binding to Java objects, for example) but more importantly, it makes the archiving of the VOEvents too fragile. A VOEvent for which the schemata have been lost is essentially unreadable (like an old tape in an obsolete format). A VOEvent with groups of params including types and UCDs can at least be decoded by humans, even if the written definition of the parameters is lost.

I suggest that certain standard kinds of packets will emerge in normal use. These can be formalized in a later version of the VOEvent schema. The machine-readablity can be improved over time.

Therefore, I would approve this document if it were supported by a record of successful trials. It's rumoured that the trials were wildly successful and the format does work; but I have no way of being sure of this. If the exec is satisfied with the trial results and the lack of docmentation, then I have no objection to VOEvent becoming a recommendation. However, I would then expect a similar derogation of process for other pending standards.

Doug Tody (Data Access Layer WG)

The VOEvent group has done an outstanding job on both this specification and on engaging the astronomical community in the project. It is difficult to strike the right balance between technology and what we can expect the science community (as well as the broader federated astronomical software community) to actually use. I think the VOEvent effort has got it about right, hence I approve the PR. In fact I think VOEvent has been one of the more successful things we have done within the IVOA recently. Regarding the PARAM issue specifically, I strongly support this construct and the position taken by the specification and by the VOEvent working group.

The recent discussions on the VOEvent list and again here have been interesting; rarely have I seen such a strong difference of opinion expressed within the IVOA. It should be remembered that with VO, we are dealing mainly with science software - software and infrastructure that astronomers will use to do their research, including often writing their own software. This is different than a typical software engineering project where we are building a single top-down engineered system and it just has to work. Tony argues for the latter but what we are really doing in VO is the former. There is not a problem in the IVOA with too little technical expertise. The real risk is in not understanding the problem domain, and building something which may be technically wonderful but which is in the end not used. It happens all the time.

Regarding the technical points raised - I have hesitated to get into this due to the strong opinions expressed as it may just inflame the argument further to no end. But it is nonsense to claim that a general container-based approach like PARAM (or VOTable, a RDBMS, FITS, a parameter set, SRB, etc.) cannot be used for anything except being looked at by humans, as has been suggested. While XML is a wonderful technology, an awful lot of software was written in the world before XML and XML Schema came along. While XML schema can verify XML documents, it cannot be used to semantically verify or manipulate a complex object such as we routinely deal with in astronomy software; in general only custom class code can do this. On the other hand, most astronomy software does not want to know anything about complex low level technologies such as XML.

Regarding UTYPE: I think a major misunderstanding here is that UTYPE has something to do with XML (hence the claim that VOTable+UTYPE is a kludgy way to do something that native XML plus a schema can do better). UTYPE provides a simple means to deal with the elements of a data model in an abstract fashion, independent of implementation. Go back and look at the RSM document for example - it is full of UTYPEs. The first paragraph in section 3 is basically explaining what UTYPE is.

Nic Walton (GGF Astro-RG)

Voevent looks fine to go from my point of view.

Roy Williams (VOEvent WG)

(no comment as this is the submitting WG)