VOEventStream: Metadata for Collections of Events
Version 0.1

IVOA Working Draft

Status of This Document: This is a Working Draft produced by the IVOA VOEvent Working Group.
Latest version of this document:
http://www.ivoa.net/Documents/latest/VOEvent.html

Authors:: Alasdair Allan, University of Exeter, UK; Matthew Graham, California Institute of Technology, USA; Ray Plante, University of Illinois, USA; Rob Seaman, National Optical Astronomy Observatory, USA; Roy Williams, California Institute of Technology, USA
Editor:: Roy Williams, roy@caltech.edu

Abstract

The VOEvent format has been a recommendation of the IVOA since 2006, and is gaining wide acceptance. The purpose is to announce celestial transient events that may need rapid follow-up observation, and it is designed so that automated systems, as well as humans, can make decisions about VOEvent packets. This document defines a way in which the Registry infrastructure of the virtual observatory can be used to allow publication, discovery, and utilization of VOEvent resources. We define two types of resource as registry extension schemas: VOEventStream and a VOEventServer. The stream schema describes a scientifically coherent collection of events; that come from the same motivation, team, project, or experiment; each event of a stream uses the same vocabulary (parameters) to describe events; while the primary thrust of the VOEvent work is responding to future events, a stream can also describe events that have already happened. The server schema describes which computers and interfaces can be used to send out future events (subscription capability) and/or can run queries on past events (query capability); it is described by the streams that it knows, and then by the capabilites it offers for each of those streams.

1. Introduction

"Real-time astronomy" is an increasingly important component of modern astronomy, the study of rapid phenomena in the sky such as supernovae, cataclysmic variables, blazars, gamma-ray bursts, microlensing, etc. Traditional event reporting systems include the Central Bureau for Astronomical Telegrams and the Astronomers Telegram, written in natural language, and inform astronomers so that they can personally run follow-up observations. However, real-time astronomy can discover more science by automation: for rapid decisions in seconds without the need to wait for humans to wake up. Automation is also necessary to deal with large numbers of events, such as from LSST in years to come, and automation also allows machines to fetch and collate other relevant data that may elucidate the reason for an event, and also improve the ability to decide and prioritize follow-up observation.

The primary conceptual idea of this document is the VOEventStream. It is analogous to an astronomical catalog, in the sense that all the entries of the stream are uniformly observed, with each entry having the same parameters. A stream is also like a catalog in that it has authors who take scientific responsibility for the data. The most obvious difference, however, is that the catalog entries come from observations in the past, whereas an event stream may have both past and future events. An example of a stream is the set of events published by the Catalina Real-Time Sky Survey: by repeatedly scanning the sky, this project finds and announces each night a handful of sky positions where the source is much brighter than previous observations. Another example could be the stream of events from the NASA Swift orbiting observatory.

Now that the structure of the VOEvent packet is converging, interoperation is enabled, and we should turn attention to the infrastructure that carries the events: network protocols, standard services, and how a user can discover, find, and utilize VOEvent resources. We will achieve this with the Registry infrastructure of the virtual observatory, a distributed, international metadata repository. In this document we will define how VOEventStreams are recorded in the registry, and also the VOEventServers, containing access points for getting event data in various ways.

In forming these collections of events as streams, the question of granularity arises: should we, for example, make all the high-energy burst events into a single stream (NASA/Swift + ESA/Integral + NASA/Fermi etc)? Or should each satellite have its own event stream? Or should each instrument of each satellite have its own stream? This is of course a matter of judgement; but there are some guidelines. First, a stream should have single point of leadership and scientific responsibility, so the idea of combining NASA and ESA resources into a single stream seems inappropriate. Second, a stream should have a single set of parameters for the event vocabulary, and the greater granularity implies a much larger number of Params, most of which would not be used in any given event. However, splitting into many small event streams is also inappropriate, since there would be much more metadata to define for the Author (once of each stream), and much more work for the Subscriber (once for each stream).

1.1 Using the Registry for VOEvents

The registry enhancements proposed here will make it easier to work with VOEvents. Some use cases are:

Resolving an event: the unique identifier for an event is a concatenation of a stream identifier and an event number within that stream. A question to the Registry finds which VOEventServer has query capability for that stream, and then that server can be probed to get the full event metadata.
What is this event: the event identifier contains the VOEventStream identifier, from which the Registry can produce the descriptive text, contact information for the responsible people, and so on.
Subscription to a stream: VOEventServers are being built to push events to those who want them by a variety of messaging protocols; in order for an astronomer to set this up, the Registry is asked which servers provide the "subscription capability" for the given stream, so that the astronomer can work with that server to set up future delivery of events.
Creating a Stream: A survey or project may wish to publish am event stream. The Registry will provide a way to define the nature of the coming events (VOEventStream) and where to get them (VOEventServer), and thus allow others to find and subscribe to those events.
What does this Parameter mean: Each event has a set of keyword parameters; the Registry can be used to find out their meanings.

2. Identifiers and VOEvents

The IVOA has established a system of identifiers for resources, meaning a way in which a metadata record can be retrieved about that resource; in other words, a global digital library for astronomy resources. The client may be in any country of the world, contacting a local registry node, which replicates all resources -- every node knows what every other node knows. Resource identifiers for the VO begin with ivo:// and are also called "IVORN" for International Virtual Observatory Resource Name. The identifier syntax is set up in a precise way to give control to both supplier and registry maintainer:

ivo://my.datacenter.here/resourceName#localName

By splitting this into pieces, we can find other IVORNs buried inside. The IVORN rules say there can only be one pound (#) sign (or none) in the IVORN. If so, the localName (after the #) is something not interpreted to the registry system. The part before the #, or the whole IVORN if there is no #, is like a call-number in the library, it is a reference to something known to the IVOA registry system: a dataset, a cone-search service, a workflow component, an organization. In order to create your own IVORNs, you will need an Authority ID, (for example ivo://my.datacenter.here), which most astronomy data centers already have.

We propose in this document to add two new kinds of resource (also known as extension schema) to those that the registry handles already. We do not register VOEvents themselves, but rather a coherent collection of events that we call VOEventStream. Each VOEvent is thus a representative of a class of events (the VOEventStream). The other extension scheme is VOEventServer, which lists one or more streams, and how to either pull event data from the past, or have event data pushed as soon as events occur.

The proposed syntax for the IVORN for a VOEvent is:

ivo://my.datacenter.here/streamName#eventName

An event is resolved at the registry as follows: By splitting on the # sign, we can look up the stream in the registry and find the meaning of the quoted parameters. We can query the registry for services which handle that stream, in order to probe the event repositories for the event primary data and annotations that may have arisen about that event. Different repositories can add new data as they wish.

3. The VOEventStream Metadata Record

3.1. Identifier

As noted above, each VOEventStream will get an IVORN upon registration, that can be used as the root for the IVORNs for events that are members of that stream.

3.2. Curation, Content, Coverage

Since the VOEventStream is defined as an extension of the current registry schema, much of this information is inherited from there. Therefore we describe only briefly, and refer readers to the registry documents and the examples below.

Title and Shortname: the first is 3-10 words describing the nature of this stream, and the second is a single abbreviation word for the stream.
Curation: This section contains the IVORN for the publisher, with creator, date, and contact information.
Content: This section defines the nature of this content as a set of subject categories (eg "radio astronomy", "gamma-ray bursts"), and it has description, a text description of the stream, in terms of meaning, science, team, experimental method, relation to other streams and surveys, etc etc. If HTML is included here, it should be quoted with an XML CDATA section. Also here is a reference URL, which points to web pages for further information about the stream.
Coverage: This means the sky coverage, expressed in STC, the simplest representation being AllSky, or another being a cone of the sky. The temporal coverage of the stream can also be expressed here: for the past there can be arbitrary detail about observation periods, and for the future an expectation of observing periods. Spectral coverage can also be expressed here.

3.3. Parameters in VOEvent

A key feature of the concept of VOEventStream is that each stream has a defined set of named "parameters", and each event that is a member of the stream should use only parameters that are selected from the list in the stream definition. The parameter set of an event is analogous to a FITS header: keyword-value pairs that convey specific metadata about the binary part of the FITS file. However in the case of events (unlike most FITS files), the set of possible paramteters and their meanings is not only well-defined (description, units, links etc), but also machine-readable.

In VOEvent, the parameter definition follows, to a great extent, the VOTable standard. Each Param element has:

name: the name of the parameter, eg "JulianDay", "Rmagnitude"
description: a short description in natural language of the parameter.
ucd: the semantic type of the parameter, for example "time.epoch" for a Julian Day, or "phot.mag;em.opt.R" for an R magnitude.
unit: the units of the quantity, if appropriate, for example "day", or "mag" for the examples above
dataType: the kind of quantity that the value has, for example "float", "char" What about enumerated values eg TelecopeWorkingProperly could be "yes" or "no"
arraySize: if there are multiple primitives in the value, this field should either be the number of primitives or "*" for variable-length. The default value of this is 1.
value: the value of the parameter, a number or string, or an arry of such, as defined by the arraySize attribute.

Here is an example of a parameter element from the VOTable specification, which is also valid in the <What>; section of the VOEvent:

<Param name="max" ucd="meta.number" datatype="int" value="50">
    <Description>Maximal number of records to retrieve</Description>
</Param>

In the next section, we split the defintion of the parameter from its value. The definition resides in the regsitry with the VOEventStream, and the value resides in the VOEvent itself.

3.4. Parameter Templates in VOEventStream

Now we come to the new version, where the definition of a parameter's meaning is separate from its value. In the VOEventStream, we define a ParamTemplate to be like the Param shown above but without the value of the parameter.

<Param name="max" ucd="meta.number" datatype="int">
    <Description>Maximal number of records to retrieve</Description>
</Param>

and therefore all that is needed in the event itself is the name-value association:

<Param name="max" value="50">

All of the other information can be added if the author desires, so that the VOEvent can stand by itself. But when the stream from which the event comes has been properly registered, only the name and value are strictly necessary.

Parameters can be collected into groups in VOEvent, for example a measurement and its error estimate can be combined:

<Group name="Rmag">
    <Param name="value" ucd="phot.mag;em.opt.R" datatype="float" value="17.3">
    <Param name="error" ucd="phot.mag;em.opt.R;stat.err" datatype="float" value="0.1">
</Group>

To build a ParamTemplate for a VOEventStream, the same structure can be used, but without the "value" attibutes. Note also that VOEvents and VOEventStreams may not have Groups within Groups: a Group can only contain Param elements, not other Groups.

The same rules of uniqueness are in force for ParamTemplate and Groups in VOEventStream, as for Params and Groups in VOEvent:

Each Param must have a name. A Group without a name is equivalent to having a name which is the null string.
For Params not in a Group, names must be unique among them.
For Params in a given Group, names must be unique among them.
Groups must have unique names, (meaning in particular that only one Group can have no name).

4. The VOEventServer Metadata Record

Once the stream is defined in the registry, the question is how to find it, with the kinds of services that might be wanted. These are known as Capabilities in the VO registry model. Three kinds of capability that we can mention are subscription, formal query, and informal query. If a server supports a subscription capability, it means that a client can submit a criterion (for example "R magnitude brighter than 17"), and events will be sent by the server in the future which satisfy that criterion. A formal query capability implies the existence of a formal request-response protocol by which queries can be made and results returned; this will be the "Simple Event Access Protocol" when it becomes an IVOA recommendation. An informal query capability means a web page with forms by which a collection of events can be browsed in some way.

The capability of a server is split further into Interfaces, which express precisely how the capability can be accessed by a client. For example, subscription capability could be achieved by having email, instant messaging, or text message sent; these are three different interfaces to the subscription capability, and there is different metadata associated with each (email address vs phone number for example). Having mentioned these basic concepts of the VO registry, we cna now define the parts of the VOEventServer definition:

4.1. Curation, Content

This is about who is running the service, description and links about the service, and other information. It is precisely the same kind of XML as all registry resopuirces have, as alluded to in the section above.

4.2 VOEventStreams

Here the server record lists the streams to which it can provide access, with a set of <VOEventStream> elements, each of which has the IVORN of a stream. Further into the VOEventServer document is where ways of accessing those streams are defined.

4.3. Capabilities

Capabilities are expressed in the VOEventServer as one of either "Subscription", "SimpleEventAccess" or "Query". Each of these capabilities can be available for each stream, perhaps a subscription capability for stream X and a SimpleEventAccess capability for stream Y.

4.4. Interfaces of a Capability

A capability is a somewhat abstract notion, for example the general idea of subscribing to event streams. To make this concrete, each capability lists the "interfaces" that it supports. For example a subscription may be implemented through email, or through text messages, or through a robot-voice calling a cell phone: each of these is an interface.

5. References

6. Examples

6.1. Example of a VOEventStream


<?xml version="1.0" encoding="UTF-8"?>

<Resource xsi:type="voe:VOEventStream" created="2008-09-23T00:00:00Z" updated="2008-09-23T00:00:00Z" status="active" xsi:schemaLocation="http://www.ivoa.net/xml/VOEventService/v0.1 VOEventService.xsd" xmlns:vr="http://www.ivoa.net/xml/VOResource/v1.0" xmlns:voe="http://www.ivoa.net/xml/VOEventService/v0.1" xmlns:stc="http://www.ivoa.net/xml/STC/stc-v1.30.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">

  <validationLevel validatedBy="ivo://nvo.caltech/registry">1</validationLevel>
  <title>Catalina Real-time Transients Survey VOEvent Stream</title>
  <shortName>catot</shortName>
  <identifier>ivo://nvo.caltech.voeventnet/catot</identifier>
  <curation>
    <publisher ivo-id="ivo://nvo.caltech/voeventnet">VOEventNet</publisher>
    <creator>
      <name>Andrew Drake</name>
    </creator>
    <date role="creation">2007-11-01</date>
    <contact>
      <name>Andrew Drake</name>
      <email>ajd@cacr.caltech.edu</email>
    </contact>
  </curation>
  <content>
    <subject>optical</subject>
    <subject>transient</subject>
    <description>The Catalina Real-time Transient Survey (CRTS) uses data obtained by the Catalina Sky Survey (CSS) to search 
    for optical transients. CSS observations are usually taken 21 nights per lunation.</description>
    <referenceURL>http://www.voeventnet.org</referenceURL>
    <type>Other</type>
    <contentLevel>Research</contentLevel>
    <contentLevel>Amateur</contentLevel>
  </content>
  <facility>CSS Schmidt Telescope, Mt Bigelow, AZ</facility>
  <coverage>
    <stc:STCResourceProfile>
      <stc:AstroCoordSystem xlink:type="simple" xlink:href="ivo://STClib/CoordSys#UTC-ICRS-TOPO" id="VOEvent_UTC_ICRS_TOPO"/>
      <stc:AstroCoordArea coord_system_id="VOEvent_UTC_ICRS_TOPO">
        <stc:AllSky/>
      </stc:AstroCoordArea>
    </stc:STCResourceProfile>
    <waveband>Optical</waveband>
  </coverage>
  <ParamTemplates>
    <Group name="Detection01">
      <Param name="magnitude"       ucd="phot.mag;em.opt.R"   unit="mag"  dataType="float"/>
      <Param name="mag_error"       ucd="phot.mag;stat.error" unit="mag"  dataType="float"/>
      <Param name="ID"              ucd="meta.id"                         dataType="long" />
      <Param name="average seeing"     ucd="instr.det.psf"    unit="pixels" dataType="float"/>
      <Param name="average background" ucd="instr.background" unit="cts"    dataType="float"/>
      <Param name="JD"              ucd="time.epoch"          unit="days" dataType="float"/>
    </Group>
    <Group name="Detection02">  ....  </Group>
    <Group name="Asteroid params">
      <description>If this were an asteroid, these would be the orbit params</description>
      <Param name="Opposition angle"   ucd="pos.posAng"    unit="deg" dataType="float"/>
      <Param name="Inner motion"       ucd="pos.pm"        unit="arcsec/min" dataType="float"/>
      <Param name="Outer motion"       ucd="pos.pm"        unit="arcsec/min" dataType="float"/>
      <Param name="Apparent motion"    ucd="pos.pm"        unit="arcsec"     dataType="float"/>
      <Param name="Timespan"           ucd="time.interval" unit="min"   dataType="float"/>
      <Param name="Ecliptic Longitude" ucd="pos.ecliptic.lon" unit="deg"     dataType="float"/>
      <Param name="Ecliptic Latitude"  ucd="pos.ecliptic.lat" unit="deg"     dataType="float"/>
      <Param name="Ecliptic Inclination" ucd="pos.posAng"  unit="deg"   dataType="float"/>
      <Param name="Motion uncertainty RA" ucd="stat.error.sys" unit="arcsec/min" dataType="float"/>
      <Param name="Motion uncertainty Dec"ucd="stat.error.sys" unit="arcsec/min" dataType="float"/>
    </Group>
  </dictionary>
</Resource>

6.2. Example of a VOEventServer

<?xml version="1.0" encoding="UTF-8"?>

<Resource xsi:type="voe:VOEventServer" created="2008-09-23T00:00:00Z" 
updated="2008-09-23T00:00:00Z" status="active" 
xsi:schemaLocation="http://www.ivoa.net/xml/VOEventService/v0.1 VOEventService.xsd" 
xmlns:vr="http://www.ivoa.net/xml/VOResource/v1.0" 
xmlns:voe="http://www.ivoa.net/xml/VOEventService/v0.1" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

  <validationLevel validatedBy="ivo://nvo.caltech/registry">1</validationLevel>
  <title>VOEventNet VOEvent Server</title>
  <shortName>voeventnet</shortName>
  <identifier>ivo://nvo.caltech/voeventnet</identifier>
  <curation>
    <publisher ivo-id="ivo://nvo.caltech/CACR">Center for Advanced Computing Research</publisher>
    <creator>
      <name>Roy Williams</name>
    </creator>
    <date role="creation">2006-04-01</date>
    <contact>
      <name>Andrew Drake</name>
      <email>ajd@cacr.caltech.edu</email>
    </contact>
  </curation>
  <content>
    <subject>VOEvent</subject>
    <subject>transient astronomy</subject>
    <description>VOEventNet gathers streams of astronomical alerts and reports in a common format, and acts as a 
    clearinghouse, so that both people and robotic systems can get the alerts quickly enough to respond with 
    follow-up observations.</description>
    <referenceURL>http://www.voeventnet.org</referenceURL>
    <type>Other</type>
    <contentLevel>University</contentLevel>
    <contentLevel>Amateur</contentLevel>
    <contentLevel>Research</contentLevel>
  </content>
  <voeventStream>ivo://nvo.caltech.voeventnet/catot</voeventStream>
  <capability xsi:type="voe:SimpleEventAccess" standardID="ivo://ivoa.net/std/VOEventSEAP"/>
  <capability xsi:type="voe:Subscription" standardID="ivo://ivoa.net/std/VOEventSubscribe">
    <validationLevel validatedBy="ivo://nvo.caltech/registry">1</validationLevel>
    <interface xsi:type="voe:Jabber">
      <accessURL>http://moriori.cacr.caltech.edu</accessURL>
      <securityMethod standardID="ivo://ivoa.net/std/UserPass"/>
      <feedNode streamId="ivo://nvo.caltech.voeventnet/catot">home/moriori.cacr.caltech.edu/catalina</feedNode>
    </interface>
    <interface xsi:type="voe:RSS">
      <accessURL>http://www.voeventnet.org</accessURL>
      <feed streamId="ivo://nvo.caltech.voeventnet/catot">http://www.voeventnet.org/feeds/Catfeed.xml</feed>
    </interface>
    <supportsFilters>true</supportsFilters>
  </capability>
</Resource>

VOEventStream: Metadata for Collections of Events Version 0.1