Minutes of VOEvent and GCN discussion 20 Sep 2022

Attendees: Tess Jaffe, Fabian Schussler, Leo Singer, Dave Morris, Henri Louvin, Judy Racusin, Marco Molinaro, Timothé Roland

• Greg’s comment from Slack: "Maybe in this generation the IVOA could work with the many communities interested in time-domain to see if the VOEvent data model itself - decoupled from XML serialization - is still able to meet their needs."

• Dave’s comment from Slack: I'd like to add to that, we should consider two types of alert/event.

  • The raw Rubin/LSST alert data describe a change in the brightness of a point in the sky, which are generated at 100,000/sec.

  • Then after post processing and filtering, there are astronomy/science events, like a supernovae, which have much richer metadata and linked data associated with them, which occur at a lower data rate

  • Kafka/Avro are probably the right infrastructure for both cases, but with different data models.

• GCN’s VOEvent broker is custom code base and we don't want to maintain it

• Fink and Ample (sp?) agree that they can do VOEvent TP as well

• Leo summarizes issues with VOTP:

  • Problem 1: few (none?) healthy open source implementations of VOTP

  • Problem 2: no users outside of astro, but more general alternatives with multiple open source implementations

  • Problem 3: no authentication built in, e.g., a TLS layer. Proposals to add it, but not implemented in Comet and GPG key distribution is a problem. TLS lack is a big issue.

• Leo in re VOTable data model and serialization

  • GCN deeply integrated into several NASA missions (Fermi and Swift), and we want to unify their alert formats. Makes sense to make this more generic.

  • US optical community moving away from VOEvent and specifically XML.

  • Prefer to focus discussion on VOTP

• Fabian:

  • Working with HESS and CTA, etc.

  • Been playing with Kafka streams.

  • Listener to a Kafka stream is straightforward. <10min

  • Issue is running that long-term. We have one central Comet broker listening to a bunch of VOEvent-based streams. Partial switch to Kafka would be a pain.

  • FRB community is growing and using VOEvent streams. So that is not going away. (TJ: we should be able to convince them to put a bit of effort into producing to Kafka as well, no?)

  • Not impossible to implement both in his system, but only large collaborations would have staff to do it.

  • Lots of followup instruments would have difficulty.

  • Authentication most crucial of Leo’s issues with VOTP. But have never had any issues.

• Dave:

  • Listening to multiple streams doesn’t require multiple brokers, but multiple listeners. One machine.

  • Somebody write something to listen to Kafka streams and redistribute as VOTP streams. [Judy’s notes show she understood “write” to apply only to a description, not an implementation]

  • Just one group to implement it and write a note for the IVOA and kick it off.

• Judy ask Leo to expand on brokers vs listeners.

  • Comet can handle multiple TCP/IP connections to multiple servers,

  • Kafka allows one client to subscribe to any number of topics from that broker.

  • Fabian: Comet also filters out duplicates

  • Leo - that redundancy is addressed by Kafka itself (multiple kafka brokers, load balancing, etc.)

• Judy:

  • Scott Barthelmy did a lot of work to enable GCN to work with a variety of data formats/models.

  • We want to lower the barriers.

• Henri (Timothé working on SVOM):

  • Problems with idea of having a single broker for everybody

    • Development phase, we’re trying to automate notices, and difficult to not have our own dedicated broker for testing. Even if we sent them to a broker that doesn’t release them to the public, it’s a binding that we don’t need and will slow us down.

    • Secondly, during regular ops, SVOM has notices that are internal to the mission and not available to the public. VOTP allows you to send them to a dedicated broker for SVOM only and the others to a public broker.

• Leo:

  • Docker image we provide?

  • Henri: yes, that would work. All our stuff is microservices on Docker

  • Dave: Kafka Docker containers already exist.

  • Apache and Confluent. SCIMMA has their own. GCN doesn’t currently see a need for another one

  • Henri: don’t need a “production” system. But still needs to have same interface so that switch between the two is only a URL address. This is the beauty of Comet.

  • Kafka could do this too

• Dave:

  • True that there is a group for whom Comet is fast and easy.

  • Could build this up for Kafka, though it appears more complicated than Comet.

  • Do we want to make that step now?

• Fabian:

  • Huge user community of Kafka in contrast to Comet, but worried about depending on a tech that we do not control.

• Leo:

  • Changes to Kafka are covered by a KIP process (Kafka Improvement Process)

  • History of backward compatibility with older versions

  • GCN also does have some influence with Kafka project because we are customers of Confluent, and Confluent are the largest contributors to Kafka project (original authors even). We talk with them almost weekly.

• Dave:

  • Think Kafka is now where Docker was a few years ago, original commercial, became open source, and spec now run by an independent standard group.

  • Leo: Confluent relationship to Kafka better than that case for Docker.

  • Fabian: But at some point in future, they could make significant changes and we’d have to change. But reassuring to hear they have remained backward compatible.

• Fabian:

  • Helpful to community once they’ve decided to switch to VOEvent over Kafka, to provide tools to make that switch.

  • Can you use Comet on your new infrastructure to ease the transition?

  • Judy: yes, need to look into cost-benefit

  • Poll the current GCN community?

  • Judy: Scott said dominated by anonymous connections. So we announce via GCN circular.

• Dave:

  • If you have a converter from Kafka to a VOTP stream but demonstrate the improvement of using Kafka, and all new missions easily start with Kafka, maybe more people would switch to Kafka.

• Judy:

  • Our plan is new missions will only be available in our new formats.

  • We could redistribute whatever format we get but there would be chaos.

  • We want to build something for Fermi & Swift and hope others will adopt due to technical merit