Parquet in IVOA

VOParquet Note

Implementation Status

• TOPCAT/STILTS/STIL: protototype implementation at topcat-extra_metaparq.jar.

• • Writes VOParquet metadata into output parquet files by default; uses VOParquet metadata from input files when reading if present.
  • Believed fully compliant with VOParquet note as at 17 Dec 2024.
  • A small example output file is attached: skysim10.parquet

Documents and Presentations

• Presentation from Tucson Interop Nov 2023 (Gregory D-F)
• DPAC/Gaia policy on bulk downloads Oct 2024 (Jos de Bruijne)
• Presentation from Malta Interop Nov 2024 (Mark T)

Meetings

5 November 2024 19:00 UTC Online meeting

Agenda

• Around the table short description of the groups and Parquet-related projects

• Mark Taylor presentation on VOTable metadata with Parquet files
• Jos de Bruijne Parquet Compression algorithms in Parquet
• Path forward - Malta Interop

Meeting minutes

• Jeff Burke (CADC) - Parquet format for TAP
• Mario Juric (U of Washington) - Upload and download large Rubin catalogues
• Vandana Desai (Caltech/IPAC) - Science with large catalogues inspired by Mario's work.
• Jos De Bruijne (ESA) - ESA GAIA DR4 - Use Parquet format (instead of the current CSV format)
• Pierre Le Sidaner (Observatoire de Paris) - Also GAIA mission. Appreciate data access by column
• Gregory Dubois-Felsmann (Rubin) - Rubin very heavy use of Parquet internally and externally. Catalogue releases. Rich metadata in VOTable goes beyond UCDs and UTypes
• Trey Roby (IPAC) - Firefly read and write Parquet file. Very good file. Pushing the edge of how big the files can be.
• Mark Taylor presentation on adding VOTable rich metadata in Parquet

Questions and comments on the presentation

• Gregory D-F: VOTable metadata is important for DataLink, no objection for adding encoding, version support is a must. Valid VOTable medata must comply with the spec. Parquet is not great to work multiple datasets but VOTable should support multiple tables. What happens if there’s a conflict between VOTable metadata nd Parquet metadata.
• Mark T.: Cannot have an empty data element. VOTable Resource has a flag that can be used. Main table can be of type result and other resources can have other types (medatadat).

• Trey R.: How to resolve inconsistencies between Parquet and VOTable. Parquet should have the last word.
• Mario J: Important to specify how to do deal with inconsistencies between VOTabe metadata and Parquet metadata
• Brigitta S: Prefers Parquet support first before generalization (FITS) Example files would be useful for implementation such as astroquery - ex multiple tables with the same file.
• Jos DB: Compression algs. Should start thinking about this as it's one of the core strengths of Parquet
• Gregory D-F. We need to define new TAP result formats. Is there a Parquet MIME type?
• A few participants stressed the importance of moving fast with the adoption of Parquet in IVOA as a lot of projects are currently under way.
• Marco M: Proposed the faster route of writing an IVOA Note for VOTable and Parquet and then explore generalizing it for other types (FITS) and turn it into a standard.

Actions:

• Mark T. and Gregory D-F will start drafting the note

• Mark T. will make his presentation available so that others can comment on it - done
• Mark T. will present progress during the Apps session in Malta
• Apps WG to support the efforst: create Twiki page with useful info, organize meetings post Interop as required, etc.