*IVOA DAL WG Running Meeting #9 *Wednesday 24 March 2021 - 10:00 UTC - vconf Participants: (Marco Molinaro, James Dempsey, Markus Demleitner, Brent Miszalski, Mark Taylor, Ada Nebot, Dave Morris, Jesus Salgado, Francois Bonnarel) Agenda: 1. (AAO Data Central) Optical Data Center SSA service 2. DAL related sessions at next Interop (brainstorming) *Notes: *1. ODC SSA service oveview (Brent) on datacentral SSA service Hosted Datasets: Optical spectrascopic surveys (AAT and UKST) Python 3 (Django) * SIA 2 (since July 2020) for cutouts * SSA (Feb 21) * TAP Have loaders for specutils to load the ODC data Metadata is important * Obscore django model which extends base ODC model * Hetereogenous data - multiple spectra, mix of spectra and images in one file * Needed to have a consittent way to ineract wiht the data * Created 19 astropy spectutils loaders to handle reach format in a standard way and to extract obscore metadata Challenges/approaches * Validate SSAP params with regexp * Have added BANDREST keyword for working with data with a known redshift * Query using filters on the obscore datamodel * Score the results and order results by score * Have extra data stored in yaml file Output data format * Respond in VOTable by default * Can reuqest FITS easily * 1D spectra from access_url, added headers from Data Central obscore metadata * Original spectra - full_data_url - may be non standard format, more complex to work with Many examples, using TOPCAT, PyVO, Astropy SSA+HIPS - spectra and cutouts * Image cutouts using HPS2FITS at CDS to show against spectra * use astropy to drie display SSA+API * Combine catalogue query to produce interactive plot to drive SSA requests TOPCAT and SPLAT * Lots of testing in TOPCAT Future * Pipeline as a web service * See ADASS 2020 Poster * Python makes it easier to expand * e.g. MOC as input using MOCpy * IFU survey would need new SODA/datalink service to create maps etc * Obscore is inside django so not externally accessible, would like to add TAP access to it * More towards spectra in SIA 2 sounds good * Register services - discussed with Mark Allen. MD has offered to help Data Aggregation Service * Combine aladin lite sky view with SSA query and image query to display on Aladin Lite [FB] Why would SIA and SSA merging be a good thing? [BM] Maintenance - obscore has the datatype already there. Each service has a cost to the user- have to track it down, learn interface. New datatypes (e.g. time domain surveys) will also need support - might be easier in a sngle protocol rather than creating new protocols for each one [MM] Dataset discovery protocol is the aim. Can't constantly create new protocols. Saves database field prolifieration [MM] You have both iriginal and standarised formats. How hard was it to do this? [BM] Not too hard, specutils is quite advanced and made it easier. Some formats (e..g SDSS) harder than others. Have built robust loaders. Much of the effort has been how to map to django obscore model. Have an example on how to load 6DF spectra. [MM] How reusable is the implementation? COuld someone else use it to setup an SSA instance? [BM] Open to helping. Loader code is fairly open but ObsCore code isn't so some work would be needed there. Might be easier if Obscore was available via TAP Demo of matplotlib tool to allow selection of spectra from CM diagram * CM diagram generated from ADQL query * Select source, shows spectra Demo of Data aggregation service * Page combining quryies to panstaars and skymapper SIA services * RACS HiPS overlays * Have hacked SAMP connector from visier to allow use from TOPCAT *2. DAL sessions brainstorming [MD] Would like to bring up role of capabilities. WOuld like to tackle availability part of VOSI. Probably a DAL task. WOuld liek some time to discuss. Will probably introduce on mailing list first. [MM] VOSI in GWS, but they;ll be focussing on science platforms and A+A, so fine for DAL to join with them on this topic. Can try to set up a discussion in next runing meeting including GWS if there is interest. [MM] Have a partial mirror of some of our data [MD] Partial mirrors aren;t supported [MM] Can split service to allow registration in that case [FB] DAL slot for templating - have use cases at CDS and working on intregrating with Aladin - may not be ready for May though [FB] Could have a discussion on extending SIA to support spetcra, time series based on obscore extensions Ada and Mireille Louys are working on [FB] SODA and datalink likely not much to discuss [FB] Common wiht Apps - descriptor mechanism in votable rather than leaving in datalink [MM] Would need to chat with apps to find out others to contribute to these [AN] Would be great to have extra help with ObsCore extension https://volute.g-vo.org/svn/trunk/projects/time-domain/time-series/note/TSSerializationNote.pdf I would like to take parts of this document to create an extension of obscore for TS. Minimum key ingredients are there. [BM] Would be happy to help - had already been talking to Mark A. Looking at time series soon [AN] WIll send draft by email [MT] Joint activity Ops and DAL session on improving standards compliance of services. Sent out email offer of taplint help but didn't get any responses [JD] WIll send resiults from checking CASDA [JS] Discovery along with format of files in the service to be consumed by client. Important for spectra and time series (a basic common format can be used by general tools, expert users can then fall back to full ortiginal spectra) [AN] Good point. Note for VOTable aimed at giving easy format https://github.com/AdaNebot/TimeSeries [FB] Could minimum format be discussed in the DM workshop [MM] Multiple access URLs in the response without resorting to datalink?