*IVOA November 2020 Interoperability Meeting *Data Access Layer - session 2 *Wednesday 18 November 2020 - 22:00 UTC Participants: 68 *Margarida Castro Neves *LineTAP: a proposal for an IVOA relational model for spectral lines Motivation * Few services taking up SLAP/SSLDM * Unsure if new version of SLAP/SSLDM would improve things * Looking at a data model and serving using TAP Goal * One table data model * Use quamntiies to cover as many use cases as possible Main Use Case * A user with a rough idea of the kind of physics in an emitting or absorbing region wants to identify a line in a spectrum * Extending from just atoms to include molecules Current State * Have use cases but need more * Source quantities (i.e. fields/columns) from VAMDC DM * Mapping from VAMDC DM to LineTAP * Started drafting IVOA note * Have a list of ~20 quanities/columns Open Questions * VAMDC has vaccuum and air information, would prefer to have just vacuum in LineTAP and provide errors and some conversion fucntions * What unit to use? Leaning towards Angstrom or m * How to refer to a molecule in the query? Map inchi to name/formula in a table or use an external service * https://github.com/mmpcn/slapvamdc - early phase draft note [François B] Is this single table a view on other tables? [Margarida CN] No this is starting from scratch MD: Well, it's building on VAMDC, which is a distant relationship of SSLDM1... [Marco M] As the communities are the same this will likely replace SLAP/SSLDM [Margarida CN] Yes muchthe same community [Nicolas M] One use case is to have a single query to get the ... *Vandana Desai *Implementation of the IVOA Spectral Data Model at IPAC IPAC has a wide variety of spectroscopy data Adding EUCLID and SPHEREx which are spectra heavy IRSA started project to convert in preparation for these missions NED also has wide range Many (million to billion) spectra meaning DM becomes really important SpectralDM - 1.1 from 2011 but PR for v2.0 in 2016 but stalled - would like suppoort for eschelle spectra Use cases * Search via web/python * Visualise (e.g. firefly) - single or many spectra * Analyse via mission software or open source * Download - can be many 1000s 1. Identify a spectral data product - covered by utype 2. Plot flux v wavelength with errors - can identify columns by param utypes 3. Connect points by spectral order - not in current model - important to IPAC - add column with ucd "instr.order" 4. Change units on both axes - model doesn't use VO units - should this be updated? WOuld like to group flux with flux error to assist this 5. Overplot photometery or SEDs - could be synthetic - can require PhotDM - open question how to do this 6. Plot upper and lower limits for SEDs - no utype, UCD so have some suggestions for ucd and uttype (see talk) 7. Error bars for SEDs - inconsistenty between UCD1+ and SpecDM 1.1 - E.* seems to allow only one following word, but DM uses two - just an update to UCD text? 8. How to represent millions of spectra - can we use something different to VOTable to accomodate common metadata not being repeated. Can SSA satisfy queries for 10k spectra at a time? [Sebastien D] For point 7, maybe just a wording issue in UCD1+ document, but having a UCD1+ composed of several individual words with syntactic flags ESS, or PESS, CSSS is not forbidden [Markus D] Would be good to document common practices and issues in implementation note (and he'd be happy to collaborate on that). [Markus D] See note on time series by Ada for a potential solutions for multiple spectra [Dave M] https://ivoa.net/documents/Notes/LightCurveTimeSeries//index.html [A N] That note is for annotation of lightcurves in one or multiple filters [Markus D] Storing low res spectra in rows, see spectra in table rows: http://www.g-vo.org/tutorials/dfbs.pdf [Jesus S] Units conversion - ... [Jesus S] UCDs and utypes in photDM and associated note may be useful [Sebastien D] see https://wiki.ivoa.net/twiki/bin/view/IVOA/PhotometryDataModel or Note https://wiki.ivoa.net/internal/IVOA/PhotometryDataModel/NOTE-PPDMDesc-0.2-20110512.pdf [Jesus S] Combine [Jesus S] GAIA using tap + datalink - millions of spectra overwhelms SSA [Mark CD] Good to have implementors fo the spectra DM. May also want to have look at the cube model (in WD) [Gregory D-F] would like to send back photometry with spectra - should this be a separate PhotDM table in result? Would like a deeper DM WG telecon to discuss this topic in detail. [Markus D] Well, I´d say SSA works fine for millions of spectra -- I´m running several services of that size.
It´s just not enough...
It´s good when people what to find spectra for a RoI.
It´s not good if people want to do server-side analyses.
 [Emmanual J] SSA maybe works for a specific database schema or constraints, but might be difficult to make it “generic” across different metadata/datasets ?
 .[François B] I would like to review utypes you created for the SED. is it really Data utypes or could Char Utypes fit better ?(FB aga --> data utype / error oin)n each measurement. Char utype error : global error for the whole spectrum *Judith Silverman *Enriched Query-Server Response: DataLinks and Column Groups Talking about enriching VOTable response with groups and service descriptors See https://wiki.ivoa.net/twiki/bin/view/IVOA/InterOpNov2020DAL for slides What do VOTable elements corresponding to Column Groups and Service Descriptors look like? * Add group to associate the position - most of this is same for any query, only the refs need to change depending on select clause * Likewise service descriptor also describes a relationship with the database colmuns - largely not query dependant. How do we introduce a new Column Group or Service Descriptor to the query-server’s repertoire? * have irsa_group and irsa_group_columns tables to control these * have irsa_serv_descriptors and irsa_serv_descriptor_cols for service descs How does the query-server decide whether and with what content to add elements corresponding to Column Groups or Service Descriptors to a given VOTable response? * load new tables along with the tap_schema at startup * populate lookups * Add extra infromation to column defs to include group ids it is a member of , same for service descs * Add count of cols to group/service defs - can then hide group/service unless all columns present * At runtime have position based ids for each column * Count columns in each group/service to determine if group/service to include in response * Then build group/service linking to column ids [Laurent M] Very interesting - could be widely used by others [Marco M, Francois B, James D] +1 [Gregory D-F] +1 from Rubin Observatory/LSST as well - these are exactly capabilities that we need