DAL_DM_TDIG_ShanghaiSession2017 < IVOA

IVOA Web>IvoaVOEvent>DAL_DM_TDIG_ShanghaiSession2017 (2017-06-09, AdaNebot) (raw view)
---+ Summary from Shanghai DAL/DM/TDIG session:

---

<br />%TOC%

---

---++ [[%PUBURL%/IVOA/InterOpMay2017-TD/TD_Cube.pdf][ NDCube Time domain application (M. Cresitello-Dittmar) ]]

   * Models: 
      * Coordinates: coo, frame, systems
      * Measurements: measure values + errors
      * DatasetMetadata: provides high level data description and supports access & discovery
      * Cube
   * Example Serialisation: 
      * real-world datasets
      * [[https://volute.g-vo.org/svn/trunk/projects/dm/CubeDM-1.0/examples/][example files]]
      * validated against schema
      * validation against model spec in progress
   * Time Series in Cube 
      * Note delivered to DM (Feb 2017): high level of correlation with cube model, productive discussion
      * Concerns fit into 2 categories: 
         * modeling (concepts not properly related)
         * annotation (not meeting user expectations)
      * Model changes: 
         * added concept of _dependent_ to DataAxis
         * strict relation of Dataset/Cube loosened
         * TimeSeries of spectra &mdash;&gt; abstracting DataAxis tree
         * Requirements added: &ldquo;model agnostic clients should be able to find basic element with minimal specialised knowledge&rdquo;
         * Moved coords/meas from pattern model to realised base model
---++ [[%PUBURL%/IVOA/InterOpMay2017-TD/MMolinaro_TS_ShanghaiTDDALDM.pdf][Time Series use cases at INAF (M. Molinaro)]]

   * Two science cases 
      * GAPS Exoplanets: radial velocity series, plus other type of data 
         * Dataset and content description
         * Simple use case: 
            * datasets discovery: date, rv & delta_rv, CCF param, stellar activity indexes.
            * access/retrieve time series: including all instrumental details and host star characteristics.
            * link points to original / reduced spectra
      * TSRS Space Weather: polarimetric fluxes 
         * Dataset metadata and content description
         * Simple use case: 
            * Search by observation date/time
            * Retrieve/cut available data
   * Probably not issue at model / serialisation level.
   * Discovery and access to check.
---++ [[%PUBURL%/IVOA/InterOpMay2017-TD/TimeSeriesSVO.pdf][Time Series and the VO (M. Alacid & E. Solano):]]

   * Data Access: 
      * SSAP: SSA is capable of describing most tabular spectrophotometric data, including time series and spectral energy distributions (SEDs) as well as 1-D spectra.
      * According to the SSAP Protocol, the Dataset.Type value must be &ldquo;Spectrum&rdquo;
   * Data Model: 
      * Spectral DM: Metadata information: position, time frame, flux, spectral frame, DataID and Curation, data serialisation as FITS. CoRoT light curves described with the Spectral Data Model can be managed with VO tools like SPLAT.
      * Time Series Cube Data Model: For our case (simple LCs) this new data model does not bring any improvement.
   * Data Discovery 
      * Time Series cannot be discovered at Registry level.
      * Time Series could be discovered using ObsCore / TAP.
      * The discovery of SSAP Time Series services has to be a must. DataType = TimeSeries
      * TOPCAT: But Datatype is not included in the Match Fields options of TOPCAT.
      * Splat-VO does not have Time Series option
   * Conclusions: it is necessary to fix the problems related to 
      * data discovery (registry)
      * data representation (data model).
---++ [[%PUBURL%/IVOA/InterOpMay2017-TD/TDIG-temporal-reqs-SSIG-BCecconi.pdf][ SSIG inputs on Time Domain (B. Cecconi, M. Louys, S. Erard) ]]

   * Solar System science cases involve photometry, spectra, images, polarisation.
   * Axes: 
      * Time scales/formats: SCLK, SCET, UTC,&hellip;
      * In addition to temporal axes: rotation, period,
   * Data discovery or catalogue mining: EpnCore temporal queries 
      * Use of temporal parameters for data selection / discovery 
         * a specific epoch or interval
         * a specific sampling step condition
         * a specific exposure time (int. time) condition
         * select a time scale and/or time origin
         * &hellip;
   * EpnCore / ObsCore comparison of keyword, description, Unit, UCD, utype 
      * Temporal time interval
      * Temporal sampling step
      * Observation exposure time
      * Observation time resolution
   * Time data products with time information: 
      * Time-series (tables)
      * Spectrograms
      * Cubes (images/spectra)
      * Events
   * Data product content: 
      * Often sparse multiD data with _time_ as main axis
      * Publishing time-series as: VOTable, FITS, CDF, &hellip;
      * Mandatory File + search metadata: 
         * Time scale (+starting point/epoch), time origin (location)
         * Time coverage + sampling/resolution/exposure
   * Time-series: 
      * Plotting tools: AMDA, TOPCAT (Time plot option), Autoplot (with SAMP)
      * Analysis: wide variety of methods (FFT, periodograms, lomb scargle,&hellip;). Analyse with IDL, python?
      * Data service? NASA/Heliophysics is working on an API for time series (HAPI).
      * Complexity? 
         * Astronomy & Solar System: observational parameters can change over time
         * Use same time axis / time scale references
---++ [[%PUBURL%/IVOA/InterOpMay2017-TD/VizieR_timeSeries.pdf][Simple (?) Time Series in VizieR (S. Derriere)]]

   * Content of VizieR: &gt; 10% contain "time series&rdquo; flag
   * SED-viewer like tool for time series is difficult: 
      * Heterogeneous formats
      * Missing characterisation / metadata or only in human-readable form
   * Catalogues: 
      * Big missions: HIPPARCOS & Tycho light curves, Kepler, CoRoT, OGLE,MACHO, EROS
      * Variability surveys
      * Tables dedicated to individual object
      * Solar data
   * Kind of time series: 
      * 70% are light curves
      * 23% are radial velocities
      * &hellip;
   * Several examples reflecting how heterogeneous data in VizieR is 
      * Example 1: 
         * One table for one target, JD versus RV (no coordinates)
         * Easily exported to VOTable, SAMP,&hellip;
      * Example 2: CoRoT 
         * One catalogue row per target
         * Thumbnails
         * Link to FITS file
      * Example 3: 
         * 8 targets in same table (no coordinates)
         * JD-offser & Phase (T_0 + Period)
      * Example 4: 
         * Coordinates and target
         * Link to plot + ascii file
         * Missing metadata
   * Parameters: Param = f(time) 
      * Time (JD, MJD, HJD, JD-xxxxx, phase)
      * Param = Y-axis (flux, magnitude, differential magnitude, color, counts, radial velocity&hellip;)
   * Extracting time series data: 
      * cone search not sufficient (some cats. only have target name&hellip;)
      * add standardised parameters (TIME) to output, but nee to keep all original params & description (provenance)
   * Data Provider: 
      * Some metadata and mapping will have to be added to VizieR
      * Only deal with large missions? but dedicated surveys to one source are very important. 90/10 rule (use popularity).
   * A time series standard should: 
      * not mandate too many metadata (or it won&rsquo;t be characterised properly)
      * allow for dataset-specific parameters: flags, S/N,&hellip;
---++ [[%PUBURL%/IVOA/InterOpMay2017-TD/TimeSeries.pdf][Asterics hackathon report (F. Bonnarel)]]

<strong><br /></strong>

   * Metadata needed to describe and discover TimeSeries: 
      * Spatial coordinate system
      * Time coordinate system: scale, ref. position, representation 
         * Time, spectral, space and polarisation characterisation and statistics (raw or mean positions, raw bounding limits, std deviation)
      * Time sampling characterisation and statistics: 
         * Mean sampling step
         * Sampling step limits
         * Sampling step standard deviation
         * Total exposure time
      * Exposure time characterisation and statistics: 
         * Mean total exposure time
         * Mean exposure time per step
         * Min, max and std deviation of exposure time per step
      * Characterisation on the time frequency axis: 
         * Periodograms, period (s), Fourier coefficients and frequencies, phase,&hellip;
      * What are the dependant and indépendant quantities and what is their nature
      * Which mode are the data? Transient or periodic? Seen by periodogram or by Target class
      * Target name, class, subclass (SN, SB,&hellip;)
   * Type of data we call TimeSeries: 
      * Temporal sequence of measurement points containing 
         * A time coordinate + flux (or similar) / RV / position / spectra / images
   * Representation of these data: time + spectra / images are better represented as a regular cube with only one sparse axis or as an event list
   * Recommend a time representation for standard output? MJD?
   * Relative time for theoretical data?
   * DAL chair view for a TS discovery & access: 
      * Extend ObsCore with a new TimeSeriesCore table
      * ObsTAP-TS can query both tables together
      * Extend &ldquo;SIAV2&rdquo; query interface to new TimeSeries specific query parameters
      * Archived TimeSeries retrieval or DataLink
      * Virtual data discovery (=TimeSeries produced on the fly) in SIAV2 = DsDisc. Access.reference is a SODA url
      * SODA expansions to TS 
         * Cutout or time selection
         * Selection on time frequencies
         * Selection in exposure times
         * Time binning
         * Frequency or phase output
---++ [[%PUBURL%/IVOA/InterOpMay2017-TD/Time_Series_Cube_-_DM.pdf][TimeSeries Cube DM (J. Nadvornik & P. Skoda)]]

<strong><br /></strong>

   * Introduction on OLAP cubes
   * Definition of Sparse Cube DM: 
      * Can describe any time series axes
      * Is flexible
      * Is extensible
   * Time Series Cube DM: Separating Data & Information: 
      * Describing meaning (Information layer)
      * Cube DM describes meaning on axes (data layer) without knowing all physical domain model
      * Changes to physical domain models (STC, Phot DM, provenance) wont require Cube DM to change.
   * Time Series Cube UML: 
      * Cube DM: 
         * SparseCube
         * TimeSeriesCube &mdash; data
      * ColumnRef 
         * Quantity (holds stats for ColumnRef)
         * Axis Domain DMs (holds context for ColumnRef) 
            * STC DM
            * Photometry DM
            * Characterisation DM
            * Gravitational Wave DM
            * Probability Distribution DM
   * Science use Cases (from E. Solano)
   * Discovery & Access 
      * ObsCore
      * SSA
      * SPLAT-VO / TOPCAT / Aladin 
         * Light-curve
         * Datalink
         * Cutouts
   * Open questions: 
      * Add datalink to ObsCore
      * What to put into Quantity DM
      * Two kinds of models (real world model & application of data model for publishing the data)
      * What do I need to discover about the data cube
      * Datalink for cutouts of cubes seems like best option
      * Use cases&hellip;
---++ [[%PUBURL%/IVOA/InterOpMay2017-TD/dachs.pdf][Time Series, VO-DML, DaCHS (M. Demleitner)]]

<strong><br /></strong>

   * DaCHS Annotation: 
      * DaCHS is a general VO publishing framework.
      * Each resource is described using a resource descriptor (RD) with (among others) table metadata
      * Before VO-DML, DaCHS understood one DM: STC. Annotation used a variant of STC-S.
      * Plan is to move into the VO-DML age: 
         * Independent task-specific annotations
         * Ad-hoc annotation language specific to DaCHS rather than XML on input
         * Not backed up by VO-DM&hellip;
   * Proposed NDCube Processing: 
      * Client parses STC annotations
      * Client looks for ndcube:Cube-typed annotation.
      * Client inspects existing STC annotations on hjd
      * Client pulls the set of dependent axes from ndcube: Cube annotations. Let the user which to plot?
      * Plotting component looks or ivoa:Measurement-typed annotation of df to work out what to use as error in the plot.
---++ [[%PUBURL%/IVOA/InterOpMay2017-TD/TS_Gaia_Archive_JuanGonzalez_InteropChina2017.pdf][Status and plans for the TS and spectra distribution for Gaia DR2 & DR3 (J. Gonzalez)]]

   * Gaia DR2 content: 
      * Main catalogue
      * X-matches with &ldquo;main&rdquo; catalogues
      * SSOs
      * Epoch photometry
   * Gaia DR3 content: also adds spectra
   * SSOs integration 
      * TAP service adaptation for SSO handling: 
         * UDFs def. for SSOs
         * Based on IMCCE Eproc integration performed for ESASky SSO search service
         * Possible ADQL extension with datatypes and functions to EPN-TAP?
      * SSO Data Model: 
         * EPNCore V2 draft: Orbital params &mdash;&gt; ephemerides table &mdash;&gt; TAP interface
   * Time Series & Spectra: 
      * DAL side: 
         * Extension of DataLink through Custom Access Data Service 
            * No protocol extension needed
            * Archive data distribution has mission specific features
            * DataLink recognition
         * On the fly data serialisation to requested formats
         * SSAP, ObsCore TS vs SVO TS-SSAP ?
      * DM side: 
         * OK for spectra, pending Time Series
         * DR ~ April 2018
         * Need to base in and extend current WIP (Jiri&rsquo;s TS draft)
      * Applications: 
         * Spectra (VOSpec, SPLAT-VO)
         * Time Series
Topic revision: r1 - 2017-06-09 - AdaNebot
IVOA
Log in or Register
IVOA.net
Wiki Home
WebChanges
WebTopicList
WebStatistics
Twiki Meta & Help
IVOA
Know
Main
Sandbox
TWiki
TWiki intro
TWiki tutorial
User registration
Notify me
Working Groups
Interest Groups
Committees
Stds&Procs
www.ivoa.net
Documents
Events
Members
XML Schema