(r9) ObsCoreExtensionForRadioData < IVOA

IVOA Web>WebPreferences>IvoaRadio>ObsCoreExtensionForRadioData (revision 9) (raw view)~~EditAttach~~
---+ ObsCore for radio data: 

[[https://github.com/ivoa-std/ObsCoreExtensionForRadioData][This project also discussed on github]] 
---++ Some considerations: Authors: A. Zanichelli, V. Galluzzi, M. Molinaro (INAF)</p>

<p dir="ltr">Here below we report some considerations mainly focused on single dish data with respect to the two draft documents: IVOA Obscore Extension for Radio data Version 1.0 (IVOA Note 2022-10-14) and Pulsar and FRB Radio Data Discovery and Access Version 1.0 (IVOA Note 2022-09-22).</p> <p dir="ltr">These comments are intended for further discussion within the IVOA Radio Interest Group.</p> <h3 dir="ltr">dataproduct_type</h3> <p dir="ltr">Current values for dataproduct_type as in the preliminary document [[https://www.ivoa.net/rdf/product-type/2021-11-18/product-type.html][Data Product Type vocabulary]] do not seem suitable to describe single dish observational products in order to allow efficient/successful data discovery.</p> <p dir="ltr">Following the same “parent / narrower Term” classification, we propose the value “sdradio” to be used 1) as a parent Term for any type of single dish data or 2) as parent Term associated with a set of more specific, narrower terms identifying more precisely the various data products coming from the possible observing modes.</p> <p dir="ltr">The value “sdradio” identifies the electromagnetic domain of the data product. We would prefer not to use a more generic “singledish” which would be more strictly related to the instrument more than the physical observable (also, single dish instruments are not used in the radio domain only).</p> 

*FB answer* _I think you are right that the data predict type of single dish data has to be discussed. However i don't think creating a specific data product type for radio single dish data is consistent with the current concept of data_product_type which is more on the dataset structure with respect to the axes nature; (because it is important for the tools which may display, render or analyse them) so i think according to following discussion things like "cube", "spectra" , "timeseries" or "dynamic spectra" should be ok. We are still looking for some term for spectropolarimetry product. My feeling is that we need to better describe single dish datasets is new parameters such as observing type or modes or scan modes or whatever;_ FrancoisBonnarel - 2023-02-09

*BC answer* _I agree with François, the dataproduct_type is about the organization of the data dimensionalities, its axes, etc, not the way it is recorded, nor the spectral range. Mixing the "radio domain", "instrument type" and "data dimensionalities" will make things difficult to separate the various semantic components._  BaptisteCecconi - 2023-02-09

<p dir="ltr">The optional, free-text dataproduct_subtype parameter could be used for a more detailed description of the data content. A better solution could be to use the sky_scan_mode parameter proposed in Table 1 of IVOA Obscore Extension for Radio data Version 1.0 (IVOA Note 2022-10-14). This last parameter offers the advantage of using a predefined vocabulary, thus avoiding the use of free text.</p>

*FB answer* _yes dataproduct_subtype is free text; As written above we probably have to introduce new paramters in this extension._ FrancoisBonnarel - 2023-02-09

*ML answer* _about scan mode , and other information about the way the data were obtained: These metadata belong to the observing configuration applied in the instrument to obtain the data._
_It makes a category by itself. This need is also caracterised for the high energy data,  and it is worth to describe those parameters separately from the data producttype._
_It is no longer a core property in terms of data discovery , but it is very useful to radio astronomers._ MireilleLouys - 2023-02-09

*BC answer* _There are already 3 types of pointing listed in the "ObsLocTAP" standard, in a "tracking_type" keyword. The current values are: "tracking", "solar-system-object-tracking", "fixed-az-el-transit"._
_This seems to call for a list of an external terms, which would be maintained in with Semantics WG._ BaptisteCecconi - 2023-02-09


<p dir="ltr">The following figure shows the main single dish observing mode:</p> <p dir="ltr"><img alt="" height="339" src="https://lh4.googleusercontent.com/Yro2Pj6Q_NLdOCsmFI0VfgybZOLnBMFoaJitspWbK5hdosIvcdKuwQo_EaAkRfo8cMxwXQDU6IOm9MPPPjsi8_2bBFdpcN7XyD7UN1_mOGkmoJc0-TNV-ec7rJ8ZjnfQ4ZIdFXXLt0fBsd77ScLTwEs" width="602" /></p>

<b><br /><br /></b>
<p dir="ltr">The following table summarizes the possible values for dataproduct_type and narrower/parent terms associations.</p>

<div align="left" dir="ltr"> <table><col width="189"></col> <tbody> <tr> <td> <p dir="ltr">Term                                 </p> </td> <td> <p dir="ltr">Parent</p> </td> <td> <p dir="ltr">dataproduct_subtype</p> </td> </tr> <tr> <td> <p dir="ltr">sdradio </p> </td> <td> </td> <td> </td> </tr> <tr> <td> <p dir="ltr">spatial-profile</p> </td> <td> <p dir="ltr">#sdradio</p> </td> <td> <p dir="ltr">skydip</p> </td> </tr> <tr> <td> <p dir="ltr">map (*)</p> </td> <td> <p dir="ltr">#cube, #sdradio</p> </td> <td> <p dir="ltr">on-the-fly map, raster map</p> </td> </tr> <tr> <td> <p dir="ltr">on-source</p> </td> <td> <p dir="ltr">#spectrum, #sdradio</p> </td> <td> <p dir="ltr">frequency switching, position switching,</p> <p dir="ltr">tracking</p> </td> </tr> <tr> <td> <p dir="ltr">crosscan (**)</p> </td> <td> <p dir="ltr">#spectrum, #sdradio</p> </td> <td> <p dir="ltr">on-the-fly cross scan, raster cross scan</p> </td> </tr> </tbody> </table> </div> <p dir="ltr">             </p> <p dir="ltr">

*FB answer* _for sdradio : i would say this is more something like an "observation_type" . sdradio will differ from interferometry_ FrancoisBonnarel - 2023-02-09"

(*) Single dish radio maps cannot be considered as “image” dataproducts. Data are typically written in (a) table(s), each row containing coordinate positions, timestamp and raw intensity (raw counts) and further processing is required to obtain a proper image. Also, in the more general case data are not acquired on a regular 2D grid in a single map. Typical observations consist of more than one map, to be combined to recover the final image. Maps can be obtained in spectropolarimetric mode, so the most appropriate parent term seems to be “cube”.</p> <p dir="ltr">

*FB answer* _sure this looks like a (sparse) cube;_  FrancoisBonnarel - 2023-02-10

(**) In principle the crosscan can be executed in raster mode instead of on-the-fly. For this reason the narrower term has been left more generic and the specific description is demanded to dataproduct_subtype.</p>



<p dir="ltr">Note that INAF has no Phased Array Feed receivers onboard the radio telescopes so we are not taking into account cases specific to beamforming techniques. Thus, more values could be needed. Do we have any PAF expert in the RadioIG?</p>

*FB answer* _i think LOFAR and Nenufar people do have this; Yan ? Baptiste ? Alan ?_ FrancoisBonnarel - 2023-02-10

*BC answer* _About phased arrays, yes, I think Alan and myself can give some inputs._ BaptisteCecconi - 2023-02-10

<p dir="ltr">This approach has some advantages: the narrower terms are in principle usable also in other spectral domains, associated with appropriate parent values/dataproduct_subtype. A query may happen in a two-level mode: a generic one can be done on “sdradio” getting back all the data products associated to any narrower term; alternatively a more detailed query can be done directly on one of the narrower terms.</p>

*FB answer* _if we consider all this is done by a (some) new parameter(s) to describe the observation (and not the product type) do we prefer several parameters or one single parameter with a hierarchy of terms ?_ FrancoisBonnarel - 2023-02-10
 <p dir="ltr">We are aware that this proposal is somehow different from the general VO approach because it is strongly related to a particular instrument/telescope design. However, we are motivated by the need to make single dish data discoverable in an effective manner, which could be hardly achieved by using the current ObsCore dataproduct_type values.</p>

*FB answer* _In other words : we have to distinguish the description of the observation (which is something like a provenance) from the type of the data which is important for the usage of the data; so really, again, I think we need a (some) new parameters in the extension_ FrancoisBonnarel - 2023-02-10

*ML answer* _I suggest having a special extension for observing configuration. this would also fit to other domain like X rays, high energy , etc ._ MireilleLouys  - 2023-02-10 

<b><br /><br /></b>
<h3 dir="ltr">em_xel and spectral MOCs</h3> <p dir="ltr">Single dish radio data may contain a multifrequency setup, that is many spectral windows disjointed on the spectral axis and with different resolutions (i.e. different numbers of spectral channels). In such a case em_xel can be computed but could lead to an incorrect interpretation of the actual spectral sampling of the dataset. In this respect, current efforts towards the creation of energy/frequency MOCs by the IVOA Applications Working Group could represent a solution. We note that frequency MOCs would also offer a comprehensive representation of the em_min, em_max and overall spectral coverage/properties of the dataset.</p> <p dir="ltr">A further extension of the work on energy/frequency MOCs could be developed to include also the polarimetric information. In fact, two polarisation states in the same dataset could in principle have different frequency setups.</p>

*FB answer* _From the  ObsCore/Characterisation point of view what you describe is the "support" concept which is more accurate than "bounds" (= min/max). A support is either a spatial detailed field og view or a set of intervals. MOC can be used to render those things. It would not be a new extension concept but a coding format for it. In VOTable would be rendered by the xtype attribute (xtype="moc" or "stmoc" or "emoc")_ FrancoisBonnarel - 2023-02-10

*BC answer* _May be a solution would be to have a generic "multi_dim_coverage" column (or just "coverage"); which there would be a MOC (with spatial, temporal and/or spectral domains, defined by an xtype in the column header). This may cover the other comment later in the text about the variation of the field of view across the spectral window. This should be further discussed, of course, but it might be useful here._   Baptiste Cecconi - 2023-02-10

<p dir="ltr">Therefore, ObsCore mapping on radio data in general represents a proof of concept for current developments on the MOC standard.</p>

<p dir="ltr">An example of multifrequency setup is shown in the following figure: a spectroscopic observation in the so-called “zoom mode” with the Xarcos spectrograph, delivering the two circular polarisations for each spectral window.</p> <p dir="ltr"><img alt="" height="241" src="https://lh6.googleusercontent.com/s5vDJd7oAKtW8l1xOOYiwcZig18p-TL2vup0NpaPk_Q2aCTimqgMgByWnfzxMTlvNYh_ZS6aZ_k2j6hMVxHXpY3ENxAN2w4_WUSLWBduSShcCvSQ09SYTY10XnMEiXK2UGLDBZaK9TiizzA-Ldn0zKQ" width="602" /></p>

<b><br /><br /></b>
<h2 dir="ltr">Comments on other ObsCore parameters</h2> <p dir="ltr">We collected two examples of ObsCore fields whose interpretation appears to be different from the original IVOA prescriptions.</p> <h3 dir="ltr">Facility_name and instrument</h3> <p dir="ltr">These parameters in our view should have the same meaning irrespectively if they are referred to space- atmosphere- or ground-based instrumentation. Typically a spacecraft hosts a number of different instruments similarly to what happens with ground-based telescopes. The same applies to modern balloon-borne experiments. Facilities like the ISS can be seen as equivalent to ground-based Observatories in the sense that they host different telescopes/experiments.</p> <p dir="ltr">We propose that facility_name identifies the (observatory + telescope) hosting the instrument used to acquire the dataset, while instrument describes the acquisition system used among all those available on that telescope.</p> <p dir="ltr">For instance: facility_name=ESO-VISTA, instrument=VIRCAM</p> <p dir="ltr">For the radio domain, generally speaking an instrument would be composed by a number of tokens, e.g. specific filters between the frontend and the backend, the frontend and backend themselves, as well as the beamformer/correlator used.</p>

<p dir="ltr">For single dish data we are currently describing the acquisition system with the combination frontend+backend.</p> 

*FB answer* _I think we have to follow what you recommend here;_ FrancoisBonnarel - 2023-02-10

<h3 dir="ltr">s_fov</h3> <p dir="ltr">In the draft IVOA Obscore Extension for Radio data Version 1.0 s_fov is defined as “A typical value for the field of view size … &lambda;/D where &lambda; is the mid value of the spectral range and D is the diameter of the telescope or the largest diameter of the array antennae or telescopes.”</p> <p dir="ltr">This appears in contrast with the ObsCore definition: “The s_fov column (1D size of the field of view) contains the approximate size of the region covered by the data product. For a circular region, this is the diameter (not the radius). For most data products the value given should be large enough to include the entire area of the observation; coverage within the bounded region need not be complete, for example if the specified FOV encompasses a rotated rectangular region. For observations which do not have a well-defined boundary, e.g. radio or high energy observations, a characteristic value should be given.”</p>

*FB answer* _this definition is loose enough that the mid value could be also valid. The result point to discussion would be "what we need this value for"; then we can decide which definition we take in our radio case;_ FrancoisBonnarel - 2023-02-10

*BC answer* _The radio extension of ObsTAP already includes s_fov_min s_fov_max, which is good. For compatibility and consistency with other services, the s_fov should be filled with the "representative" value. I mean, that it is the value, that the provider considers as most representative as possible of the fov value. The provider should have a good idea of the typical value that a user would use to filter data._ BaptisteCecconi - 2023-02-10
 <p dir="ltr">The former is in fact computed by means of the mid value of the wavelength range, while the latter is computed by means of the maximum value of the wavelength range.</p> <p dir="ltr">Also, we note that for low frequency aperture arrays whose stations are composed of dipole antennas the diameter D could not be given simply as a dish size but may be defined as a function of the number of dipoles, geometry, etc.</p>

<h3 dir="ltr">o_ucd </h3>  <p dir="ltr">In the current UCD vocabulary (UCD1+ Controlled Vocabulary 1.4 https://ivoa.net/documents/UCD1+/20210616/index.html) there appear to be no primary words suitable to describe raw single dish radio data.</p> <p dir="ltr">For pulsar data and transient radio data, o_ucd=stat.Fourier could be used, as proposed for visibility data in the Obscore extension for radio data document (v 1.0).</p> <p dir="ltr">The use of o_ucd=phot.flux.density for raw single dish data does not seem appropriate, since the single dish measured quantity is expressed in raw counts. These counts come from the digitisation of a voltage signal generated in the receiver chain by the incoming electromagnetic field.</p>

*FB answer* _there is phot.count : wouldn't that be ok for raw single dish data ?_ FrancoisBonnarel - 2023-02-10

*MM answer* _I think it doesn't because it's not photons that are recorded by the ADC conversion of the EM field. This looks semantically different. But I need @alessandra.zanichelli@inaf.it to check if my comment is right._ MarcoMolinaro - 2023-02-10

*BC answer* _there might indeed be some semantics issue here. We had this discusssion a few years ago in the Semantics WG, and the proposed solution was to use "phot.flux.density" for both photometric flux density and EM wave flux density, since there would be no sense to have 2 terms in this case. I would say that the raw counts issue is different: "phot.count" means "Flux expressed in counts" (and this is really counting photon hits), whereas the output of an RF ADC is not photon hit counts. I'm submitting a new term for ADU (i.e., analogue to digital converter units) to the UCD group._ Baptiste Cecconi - 2023-02-10

<b><br /><br /><br /><br /></b>
<h2 dir="ltr">Specific comments on IVOA Obscore Extension for Radio data Version 1.0 (IVOA Note 2022-10-14)</h2>
   * <p dir="ltr">Introduction (page 3) and thereon: when referring to single dish data format we would avoid to be too specific in referring to the SDFITS format. SDFITS is a registered standard but is not the only possible data format. Other FITS flavours which follow the FITS prescriptions (without being registered standards) are delivered by the various telescopes.</p>

_it was given as an example, we may add others to be more realistic. FrancoisBonnarel - 2023-02-10"_

   * <p dir="ltr">Sect 2.1:</p>
   1 <p dir="ltr">single dish may be equipped with multifeed receivers, we would avoid mentioning the acquisition of signal through a central beam only because this is a particular case of a more general scenario.</p>
   1 <p dir="ltr">there seems to be a mix between the definitions of s_fov and primary beam. In ObsCore s_fov is defined as the “approximate size of the region covered by the data product”. This is not the primary beam size of the antenna, for instance a map observation has an s_fov described by the approximate size of the mapped region on the sky. See also note on Sect 4.1 below.</p>

_This part has to be rewriten FrancoisBonnarel - 2023-02-10_
   1 <p dir="ltr">the typical case is not the one stated at the beginning of the section (spectrum), since the acquisition of emission for each spectral sample in the spectral band and polarization can be generally done in more complex modes (for instance, map). The following sentences well summarizes the variety of cases. We should rephrase the content, starting from the general statement and later describing specific cases. </p>

_OK FrancoisBonnarel - 2023-02-10"_

   1 <p dir="ltr">single dish multifeed receivers typically allow to cover larger spatial regions, acquiring simultaneously spectra from different positions on the sky still sharing the same spectral setup.</p>

_OK FrancoisBonnarel - 2023-02-10"_
   * <p dir="ltr">Sect 3: a typo in the first sentence: for radio data (not only for visibilities)</p>

_OK FrancoisBonnarel - 2023-02-10"_

   * <p dir="ltr">Sect 3.2: a correspondent definition for single dish data should be given.
_of course FrancoisBonnarel - 2023-02-10"_

 Also: with ultrawide band receivers (for instance: 20 GHz bandwidth at 7mm) it may happen that the number of spectral windows (each with its own setup) largely increases thus translating in a multiplication of entry lines in ObsCore for the same observation. How do we plan to deal with these cases? Are we happy to have a large number of records in such cases?</p>

_I think it's difficult to avoid. Or we have to group together several spectral windows and use the multi-interval support concept FrancoisBonnarel - 2023-02-10"_

_Something like a spectral-coverage MOC could be useful here, but that would be something for a future ObsCore update as it isn't really radio-specific. I think it makes sense to describe data with small "holes" in the spectral coverage using a single ObsCore entry, (i.e. 16 MHz "bands" separated by something like 1 MHz), but describe data with larger holes by multiple entries (i.e. 1GHz@4GHz + 1GHz@15GHz + 1GHz@22GHz). That would help discovery because people may discard individual entries with a small amount of bandwidth on grounds of not providing enough sensitivity. MarkKettenis -2023-03-01_


   * <p dir="ltr">Sect 3.3: the definition of D as the largest diameter of the array antennae or telescopes is not correct for any telescope type. In fact, there is no main mirror size in the case of dipole arrays or beamformed data. Instead, D should be defined as the measuring system aperture scale, as already written in Sect 2. </p>

_OK FrancoisBonnarel - 2023-02-10"_

   * <p dir="ltr">Sect 3.4: it should be clarified that s_resolution is the best (smallest) spatial resolution achieved during your observation. The minimum of the spectral range should be used instead. this holds true for any radio data. See also note on Sect 4.1 below.</p>

_i think it's the same discussion than for the s_fov family. What do we expect from such a value ? FrancoisBonnarel - 2023-02-10"_

   * <p dir="ltr">Sect 3.5: last sentence could be changed to mention MOCs: “the use of MOC for s_region is strongly encouraged to gather the more accurate description of the spatial coverage”</p>

_See my above comment : MOC is an xtype FrancoisBonnarel - 2023-02-10"_ 



   * <p dir="ltr">Sect 3.6: see note above on single dish values for o_ucd.</p>

_OK FrancoisBonnarel - 2023-02-10"_

   * <p dir="ltr">Sect 4.1: by ObsCore definition s_fov coincides with s_fov_max. We suggest to introduce s_fov_mid and s_fov_min. Similarly, s_resolution coincides with s_resolution_min and we suggest introducing s_resolution_min and s_resolution_max.</p>_

_we agree that in the radio case we need three concepts for fov and resolution. Which we call s_resolution and s_fov has to be discussed according to what usage we want to make of it (discovery or description ) FrancoisBonnarel - 2023-02-10"_

---++ Comments on ObsCore extension and Pulsar/FRB draft documents . John Tobin

See also [[%ATTACHURL%/ObsCoreExtensionForRadioData_jt.pdf][commented document]]

1. spelling of names/definintions/utypes - antennae -> antennas, excentricity -> eccentricity or ellipticity hence uv_distribution_exc might not be appropriately named.

_OK thank FrancoisBonnarel - 2023-02-10_


2. uv_distribution_fill - this definition seemed quite confusing to me as written it seems like you would always get an answer of 1/N_samples, I think it needs to be summation over i,j for n_cells with n_points >=1/n_cells. Maybe I am reading the definition incorrectly. The other issue with this definition is that it does not account for the fact that a dataset can have a large number of channels, where each is actually is own uv point. Each entry in a VO Table will be split into some number of channels, so this might need to be addressed and perhaps requires its own field. Finally, the uv filling factor will also be different depending on whether a user has continuum or spectral line observations in mind, continuum will have multi-frequency synthesis which implicitly increases its uv-coverage, while a spectral line applications will have worse uv-coverage implicitly.

_I see your point, and would like to have comments coming from our Astron and JIVE colleagues who originally proposed to characterize the uv coverage this way.FrancoisBonnarel - 2023-02-10"_

3. uv_distance_max, uv_distance_min; This might not quite be fine-grained enough because you might have one really long baseline and one very short baseline, but an array is actually configured somewhere in between. Perhaps also adding a 75th percentile baseline and 50th percentile baseline distance would be useful to add to this since those values would provide more information about where most of the uv-coverage is concentrated.

_Good point, we were already wondering how to estimate "effective numbers" for these two quantities in order to avoid "outliers". Your percentile is an interesting proposal to investigate. Or can we find another significant minimum and maximum estimation ? FrancoisBonnarel - 2023-02-10"_

_Well, for dense-core arrays, there might be very few "outlier" baselines, but those are a very significant addition to the core. Hence, we (NenuFAR team) would like to keep the min and max values as they are. Remember that this metadata should be filled for each observations, hence those values should contain the actual baseline min and max values for an observation, not a generic value for the instrument. Since we are building data discovery metadata, the uv coverage keywords should be consistent with each shared dataset. BaptisteCecconi - 2023-02-10_

_The 75 percentile uv distance is not a generic value, but could be calculated for each dataset. I get your point that dense core arrays will have fewer outlier baselines, but the density of uv points will be such that the beam one gets from imaging a dataset would be more reliably characterized by something like the 75th percentile baseline rather than the max uv distance. I think there would be value in min, max, and something in between like 75th percentile.  JohnTobin - 2023-02-10_

4. sky_scan_mode - this has applications beyond just single-dish as listed because interferometry data also have single-pointing, mosaic, and on the fly mosaics. Also, for single-dish there may be other modes that are not covered like drift-scan.

_OK. thanks for that. In that case we have to restructure the text. FrancoisBonnarel - 2023-02-10"_

5. s_resolution_beam_dirty - it is unclear what is intended here, whether it's to be a map of the dirty beam or the resolution of the dirty beam. If it's the resolution, this is somewhat redundant because the resolution of a cleaned map is derived from a Gaussian fit to the central core of the dirty beam, and there would be a degeneracy with what is provided by s_resolution. The dirty beam image, or psf image as CASA refers to it, is not always archived. ALMA and the NRAO do not include it in their standard image products for instance, so it is unclear how readily available this information would be for most archives. 

_Obviously the idea was to add a FIELD containing a link to the dirty beam map. The idea was to display it to help the user to figure out the level of quality of data. And this not a queryable column of course.FrancoisBonnarel - 2023-02-10"_

---++ Comments on ObsCore extension . Andreas Wicenec

I  agree with the comments JohnTobin made above, in particular wrt. uv stats. In addition I would  like to raise the following points:

1. Just a very general comment on use cases: I'm not quite sure whether this was discussed, but what are use cases to search for a visibility data set using an antenna diameter or the minimum and maximum uv distance? In many interferometric arrays there are multiple different antennas and specifying or searching for a single diameter is thus not really useful. The uv distances (as well as eccentricity) are highly variable even within a single, long observation and yes, it seems straight forward to calculate them, but in fact it is not, if a dataset is of the order of many TB or even PB. Thus the question about the usefulness of providing these values in the first place. Who would use these values for queries and why? The explanations in the text are kind of fine for UV snapshots, but for observations spanning many hours, many channels and an in-homogeneous distribution of baseline length, they get far less useful to the degree that they might be misleading or plainly not comparable between datasets from different arrays.

2. Related to this, the description of the antenna diameter in the table should also mention that this might be the maximum diameter in case of multiple different ones, like it is in the text (2.2)

3. Another, high level question is more about wording: When reducing a visibility data set it is possible to change the resolution and sacrifice frequency for spatial resolution to a certain degree. It is also possible to change the phase centre as well as the FOV, i.e. the pointing of the final data product. Thus quite a number of the values are more or less just boundaries and in some cases not even very strict ones. I guess one example would be the FOV, since that depends a lot on how far out you are performing the imaging, 1st, 2nd, 3rd null and that is pretty much up to the user to decide. In that sense I think we need to have more constraint descriptions, else these values will not be consistent and comparable, even if we allow for min and max. 

4. In the second sentence of 2 the case wavelength vs frequency is correctly made, but then, in particular for the FOV and resolution description the document is referring to wavelength. I think all of this should be described in terms of frequency.

5. Measurement sets allow for an almost insane flexibility and this extension will never be able to account for all of the possible variations, else it would need to replicate the MS data model. Thus I would opt for an extension which is as lightweight as possible and fully driven by actual real-life use cases of queries people would be performing on visibility data.


---++ New vocab proposal for  ObsCore extension. Authors: A. Zanichelli, V. Galluzzi, M. Molinaro (INAF)

We report some considerations mainly focused on single dish data with respect to the draft documents: IVOA Obscore Extension for Radio data Version 1.0 (IVOA Note 2023-02-13).
These considerations are intended for further discussion within the IVOA Radio Interest Group.

dataproduct_type and sky_scan_mode
We are proposing a new schema that, while trying to better follow IVOA ObsCore prescriptions, still poses some open questions to be discussed.

Current values for dataproduct_type are taken from the preliminary document Data Product Type vocabulary.  Note that the current definition of spectrum in the Data Product Type vocabulary should be extended to include raw counts (not only flux or magnitudes). For instance: “A scalar observable given as a function of a spectral coordinate”.

We assume that multi-feed/multi-beam capabilities can be exploited in case of imaging observations (raster map, otf map). For non-imaging observing modes (ON, ON-OFF, otf cross scan, skydip) single feed capabilities are typically used (in the case of a multi feed receiver, data are recorded from a single/central feed).

In the following table the proposed new values for dataproduct_type and sky_scan_mode are marked in yellow. Note: for dataproduct_type we are using the new definition of “measurements” from the Data Product Type vocabulary (preliminary): https://www.ivoa.net/rdf/product-type/2021-11-18/product-type.html as “Generic tabular data not fitting any of the other terms. Because of its lack of specificity, this term should generally be avoided, and new, more precise terms should be introduced instead.”. 
The definition of “measurements”  in ObsCore-1.1 (REC): https://ivoa.net/documents/ObsCore/20170509/index.html as “A list of derived measurements gathered in a particular original dataset of one of the previous sort after some analysis processing, like a source list, or more generally a list of ‘results’ attached to such datasets” would not be appropriate and would imply the definition of a further value for dataproduct_type.
We are also supporting the use of  the value “spatial profile” for dataproduct_type (currenlty under discussion).
Both “measurements” and  “spatial_profile” are written in square brackets to account for their preliminary status/meaning.

We note that sky_scan_mode applies to all radio data, not only single-dish ones. Also, sky_scan_mode cannot describe the frequency switching mode. We propose to change from sky_scan_mode to a more general scan_mode. Alternatively, in order to consider the  space and frequency domains separately we propose to keep sky_scan_mode and to add a further term frequency_scan_mode (equal to 'fixed' or 'switching' depending on the cases).

<p class="c4 c8"><span class="c0"></span></p><p class="c4 c8"><span class="c0"></span></p><a id="t.d47a711861f89153e40c18a1c37cdeb8a1e262f2"></a><a id="t.0"></a><table class="c12"><thead><tr class="c15"><td class="c1" colspan="1" rowspan="1"><p class="c4"><span class="c2">mode</span></p></td><td class="c13" colspan="1" rowspan="1"><p class="c4"><span class="c2">axes</span></p></td><td class="c3" colspan="1" rowspan="1"><p class="c4"><span class="c2">dataproduct_type</span></p></td><td class="c19" colspan="1" rowspan="1"><p class="c4"><span class="c2">note</span></p></td><td class="c7" colspan="1" rowspan="1"><p class="c4"><span class="c2">sky_scan_mode</span></p></td><tbody></tbody></tr><tr class="c15"><td class="c1" colspan="1" rowspan="1"><p class="c4"><span class="c24">ON total power</span></p></td><td class="c13" colspan="1" rowspan="1"><p class="c4"><span>degenerate in: x, y, freq</span></p></td><td class="c3" colspan="1" rowspan="1"><p class="c4"><span>[measurements]</span></p></td><td class="c19" colspan="1" rowspan="1"><p class="c4"><span>(a)</span></p></td><td class="c7" colspan="1" rowspan="1"><p class="c5"><span class="c10">on-source </span><span class="c0">(g)</span></p></td></tr><tr class="c15"><td class="c1" colspan="1" rowspan="1"><p class="c4"><span class="c24">ON spectral</span></p></td><td class="c13" colspan="1" rowspan="1"><p class="c4"><span>freq, degenerate in: x, y</span></p></td><td class="c3" colspan="1" rowspan="1"><p class="c4"><span>spectrum</span></p></td><td class="c19" colspan="1" rowspan="1"><p class="c4 c8"><span class="c0"></span></p></td><td class="c7" colspan="1" rowspan="1"><p class="c5"><span class="c10">on-source </span><span class="c0">(g)</span></p></td></tr><tr class="c15"><td class="c1" colspan="1" rowspan="1"><p class="c4"><span class="c24">ON-OFF &nbsp;total power</span></p></td><td class="c13" colspan="1" rowspan="1"><p class="c4"><span>x,y, degenerate in: freq</span></p></td><td class="c3" colspan="1" rowspan="1"><p class="c4"><span>[measurements]</span></p></td><td class="c19" colspan="1" rowspan="1"><p class="c4"><span>(b)</span></p></td><td class="c7" colspan="1" rowspan="1"><p class="c5"><span class="c0">on-off</span></p></td></tr><tr class="c15"><td class="c1" colspan="1" rowspan="1"><p class="c4"><span class="c24">ON-OFF spectral</span></p></td><td class="c13" colspan="1" rowspan="1"><p class="c4"><span>x,y,freq</span></p></td><td class="c3" colspan="1" rowspan="1"><p class="c4"><span>cube</span></p></td><td class="c19" colspan="1" rowspan="1"><p class="c4"><span>(c)</span></p></td><td class="c7" colspan="1" rowspan="1"><p class="c5"><span class="c0">on-off</span></p></td></tr><tr class="c15"><td class="c1" colspan="1" rowspan="1"><p class="c4"><span class="c24">frequency switching</span></p></td><td class="c13" colspan="1" rowspan="1"><p class="c4"><span>freq, degenerate in: x and y</span></p></td><td class="c3" colspan="1" rowspan="1"><p class="c4"><span>spectrum </span></p></td><td class="c19" colspan="1" rowspan="1"><p class="c4"><span>(d)</span></p></td><td class="c7" colspan="1" rowspan="1"><p class="c5"><span class="c0 c10">on-source (g)</span></p></td></tr><tr class="c15"><td class="c1" colspan="1" rowspan="1"><p class="c4"><span class="c24">on-the-fly cross scan total power</span></p></td><td class="c13" colspan="1" rowspan="1"><p class="c4"><span class="c0">x,y, degenerate in: freq</span></p></td><td class="c3" colspan="1" rowspan="1"><p class="c4"><span>[spatial profile]</span></p></td><td class="c19" colspan="1" rowspan="1"><p class="c4 c8"><span class="c0"></span></p></td><td class="c7" colspan="1" rowspan="1"><p class="c5"><span class="c0 c10">on-the-fly cross scan</span></p></td></tr><tr class="c15"><td class="c1" colspan="1" rowspan="1"><p class="c4"><span class="c24">on-the-fly cross scan spectral</span></p></td><td class="c13" colspan="1" rowspan="1"><p class="c4"><span>x,y,freq</span></p></td><td class="c3" colspan="1" rowspan="1"><p class="c4"><span>cube</span></p></td><td class="c19" colspan="1" rowspan="1"><p class="c4"><span>(c)</span></p></td><td class="c7" colspan="1" rowspan="1"><p class="c5"><span class="c0 c10">on-the-fly crosscan</span></p></td></tr><tr class="c15"><td class="c1" colspan="1" rowspan="1"><p class="c4"><span class="c32">raster map total power</span></p></td><td class="c13" colspan="1" rowspan="1"><p class="c4"><span>x,y, degenerate in: freq</span></p></td><td class="c3" colspan="1" rowspan="1"><p class="c4"><span class="c10">map </span></p></td><td class="c19" colspan="1" rowspan="1"><p class="c4"><span>(e)</span></p></td><td class="c7" colspan="1" rowspan="1"><p class="c5"><span class="c0">raster map</span></p></td></tr><tr class="c15"><td class="c1" colspan="1" rowspan="1"><p class="c4"><span class="c32">raster map spectral</span></p></td><td class="c13" colspan="1" rowspan="1"><p class="c4"><span>x,y,freq</span></p></td><td class="c3" colspan="1" rowspan="1"><p class="c4"><span>cube</span></p></td><td class="c19" colspan="1" rowspan="1"><p class="c5"><span class="c0">(c)</span></p></td><td class="c7" colspan="1" rowspan="1"><p class="c5"><span class="c0">raster map</span></p></td></tr><tr class="c15"><td class="c1" colspan="1" rowspan="1"><p class="c4"><span class="c32">otf map total power</span></p></td><td class="c13" colspan="1" rowspan="1"><p class="c4"><span>x,y, degenerate in: freq</span></p></td><td class="c3" colspan="1" rowspan="1"><p class="c4"><span class="c10">map</span></p></td><td class="c19" colspan="1" rowspan="1"><p class="c4"><span>(e)</span></p></td><td class="c7" colspan="1" rowspan="1"><p class="c5"><span class="c0">on the fly map</span></p></td></tr><tr class="c15"><td class="c1" colspan="1" rowspan="1"><p class="c4"><span class="c32">otf map spectral</span></p></td><td class="c13" colspan="1" rowspan="1"><p class="c4"><span>x,y,freq</span></p></td><td class="c3" colspan="1" rowspan="1"><p class="c4"><span>cube </span></p></td><td class="c19" colspan="1" rowspan="1"><p class="c4"><span>(c)</span></p></td><td class="c7" colspan="1" rowspan="1"><p class="c5"><span class="c0">on the fly map</span></p></td></tr><tr class="c15"><td class="c1" colspan="1" rowspan="1"><p class="c4"><span class="c24">skydip total power</span></p></td><td class="c13" colspan="1" rowspan="1"><p class="c4"><span>x, y, degenerate in: freq</span></p></td><td class="c3" colspan="1" rowspan="1"><p class="c4"><span>[spatial profile] </span></p></td><td class="c19" colspan="1" rowspan="1"><p class="c5 c8"><span class="c0"></span></p></td><td class="c7" colspan="1" rowspan="1"><p class="c5"><span class="c0 c10">skydip</span></p></td></tr><tr class="c15"><td class="c1" colspan="1" rowspan="1"><p class="c4"><span class="c24">skydip spectral &nbsp; &nbsp; &nbsp; &nbsp; </span></p></td><td class="c13" colspan="1" rowspan="1"><p class="c4"><span>x,y,freq</span></p></td><td class="c3" colspan="1" rowspan="1"><p class="c4"><span>[spatial profile]</span></p></td><td class="c19" colspan="1" rowspan="1"><p class="c4"><span class="c0">(f)</span></p></td><td class="c7" colspan="1" rowspan="1"><p class="c5"><span class="c0 c10">skydip</span></p></td></tr></thead></table><p class="c4 c8"><span class="c0"></span></p><p class="c4"><span>(


(a) in this case “measurements” seems to be the most appropriate value for dataproduct_type. 

(b) ON-OFF total power: two points are measured, no spectral info. One of the two measurements (OFF) may have very little scientific content if used by itself.

(c) sparse cube. Only the cross in a cross scan (or the ON and OFF positions in an ON-OFF) is sampled. An otf spectropolarimetric map taken with a peculiar scanning strategy (for instance a spiral pattern) may result in a sparse cube. A spectral skydip may be considered a sparse cube, but see note (f)

(d) frequency switching has dataproduct_type=spectrum. The resulting spectrum may contain gap(s) if the frequency switching encompasses non-overlapping bandwidths.

(e) In the case of total power raster or otf map we propose to introduce a new value dataproduct_type=map. In fact the frequency axis is degenerated so that `”cube'' is not appropriate and at the same time this is not an “image” as it is intended in the VO dataproduct_type Document. A single dish total power radio map cannot be considered an “image” dataproduct because data are typically written in table(s), each row containing spatial coordinates, timestamp and raw intensity (raw counts). Further processing is required to obtain a proper 2D image. Also, in the more general case data are not acquired on a regular 2D grid (for instance, if one uses a spiral scanning pattern). Map should be the parent term for “image”.

(f) in principle, a spectral skydip could be considered as a collection of spectra (e.g a sparse cube). However, the relevant information (atmospheric opacity) is contained in the spatial profile derived from the observation, thus the choice of dataproduct_type=spatial profile also for spectral skydip.

(g) sky_scan_mode value “on-source” should be added for ON and frequency switching observations.	


The following figure illustrates the main single dish observing mode.

<img alt="" height="339" src="https://lh4.googleusercontent.com/Yro2Pj6Q_NLdOCsmFI0VfgybZOLnBMFoaJitspWbK5hdosIvcdKuwQo_EaAkRfo8cMxwXQDU6IOm9MPPPjsi8_2bBFdpcN7XyD7UN1_mOGkmoJc0-TNV-ec7rJ8ZjnfQ4ZIdFXXLt0fBsd77ScLTwEs" width="602" />

	     	           

Note that INAF has no Phased Array Feed receivers onboard the radio telescopes so we are not taking into account cases specific to beamforming techniques. Thus, more values could be needed. Do we have any PAF expertsexpert in the RadioIG?



<h3 dir="ltr">s_fov</h3> <p dir="ltr">

The field of view of an interferometric array can be further restricted by bandwith and/or time smearing, that is the frequency resolution and integration time chosen when correlating the data. This is especially true for VLBI where these correlator parameters almost always restrict the field of view to only part of the primary beam of the primary beam of the individual antennas. This is probably worth adding to the description of s_fov.

-- IVOA.MarkKettenis -2023-08-29
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pdf	ObsCoreExtensionForRadioData_jt.pdf	r1	manage	556.0 K	2023-05-10 - 05:16	FrancoisBonnarel	John Tobin comments inside document
Topic revision: r9 - 2023-08-30 - FrancoisBonnarel
IVOA
Log in or Register
IVOA.net
Wiki Home
WebChanges
WebTopicList
WebStatistics
Twiki Meta & Help
IVOA
Know
Main
Sandbox
TWiki
TWiki intro
TWiki tutorial
User registration
Notify me
Working Groups
Interest Groups
Committees
Stds&Procs
www.ivoa.net
Documents
Events
Members
XML Schema