Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. B. Not only flat serialisation to consider Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements:
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC serialization in a VOTable)"> (STC serialization in a VOTable)Since right now, the only non-deprecated way to include STC metadata in VOTables relies on utypes, it is particularly unfortunate we have no way of doing this that's REC. I'm counting this as a separate use case since IMHO it's a particular shame something as basic is almost undefined in our recommended format -- the format we, as the VO community, actually control. Also, IMHO ideally clients should not have to worry about what (if any) data model some data within a VOTable conforms to. Just as any VOTable library could support (the now deprecated) COOSYS element, support for "modern" STC metadata should be "generic". Sorry for littering the use case with discussions on practice; if you have a better place for this stuff, please do move it there. One plan to do this is described in Referencing STC in VOTable. The basic idea is to collect all information pertaining to STC (or even some other data model) in one group, like this:<GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> <PARAM name="CoordRefFrame" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordRefFrame" value="ICRS"/> <PARAM name="ReferencePosition" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.ReferencePosition" value="BARYCENTER"/> <PARAM name="TimeScale" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.TimeScale" value="TT"/> <PARAM name="Epoch" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch" value="2010.2"/> <PARAM name="yearDef" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch.yearDef" value="J"/> <PARAM name="TimeInstant" datatype="char" arraysize="*" utype="stc:AstroCoords.Time.TimeInstant" value="2002-01-28T09:30:00"/> <PARAM name="URI" datatype="char" arraysize="*" utype="stc:DataModel.URI" value="http://www.ivoa.net/xml/STC/stc-v1.30.xsd"/> <FIELDref ref="raErr" utype="stc:AstroCoords.Position2D.Error2.C1"/> <FIELDref ref="deErr" utype="stc:AstroCoords.Position2D.Error2.C2"/> <FIELDref ref="ra" utype="stc:AstroCoords.Position2D.Value2.C1"/> <FIELDref ref="de" utype="stc:AstroCoords.Position2D.Value2.C2"/> <FIELDref ref="pmra" utype="stc:AstroCoords.Velocity2D.Value2.C1"/> <FIELDref ref="pmde" utype="stc:AstroCoords.Velocity2D.Value2.C2"/> </GROUP> <FIELD ID="ra" name="ra" datatype="float"/> <FIELD ID="de" name="de" datatype="float"/> <FIELD ID="raErr" name="raErr" datatype="float"/> <FIELD ID="deErr" name="deErr" datatype="float"/> <FIELD ID="pmra" name="pmra" datatype="float"/> <FIELD ID="pmde" name="pmde" datatype="float"/>One advantage of this scheme is that it's fairly easy to isolate the STC parsing/unparsing code from the rest of the VOTable handling since the stuff it has to operate on is "just an element", not many elements spread out over the entire document. This scheme also lets you embed multiple data models in a single VOTable. The "primary" data model (e.g., obscore or spectrum in ObsTAP or SSAP, respectively) could still use the FIELD's utype attributes; even though the "primary" data models only have crippled STC metadata (and would not, e.g., support proper motions), such information can still be transmitted in the VOTable and evaluated by clients. Here's an example that could be part of an obscore response in which the server also provides information on SSA: <GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype=... etc, as above </GROUP> <GROUP utype="spec:Spectrum"> <FIELDref ref="ra" utype="spec:Target.pos.ra"/> <FIELDref ref="dec" utype="spec:Target.pos.dec"/> ... (whatever else is in the table for Spectrum) ... </GROUP> <FIELD ID="ra" name="ra" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c1"/> <FIELD ID="de" name="de" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c2"/> ...(ignoring the fact that I've probably made up spec utypes and obscore talks about observation rather than target; I guess you get the drift anyway). Of course, it would be much nicer if we could agree on throwing overboard most of the existing practice and could just say <GROUP utype="stc:CatalogEntryLocation" id="targetPos"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> ... etc, as before </GROUP> <GROUP utype="spec:Spectrum"> <GROUPref ref="targetPos" utype="spec:Target.pos"/> </GROUP>But it's probably too late for that, we don't have a GROUPref element in the first place, and less referencing is better as a rule. -- MarkusDemleitner - 2012-09-20 STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
VO-URP and UTYPE-sSee here for a discussion on how VO-URP can support UTYPE-s discussion.Minutes of telecons
| ||||||||
Added: | ||||||||
> > |
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. B. Not only flat serialisation to consider Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements:
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC serialization in a VOTable)"> (STC serialization in a VOTable)Since right now, the only non-deprecated way to include STC metadata in VOTables relies on utypes, it is particularly unfortunate we have no way of doing this that's REC. I'm counting this as a separate use case since IMHO it's a particular shame something as basic is almost undefined in our recommended format -- the format we, as the VO community, actually control. Also, IMHO ideally clients should not have to worry about what (if any) data model some data within a VOTable conforms to. Just as any VOTable library could support (the now deprecated) COOSYS element, support for "modern" STC metadata should be "generic". Sorry for littering the use case with discussions on practice; if you have a better place for this stuff, please do move it there. One plan to do this is described in Referencing STC in VOTable. The basic idea is to collect all information pertaining to STC (or even some other data model) in one group, like this:<GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> <PARAM name="CoordRefFrame" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordRefFrame" value="ICRS"/> <PARAM name="ReferencePosition" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.ReferencePosition" value="BARYCENTER"/> <PARAM name="TimeScale" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.TimeScale" value="TT"/> <PARAM name="Epoch" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch" value="2010.2"/> <PARAM name="yearDef" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch.yearDef" value="J"/> <PARAM name="TimeInstant" datatype="char" arraysize="*" utype="stc:AstroCoords.Time.TimeInstant" value="2002-01-28T09:30:00"/> <PARAM name="URI" datatype="char" arraysize="*" utype="stc:DataModel.URI" value="http://www.ivoa.net/xml/STC/stc-v1.30.xsd"/> <FIELDref ref="raErr" utype="stc:AstroCoords.Position2D.Error2.C1"/> <FIELDref ref="deErr" utype="stc:AstroCoords.Position2D.Error2.C2"/> <FIELDref ref="ra" utype="stc:AstroCoords.Position2D.Value2.C1"/> <FIELDref ref="de" utype="stc:AstroCoords.Position2D.Value2.C2"/> <FIELDref ref="pmra" utype="stc:AstroCoords.Velocity2D.Value2.C1"/> <FIELDref ref="pmde" utype="stc:AstroCoords.Velocity2D.Value2.C2"/> </GROUP> <FIELD ID="ra" name="ra" datatype="float"/> <FIELD ID="de" name="de" datatype="float"/> <FIELD ID="raErr" name="raErr" datatype="float"/> <FIELD ID="deErr" name="deErr" datatype="float"/> <FIELD ID="pmra" name="pmra" datatype="float"/> <FIELD ID="pmde" name="pmde" datatype="float"/>One advantage of this scheme is that it's fairly easy to isolate the STC parsing/unparsing code from the rest of the VOTable handling since the stuff it has to operate on is "just an element", not many elements spread out over the entire document. This scheme also lets you embed multiple data models in a single VOTable. The "primary" data model (e.g., obscore or spectrum in ObsTAP or SSAP, respectively) could still use the FIELD's utype attributes; even though the "primary" data models only have crippled STC metadata (and would not, e.g., support proper motions), such information can still be transmitted in the VOTable and evaluated by clients. Here's an example that could be part of an obscore response in which the server also provides information on SSA: <GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype=... etc, as above </GROUP> <GROUP utype="spec:Spectrum"> <FIELDref ref="ra" utype="spec:Target.pos.ra"/> <FIELDref ref="dec" utype="spec:Target.pos.dec"/> ... (whatever else is in the table for Spectrum) ... </GROUP> <FIELD ID="ra" name="ra" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c1"/> <FIELD ID="de" name="de" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c2"/> ...(ignoring the fact that I've probably made up spec utypes and obscore talks about observation rather than target; I guess you get the drift anyway). Of course, it would be much nicer if we could agree on throwing overboard most of the existing practice and could just say <GROUP utype="stc:CatalogEntryLocation" id="targetPos"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> ... etc, as before </GROUP> <GROUP utype="spec:Spectrum"> <GROUPref ref="targetPos" utype="spec:Target.pos"/> </GROUP>But it's probably too late for that, we don't have a GROUPref element in the first place, and less referencing is better as a rule. -- MarkusDemleitner - 2012-09-20 STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
VO-URP and UTYPE-sSee here for a discussion on how VO-URP can support UTYPE-s discussion.Minutes of telecons
| ||||||||
Added: | ||||||||
> > |
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. B. Not only flat serialisation to consider Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements:
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC serialization in a VOTable)"> (STC serialization in a VOTable)Since right now, the only non-deprecated way to include STC metadata in VOTables relies on utypes, it is particularly unfortunate we have no way of doing this that's REC. I'm counting this as a separate use case since IMHO it's a particular shame something as basic is almost undefined in our recommended format -- the format we, as the VO community, actually control. Also, IMHO ideally clients should not have to worry about what (if any) data model some data within a VOTable conforms to. Just as any VOTable library could support (the now deprecated) COOSYS element, support for "modern" STC metadata should be "generic". Sorry for littering the use case with discussions on practice; if you have a better place for this stuff, please do move it there. One plan to do this is described in Referencing STC in VOTable. The basic idea is to collect all information pertaining to STC (or even some other data model) in one group, like this:<GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> <PARAM name="CoordRefFrame" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordRefFrame" value="ICRS"/> <PARAM name="ReferencePosition" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.ReferencePosition" value="BARYCENTER"/> <PARAM name="TimeScale" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.TimeScale" value="TT"/> <PARAM name="Epoch" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch" value="2010.2"/> <PARAM name="yearDef" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch.yearDef" value="J"/> <PARAM name="TimeInstant" datatype="char" arraysize="*" utype="stc:AstroCoords.Time.TimeInstant" value="2002-01-28T09:30:00"/> <PARAM name="URI" datatype="char" arraysize="*" utype="stc:DataModel.URI" value="http://www.ivoa.net/xml/STC/stc-v1.30.xsd"/> <FIELDref ref="raErr" utype="stc:AstroCoords.Position2D.Error2.C1"/> <FIELDref ref="deErr" utype="stc:AstroCoords.Position2D.Error2.C2"/> <FIELDref ref="ra" utype="stc:AstroCoords.Position2D.Value2.C1"/> <FIELDref ref="de" utype="stc:AstroCoords.Position2D.Value2.C2"/> <FIELDref ref="pmra" utype="stc:AstroCoords.Velocity2D.Value2.C1"/> <FIELDref ref="pmde" utype="stc:AstroCoords.Velocity2D.Value2.C2"/> </GROUP> <FIELD ID="ra" name="ra" datatype="float"/> <FIELD ID="de" name="de" datatype="float"/> <FIELD ID="raErr" name="raErr" datatype="float"/> <FIELD ID="deErr" name="deErr" datatype="float"/> <FIELD ID="pmra" name="pmra" datatype="float"/> <FIELD ID="pmde" name="pmde" datatype="float"/>One advantage of this scheme is that it's fairly easy to isolate the STC parsing/unparsing code from the rest of the VOTable handling since the stuff it has to operate on is "just an element", not many elements spread out over the entire document. This scheme also lets you embed multiple data models in a single VOTable. The "primary" data model (e.g., obscore or spectrum in ObsTAP or SSAP, respectively) could still use the FIELD's utype attributes; even though the "primary" data models only have crippled STC metadata (and would not, e.g., support proper motions), such information can still be transmitted in the VOTable and evaluated by clients. Here's an example that could be part of an obscore response in which the server also provides information on SSA: <GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype=... etc, as above </GROUP> <GROUP utype="spec:Spectrum"> <FIELDref ref="ra" utype="spec:Target.pos.ra"/> <FIELDref ref="dec" utype="spec:Target.pos.dec"/> ... (whatever else is in the table for Spectrum) ... </GROUP> <FIELD ID="ra" name="ra" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c1"/> <FIELD ID="de" name="de" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c2"/> ...(ignoring the fact that I've probably made up spec utypes and obscore talks about observation rather than target; I guess you get the drift anyway). Of course, it would be much nicer if we could agree on throwing overboard most of the existing practice and could just say <GROUP utype="stc:CatalogEntryLocation" id="targetPos"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> ... etc, as before </GROUP> <GROUP utype="spec:Spectrum"> <GROUPref ref="targetPos" utype="spec:Target.pos"/> </GROUP>But it's probably too late for that, we don't have a GROUPref element in the first place, and less referencing is better as a rule. -- MarkusDemleitner - 2012-09-20 STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
VO-URP and UTYPE-sSee here for a discussion on how VO-URP can support UTYPE-s discussion.Minutes of telecons
| ||||||||
Added: | ||||||||
> > |
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. B. Not only flat serialisation to consider Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements:
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC serialization in a VOTable)"> (STC serialization in a VOTable)Since right now, the only non-deprecated way to include STC metadata in VOTables relies on utypes, it is particularly unfortunate we have no way of doing this that's REC. I'm counting this as a separate use case since IMHO it's a particular shame something as basic is almost undefined in our recommended format -- the format we, as the VO community, actually control. Also, IMHO ideally clients should not have to worry about what (if any) data model some data within a VOTable conforms to. Just as any VOTable library could support (the now deprecated) COOSYS element, support for "modern" STC metadata should be "generic". Sorry for littering the use case with discussions on practice; if you have a better place for this stuff, please do move it there. One plan to do this is described in Referencing STC in VOTable. The basic idea is to collect all information pertaining to STC (or even some other data model) in one group, like this:<GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> <PARAM name="CoordRefFrame" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordRefFrame" value="ICRS"/> <PARAM name="ReferencePosition" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.ReferencePosition" value="BARYCENTER"/> <PARAM name="TimeScale" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.TimeScale" value="TT"/> <PARAM name="Epoch" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch" value="2010.2"/> <PARAM name="yearDef" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch.yearDef" value="J"/> <PARAM name="TimeInstant" datatype="char" arraysize="*" utype="stc:AstroCoords.Time.TimeInstant" value="2002-01-28T09:30:00"/> <PARAM name="URI" datatype="char" arraysize="*" utype="stc:DataModel.URI" value="http://www.ivoa.net/xml/STC/stc-v1.30.xsd"/> <FIELDref ref="raErr" utype="stc:AstroCoords.Position2D.Error2.C1"/> <FIELDref ref="deErr" utype="stc:AstroCoords.Position2D.Error2.C2"/> <FIELDref ref="ra" utype="stc:AstroCoords.Position2D.Value2.C1"/> <FIELDref ref="de" utype="stc:AstroCoords.Position2D.Value2.C2"/> <FIELDref ref="pmra" utype="stc:AstroCoords.Velocity2D.Value2.C1"/> <FIELDref ref="pmde" utype="stc:AstroCoords.Velocity2D.Value2.C2"/> </GROUP> <FIELD ID="ra" name="ra" datatype="float"/> <FIELD ID="de" name="de" datatype="float"/> <FIELD ID="raErr" name="raErr" datatype="float"/> <FIELD ID="deErr" name="deErr" datatype="float"/> <FIELD ID="pmra" name="pmra" datatype="float"/> <FIELD ID="pmde" name="pmde" datatype="float"/>One advantage of this scheme is that it's fairly easy to isolate the STC parsing/unparsing code from the rest of the VOTable handling since the stuff it has to operate on is "just an element", not many elements spread out over the entire document. This scheme also lets you embed multiple data models in a single VOTable. The "primary" data model (e.g., obscore or spectrum in ObsTAP or SSAP, respectively) could still use the FIELD's utype attributes; even though the "primary" data models only have crippled STC metadata (and would not, e.g., support proper motions), such information can still be transmitted in the VOTable and evaluated by clients. Here's an example that could be part of an obscore response in which the server also provides information on SSA: <GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype=... etc, as above </GROUP> <GROUP utype="spec:Spectrum"> <FIELDref ref="ra" utype="spec:Target.pos.ra"/> <FIELDref ref="dec" utype="spec:Target.pos.dec"/> ... (whatever else is in the table for Spectrum) ... </GROUP> <FIELD ID="ra" name="ra" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c1"/> <FIELD ID="de" name="de" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c2"/> ...(ignoring the fact that I've probably made up spec utypes and obscore talks about observation rather than target; I guess you get the drift anyway). Of course, it would be much nicer if we could agree on throwing overboard most of the existing practice and could just say <GROUP utype="stc:CatalogEntryLocation" id="targetPos"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> ... etc, as before </GROUP> <GROUP utype="spec:Spectrum"> <GROUPref ref="targetPos" utype="spec:Target.pos"/> </GROUP>But it's probably too late for that, we don't have a GROUPref element in the first place, and less referencing is better as a rule. -- MarkusDemleitner - 2012-09-20 STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
VO-URP and UTYPE-sSee here for a discussion on how VO-URP can support UTYPE-s discussion.Minutes of telecons
| ||||||||
Added: | ||||||||
> > |
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. B. Not only flat serialisation to consider Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements:
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC serialization in a VOTable)"> (STC serialization in a VOTable)Since right now, the only non-deprecated way to include STC metadata in VOTables relies on utypes, it is particularly unfortunate we have no way of doing this that's REC. I'm counting this as a separate use case since IMHO it's a particular shame something as basic is almost undefined in our recommended format -- the format we, as the VO community, actually control. Also, IMHO ideally clients should not have to worry about what (if any) data model some data within a VOTable conforms to. Just as any VOTable library could support (the now deprecated) COOSYS element, support for "modern" STC metadata should be "generic". Sorry for littering the use case with discussions on practice; if you have a better place for this stuff, please do move it there. One plan to do this is described in Referencing STC in VOTable. The basic idea is to collect all information pertaining to STC (or even some other data model) in one group, like this:<GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> <PARAM name="CoordRefFrame" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordRefFrame" value="ICRS"/> <PARAM name="ReferencePosition" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.ReferencePosition" value="BARYCENTER"/> <PARAM name="TimeScale" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.TimeScale" value="TT"/> <PARAM name="Epoch" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch" value="2010.2"/> <PARAM name="yearDef" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch.yearDef" value="J"/> <PARAM name="TimeInstant" datatype="char" arraysize="*" utype="stc:AstroCoords.Time.TimeInstant" value="2002-01-28T09:30:00"/> <PARAM name="URI" datatype="char" arraysize="*" utype="stc:DataModel.URI" value="http://www.ivoa.net/xml/STC/stc-v1.30.xsd"/> <FIELDref ref="raErr" utype="stc:AstroCoords.Position2D.Error2.C1"/> <FIELDref ref="deErr" utype="stc:AstroCoords.Position2D.Error2.C2"/> <FIELDref ref="ra" utype="stc:AstroCoords.Position2D.Value2.C1"/> <FIELDref ref="de" utype="stc:AstroCoords.Position2D.Value2.C2"/> <FIELDref ref="pmra" utype="stc:AstroCoords.Velocity2D.Value2.C1"/> <FIELDref ref="pmde" utype="stc:AstroCoords.Velocity2D.Value2.C2"/> </GROUP> <FIELD ID="ra" name="ra" datatype="float"/> <FIELD ID="de" name="de" datatype="float"/> <FIELD ID="raErr" name="raErr" datatype="float"/> <FIELD ID="deErr" name="deErr" datatype="float"/> <FIELD ID="pmra" name="pmra" datatype="float"/> <FIELD ID="pmde" name="pmde" datatype="float"/>One advantage of this scheme is that it's fairly easy to isolate the STC parsing/unparsing code from the rest of the VOTable handling since the stuff it has to operate on is "just an element", not many elements spread out over the entire document. This scheme also lets you embed multiple data models in a single VOTable. The "primary" data model (e.g., obscore or spectrum in ObsTAP or SSAP, respectively) could still use the FIELD's utype attributes; even though the "primary" data models only have crippled STC metadata (and would not, e.g., support proper motions), such information can still be transmitted in the VOTable and evaluated by clients. Here's an example that could be part of an obscore response in which the server also provides information on SSA: <GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype=... etc, as above </GROUP> <GROUP utype="spec:Spectrum"> <FIELDref ref="ra" utype="spec:Target.pos.ra"/> <FIELDref ref="dec" utype="spec:Target.pos.dec"/> ... (whatever else is in the table for Spectrum) ... </GROUP> <FIELD ID="ra" name="ra" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c1"/> <FIELD ID="de" name="de" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c2"/> ...(ignoring the fact that I've probably made up spec utypes and obscore talks about observation rather than target; I guess you get the drift anyway). Of course, it would be much nicer if we could agree on throwing overboard most of the existing practice and could just say <GROUP utype="stc:CatalogEntryLocation" id="targetPos"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> ... etc, as before </GROUP> <GROUP utype="spec:Spectrum"> <GROUPref ref="targetPos" utype="spec:Target.pos"/> </GROUP>But it's probably too late for that, we don't have a GROUPref element in the first place, and less referencing is better as a rule. -- MarkusDemleitner - 2012-09-20 STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
VO-URP and UTYPE-sSee here for a discussion on how VO-URP can support UTYPE-s discussion.Minutes of telecons
| ||||||||
Added: | ||||||||
> > |
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. B. Not only flat serialisation to consider Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements:
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC serialization in a VOTable)"> (STC serialization in a VOTable)Since right now, the only non-deprecated way to include STC metadata in VOTables relies on utypes, it is particularly unfortunate we have no way of doing this that's REC. I'm counting this as a separate use case since IMHO it's a particular shame something as basic is almost undefined in our recommended format -- the format we, as the VO community, actually control. Also, IMHO ideally clients should not have to worry about what (if any) data model some data within a VOTable conforms to. Just as any VOTable library could support (the now deprecated) COOSYS element, support for "modern" STC metadata should be "generic". Sorry for littering the use case with discussions on practice; if you have a better place for this stuff, please do move it there. One plan to do this is described in Referencing STC in VOTable. The basic idea is to collect all information pertaining to STC (or even some other data model) in one group, like this:<GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> <PARAM name="CoordRefFrame" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordRefFrame" value="ICRS"/> <PARAM name="ReferencePosition" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.ReferencePosition" value="BARYCENTER"/> <PARAM name="TimeScale" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.TimeScale" value="TT"/> <PARAM name="Epoch" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch" value="2010.2"/> <PARAM name="yearDef" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch.yearDef" value="J"/> <PARAM name="TimeInstant" datatype="char" arraysize="*" utype="stc:AstroCoords.Time.TimeInstant" value="2002-01-28T09:30:00"/> <PARAM name="URI" datatype="char" arraysize="*" utype="stc:DataModel.URI" value="http://www.ivoa.net/xml/STC/stc-v1.30.xsd"/> <FIELDref ref="raErr" utype="stc:AstroCoords.Position2D.Error2.C1"/> <FIELDref ref="deErr" utype="stc:AstroCoords.Position2D.Error2.C2"/> <FIELDref ref="ra" utype="stc:AstroCoords.Position2D.Value2.C1"/> <FIELDref ref="de" utype="stc:AstroCoords.Position2D.Value2.C2"/> <FIELDref ref="pmra" utype="stc:AstroCoords.Velocity2D.Value2.C1"/> <FIELDref ref="pmde" utype="stc:AstroCoords.Velocity2D.Value2.C2"/> </GROUP> <FIELD ID="ra" name="ra" datatype="float"/> <FIELD ID="de" name="de" datatype="float"/> <FIELD ID="raErr" name="raErr" datatype="float"/> <FIELD ID="deErr" name="deErr" datatype="float"/> <FIELD ID="pmra" name="pmra" datatype="float"/> <FIELD ID="pmde" name="pmde" datatype="float"/>One advantage of this scheme is that it's fairly easy to isolate the STC parsing/unparsing code from the rest of the VOTable handling since the stuff it has to operate on is "just an element", not many elements spread out over the entire document. This scheme also lets you embed multiple data models in a single VOTable. The "primary" data model (e.g., obscore or spectrum in ObsTAP or SSAP, respectively) could still use the FIELD's utype attributes; even though the "primary" data models only have crippled STC metadata (and would not, e.g., support proper motions), such information can still be transmitted in the VOTable and evaluated by clients. Here's an example that could be part of an obscore response in which the server also provides information on SSA: <GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype=... etc, as above </GROUP> <GROUP utype="spec:Spectrum"> <FIELDref ref="ra" utype="spec:Target.pos.ra"/> <FIELDref ref="dec" utype="spec:Target.pos.dec"/> ... (whatever else is in the table for Spectrum) ... </GROUP> <FIELD ID="ra" name="ra" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c1"/> <FIELD ID="de" name="de" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c2"/> ...(ignoring the fact that I've probably made up spec utypes and obscore talks about observation rather than target; I guess you get the drift anyway). Of course, it would be much nicer if we could agree on throwing overboard most of the existing practice and could just say <GROUP utype="stc:CatalogEntryLocation" id="targetPos"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> ... etc, as before </GROUP> <GROUP utype="spec:Spectrum"> <GROUPref ref="targetPos" utype="spec:Target.pos"/> </GROUP>But it's probably too late for that, we don't have a GROUPref element in the first place, and less referencing is better as a rule. -- MarkusDemleitner - 2012-09-20 STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
VO-URP and UTYPE-sSee here for a discussion on how VO-URP can support UTYPE-s discussion.Minutes of telecons
| ||||||||
Added: | ||||||||
> > |
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. B. Not only flat serialisation to consider Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements:
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC serialization in a VOTable)"> (STC serialization in a VOTable)Since right now, the only non-deprecated way to include STC metadata in VOTables relies on utypes, it is particularly unfortunate we have no way of doing this that's REC. I'm counting this as a separate use case since IMHO it's a particular shame something as basic is almost undefined in our recommended format -- the format we, as the VO community, actually control. Also, IMHO ideally clients should not have to worry about what (if any) data model some data within a VOTable conforms to. Just as any VOTable library could support (the now deprecated) COOSYS element, support for "modern" STC metadata should be "generic". Sorry for littering the use case with discussions on practice; if you have a better place for this stuff, please do move it there. One plan to do this is described in Referencing STC in VOTable. The basic idea is to collect all information pertaining to STC (or even some other data model) in one group, like this:<GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> <PARAM name="CoordRefFrame" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordRefFrame" value="ICRS"/> <PARAM name="ReferencePosition" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.ReferencePosition" value="BARYCENTER"/> <PARAM name="TimeScale" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.TimeScale" value="TT"/> <PARAM name="Epoch" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch" value="2010.2"/> <PARAM name="yearDef" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch.yearDef" value="J"/> <PARAM name="TimeInstant" datatype="char" arraysize="*" utype="stc:AstroCoords.Time.TimeInstant" value="2002-01-28T09:30:00"/> <PARAM name="URI" datatype="char" arraysize="*" utype="stc:DataModel.URI" value="http://www.ivoa.net/xml/STC/stc-v1.30.xsd"/> <FIELDref ref="raErr" utype="stc:AstroCoords.Position2D.Error2.C1"/> <FIELDref ref="deErr" utype="stc:AstroCoords.Position2D.Error2.C2"/> <FIELDref ref="ra" utype="stc:AstroCoords.Position2D.Value2.C1"/> <FIELDref ref="de" utype="stc:AstroCoords.Position2D.Value2.C2"/> <FIELDref ref="pmra" utype="stc:AstroCoords.Velocity2D.Value2.C1"/> <FIELDref ref="pmde" utype="stc:AstroCoords.Velocity2D.Value2.C2"/> </GROUP> <FIELD ID="ra" name="ra" datatype="float"/> <FIELD ID="de" name="de" datatype="float"/> <FIELD ID="raErr" name="raErr" datatype="float"/> <FIELD ID="deErr" name="deErr" datatype="float"/> <FIELD ID="pmra" name="pmra" datatype="float"/> <FIELD ID="pmde" name="pmde" datatype="float"/>One advantage of this scheme is that it's fairly easy to isolate the STC parsing/unparsing code from the rest of the VOTable handling since the stuff it has to operate on is "just an element", not many elements spread out over the entire document. This scheme also lets you embed multiple data models in a single VOTable. The "primary" data model (e.g., obscore or spectrum in ObsTAP or SSAP, respectively) could still use the FIELD's utype attributes; even though the "primary" data models only have crippled STC metadata (and would not, e.g., support proper motions), such information can still be transmitted in the VOTable and evaluated by clients. Here's an example that could be part of an obscore response in which the server also provides information on SSA: <GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype=... etc, as above </GROUP> <GROUP utype="spec:Spectrum"> <FIELDref ref="ra" utype="spec:Target.pos.ra"/> <FIELDref ref="dec" utype="spec:Target.pos.dec"/> ... (whatever else is in the table for Spectrum) ... </GROUP> <FIELD ID="ra" name="ra" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c1"/> <FIELD ID="de" name="de" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c2"/> ...(ignoring the fact that I've probably made up spec utypes and obscore talks about observation rather than target; I guess you get the drift anyway). Of course, it would be much nicer if we could agree on throwing overboard most of the existing practice and could just say <GROUP utype="stc:CatalogEntryLocation" id="targetPos"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> ... etc, as before </GROUP> <GROUP utype="spec:Spectrum"> <GROUPref ref="targetPos" utype="spec:Target.pos"/> </GROUP>But it's probably too late for that, we don't have a GROUPref element in the first place, and less referencing is better as a rule. -- MarkusDemleitner - 2012-09-20 STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
VO-URP and UTYPE-sSee here for a discussion on how VO-URP can support UTYPE-s discussion.Minutes of telecons
| ||||||||
Added: | ||||||||
> > |
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. B. Not only flat serialisation to consider Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements:
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC serialization in a VOTable)"> (STC serialization in a VOTable)Since right now, the only non-deprecated way to include STC metadata in VOTables relies on utypes, it is particularly unfortunate we have no way of doing this that's REC. I'm counting this as a separate use case since IMHO it's a particular shame something as basic is almost undefined in our recommended format -- the format we, as the VO community, actually control. Also, IMHO ideally clients should not have to worry about what (if any) data model some data within a VOTable conforms to. Just as any VOTable library could support (the now deprecated) COOSYS element, support for "modern" STC metadata should be "generic". Sorry for littering the use case with discussions on practice; if you have a better place for this stuff, please do move it there. One plan to do this is described in Referencing STC in VOTable. The basic idea is to collect all information pertaining to STC (or even some other data model) in one group, like this:<GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> <PARAM name="CoordRefFrame" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordRefFrame" value="ICRS"/> <PARAM name="ReferencePosition" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.ReferencePosition" value="BARYCENTER"/> <PARAM name="TimeScale" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.TimeScale" value="TT"/> <PARAM name="Epoch" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch" value="2010.2"/> <PARAM name="yearDef" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch.yearDef" value="J"/> <PARAM name="TimeInstant" datatype="char" arraysize="*" utype="stc:AstroCoords.Time.TimeInstant" value="2002-01-28T09:30:00"/> <PARAM name="URI" datatype="char" arraysize="*" utype="stc:DataModel.URI" value="http://www.ivoa.net/xml/STC/stc-v1.30.xsd"/> <FIELDref ref="raErr" utype="stc:AstroCoords.Position2D.Error2.C1"/> <FIELDref ref="deErr" utype="stc:AstroCoords.Position2D.Error2.C2"/> <FIELDref ref="ra" utype="stc:AstroCoords.Position2D.Value2.C1"/> <FIELDref ref="de" utype="stc:AstroCoords.Position2D.Value2.C2"/> <FIELDref ref="pmra" utype="stc:AstroCoords.Velocity2D.Value2.C1"/> <FIELDref ref="pmde" utype="stc:AstroCoords.Velocity2D.Value2.C2"/> </GROUP> <FIELD ID="ra" name="ra" datatype="float"/> <FIELD ID="de" name="de" datatype="float"/> <FIELD ID="raErr" name="raErr" datatype="float"/> <FIELD ID="deErr" name="deErr" datatype="float"/> <FIELD ID="pmra" name="pmra" datatype="float"/> <FIELD ID="pmde" name="pmde" datatype="float"/>One advantage of this scheme is that it's fairly easy to isolate the STC parsing/unparsing code from the rest of the VOTable handling since the stuff it has to operate on is "just an element", not many elements spread out over the entire document. This scheme also lets you embed multiple data models in a single VOTable. The "primary" data model (e.g., obscore or spectrum in ObsTAP or SSAP, respectively) could still use the FIELD's utype attributes; even though the "primary" data models only have crippled STC metadata (and would not, e.g., support proper motions), such information can still be transmitted in the VOTable and evaluated by clients. Here's an example that could be part of an obscore response in which the server also provides information on SSA: <GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype=... etc, as above </GROUP> <GROUP utype="spec:Spectrum"> <FIELDref ref="ra" utype="spec:Target.pos.ra"/> <FIELDref ref="dec" utype="spec:Target.pos.dec"/> ... (whatever else is in the table for Spectrum) ... </GROUP> <FIELD ID="ra" name="ra" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c1"/> <FIELD ID="de" name="de" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c2"/> ...(ignoring the fact that I've probably made up spec utypes and obscore talks about observation rather than target; I guess you get the drift anyway). Of course, it would be much nicer if we could agree on throwing overboard most of the existing practice and could just say <GROUP utype="stc:CatalogEntryLocation" id="targetPos"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> ... etc, as before </GROUP> <GROUP utype="spec:Spectrum"> <GROUPref ref="targetPos" utype="spec:Target.pos"/> </GROUP>But it's probably too late for that, we don't have a GROUPref element in the first place, and less referencing is better as a rule. -- MarkusDemleitner - 2012-09-20 STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
VO-URP and UTYPE-sSee here for a discussion on how VO-URP can support UTYPE-s discussion.Minutes of telecons | ||||||||
Added: | ||||||||
> > |
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. B. Not only flat serialisation to consider Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements:
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC serialization in a VOTable)"> (STC serialization in a VOTable)Since right now, the only non-deprecated way to include STC metadata in VOTables relies on utypes, it is particularly unfortunate we have no way of doing this that's REC. I'm counting this as a separate use case since IMHO it's a particular shame something as basic is almost undefined in our recommended format -- the format we, as the VO community, actually control. Also, IMHO ideally clients should not have to worry about what (if any) data model some data within a VOTable conforms to. Just as any VOTable library could support (the now deprecated) COOSYS element, support for "modern" STC metadata should be "generic". Sorry for littering the use case with discussions on practice; if you have a better place for this stuff, please do move it there. One plan to do this is described in Referencing STC in VOTable. The basic idea is to collect all information pertaining to STC (or even some other data model) in one group, like this:<GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> <PARAM name="CoordRefFrame" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordRefFrame" value="ICRS"/> <PARAM name="ReferencePosition" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.ReferencePosition" value="BARYCENTER"/> <PARAM name="TimeScale" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.TimeScale" value="TT"/> <PARAM name="Epoch" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch" value="2010.2"/> <PARAM name="yearDef" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch.yearDef" value="J"/> <PARAM name="TimeInstant" datatype="char" arraysize="*" utype="stc:AstroCoords.Time.TimeInstant" value="2002-01-28T09:30:00"/> <PARAM name="URI" datatype="char" arraysize="*" utype="stc:DataModel.URI" value="http://www.ivoa.net/xml/STC/stc-v1.30.xsd"/> <FIELDref ref="raErr" utype="stc:AstroCoords.Position2D.Error2.C1"/> <FIELDref ref="deErr" utype="stc:AstroCoords.Position2D.Error2.C2"/> <FIELDref ref="ra" utype="stc:AstroCoords.Position2D.Value2.C1"/> <FIELDref ref="de" utype="stc:AstroCoords.Position2D.Value2.C2"/> <FIELDref ref="pmra" utype="stc:AstroCoords.Velocity2D.Value2.C1"/> <FIELDref ref="pmde" utype="stc:AstroCoords.Velocity2D.Value2.C2"/> </GROUP> <FIELD ID="ra" name="ra" datatype="float"/> <FIELD ID="de" name="de" datatype="float"/> <FIELD ID="raErr" name="raErr" datatype="float"/> <FIELD ID="deErr" name="deErr" datatype="float"/> <FIELD ID="pmra" name="pmra" datatype="float"/> <FIELD ID="pmde" name="pmde" datatype="float"/>One advantage of this scheme is that it's fairly easy to isolate the STC parsing/unparsing code from the rest of the VOTable handling since the stuff it has to operate on is "just an element", not many elements spread out over the entire document. This scheme also lets you embed multiple data models in a single VOTable. The "primary" data model (e.g., obscore or spectrum in ObsTAP or SSAP, respectively) could still use the FIELD's utype attributes; even though the "primary" data models only have crippled STC metadata (and would not, e.g., support proper motions), such information can still be transmitted in the VOTable and evaluated by clients. Here's an example that could be part of an obscore response in which the server also provides information on SSA: <GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype=... etc, as above </GROUP> <GROUP utype="spec:Spectrum"> <FIELDref ref="ra" utype="spec:Target.pos.ra"/> <FIELDref ref="dec" utype="spec:Target.pos.dec"/> ... (whatever else is in the table for Spectrum) ... </GROUP> <FIELD ID="ra" name="ra" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c1"/> <FIELD ID="de" name="de" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c2"/> ...(ignoring the fact that I've probably made up spec utypes and obscore talks about observation rather than target; I guess you get the drift anyway). Of course, it would be much nicer if we could agree on throwing overboard most of the existing practice and could just say <GROUP utype="stc:CatalogEntryLocation" id="targetPos"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> ... etc, as before </GROUP> <GROUP utype="spec:Spectrum"> <GROUPref ref="targetPos" utype="spec:Target.pos"/> </GROUP>But it's probably too late for that, we don't have a GROUPref element in the first place, and less referencing is better as a rule. -- MarkusDemleitner - 2012-09-20 STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
VO-URP and UTYPE-sSee here for a discussion on how VO-URP can support UTYPE-s discussion.Minutes of telecons | ||||||||
Added: | ||||||||
> > |
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. B. Not only flat serialisation to consider Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements:
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC serialization in a VOTable)"> (STC serialization in a VOTable)Since right now, the only non-deprecated way to include STC metadata in VOTables relies on utypes, it is particularly unfortunate we have no way of doing this that's REC. I'm counting this as a separate use case since IMHO it's a particular shame something as basic is almost undefined in our recommended format -- the format we, as the VO community, actually control. Also, IMHO ideally clients should not have to worry about what (if any) data model some data within a VOTable conforms to. Just as any VOTable library could support (the now deprecated) COOSYS element, support for "modern" STC metadata should be "generic". Sorry for littering the use case with discussions on practice; if you have a better place for this stuff, please do move it there. One plan to do this is described in Referencing STC in VOTable. The basic idea is to collect all information pertaining to STC (or even some other data model) in one group, like this:<GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> <PARAM name="CoordRefFrame" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordRefFrame" value="ICRS"/> <PARAM name="ReferencePosition" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.ReferencePosition" value="BARYCENTER"/> <PARAM name="TimeScale" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.TimeScale" value="TT"/> <PARAM name="Epoch" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch" value="2010.2"/> <PARAM name="yearDef" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch.yearDef" value="J"/> <PARAM name="TimeInstant" datatype="char" arraysize="*" utype="stc:AstroCoords.Time.TimeInstant" value="2002-01-28T09:30:00"/> <PARAM name="URI" datatype="char" arraysize="*" utype="stc:DataModel.URI" value="http://www.ivoa.net/xml/STC/stc-v1.30.xsd"/> <FIELDref ref="raErr" utype="stc:AstroCoords.Position2D.Error2.C1"/> <FIELDref ref="deErr" utype="stc:AstroCoords.Position2D.Error2.C2"/> <FIELDref ref="ra" utype="stc:AstroCoords.Position2D.Value2.C1"/> <FIELDref ref="de" utype="stc:AstroCoords.Position2D.Value2.C2"/> <FIELDref ref="pmra" utype="stc:AstroCoords.Velocity2D.Value2.C1"/> <FIELDref ref="pmde" utype="stc:AstroCoords.Velocity2D.Value2.C2"/> </GROUP> <FIELD ID="ra" name="ra" datatype="float"/> <FIELD ID="de" name="de" datatype="float"/> <FIELD ID="raErr" name="raErr" datatype="float"/> <FIELD ID="deErr" name="deErr" datatype="float"/> <FIELD ID="pmra" name="pmra" datatype="float"/> <FIELD ID="pmde" name="pmde" datatype="float"/>One advantage of this scheme is that it's fairly easy to isolate the STC parsing/unparsing code from the rest of the VOTable handling since the stuff it has to operate on is "just an element", not many elements spread out over the entire document. This scheme also lets you embed multiple data models in a single VOTable. The "primary" data model (e.g., obscore or spectrum in ObsTAP or SSAP, respectively) could still use the FIELD's utype attributes; even though the "primary" data models only have crippled STC metadata (and would not, e.g., support proper motions), such information can still be transmitted in the VOTable and evaluated by clients. Here's an example that could be part of an obscore response in which the server also provides information on SSA: <GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype=... etc, as above </GROUP> <GROUP utype="spec:Spectrum"> <FIELDref ref="ra" utype="spec:Target.pos.ra"/> <FIELDref ref="dec" utype="spec:Target.pos.dec"/> ... (whatever else is in the table for Spectrum) ... </GROUP> <FIELD ID="ra" name="ra" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c1"/> <FIELD ID="de" name="de" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c2"/> ...(ignoring the fact that I've probably made up spec utypes and obscore talks about observation rather than target; I guess you get the drift anyway). Of course, it would be much nicer if we could agree on throwing overboard most of the existing practice and could just say <GROUP utype="stc:CatalogEntryLocation" id="targetPos"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> ... etc, as before </GROUP> <GROUP utype="spec:Spectrum"> <GROUPref ref="targetPos" utype="spec:Target.pos"/> </GROUP>But it's probably too late for that, we don't have a GROUPref element in the first place, and less referencing is better as a rule. -- MarkusDemleitner - 2012-09-20 STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
VO-URP and UTYPE-sSee here for a discussion on how VO-URP can support UTYPE-s discussion.Minutes of telecons | ||||||||
Added: | ||||||||
> > |
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. B. Not only flat serialisation to consider Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements:
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC serialization in a VOTable)"> (STC serialization in a VOTable)Since right now, the only non-deprecated way to include STC metadata in VOTables relies on utypes, it is particularly unfortunate we have no way of doing this that's REC. I'm counting this as a separate use case since IMHO it's a particular shame something as basic is almost undefined in our recommended format -- the format we, as the VO community, actually control. Also, IMHO ideally clients should not have to worry about what (if any) data model some data within a VOTable conforms to. Just as any VOTable library could support (the now deprecated) COOSYS element, support for "modern" STC metadata should be "generic". Sorry for littering the use case with discussions on practice; if you have a better place for this stuff, please do move it there. One plan to do this is described in Referencing STC in VOTable. The basic idea is to collect all information pertaining to STC (or even some other data model) in one group, like this:<GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> <PARAM name="CoordRefFrame" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordRefFrame" value="ICRS"/> <PARAM name="ReferencePosition" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.ReferencePosition" value="BARYCENTER"/> <PARAM name="TimeScale" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.TimeScale" value="TT"/> <PARAM name="Epoch" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch" value="2010.2"/> <PARAM name="yearDef" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch.yearDef" value="J"/> <PARAM name="TimeInstant" datatype="char" arraysize="*" utype="stc:AstroCoords.Time.TimeInstant" value="2002-01-28T09:30:00"/> <PARAM name="URI" datatype="char" arraysize="*" utype="stc:DataModel.URI" value="http://www.ivoa.net/xml/STC/stc-v1.30.xsd"/> <FIELDref ref="raErr" utype="stc:AstroCoords.Position2D.Error2.C1"/> <FIELDref ref="deErr" utype="stc:AstroCoords.Position2D.Error2.C2"/> <FIELDref ref="ra" utype="stc:AstroCoords.Position2D.Value2.C1"/> <FIELDref ref="de" utype="stc:AstroCoords.Position2D.Value2.C2"/> <FIELDref ref="pmra" utype="stc:AstroCoords.Velocity2D.Value2.C1"/> <FIELDref ref="pmde" utype="stc:AstroCoords.Velocity2D.Value2.C2"/> </GROUP> <FIELD ID="ra" name="ra" datatype="float"/> <FIELD ID="de" name="de" datatype="float"/> <FIELD ID="raErr" name="raErr" datatype="float"/> <FIELD ID="deErr" name="deErr" datatype="float"/> <FIELD ID="pmra" name="pmra" datatype="float"/> <FIELD ID="pmde" name="pmde" datatype="float"/>One advantage of this scheme is that it's fairly easy to isolate the STC parsing/unparsing code from the rest of the VOTable handling since the stuff it has to operate on is "just an element", not many elements spread out over the entire document. This scheme also lets you embed multiple data models in a single VOTable. The "primary" data model (e.g., obscore or spectrum in ObsTAP or SSAP, respectively) could still use the FIELD's utype attributes; even though the "primary" data models only have crippled STC metadata (and would not, e.g., support proper motions), such information can still be transmitted in the VOTable and evaluated by clients. Here's an example that could be part of an obscore response in which the server also provides information on SSA: <GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype=... etc, as above </GROUP> <GROUP utype="spec:Spectrum"> <FIELDref ref="ra" utype="spec:Target.pos.ra"/> <FIELDref ref="dec" utype="spec:Target.pos.dec"/> ... (whatever else is in the table for Spectrum) ... </GROUP> <FIELD ID="ra" name="ra" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c1"/> <FIELD ID="de" name="de" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c2"/> ...(ignoring the fact that I've probably made up spec utypes and obscore talks about observation rather than target; I guess you get the drift anyway). Of course, it would be much nicer if we could agree on throwing overboard most of the existing practice and could just say <GROUP utype="stc:CatalogEntryLocation" id="targetPos"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> ... etc, as before </GROUP> <GROUP utype="spec:Spectrum"> <GROUPref ref="targetPos" utype="spec:Target.pos"/> </GROUP>But it's probably too late for that, we don't have a GROUPref element in the first place, and less referencing is better as a rule. -- MarkusDemleitner - 2012-09-20 STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
VO-URP and UTYPE-sSee here for a discussion on how VO-URP can support UTYPE-s discussion.Minutes of telecons | ||||||||
Added: | ||||||||
> > |
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. B. Not only flat serialisation to consider Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements:
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC serialization in a VOTable)"> (STC serialization in a VOTable)Since right now, the only non-deprecated way to include STC metadata in VOTables relies on utypes, it is particularly unfortunate we have no way of doing this that's REC. I'm counting this as a separate use case since IMHO it's a particular shame something as basic is almost undefined in our recommended format -- the format we, as the VO community, actually control. Also, IMHO ideally clients should not have to worry about what (if any) data model some data within a VOTable conforms to. Just as any VOTable library could support (the now deprecated) COOSYS element, support for "modern" STC metadata should be "generic". Sorry for littering the use case with discussions on practice; if you have a better place for this stuff, please do move it there. One plan to do this is described in Referencing STC in VOTable. The basic idea is to collect all information pertaining to STC (or even some other data model) in one group, like this:<GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> <PARAM name="CoordRefFrame" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordRefFrame" value="ICRS"/> <PARAM name="ReferencePosition" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.ReferencePosition" value="BARYCENTER"/> <PARAM name="TimeScale" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.TimeScale" value="TT"/> <PARAM name="Epoch" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch" value="2010.2"/> <PARAM name="yearDef" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch.yearDef" value="J"/> <PARAM name="TimeInstant" datatype="char" arraysize="*" utype="stc:AstroCoords.Time.TimeInstant" value="2002-01-28T09:30:00"/> <PARAM name="URI" datatype="char" arraysize="*" utype="stc:DataModel.URI" value="http://www.ivoa.net/xml/STC/stc-v1.30.xsd"/> <FIELDref ref="raErr" utype="stc:AstroCoords.Position2D.Error2.C1"/> <FIELDref ref="deErr" utype="stc:AstroCoords.Position2D.Error2.C2"/> <FIELDref ref="ra" utype="stc:AstroCoords.Position2D.Value2.C1"/> <FIELDref ref="de" utype="stc:AstroCoords.Position2D.Value2.C2"/> <FIELDref ref="pmra" utype="stc:AstroCoords.Velocity2D.Value2.C1"/> <FIELDref ref="pmde" utype="stc:AstroCoords.Velocity2D.Value2.C2"/> </GROUP> <FIELD ID="ra" name="ra" datatype="float"/> <FIELD ID="de" name="de" datatype="float"/> <FIELD ID="raErr" name="raErr" datatype="float"/> <FIELD ID="deErr" name="deErr" datatype="float"/> <FIELD ID="pmra" name="pmra" datatype="float"/> <FIELD ID="pmde" name="pmde" datatype="float"/>One advantage of this scheme is that it's fairly easy to isolate the STC parsing/unparsing code from the rest of the VOTable handling since the stuff it has to operate on is "just an element", not many elements spread out over the entire document. This scheme also lets you embed multiple data models in a single VOTable. The "primary" data model (e.g., obscore or spectrum in ObsTAP or SSAP, respectively) could still use the FIELD's utype attributes; even though the "primary" data models only have crippled STC metadata (and would not, e.g., support proper motions), such information can still be transmitted in the VOTable and evaluated by clients. Here's an example that could be part of an obscore response in which the server also provides information on SSA: <GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype=... etc, as above </GROUP> <GROUP utype="spec:Spectrum"> <FIELDref ref="ra" utype="spec:Target.pos.ra"/> <FIELDref ref="dec" utype="spec:Target.pos.dec"/> ... (whatever else is in the table for Spectrum) ... </GROUP> <FIELD ID="ra" name="ra" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c1"/> <FIELD ID="de" name="de" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c2"/> ...(ignoring the fact that I've probably made up spec utypes and obscore talks about observation rather than target; I guess you get the drift anyway). Of course, it would be much nicer if we could agree on throwing overboard most of the existing practice and could just say <GROUP utype="stc:CatalogEntryLocation" id="targetPos"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> ... etc, as before </GROUP> <GROUP utype="spec:Spectrum"> <GROUPref ref="targetPos" utype="spec:Target.pos"/> </GROUP>But it's probably too late for that, we don't have a GROUPref element in the first place, and less referencing is better as a rule. -- MarkusDemleitner - 2012-09-20 STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
VO-URP and UTYPE-sSee here for a discussion on how VO-URP can support UTYPE-s discussion.Minutes of telecons | ||||||||
Added: | ||||||||
> > |
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. B. Not only flat serialisation to consider Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements:
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC serialization in a VOTable)"> (STC serialization in a VOTable)Since right now, the only non-deprecated way to include STC metadata in VOTables relies on utypes, it is particularly unfortunate we have no way of doing this that's REC. I'm counting this as a separate use case since IMHO it's a particular shame something as basic is almost undefined in our recommended format -- the format we, as the VO community, actually control. Also, IMHO ideally clients should not have to worry about what (if any) data model some data within a VOTable conforms to. Just as any VOTable library could support (the now deprecated) COOSYS element, support for "modern" STC metadata should be "generic". Sorry for littering the use case with discussions on practice; if you have a better place for this stuff, please do move it there. One plan to do this is described in Referencing STC in VOTable. The basic idea is to collect all information pertaining to STC (or even some other data model) in one group, like this:<GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> <PARAM name="CoordRefFrame" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordRefFrame" value="ICRS"/> <PARAM name="ReferencePosition" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.ReferencePosition" value="BARYCENTER"/> <PARAM name="TimeScale" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.TimeScale" value="TT"/> <PARAM name="Epoch" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch" value="2010.2"/> <PARAM name="yearDef" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch.yearDef" value="J"/> <PARAM name="TimeInstant" datatype="char" arraysize="*" utype="stc:AstroCoords.Time.TimeInstant" value="2002-01-28T09:30:00"/> <PARAM name="URI" datatype="char" arraysize="*" utype="stc:DataModel.URI" value="http://www.ivoa.net/xml/STC/stc-v1.30.xsd"/> <FIELDref ref="raErr" utype="stc:AstroCoords.Position2D.Error2.C1"/> <FIELDref ref="deErr" utype="stc:AstroCoords.Position2D.Error2.C2"/> <FIELDref ref="ra" utype="stc:AstroCoords.Position2D.Value2.C1"/> <FIELDref ref="de" utype="stc:AstroCoords.Position2D.Value2.C2"/> <FIELDref ref="pmra" utype="stc:AstroCoords.Velocity2D.Value2.C1"/> <FIELDref ref="pmde" utype="stc:AstroCoords.Velocity2D.Value2.C2"/> </GROUP> <FIELD ID="ra" name="ra" datatype="float"/> <FIELD ID="de" name="de" datatype="float"/> <FIELD ID="raErr" name="raErr" datatype="float"/> <FIELD ID="deErr" name="deErr" datatype="float"/> <FIELD ID="pmra" name="pmra" datatype="float"/> <FIELD ID="pmde" name="pmde" datatype="float"/>One advantage of this scheme is that it's fairly easy to isolate the STC parsing/unparsing code from the rest of the VOTable handling since the stuff it has to operate on is "just an element", not many elements spread out over the entire document. This scheme also lets you embed multiple data models in a single VOTable. The "primary" data model (e.g., obscore or spectrum in ObsTAP or SSAP, respectively) could still use the FIELD's utype attributes; even though the "primary" data models only have crippled STC metadata (and would not, e.g., support proper motions), such information can still be transmitted in the VOTable and evaluated by clients. Here's an example that could be part of an obscore response in which the server also provides information on SSA: <GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype=... etc, as above </GROUP> <GROUP utype="spec:Spectrum"> <FIELDref ref="ra" utype="spec:Target.pos.ra"/> <FIELDref ref="dec" utype="spec:Target.pos.dec"/> ... (whatever else is in the table for Spectrum) ... </GROUP> <FIELD ID="ra" name="ra" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c1"/> <FIELD ID="de" name="de" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c2"/> ...(ignoring the fact that I've probably made up spec utypes and obscore talks about observation rather than target; I guess you get the drift anyway). Of course, it would be much nicer if we could agree on throwing overboard most of the existing practice and could just say <GROUP utype="stc:CatalogEntryLocation" id="targetPos"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> ... etc, as before </GROUP> <GROUP utype="spec:Spectrum"> <GROUPref ref="targetPos" utype="spec:Target.pos"/> </GROUP>But it's probably too late for that, we don't have a GROUPref element in the first place, and less referencing is better as a rule. -- MarkusDemleitner - 2012-09-20 STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
VO-URP and UTYPE-sSee here for a discussion on how VO-URP can support UTYPE-s discussion.Minutes of telecons | ||||||||
Added: | ||||||||
> > |
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. B. Not only flat serialisation to consider Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements:
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC serialization in a VOTable)"> (STC serialization in a VOTable)Since right now, the only non-deprecated way to include STC metadata in VOTables relies on utypes, it is particularly unfortunate we have no way of doing this that's REC. I'm counting this as a separate use case since IMHO it's a particular shame something as basic is almost undefined in our recommended format -- the format we, as the VO community, actually control. Also, IMHO ideally clients should not have to worry about what (if any) data model some data within a VOTable conforms to. Just as any VOTable library could support (the now deprecated) COOSYS element, support for "modern" STC metadata should be "generic". Sorry for littering the use case with discussions on practice; if you have a better place for this stuff, please do move it there. One plan to do this is described in Referencing STC in VOTable. The basic idea is to collect all information pertaining to STC (or even some other data model) in one group, like this:<GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> <PARAM name="CoordRefFrame" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordRefFrame" value="ICRS"/> <PARAM name="ReferencePosition" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.ReferencePosition" value="BARYCENTER"/> <PARAM name="TimeScale" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.TimeScale" value="TT"/> <PARAM name="Epoch" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch" value="2010.2"/> <PARAM name="yearDef" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch.yearDef" value="J"/> <PARAM name="TimeInstant" datatype="char" arraysize="*" utype="stc:AstroCoords.Time.TimeInstant" value="2002-01-28T09:30:00"/> <PARAM name="URI" datatype="char" arraysize="*" utype="stc:DataModel.URI" value="http://www.ivoa.net/xml/STC/stc-v1.30.xsd"/> <FIELDref ref="raErr" utype="stc:AstroCoords.Position2D.Error2.C1"/> <FIELDref ref="deErr" utype="stc:AstroCoords.Position2D.Error2.C2"/> <FIELDref ref="ra" utype="stc:AstroCoords.Position2D.Value2.C1"/> <FIELDref ref="de" utype="stc:AstroCoords.Position2D.Value2.C2"/> <FIELDref ref="pmra" utype="stc:AstroCoords.Velocity2D.Value2.C1"/> <FIELDref ref="pmde" utype="stc:AstroCoords.Velocity2D.Value2.C2"/> </GROUP> <FIELD ID="ra" name="ra" datatype="float"/> <FIELD ID="de" name="de" datatype="float"/> <FIELD ID="raErr" name="raErr" datatype="float"/> <FIELD ID="deErr" name="deErr" datatype="float"/> <FIELD ID="pmra" name="pmra" datatype="float"/> <FIELD ID="pmde" name="pmde" datatype="float"/>One advantage of this scheme is that it's fairly easy to isolate the STC parsing/unparsing code from the rest of the VOTable handling since the stuff it has to operate on is "just an element", not many elements spread out over the entire document. This scheme also lets you embed multiple data models in a single VOTable. The "primary" data model (e.g., obscore or spectrum in ObsTAP or SSAP, respectively) could still use the FIELD's utype attributes; even though the "primary" data models only have crippled STC metadata (and would not, e.g., support proper motions), such information can still be transmitted in the VOTable and evaluated by clients. Here's an example that could be part of an obscore response in which the server also provides information on SSA: <GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype=... etc, as above </GROUP> <GROUP utype="spec:Spectrum"> <FIELDref ref="ra" utype="spec:Target.pos.ra"/> <FIELDref ref="dec" utype="spec:Target.pos.dec"/> ... (whatever else is in the table for Spectrum) ... </GROUP> <FIELD ID="ra" name="ra" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c1"/> <FIELD ID="de" name="de" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c2"/> ...(ignoring the fact that I've probably made up spec utypes and obscore talks about observation rather than target; I guess you get the drift anyway). Of course, it would be much nicer if we could agree on throwing overboard most of the existing practice and could just say <GROUP utype="stc:CatalogEntryLocation" id="targetPos"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> ... etc, as before </GROUP> <GROUP utype="spec:Spectrum"> <GROUPref ref="targetPos" utype="spec:Target.pos"/> </GROUP>But it's probably too late for that, we don't have a GROUPref element in the first place, and less referencing is better as a rule. -- MarkusDemleitner - 2012-09-20 STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
| ||||||||
Added: | ||||||||
> > |
VO-URP and UTYPE-sSee here for a discussion on how VO-URP can support UTYPE-s discussion. | |||||||
Minutes of telecons |
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. B. Not only flat serialisation to consider Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements:
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models) | ||||||||
Deleted: | ||||||||
< < | ||||||||
STC serialization in a VOTable)"> (STC serialization in a VOTable)Since right now, the only non-deprecated way to include STC metadata in VOTables relies on utypes, it is particularly unfortunate we have no way of doing this that's REC. I'm counting this as a separate use case since IMHO it's a particular shame something as basic is almost undefined in our recommended format -- the format we, as the VO community, actually control. Also, IMHO ideally clients should not have to worry about what (if any) data model some data within a VOTable conforms to. Just as any VOTable library could support (the now deprecated) COOSYS element, support for "modern" STC metadata should be "generic". Sorry for littering the use case with discussions on practice; if you have a better place for this stuff, please do move it there. One plan to do this is described in Referencing STC in VOTable. The basic idea is to collect all information pertaining to STC (or even some other data model) in one group, like this:<GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> <PARAM name="CoordRefFrame" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordRefFrame" value="ICRS"/> <PARAM name="ReferencePosition" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.ReferencePosition" value="BARYCENTER"/> <PARAM name="TimeScale" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.TimeScale" value="TT"/> <PARAM name="Epoch" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch" value="2010.2"/> <PARAM name="yearDef" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch.yearDef" value="J"/> <PARAM name="TimeInstant" datatype="char" arraysize="*" utype="stc:AstroCoords.Time.TimeInstant" value="2002-01-28T09:30:00"/> <PARAM name="URI" datatype="char" arraysize="*" utype="stc:DataModel.URI" value="http://www.ivoa.net/xml/STC/stc-v1.30.xsd"/> <FIELDref ref="raErr" utype="stc:AstroCoords.Position2D.Error2.C1"/> <FIELDref ref="deErr" utype="stc:AstroCoords.Position2D.Error2.C2"/> <FIELDref ref="ra" utype="stc:AstroCoords.Position2D.Value2.C1"/> <FIELDref ref="de" utype="stc:AstroCoords.Position2D.Value2.C2"/> <FIELDref ref="pmra" utype="stc:AstroCoords.Velocity2D.Value2.C1"/> <FIELDref ref="pmde" utype="stc:AstroCoords.Velocity2D.Value2.C2"/> </GROUP> <FIELD ID="ra" name="ra" datatype="float"/> <FIELD ID="de" name="de" datatype="float"/> <FIELD ID="raErr" name="raErr" datatype="float"/> <FIELD ID="deErr" name="deErr" datatype="float"/> <FIELD ID="pmra" name="pmra" datatype="float"/> <FIELD ID="pmde" name="pmde" datatype="float"/> | ||||||||
Changed: | ||||||||
< < | One advantage of this scheme is that it's fairly easy to isolate the STC | |||||||
> > | One advantage of this scheme is that it's fairly easy to isolate the STC parsing/unparsing code from the rest of the VOTable handling since the stuff it has to operate on is "just an element", not many elements spread out over the entire document. | |||||||
Deleted: | ||||||||
< < | parsing/unparsing code from the rest of the VOTable handling since the stuff it has to operate on is "just an element", not many elements spread out over the entire document. | |||||||
Changed: | ||||||||
< < | This scheme also lets you embed multiple data models in a single | |||||||
> > | This scheme also lets you embed multiple data models in a single VOTable. The "primary" data model (e.g., obscore or spectrum in ObsTAP or SSAP, respectively) could still use the FIELD's utype attributes; even though the "primary" data models only have crippled STC metadata (and would not, e.g., support proper motions), such information can still be transmitted in the VOTable and evaluated by clients. Here's an example that could be part of an obscore response in which the server also provides information on SSA: | |||||||
Deleted: | ||||||||
< < | VOTable. The "primary" data model (e.g., obscore or spectrum in ObsTAP or SSAP, respectively) could still use the FIELD's utype attributes; even though the "primary" data models only have crippled STC metadata (and would not, e.g., support proper motions), such information can still be transmitted in the VOTable and evaluated by clients. Here's an example that could be part of an obscore response in which the server also provides information on SSA: | |||||||
<GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype=... etc, as above </GROUP> <GROUP utype="spec:Spectrum"> <FIELDref ref="ra" utype="spec:Target.pos.ra"/> <FIELDref ref="dec" utype="spec:Target.pos.dec"/> ... (whatever else is in the table for Spectrum) ... </GROUP> <FIELD ID="ra" name="ra" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c1"/> <FIELD ID="de" name="de" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c2"/> ... | ||||||||
Changed: | ||||||||
< < | (ignoring the fact that I've probably made up spec utypes and obscore talks | |||||||
> > | (ignoring the fact that I've probably made up spec utypes and obscore talks about observation rather than target; I guess you get the drift anyway). Of course, it would be much nicer if we could agree on throwing overboard most of the existing practice and could just say | |||||||
Deleted: | ||||||||
< < | about observation rather than target; I guess you get the drift anyway). Of course, it would be much nicer if we could agree on throwing overboard most of the existing practice and could just say | |||||||
<GROUP utype="stc:CatalogEntryLocation" id="targetPos"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> ... etc, as before </GROUP> <GROUP utype="spec:Spectrum"> <GROUPref ref="targetPos" utype="spec:Target.pos"/> </GROUP>But it's probably too late for that, we don't have a GROUPref element in the first place, and less referencing is better as a rule. -- MarkusDemleitner - 2012-09-20 STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
Minutes of telecons | ||||||||
Added: | ||||||||
> > |
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. B. Not only flat serialisation to consider Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements:
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models) | ||||||||
Added: | ||||||||
> > |
STC serialization in a VOTable)"> (STC serialization in a VOTable)Since right now, the only non-deprecated way to include STC metadata in VOTables relies on utypes, it is particularly unfortunate we have no way of doing this that's REC. I'm counting this as a separate use case since IMHO it's a particular shame something as basic is almost undefined in our recommended format -- the format we, as the VO community, actually control. Also, IMHO ideally clients should not have to worry about what (if any) data model some data within a VOTable conforms to. Just as any VOTable library could support (the now deprecated) COOSYS element, support for "modern" STC metadata should be "generic". Sorry for littering the use case with discussions on practice; if you have a better place for this stuff, please do move it there. One plan to do this is described in Referencing STC in VOTable. The basic idea is to collect all information pertaining to STC (or even some other data model) in one group, like this:<GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> <PARAM name="CoordRefFrame" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordRefFrame" value="ICRS"/> <PARAM name="ReferencePosition" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.ReferencePosition" value="BARYCENTER"/> <PARAM name="TimeScale" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.TimeFrame.TimeScale" value="TT"/> <PARAM name="Epoch" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch" value="2010.2"/> <PARAM name="yearDef" datatype="char" arraysize="*" utype="stc:AstroCoords.Position2D.Epoch.yearDef" value="J"/> <PARAM name="TimeInstant" datatype="char" arraysize="*" utype="stc:AstroCoords.Time.TimeInstant" value="2002-01-28T09:30:00"/> <PARAM name="URI" datatype="char" arraysize="*" utype="stc:DataModel.URI" value="http://www.ivoa.net/xml/STC/stc-v1.30.xsd"/> <FIELDref ref="raErr" utype="stc:AstroCoords.Position2D.Error2.C1"/> <FIELDref ref="deErr" utype="stc:AstroCoords.Position2D.Error2.C2"/> <FIELDref ref="ra" utype="stc:AstroCoords.Position2D.Value2.C1"/> <FIELDref ref="de" utype="stc:AstroCoords.Position2D.Value2.C2"/> <FIELDref ref="pmra" utype="stc:AstroCoords.Velocity2D.Value2.C1"/> <FIELDref ref="pmde" utype="stc:AstroCoords.Velocity2D.Value2.C2"/> </GROUP> <FIELD ID="ra" name="ra" datatype="float"/> <FIELD ID="de" name="de" datatype="float"/> <FIELD ID="raErr" name="raErr" datatype="float"/> <FIELD ID="deErr" name="deErr" datatype="float"/> <FIELD ID="pmra" name="pmra" datatype="float"/> <FIELD ID="pmde" name="pmde" datatype="float"/>One advantage of this scheme is that it's fairly easy to isolate the STC parsing/unparsing code from the rest of the VOTable handling since the stuff it has to operate on is "just an element", not many elements spread out over the entire document. This scheme also lets you embed multiple data models in a single VOTable. The "primary" data model (e.g., obscore or spectrum in ObsTAP or SSAP, respectively) could still use the FIELD's utype attributes; even though the "primary" data models only have crippled STC metadata (and would not, e.g., support proper motions), such information can still be transmitted in the VOTable and evaluated by clients. Here's an example that could be part of an obscore response in which the server also provides information on SSA: <GROUP utype="stc:CatalogEntryLocation"> <PARAM name="CoordFlavor" datatype=... etc, as above </GROUP> <GROUP utype="spec:Spectrum"> <FIELDref ref="ra" utype="spec:Target.pos.ra"/> <FIELDref ref="dec" utype="spec:Target.pos.dec"/> ... (whatever else is in the table for Spectrum) ... </GROUP> <FIELD ID="ra" name="ra" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c1"/> <FIELD ID="de" name="de" datatype="float" utype="obscore:char.spatialaxis.coverage.location.coord.position2d.value2.c2"/> ...(ignoring the fact that I've probably made up spec utypes and obscore talks about observation rather than target; I guess you get the drift anyway). Of course, it would be much nicer if we could agree on throwing overboard most of the existing practice and could just say <GROUP utype="stc:CatalogEntryLocation" id="targetPos"> <PARAM name="CoordFlavor" datatype="char" arraysize="*" utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" value="SPHERICAL"/> ... etc, as before </GROUP> <GROUP utype="spec:Spectrum"> <GROUPref ref="targetPos" utype="spec:Target.pos"/> </GROUP>But it's probably too late for that, we don't have a GROUPref element in the first place, and less referencing is better as a rule. -- MarkusDemleitner - 2012-09-20 | |||||||
STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
Minutes of telecons |
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. B. Not only flat serialisation to consider Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements:
| ||||||||
Deleted: | ||||||||
< < | ||||||||
Remarks for R5-R7: I added them since these were at times design goals for utypes, and if we drop them we should do so consciously. R5 is a consequence of utype columns in TAP_SCHEMA, for example, and it also helps locating data model parts within larger documents. R6 is stipulated in SSA but has been hotly debated within Obscore. It's at least painful with current ADQL, and IMHO outside of it, too, so it's at least contentious. R7, finally, has been the reason for the abuse of xmlns in SSA VOTables. -- MarkusDemleitner - 2012-09-07 Remarks on breaking out reqs 1a through 1c: Supporting 1a and 1b introduces a whole new set of constraints on the actual method(s) employed, and I'm pretty sure their cost in terms of specification complexity is rather high (e.g., 1b requires we define a DM specification language). If we go this way, we should do so seeing the cost. As to 1c, that's an el-cheapo dumb-down of 1b that's probably much easier to achieve but my just be good enough. -- MarkusDemleitner - 2012-09-12 Remarks on 8: The classic example (I'm reluctant to call it use case) is that an STC library should be able to identify and e.g., transform coordinates given in characterization metadata even if it has no idea that something like char exists. -- MarkusDemleitner - 2012-09-12 | ||||||||
Changed: | ||||||||
< < | Remark on 9: This is currently realized by way of FIELDref/PARAMref; an | |||||||
> > | Remark on 9: This is currently realized by way of FIELDref/PARAMref; an obvious -- and format-agnostic -- alternative is to allow multiple "pointers" in a single utype, e.g., by concatenating individual foo:bar.quux strings with a separator (";" was once floated as a candidate for that). Other solutions, possibly less demanding on the format (FIELDref/PARAMref) or the utype format itself (internal structure) are certainly conceivable. -- MarkusDemleitner - 2012-09-14 | |||||||
Deleted: | ||||||||
< < | obvious -- and format-agnostic -- alternative is to allow multiple "pointers" in a single utype, e.g., by concatenating individual foo:bar.quux strings with a separator (";" was once floated as a candidate for that). Other solutions, possibly less demanding on the format (FIELDref/PARAMref) or the utype format itself (internal structure) are certainly conceivable. -- MarkusDemleitner - 2012-09-14 | |||||||
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
| ||||||||
Deleted: | ||||||||
< < | Minutes of telecon 2012-08-21 here | |||||||
Changed: | ||||||||
< < | Minutes of the Utypes Telco / Wednesday 5 September CET 18:00 here | |||||||
> > | Minutes of telecons | |||||||
Added: | ||||||||
> > |
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. B. Not only flat serialisation to consider Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements:
| ||||||||
Added: | ||||||||
> > |
| |||||||
Remarks for R5-R7: I added them since these were at times design goals for utypes, and if we drop them we should do so consciously. R5 is a consequence of utype columns in TAP_SCHEMA, for example, and it also helps locating data model parts within larger documents. R6 is stipulated in SSA but has been hotly debated within Obscore. It's at least painful with current ADQL, and IMHO outside of it, too, so it's at least contentious. R7, finally, has been the reason for the abuse of xmlns in SSA VOTables. -- MarkusDemleitner - 2012-09-07 Remarks on breaking out reqs 1a through 1c: Supporting 1a and 1b introduces a whole new set of constraints on the actual method(s) employed, and I'm pretty sure their cost in terms of specification complexity is rather high (e.g., 1b requires we define a DM specification language). If we go this way, we should do so seeing the cost. As to 1c, that's an el-cheapo dumb-down of 1b that's probably much easier to achieve but my just be good enough. -- MarkusDemleitner - 2012-09-12 Remarks on 8: The classic example (I'm reluctant to call it use case) is that an STC library should be able to identify and e.g., transform coordinates given in characterization metadata even if it has no idea that something like char exists. -- MarkusDemleitner - 2012-09-12 | ||||||||
Added: | ||||||||
> > | Remark on 9: This is currently realized by way of FIELDref/PARAMref; an obvious -- and format-agnostic -- alternative is to allow multiple "pointers" in a single utype, e.g., by concatenating individual foo:bar.quux strings with a separator (";" was once floated as a candidate for that). Other solutions, possibly less demanding on the format (FIELDref/PARAMref) or the utype format itself (internal structure) are certainly conceivable. -- MarkusDemleitner - 2012-09-14 | |||||||
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
Minutes of telecon 2012-08-21 hereMinutes of the Utypes Telco / Wednesday 5 September CET 18:00 here |
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : | ||||||||
Added: | ||||||||
> > | A. there is an underlying core use-case not mentionned above : create labels to map a piece of metadata (any_name, value) to an IVOA data model field if it exists and if usage defined in this model corresponds to usage in the considered service or application . | |||||||
The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. | ||||||||
Added: | ||||||||
> > | B. Not only flat serialisation to consider | |||||||
Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
| ||||||||
Changed: | ||||||||
< < | -- MireilleLouys - 2012-09-10 | |||||||
> > | -- MireilleLouys - 2012-09-10/ updated Sept 12 | |||||||
Requirements:
Abstract Use Cases:Data Model (de)serialization
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
VO Tools
Data Model description
| ||||||||
Added: | ||||||||
> > |
| |||||||
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: | ||||||||
Changed: | ||||||||
< < | TargetName | RA | DEC | Instrument | Filter | Magnitude | Units | |||||||
> > | TargetName | RA | DEC | Instrument | Filter | Magnitude | Units | |||||||
M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG
M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG
NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy
(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and Uses | ||||||||
Changed: | ||||||||
< < | Spectrum 1.1 REC, Quality Flags: FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) | |||||||
> > | Spectrum 1.1 REC, Quality Flags: Data.FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) | |||||||
Changed: | ||||||||
< < | Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* | |||||||
> > | Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* | |||||||
Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? | ||||||||
Changed: | ||||||||
< < | Extensibility (e.g. NED SED): FluxAxis.Published.Value: is this Utype by any chance related to the standard FluxAxis or to Target.Name? (How can I infer it?) | |||||||
> > | Extensibility (e.g. NED SED): Data.FluxAxis.Published.Value: is this Utype by any chance related to the standard Data.FluxAxis or to Target.Name? (How can I infer it?) | |||||||
Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
| ||||||||
Added: | ||||||||
> > | Minutes of telecon 2012-08-21 here | |||||||
Changed: | ||||||||
< < | Minutes of telecon 2012-08-21 | |||||||
> > | Minutes of the Utypes Telco / Wednesday 5 September CET 18:00 here | |||||||
Deleted: | ||||||||
< < |
Persons present: Omar, Markus, Gerard, Matthew, Patrick
Points discussed (not in temporal order):
Starting with a clean slate as far as the current working draft is concerned (while harvesting from the current text what's useful): None of the persons present object.
But: We'll have to first come up with a report that summarizes current usage of utypes (plus, when we have a better idea of where we want to go, the impact of any changes to current practice). Matthew volunteers to edit this report. Markus will help out.
Omar will be editor of the working draft, and we assume Mireille will also want to be part of the effort.
Omar will circulate a list of the use cases over the DM list; there's already something on this page (http://wiki.ivoa.net/twiki/bin/view/IVOA/UtypesTigerTeam), but it needs smoothing and elaboration. Documents will end up in volute.
We need to make explicit what are the prime use cases driving utypes. Not all of the ones on the wiki may be achieveable.
Attempts to define utypes: Are they "pointers into a data model instance", much like xpath points into a XML document? That much seems not very contentious. As far as this function is concerned, it is seen that inheritance and embedding are not very well dealt with now.
"I should be able to serialize and deserialize a data model instance in any way I want" using utypes (Matthew). But we obviously need to pose additional constraints on what the target formats need to be able to to (e.g., represent key-value pairs).
What formats do we actually care about? VOTable obviously, FITS probably (though some say people that want FITS won't care about data models), SAMP messages. JSON? No agreement. CSV doesn't have enough of a header to allow utype mapping, probably.
We've got about 8 weeks until the interop. What do we want to get done by then? We want the use cases in a form so TCG/plenary/whoever can decide which absolutely have to be covered, where we as Tiger Team provide comments/suggest priorities. We also must have the overview of where utypes are already used. Telecons every two weeks.
Minutes of the Utypes Telco / Wednesday 5 September CET 18:00Participants: Omar Laurino, Gerard Lemson, Pat Dowler, Markus Demleitner, Matthew Graham, Mireille Louys Matthew : reports on his survey about the current state and usage of Utypes in the VO. A first version of his note has been circulated in the Tiger Team. It points to some conflicts in the existing assertions found in various VO specifications. For instance :
|
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
Requirements: | ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
Added: | ||||||||
> > |
| |||||||
| ||||||||
Added: | ||||||||
> > |
| |||||||
Remarks for R5-R7: I added them since these were at times design goals for utypes, and if we drop them we should do so consciously. R5 is a consequence of utype columns in TAP_SCHEMA, for example, and it also helps locating data model parts within larger documents. R6 is stipulated in SSA but has been hotly debated within Obscore. It's at least painful with current ADQL, and IMHO outside of it, too, so it's at least contentious. R7, finally, has been the reason for the abuse of xmlns in SSA VOTables. -- MarkusDemleitner - 2012-09-07 | ||||||||
Added: | ||||||||
> > | Remarks on breaking out reqs 1a through 1c: Supporting 1a and 1b introduces a whole new set of constraints on the actual method(s) employed, and I'm pretty sure their cost in terms of specification complexity is rather high (e.g., 1b requires we define a DM specification language). If we go this way, we should do so seeing the cost. As to 1c, that's an el-cheapo dumb-down of 1b that's probably much easier to achieve but my just be good enough. -- MarkusDemleitner - 2012-09-12 Remarks on 8: The classic example (I'm reluctant to call it use case) is that an STC library should be able to identify and e.g., transform coordinates given in characterization metadata even if it has no idea that something like char exists. -- MarkusDemleitner - 2012-09-12 | |||||||
Abstract Use Cases:Data Model (de)serialization
VO Tools | ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): FluxAxis.Published.Value: is this Utype by any chance related to the standard FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
Minutes of telecon 2012-08-21Persons present: Omar, Markus, Gerard, Matthew, Patrick Points discussed (not in temporal order): Starting with a clean slate as far as the current working draft is concerned (while harvesting from the current text what's useful): None of the persons present object. But: We'll have to first come up with a report that summarizes current usage of utypes (plus, when we have a better idea of where we want to go, the impact of any changes to current practice). Matthew volunteers to edit this report. Markus will help out. Omar will be editor of the working draft, and we assume Mireille will also want to be part of the effort. Omar will circulate a list of the use cases over the DM list; there's already something on this page (http://wiki.ivoa.net/twiki/bin/view/IVOA/UtypesTigerTeam), but it needs smoothing and elaboration. Documents will end up in volute. We need to make explicit what are the prime use cases driving utypes. Not all of the ones on the wiki may be achieveable. Attempts to define utypes: Are they "pointers into a data model instance", much like xpath points into a XML document? That much seems not very contentious. As far as this function is concerned, it is seen that inheritance and embedding are not very well dealt with now. "I should be able to serialize and deserialize a data model instance in any way I want" using utypes (Matthew). But we obviously need to pose additional constraints on what the target formats need to be able to to (e.g., represent key-value pairs). What formats do we actually care about? VOTable obviously, FITS probably (though some say people that want FITS won't care about data models), SAMP messages. JSON? No agreement. CSV doesn't have enough of a header to allow utype mapping, probably. We've got about 8 weeks until the interop. What do we want to get done by then? We want the use cases in a form so TCG/plenary/whoever can decide which absolutely have to be covered, where we as Tiger Team provide comments/suggest priorities. We also must have the overview of where utypes are already used. Telecons every two weeks.Minutes of the Utypes Telco / Wednesday 5 September CET 18:00Participants: Omar Laurino, Gerard Lemson, Pat Dowler, Markus Demleitner, Matthew Graham, Mireille Louys Matthew : reports on his survey about the current state and usage of Utypes in the VO. A first version of his note has been circulated in the Tiger Team. It points to some conflicts in the existing assertions found in various VO specifications. For instance :
|
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements-- MireilleLouys - 2012-09-10 Comments on the use-cases : The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. | ||||||||
Changed: | ||||||||
< < | Today we can distinguish more various ways to distribute data sets:( please iterate on this to enrich this section ...) 1. Separated extra metadata file: | |||||||
> > | Today we can distinguish more various ways to distribute data sets:( -- please iterate on this to enrich this section ...-- )
| |||||||
Deleted: | ||||||||
< < | The data comes along in a file in a usual file format for astronomers : FITS table , image, bin-table , tsv , other,... The metadata comes together in a separate file as an extra documentation for this data set, where, when ,how was it obtained, and what kind of information is inside. This was encouraged from the beginning because this was a strategy to enrich and describe existing data set and circulate them, with their added metadata "wrap". 1. All-in one file structure: The data are coded in a usual format for the astronomer , or an adhoc format ( see Astrowise?) . Metadata are inserted in the data file and describe : where, when , how, etc... + the way the data arranged in the files ( table names , column names , content, etc.. List here example of usage : catalogs ... 1. Complex related datafiles Compressed-tied together archive files like in X-ray tarballs: data +metadata + interpretation charts and parameters, etc. Interpreting these data sets via applications , need to identify the mapping of metadata parameters and values , with the individual parts of a data model. Publishing a data set to the VO , needs to apply IVOA data model items to local metadata values. | |||||||
The interoperability relies on this mapping layer. | ||||||||
Changed: | ||||||||
< < | --IVOA.MireilleLouys - 2012-09-10 | |||||||
> > | -- MireilleLouys - 2012-09-10 | |||||||
Requirements:
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): FluxAxis.Published.Value: is this Utype by any chance related to the standard FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
Minutes of telecon 2012-08-21Persons present: Omar, Markus, Gerard, Matthew, Patrick Points discussed (not in temporal order): Starting with a clean slate as far as the current working draft is concerned (while harvesting from the current text what's useful): None of the persons present object. But: We'll have to first come up with a report that summarizes current usage of utypes (plus, when we have a better idea of where we want to go, the impact of any changes to current practice). Matthew volunteers to edit this report. Markus will help out. Omar will be editor of the working draft, and we assume Mireille will also want to be part of the effort. Omar will circulate a list of the use cases over the DM list; there's already something on this page (http://wiki.ivoa.net/twiki/bin/view/IVOA/UtypesTigerTeam), but it needs smoothing and elaboration. Documents will end up in volute. We need to make explicit what are the prime use cases driving utypes. Not all of the ones on the wiki may be achieveable. Attempts to define utypes: Are they "pointers into a data model instance", much like xpath points into a XML document? That much seems not very contentious. As far as this function is concerned, it is seen that inheritance and embedding are not very well dealt with now. "I should be able to serialize and deserialize a data model instance in any way I want" using utypes (Matthew). But we obviously need to pose additional constraints on what the target formats need to be able to to (e.g., represent key-value pairs). What formats do we actually care about? VOTable obviously, FITS probably (though some say people that want FITS won't care about data models), SAMP messages. JSON? No agreement. CSV doesn't have enough of a header to allow utype mapping, probably. We've got about 8 weeks until the interop. What do we want to get done by then? We want the use cases in a form so TCG/plenary/whoever can decide which absolutely have to be covered, where we as Tiger Team provide comments/suggest priorities. We also must have the overview of where utypes are already used. Telecons every two weeks.Minutes of the Utypes Telco / Wednesday 5 September CET 18:00Participants: Omar Laurino, Gerard Lemson, Pat Dowler, Markus Demleitner, Matthew Graham, Mireille Louys Matthew : reports on his survey about the current state and usage of Utypes in the VO. A first version of his note has been circulated in the Tiger Team. It points to some conflicts in the existing assertions found in various VO specifications. For instance :
|
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and Requirements | ||||||||
Added: | ||||||||
> > | -- MireilleLouys - 2012-09-10 Comments on the use-cases : The very first goal why Utypes were invented is to "attach" to some metadata value the name of a data model attribute in an object oriented model describing astronomical observations and simulations. This was thought for a set of metadata , attached to an observation , and represented in a VOtable file. The idea is just to propose as general uniform 'names' a set of strings logically constructed from the names of classes and attributes built up in data models. Just looking at the variety of possible fits keywords and vocalulary extensions defined in various archives shows that the problem of defining a uniform langage for all pieces of metadata is too complex. Relying on a recommended modeled metadata arrangement ( the current set of recommended Data models for IVOA) has given a framework for that. Even restricted and bounded it covers common use-cases. Today we can distinguish more various ways to distribute data sets:( please iterate on this to enrich this section ...) 1. Separated extra metadata file: The data comes along in a file in a usual file format for astronomers : FITS table , image, bin-table , tsv , other,... The metadata comes together in a separate file as an extra documentation for this data set, where, when ,how was it obtained, and what kind of information is inside. This was encouraged from the beginning because this was a strategy to enrich and describe existing data set and circulate them, with their added metadata "wrap". 1. All-in one file structure: The data are coded in a usual format for the astronomer , or an adhoc format ( see Astrowise?) . Metadata are inserted in the data file and describe : where, when , how, etc... + the way the data arranged in the files ( table names , column names , content, etc.. List here example of usage : catalogs ... 1. Complex related datafiles Compressed-tied together archive files like in X-ray tarballs: data +metadata + interpretation charts and parameters, etc. Interpreting these data sets via applications , need to identify the mapping of metadata parameters and values , with the individual parts of a data model. Publishing a data set to the VO , needs to apply IVOA data model items to local metadata values. The interoperability relies on this mapping layer. --IVOA.MireilleLouys - 2012-09-10 | |||||||
Requirements:
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): FluxAxis.Published.Value: is this Utype by any chance related to the standard FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
Minutes of telecon 2012-08-21Persons present: Omar, Markus, Gerard, Matthew, Patrick Points discussed (not in temporal order): Starting with a clean slate as far as the current working draft is concerned (while harvesting from the current text what's useful): None of the persons present object. But: We'll have to first come up with a report that summarizes current usage of utypes (plus, when we have a better idea of where we want to go, the impact of any changes to current practice). Matthew volunteers to edit this report. Markus will help out. Omar will be editor of the working draft, and we assume Mireille will also want to be part of the effort. Omar will circulate a list of the use cases over the DM list; there's already something on this page (http://wiki.ivoa.net/twiki/bin/view/IVOA/UtypesTigerTeam), but it needs smoothing and elaboration. Documents will end up in volute. We need to make explicit what are the prime use cases driving utypes. Not all of the ones on the wiki may be achieveable. Attempts to define utypes: Are they "pointers into a data model instance", much like xpath points into a XML document? That much seems not very contentious. As far as this function is concerned, it is seen that inheritance and embedding are not very well dealt with now. "I should be able to serialize and deserialize a data model instance in any way I want" using utypes (Matthew). But we obviously need to pose additional constraints on what the target formats need to be able to to (e.g., represent key-value pairs). What formats do we actually care about? VOTable obviously, FITS probably (though some say people that want FITS won't care about data models), SAMP messages. JSON? No agreement. CSV doesn't have enough of a header to allow utype mapping, probably. We've got about 8 weeks until the interop. What do we want to get done by then? We want the use cases in a form so TCG/plenary/whoever can decide which absolutely have to be covered, where we as Tiger Team provide comments/suggest priorities. We also must have the overview of where utypes are already used. Telecons every two weeks.Minutes of the Utypes Telco / Wednesday 5 September CET 18:00Participants: Omar Laurino, Gerard Lemson, Pat Dowler, Markus Demleitner, Matthew Graham, Mireille Louys Matthew : reports on his survey about the current state and usage of Utypes in the VO. A first version of his note has been circulated in the Tiger Team. It points to some conflicts in the existing assertions found in various VO specifications. For instance :
|
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and RequirementsRequirements:
| ||||||||
Added: | ||||||||
> > | Remarks for R5-R7: I added them since these were at times design goals for utypes, and if we drop them we should do so consciously. R5 is a consequence of utype columns in TAP_SCHEMA, for example, and it also helps locating data model parts within larger documents. R6 is stipulated in SSA but has been hotly debated within Obscore. It's at least painful with current ADQL, and IMHO outside of it, too, so it's at least contentious. R7, finally, has been the reason for the abuse of xmlns in SSA VOTables. -- MarkusDemleitner - 2012-09-07 | |||||||
Deleted: | ||||||||
< < | Remarks for R5-R7: I added them since these were at times design goals for utypes, and if we drop them we should do so consciously. R5 is a consequence of utype columns in TAP_SCHEMA, for example, and it also helps locating data model parts within larger documents. R6 is stipulated in SSA but has been hotly debated within Obscore. It's at least painful with current ADQL, and IMHO outside of it, too, so it's at least contentious. R7, finally, has been the reason for the abuse of xmlns in SSA VOTables. -- MarkusDemleitner - 2012-09-07 | |||||||
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): FluxAxis.Published.Value: is this Utype by any chance related to the standard FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
Minutes of telecon 2012-08-21Persons present: Omar, Markus, Gerard, Matthew, Patrick Points discussed (not in temporal order): Starting with a clean slate as far as the current working draft is concerned (while harvesting from the current text what's useful): None of the persons present object. But: We'll have to first come up with a report that summarizes current usage of utypes (plus, when we have a better idea of where we want to go, the impact of any changes to current practice). Matthew volunteers to edit this report. Markus will help out. Omar will be editor of the working draft, and we assume Mireille will also want to be part of the effort. Omar will circulate a list of the use cases over the DM list; there's already something on this page (http://wiki.ivoa.net/twiki/bin/view/IVOA/UtypesTigerTeam), but it needs smoothing and elaboration. Documents will end up in volute. We need to make explicit what are the prime use cases driving utypes. Not all of the ones on the wiki may be achieveable. Attempts to define utypes: Are they "pointers into a data model instance", much like xpath points into a XML document? That much seems not very contentious. As far as this function is concerned, it is seen that inheritance and embedding are not very well dealt with now. "I should be able to serialize and deserialize a data model instance in any way I want" using utypes (Matthew). But we obviously need to pose additional constraints on what the target formats need to be able to to (e.g., represent key-value pairs). What formats do we actually care about? VOTable obviously, FITS probably (though some say people that want FITS won't care about data models), SAMP messages. JSON? No agreement. CSV doesn't have enough of a header to allow utype mapping, probably. We've got about 8 weeks until the interop. What do we want to get done by then? We want the use cases in a form so TCG/plenary/whoever can decide which absolutely have to be covered, where we as Tiger Team provide comments/suggest priorities. We also must have the overview of where utypes are already used. Telecons every two weeks. | ||||||||
Added: | ||||||||
> > |
Minutes of the Utypes Telco / Wednesday 5 September CET 18:00Participants: Omar Laurino, Gerard Lemson, Pat Dowler, Markus Demleitner, Matthew Graham, Mireille Louys Matthew : reports on his survey about the current state and usage of Utypes in the VO. A first version of his note has been circulated in the Tiger Team. It points to some conflicts in the existing assertions found in various VO specifications. For instance :
|
Utypes Tiger TeamUtypes Use CasesRaw use cases from Urbana-Champaign
Reviewed Use Cases and RequirementsRequirements:
| ||||||||
Added: | ||||||||
> > |
| |||||||
Abstract Use Cases:Data Model (de)serialization
VO Tools
Data Model description
Others
Concrete Use Cases:(Photometry Catalog, points in columns)Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: SDSSID | RA | DEC | U | G | R | I | Z(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): FluxAxis.Published.Value: is this Utype by any chance related to the standard FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
Minutes of telecon 2012-08-21Persons present: Omar, Markus, Gerard, Matthew, Patrick Points discussed (not in temporal order): Starting with a clean slate as far as the current working draft is concerned (while harvesting from the current text what's useful): None of the persons present object. But: We'll have to first come up with a report that summarizes current usage of utypes (plus, when we have a better idea of where we want to go, the impact of any changes to current practice). Matthew volunteers to edit this report. Markus will help out. Omar will be editor of the working draft, and we assume Mireille will also want to be part of the effort. Omar will circulate a list of the use cases over the DM list; there's already something on this page (http://wiki.ivoa.net/twiki/bin/view/IVOA/UtypesTigerTeam), but it needs smoothing and elaboration. Documents will end up in volute. We need to make explicit what are the prime use cases driving utypes. Not all of the ones on the wiki may be achieveable. Attempts to define utypes: Are they "pointers into a data model instance", much like xpath points into a XML document? That much seems not very contentious. As far as this function is concerned, it is seen that inheritance and embedding are not very well dealt with now. "I should be able to serialize and deserialize a data model instance in any way I want" using utypes (Matthew). But we obviously need to pose additional constraints on what the target formats need to be able to to (e.g., represent key-value pairs). What formats do we actually care about? VOTable obviously, FITS probably (though some say people that want FITS won't care about data models), SAMP messages. JSON? No agreement. CSV doesn't have enough of a header to allow utype mapping, probably. We've got about 8 weeks until the interop. What do we want to get done by then? We want the use cases in a form so TCG/plenary/whoever can decide which absolutely have to be covered, where we as Tiger Team provide comments/suggest priorities. We also must have the overview of where utypes are already used. Telecons every two weeks. |
| ||||||||
Added: | ||||||||
> > |
| |||||||
Utypes Tiger TeamUtypes Use Cases | ||||||||
Changed: | ||||||||
< < | ||||||||
> > | Raw use cases from Urbana-Champaign
| |||||||
Added: | ||||||||
> > |
| |||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
Added: | ||||||||
> > | Reviewed Use Cases and Requirements | |||||||
Changed: | ||||||||
< < | Current Practices and Uses | |||||||
> > | Requirements: | |||||||
Changed: | ||||||||
< < | Spectrum 1.1 REC, Quality Flags: FluxAxis.Quality.n, where n is | |||||||
> > |
| |||||||
Deleted: | ||||||||
< < | an integer. (Parsable? Anyway this is gone in Spectral 2.0) | |||||||
Changed: | ||||||||
< < | Photometry 1.0 PR, Access class: “we use the Access class defined in | |||||||
> > |
| |||||||
Deleted: | ||||||||
< < | ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* | |||||||
Changed: | ||||||||
< < | Photometry 1.0 PR, Spectrum is imported using the spec namespace | |||||||
> > |
| |||||||
Deleted: | ||||||||
< < | (notice the difference with the previous approach). | |||||||
Changed: | ||||||||
< < | Namespace (in several DMs): the namespace must be parsed out of the | |||||||
> > |
| |||||||
Deleted: | ||||||||
< < | Utype string… but then again which is the actual Utype string? | |||||||
Changed: | ||||||||
< < | Extensibility (e.g. NED SED): FluxAxis.Published.Value: is this Utype by any chance related to the standard FluxAxis or to | |||||||
> > | Abstract Use Cases:Data Model (de)serialization | |||||||
Deleted: | ||||||||
< < | Target.Name? (How can I infer it?) | |||||||
Changed: | ||||||||
< < | Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
| |||||||
> > |
| |||||||
Deleted: | ||||||||
< < |
| |||||||
Changed: | ||||||||
< < | Redefined in SSA 1.1: | |||||||
> > | VO Tools | |||||||
Deleted: | ||||||||
< < |
| |||||||
Changed: | ||||||||
< < | Utype is built with the pseudo-grammar “<component-name>”.”<field-name>” | |||||||
> > |
| |||||||
Added: | ||||||||
> > |
| |||||||
Changed: | ||||||||
< < | spec:Spectrum.Target.Name and ssa:Target.Name are the same thing. | |||||||
> > | Data Model description | |||||||
Deleted: | ||||||||
< < |
| |||||||
Changed: | ||||||||
< < | Redefined in Spectrum 1.1, also introducing Data Model inheritance:
| |||||||
> > |
| |||||||
Changed: | ||||||||
< < | Most DMs define utypes in tables, using different conventions | |||||||
> > | Others | |||||||
Changed: | ||||||||
< < | Utypes strings can change when DMs are reused. Also, the namespace | |||||||
> > |
| |||||||
Deleted: | ||||||||
< < | changes globally for each DM (spec:Target.Name, ssa:Target.Name) | |||||||
Changed: | ||||||||
< < | Utypes are only partially used in FITS serializations: they can be used for columns, not for parameters: in this case, an arbitrary 8 char string is | |||||||
> > | Concrete Use Cases:(Photometry Catalog, points in columns) | |||||||
Deleted: | ||||||||
< < | provided by the DM document. | |||||||
Changed: | ||||||||
< < | DMs do not define an “xmlns” link to the DM URI | |||||||
> > | Represent a Photometry Catalog with a definite number of Magnitudes expressed in columns and astronomical sources in rows. For example, an SDSS catalog with the following columns: | |||||||
Added: | ||||||||
> > | SDSSID | RA | DEC | U | G | R | I | Z
(Photometry Catalog, points in rows)A Photometry Catalog could refer to a single object observed in a number of filters, or to different objects observed in a number of filters, and the filters could be an arbitrary number. Employing an efficient relational approach would suggest to represent this as a table where each magnitude is expressed in a different row, and the other information (object name, coordinates, instrument, filter, etc) are in columns, or are factored out in the table header if they are common to all points. For instance, here is a (simple) example of an (unnormalized) catalog for different sources. Notice that this table doesn't use any controlled vocabulary for filters, target names and instruments, while VO documents should: TargetName | RA | DEC | Instrument | Filter | Magnitude | Units M51 | xx.yy | xx.yy | SDSS | u | xx | ABMAG M51 | xx.yy | xx.yy | SDSS | g | xx | ABMAG NGC1068 | xx.yy | xx.yy | GALEX | nuv | Jy(Aggregated SED)An Aggregated SED is defined as an aggregation of different segments of spectro-photometric data, where each segment can be a Photometry Catalog, a Spectrum or an entire SED itself. It should be possible to serialize an SED as a list of several tables, each table representing a segment. Complex formats like VOTable and FITS can allow the tables to be stored in the same file.STC serialization)"> (STC serialization)It has to be possible to embed STC instances in tables and attach these instances to other objects in the table. This is a generic example of Model reuse (the STC model is reused by other models)STC in a Spectrum)"> (STC in a Spectrum)This is a more specific example of model reuse. Spectral DM uses some parts of the STC DM to describe the reference frame of the observations: the frame can include several axes: time, spectral coordinate, flux, photometry filters. Some of these axes could be different instance of the same STC or STC-derived class, like the photometry filters. This has to be represented in a generic tabular format using Utypes to describe the structure of the serialized instances. Different instances of the same class must somehow be disentangled from the others.(Create compliant SSA service using a database and a non-compliant archive)Given a database and an archive containing spectra serialized in a non-compliant way (but in a supported format, like FITS), a data publisher might want to create a VO service: in principle this would require the creation of a new database (or a view on the original one) and to copy and change the headers of the non-compliant files. A more efficient solution would be to leave the archive and the database untouched and to add an additional layer on top of them: the layer would add the required metadata to the original files on the fly (see R #4). For example, the service can read the information in the database and fill a VOTable compliant header (putting together the database values with the predefined Utypes) that will wrap the original FITS file in the files response.Current Practices and UsesSpectrum 1.1 REC, Quality Flags: FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): FluxAxis.Published.Value: is this Utype by any chance related to the standard FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
| |||||||
Minutes of telecon 2012-08-21Persons present: Omar, Markus, Gerard, Matthew, Patrick Points discussed (not in temporal order): | ||||||||
Changed: | ||||||||
< < | Starting with a clean slate as far as the current working draft is | |||||||
> > | Starting with a clean slate as far as the current working draft is concerned (while harvesting from the current text what's useful): None of the persons present object. | |||||||
Deleted: | ||||||||
< < | concerned (while harvesting from the current text what's useful): None of the persons present object. | |||||||
Changed: | ||||||||
< < | But: We'll have to first come up with a report that summarizes current | |||||||
> > | But: We'll have to first come up with a report that summarizes current usage of utypes (plus, when we have a better idea of where we want to go, the impact of any changes to current practice). Matthew volunteers to edit this report. Markus will help out. | |||||||
Deleted: | ||||||||
< < | usage of utypes (plus, when we have a better idea of where we want to go, the impact of any changes to current practice). Matthew volunteers to edit this report. Markus will help out. | |||||||
Changed: | ||||||||
< < | Omar will be editor of the working draft, and we assume Mireille will | |||||||
> > | Omar will be editor of the working draft, and we assume Mireille will also want to be part of the effort. | |||||||
Deleted: | ||||||||
< < | also want to be part of the effort. | |||||||
Changed: | ||||||||
< < | Omar will circulate a list of the use cases over the DM list; there's | |||||||
> > | Omar will circulate a list of the use cases over the DM list; there's already something on this page (http://wiki.ivoa.net/twiki/bin/view/IVOA/UtypesTigerTeam), but it needs smoothing and elaboration. Documents will end up in volute. | |||||||
Deleted: | ||||||||
< < | already something on this page (http://wiki.ivoa.net/twiki/bin/view/IVOA/UtypesTigerTeam), but it needs smoothing and elaboration. Documents will end up in volute. | |||||||
Changed: | ||||||||
< < | We need to make explicit what are the prime use cases driving utypes. | |||||||
> > | We need to make explicit what are the prime use cases driving utypes. Not all of the ones on the wiki may be achieveable. | |||||||
Deleted: | ||||||||
< < | Not all of the ones on the wiki may be achieveable. | |||||||
Changed: | ||||||||
< < | Attempts to define utypes: Are they "pointers into a data model | |||||||
> > | Attempts to define utypes: Are they "pointers into a data model instance", much like xpath points into a XML document? That much seems not very contentious. As far as this function is concerned, it is seen that inheritance and embedding are not very well dealt with now. | |||||||
Deleted: | ||||||||
< < | instance", much like xpath points into a XML document? That much seems not very contentious. As far as this function is concerned, it is seen that inheritance and embedding are not very well dealt with now. | |||||||
Changed: | ||||||||
< < | "I should be able to serialize and deserialize a data model instance in | |||||||
> > | "I should be able to serialize and deserialize a data model instance in any way I want" using utypes (Matthew). But we obviously need to pose additional constraints on what the target formats need to be able to to (e.g., represent key-value pairs). | |||||||
Deleted: | ||||||||
< < | any way I want" using utypes (Matthew). But we obviously need to pose additional constraints on what the target formats need to be able to to (e.g., represent key-value pairs). | |||||||
Changed: | ||||||||
< < | What formats do we actually care about? VOTable obviously, FITS | |||||||
> > | What formats do we actually care about? VOTable obviously, FITS probably (though some say people that want FITS won't care about data models), SAMP messages. JSON? No agreement. CSV doesn't have enough of a header to allow utype mapping, probably. | |||||||
Deleted: | ||||||||
< < | probably (though some say people that want FITS won't care about data models), SAMP messages. JSON? No agreement. CSV doesn't have enough of a header to allow utype mapping, probably. | |||||||
Changed: | ||||||||
< < | We've got about 8 weeks until the interop. What do we want to get done | |||||||
> > | We've got about 8 weeks until the interop. What do we want to get done by then? We want the use cases in a form so TCG/plenary/whoever can decide which absolutely have to be covered, where we as Tiger Team provide comments/suggest priorities. We also must have the overview of where utypes are already used. Telecons every two weeks. | |||||||
Deleted: | ||||||||
< < | by then? We want the use cases in a form so TCG/plenary/whoever can decide which absolutely have to be covered, where we as Tiger Team provide comments/suggest priorities. We also must have the overview of where utypes are already used. Telecons every two weeks. |
Utypes Tiger TeamUtypes Use Cases
Current Practices and UsesSpectrum 1.1 REC, Quality Flags: FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): FluxAxis.Published.Value: is this Utype by any chance related to the standard FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
| ||||||||
Added: | ||||||||
> > |
Minutes of telecon 2012-08-21Persons present: Omar, Markus, Gerard, Matthew, Patrick Points discussed (not in temporal order): Starting with a clean slate as far as the current working draft is concerned (while harvesting from the current text what's useful): None of the persons present object. But: We'll have to first come up with a report that summarizes current usage of utypes (plus, when we have a better idea of where we want to go, the impact of any changes to current practice). Matthew volunteers to edit this report. Markus will help out. Omar will be editor of the working draft, and we assume Mireille will also want to be part of the effort. Omar will circulate a list of the use cases over the DM list; there's already something on this page (http://wiki.ivoa.net/twiki/bin/view/IVOA/UtypesTigerTeam), but it needs smoothing and elaboration. Documents will end up in volute. We need to make explicit what are the prime use cases driving utypes. Not all of the ones on the wiki may be achieveable. Attempts to define utypes: Are they "pointers into a data model instance", much like xpath points into a XML document? That much seems not very contentious. As far as this function is concerned, it is seen that inheritance and embedding are not very well dealt with now. "I should be able to serialize and deserialize a data model instance in any way I want" using utypes (Matthew). But we obviously need to pose additional constraints on what the target formats need to be able to to (e.g., represent key-value pairs). What formats do we actually care about? VOTable obviously, FITS probably (though some say people that want FITS won't care about data models), SAMP messages. JSON? No agreement. CSV doesn't have enough of a header to allow utype mapping, probably. We've got about 8 weeks until the interop. What do we want to get done by then? We want the use cases in a form so TCG/plenary/whoever can decide which absolutely have to be covered, where we as Tiger Team provide comments/suggest priorities. We also must have the overview of where utypes are already used. Telecons every two weeks. |
Utypes Tiger TeamUtypes Use Cases
Current Practices and UsesSpectrum 1.1 REC, Quality Flags: FluxAxis.Quality.n, where n is an integer. (Parsable? Anyway this is gone in Spectral 2.0) Photometry 1.0 PR, Access class: “we use the Access class defined in ObsTAP and inherited from SSA” -> PhotometryFilter.transmissionCurve.Access.* Photometry 1.0 PR, Spectrum is imported using the spec namespace (notice the difference with the previous approach). Namespace (in several DMs): the namespace must be parsed out of the Utype string… but then again which is the actual Utype string? Extensibility (e.g. NED SED): FluxAxis.Published.Value: is this Utype by any chance related to the standard FluxAxis or to Target.Name? (How can I infer it?) Introduced as an attribute for FIELD and PARAM in VOTable 1.2:
|