Issues with VOTable version 1.3

This page contains notes and discussions of items in VOTable version 1.3 that may need attention or changes in some future version of the VOTable standard. It is a discussion forum, and inclusion here does not guarantee that any changes will be made. The existence of this page does not necessarily indicate that there will be a new revision of VOTable in the near or distant future, but it makes sense to gather issues here rather than to let them get lost in old mailing list threads.

See also VOTableIssues, which discusses items in VOTable version 1.2 that were addressed in version 1.3 (mostly to do with representing null values).

Unicode primitive character data representation

The discussion of the unicodeChar primitive data type in section 2.1 of the standard says:

VOTables support two kinds of characters: ASCII 1-byte characters and Unicode (UCS-2) 2-byte characters. Unicode is a way to represent characters that is an alternative to ASCII. It uses two bytes per character instead of one, it is strongly supported by XML tools, and it can handle a large variety of international alphabets. Therefore VOTable supports not only ASCII strings (datatype="char"), but also Unicode (datatype="unicodeChar").

This is a bit confused. Unicode itself does not specify how many bytes are used to encode each character. UCS-2 (a Unicode encoding scheme) does specify 2 bytes per character, but is very out of date. It resembles UTF-16, which uses two bytes for most characters, but more bytes for some exotic characters. Moreover the encoding used for Unicode character data inline (e.g. in TABLEDATA/TD elements) is determined by the encoding of the XML document hosting the VOTable, so it makes no sense to mandate an encoding (UCS-2 or anything else) for inline unicode character data. The encoding does however need to be determined somehow for BINARY or BINARY2-serialized data, which is not considered to form part of an XML stream. Another complication is that if UCS-2 really is intended (for BINARY) there should be some discussion of endianness. This topic was discussed in the "Unicode in VOTable" thread started by Walter Landry on the apps mailing list here.

What to do?

Proposals include, in ascending order of disruptiveness:

  1. Rewording the definition of the unicodeChar type to clarify that inline text uses the document's encoding, but retaining (the obsolete and deprecated) UCS-2 for binary serializations, and adding discussion of endianness. This just clarifies the existing text so it makes sense.
  2. Rewording and clarifying as above, but redefining the unicodeChar type to use UTF-16 rather than UCS-2. Given the characters likely to appear in VOTables, the encodings are likely to be identical in most or all cases.
  3. Changing char type to mean unicode character, with its encoding for BINARY serialization being UTF-8, while allowing inline char arrays to contain any Unicode character encoded as per the VOTable document's encoding. In practice it may be the case that some existing software in Unicode-friendly languages such as Java inadvertently treats char data as UTF-8 already. Another argument in favour is that strictly speaking ASCII characters are 7-bit and should have the same interpretation in UTF-8 as in ASCII, but it's probably the case that various extended ASCII character sets are in use in some VOTables which would cause trouble under such a redefinition.
My take is that, for maximum backward compatibility (including with non-Unicode-friendly formats and languages), char should continue to represent 1-byte characters (ASCII or extended ASCII), while unicodeChar should be redefined to use UTF-16 in binary contexts. People who want to write Unicode now should do it using the unicodeChar type, employing UTF-16 with a BOM in BINARY contexts. -- MarkTaylor - 2014-03-31

RESOURCE type attribute

The type attribute of the RESOURCE element is defined like this:

  <xs:attribute name="type" default="results">
    <xs:simpleType>
      <xs:restriction base="xs:NMTOKEN">
        <xs:enumeration value="results"/>
        <xs:enumeration value="meta"/>
      </xs:restriction>
    </xs:simpleType>
  </xs:attribute>

The two possible values seem a bit ad-hoc, and this restriction has been a cause of surprise (see the thread "Datalink feedback II: RESOURCE type" on the DAL list in March 2014 here). Does it make sense to change it? -- MarkTaylor - 2014-03-31


Edit | Attach | Print version | History: r7 | r5 < r4 < r3 < r2 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r3 - 2014-04-02 - MarkusDemleitner
 
This site is powered by the TWiki collaboration platformCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback