Skip to Content.
Sympa Menu

xom-interest - Re: [XOM-interest] Is XML Text or CDATA that includes the zerocharacter legal?

xom-interest AT lists.ibiblio.org

Subject: XOM API for Processing XML with Java

List archive

Chronological Thread  
  • From: Wolfgang Hoschek <whoschek AT lbl.gov>
  • To: John Cowan <cowan AT ccil.org>
  • Cc: xom-interest AT lists.ibiblio.org
  • Subject: Re: [XOM-interest] Is XML Text or CDATA that includes the zerocharacter legal?
  • Date: Thu, 28 Jul 2005 12:47:32 -0700

Mike and John, thanks for the clarifications.

To summarize, the zero character can't appear anywhere inside an XML document. Not even in a Unicode surrogate.

Why is this a nice property?

Consider that UTF-8 encoding has the property that it never ever produces a zero *byte* as part of the output byte stream, unless a zero *character* is part of the input. It follows that an UTF-8 encoded XML document never ever contains a zero byte.

This corrolary can be exploited in a variety of applications, in particular for unambigous end-of-document or end-of-fragment demarkation in data containers. For example, if one was to define the simplest conceivable XML based multicast protocol, continously sending and receiving N messages, it would go along the following lines:

- A message is an XML document
- All XML documents are UTF-8 encoded
- Sender writes UTF-8 encoded XML document onto multicast stream, appending a zero byte terminator after each document
- Receivers listen in on the stream until they see a zero byte, after which they parse an XML document until the next zero byte is seen
- Receiver acts on parsed document, as deemed appropriate
- Repeat forever

This is an unambigous, simple and efficient framing protocol, making it straighforward for receivers to join the multicast stream at any point in time, making it straightforward to recover and resync from errors such as disconnects, without requiring any further higher level or out-of-band knowledge. To sync or resync, simply wait for the next zero byte. For example, it becomes unnecessary to frame messages with "byte stuffing" mechanisms such as COBS [1] or similar.

Another example application is the "safe, simple and efficient" usage of zero byte terminated strings in a persistent QName and text pool for binary XML schemes such as bnux.

[To the curious: XML does not define an end-of-document marker, leaving multi-document handling up to higher level framing protocols, with predictably miserable interop implications at the real-world application level. Heck, even networked single document stream handling is undefined, considering there's no definition of what it means to parse until "end-of-stream", unless one is parsing a single document from a single file. Evasion to HTTP(S), SOAP or similar is often like shooting pigeons with cannons...]

Wolfgang.

[1] COBS - In my experiece, a proven and incredibly efficient general- purpose byte stuffing scheme: http://www.stuartcheshire.org/papers/ COBSforToN.pdf

On Jul 28, 2005, at 5:40 AM, John Cowan wrote:

Michael Kay scripsit:


In XML 1.0 no C0 characters other than x9, xA, or xD are permitted.

In XML 1.1 all C0 characters other than X0 are permitted. A concession,
apparently, to people still writing in archaic programming languages that
treat 0 as a string terminator...


To clarify:

In XML 1.0, the characters #x0..#x1F (except #x9, #xA, and #xD) are
forbidden unconditionally; the characters #x7F..#x9F are permitted
unconditionally.

In XML 1.1, the characters #x1..#x1F and #x7F..#x9F are permitted as
character references, but not as raw characters (except #x9, #xA, and
#xD, which are permitted either way). #x0 is still forbidden either
raw or as a reference. (Technically this makes XML 1.1 not backward
compatible with XML 1.0. Whoop.)

The point of allowing these as references is to permit straightforward
XML serialization of database fields, which typically cannot contain #x0,
while preventing outright binary from being serialized.



Further, am I right in that a zero character must not appear in a
surrogate?


What do you mean by a "surrogate"? If you mean a Unicode surrogate pair, then a
#x0 character cannot be part of such a pair. Zero *bytes* can of course appear
as part of the representation of many Unicode characters in UTF-16 or UTF-32.


Yes, (Java) Unicode surrogate is what I mean.





Archive powered by MHonArc 2.6.24.

Top of Page