XOM API for Processing XML with Java

[Wolfgang Hoschek <whoschek AT lbl.gov>]

Mike and John, thanks for the clarifications.

To summarize, the zero character can't appear anywhere inside an XML  
document. Not even in a Unicode surrogate.

Why is this a nice property?

Consider that UTF-8 encoding has the property that it never ever  
produces a zero *byte* as part of the output byte stream, unless a  
zero *character* is part of the input. It follows that an UTF-8  
encoded XML document never ever contains a zero byte.

This corrolary can be exploited in a variety of applications, in  
particular for unambigous end-of-document or end-of-fragment  
demarkation in data containers. For example, if one was to define the  
simplest conceivable XML based multicast protocol, continously  
sending and receiving N messages, it would go along the following lines:

- A message is an XML document
- All XML documents are UTF-8 encoded
- Sender writes UTF-8 encoded XML document onto multicast stream,  
appending a zero byte terminator after each document
- Receivers listen in on the stream until they see a zero byte, after  
which they parse an XML document until the next zero byte is seen
- Receiver acts on parsed document, as deemed appropriate
- Repeat forever

This is an unambigous, simple and efficient framing protocol, making  
it straighforward for receivers to join the multicast stream at any  
point in time, making it straightforward to recover and resync from  
errors such as disconnects, without requiring any further higher  
level or out-of-band knowledge. To sync or resync, simply wait for  
the next zero byte. For example, it becomes unnecessary to frame  
messages with "byte stuffing" mechanisms such as COBS [1] or similar.

Another example application is the "safe, simple and efficient" usage  
of zero byte terminated strings in a persistent QName and text pool  
for binary XML schemes such as bnux.

[To the curious: XML does not define an end-of-document marker,  
leaving multi-document handling up to higher level framing protocols,  
with predictably miserable interop implications at the real-world  
application level. Heck, even networked single document stream  
handling is undefined, considering there's no definition of what it  
means to parse until "end-of-stream", unless one is parsing a single  
document from a single file. Evasion to HTTP(S), SOAP or similar is  
often like shooting pigeons with cannons...]

Wolfgang.

[1] COBS - In my experiece, a proven and incredibly efficient general- 
purpose byte stuffing scheme: http://www.stuartcheshire.org/papers/ 
COBSforToN.pdf

On Jul 28, 2005, at 5:40 AM, John Cowan wrote:

> Michael Kay scripsit:
>
>
> > In XML 1.0 no C0 characters other than x9, xA, or xD are permitted.
> >
> > In XML 1.1 all C0 characters other than X0 are permitted. A  
> > concession,
> > apparently, to people still writing in archaic programming  
> > languages that
> > treat 0 as a string terminator...
>
> To clarify:
>
> In XML 1.0, the characters #x0..#x1F (except #x9, #xA, and #xD) are
> forbidden unconditionally; the characters #x7F..#x9F are permitted
> unconditionally.
>
> In XML 1.1, the characters #x1..#x1F and #x7F..#x9F are permitted as
> character references, but not as raw characters (except #x9, #xA, and
> #xD, which are permitted either way).  #x0 is still forbidden either
> raw or as a reference.  (Technically this makes XML 1.1 not backward
> compatible with XML 1.0.  Whoop.)
>
> The point of allowing these as references is to permit straightforward
> XML serialization of database fields, which typically cannot  
> contain #x0,
> while preventing outright binary from being serialized.
>
>
>
> > Further, am I right in that a zero character must not appear in a
> > surrogate?
>
> What do you mean by a "surrogate"?  If you mean a Unicode surrogate  
> pair, then a
> #x0 character cannot be part of such a pair.  Zero *bytes* can of  
> course appear
> as part of the representation of many Unicode characters in UTF-16  
> or UTF-32.

Yes, (Java) Unicode surrogate is what I mean.