XOM API for Processing XML with Java

[Elliotte Harold <elharo AT metalab.unc.edu>]

I had planned to add support for several East Asian character sets to 
XOM, including Big5, SJIS, and a few others. In fact, I've actually 
added EUC-JP and Big5 to CVS. However in doing so I've discovered that 
this is quite a bit trickier than I thought. In particular, there is no 
1-1 mapping between these character sets and Unicode. It is not always 
obvious (or even decidable) which Unicode code point matches which SJIS 
code point. The problems are elaborated in detail in the W3C Note "XML 
Japanese Profile" <http://www.w3.org/TR/2000/NOTE-japanese-xml-20000414> 
by Murata Makoto et al. The bottom line is that "This technical report 
recommends the use of UTF-8 (or UTF-16)."; and since the authors 
certainly know a lot more about Japanese than I ever will, I'm inclined 
to follow their lead.

I haven't found anything that lays out the issues quite so clearly for 
Chinese character sets such as Big5 but reading between the lines of 
various specs, I suspect the same issues apply. Most of the specs of 
interest (except GB18030)  predate Unicode and do not have clear and 
unambiguous mappings to Unicode characters in all cases.

Therefore, in keeping with XOM's principle of making it easy to do the 
right thing and hard to do the wrong thing, and trusting Muratasan's 
judgement that the right thing for Japanese XML is UTF-8 or UTF-16, my 
current plan is not to do any further work on supporting these legacy 
character sets for East Asian characters. Furthermore I plan to remove 
the explicit support I've recently added for EUC-JP and Big5.

I should add that I have recently hacked together a GenericWriter class 
outside the public API that works for any character set the local VM 
recognizes, irrespective of whether XOM has explicit support for it. 
Thus it will still be possible for the Serializer to output characters 
in SJIS, Big5, EUC-JP and other non-supported encodings such as MacRoman 
and Cp1252; and such output shouldn't produce any unnecessary numeric 
character references at least to the limit of what the VM understands. 
(One thing I've noticed in all this work is that Java's built in mapping 
tables are often quite buggy.) However, using an unsupported character 
set carries a lot of extra overhead and is likely to be quite a bit 
slower than using a supported encoding such as UTF-8 or ISO-8859-1. (I 
haven't yet measured exactly how much slower it actually is. I could be 
pleasantly surprised. Eyeballing it's probably not more than one order 
of magnitude and maybe less.) But it will still be possible to use 
Big5/SJIS/EUC-JP etc. if someone really wants to do that.

Comments? Thoughts? Suggestions? I'd be particularly interested in 
hearing from anyone who knows if these issues really do apply to Chinese 
and/or Korean character sets. I'm guessing the same issues arise, but I 
could be wrong about that, especially for Korean which I haven't looked 
into yet.

--
Elliotte Rusty Harold