Skip to Content.
Sympa Menu

xom-interest - [XOM-interest] East Asian Character Sets

xom-interest AT lists.ibiblio.org

Subject: XOM API for Processing XML with Java

List archive

Chronological Thread  
  • From: Elliotte Harold <elharo AT metalab.unc.edu>
  • To: xom-interest <xom-interest AT lists.ibiblio.org>
  • Subject: [XOM-interest] East Asian Character Sets
  • Date: Wed, 23 Jun 2004 20:39:24 -0400

I had planned to add support for several East Asian character sets to XOM, including Big5, SJIS, and a few others. In fact, I've actually added EUC-JP and Big5 to CVS. However in doing so I've discovered that this is quite a bit trickier than I thought. In particular, there is no 1-1 mapping between these character sets and Unicode. It is not always obvious (or even decidable) which Unicode code point matches which SJIS code point. The problems are elaborated in detail in the W3C Note "XML Japanese Profile" <http://www.w3.org/TR/2000/NOTE-japanese-xml-20000414> by Murata Makoto et al. The bottom line is that "This technical report recommends the use of UTF-8 (or UTF-16)."; and since the authors certainly know a lot more about Japanese than I ever will, I'm inclined to follow their lead.

I haven't found anything that lays out the issues quite so clearly for Chinese character sets such as Big5 but reading between the lines of various specs, I suspect the same issues apply. Most of the specs of interest (except GB18030) predate Unicode and do not have clear and unambiguous mappings to Unicode characters in all cases.

Therefore, in keeping with XOM's principle of making it easy to do the right thing and hard to do the wrong thing, and trusting Muratasan's judgement that the right thing for Japanese XML is UTF-8 or UTF-16, my current plan is not to do any further work on supporting these legacy character sets for East Asian characters. Furthermore I plan to remove the explicit support I've recently added for EUC-JP and Big5.

I should add that I have recently hacked together a GenericWriter class outside the public API that works for any character set the local VM recognizes, irrespective of whether XOM has explicit support for it. Thus it will still be possible for the Serializer to output characters in SJIS, Big5, EUC-JP and other non-supported encodings such as MacRoman and Cp1252; and such output shouldn't produce any unnecessary numeric character references at least to the limit of what the VM understands. (One thing I've noticed in all this work is that Java's built in mapping tables are often quite buggy.) However, using an unsupported character set carries a lot of extra overhead and is likely to be quite a bit slower than using a supported encoding such as UTF-8 or ISO-8859-1. (I haven't yet measured exactly how much slower it actually is. I could be pleasantly surprised. Eyeballing it's probably not more than one order of magnitude and maybe less.) But it will still be possible to use Big5/SJIS/EUC-JP etc. if someone really wants to do that.

Comments? Thoughts? Suggestions? I'd be particularly interested in hearing from anyone who knows if these issues really do apply to Chinese and/or Korean character sets. I'm guessing the same issues arise, but I could be wrong about that, especially for Korean which I haven't looked into yet.

--
Elliotte Rusty Harold





Archive powered by MHonArc 2.6.24.

Top of Page