XOM API for Processing XML with Java

[Elliotte Rusty Harold <elharo AT metalab.unc.edu>]

At 9:00 PM -0400 10/20/03, John Cowan wrote:


> Neither.  The righteous thing is to normalize first and then use your
> smart transcoding algorithm.  So if an attribute value or character
> content contains U+0041 U+0301 U+0042 U+0301, normalization will transform
> it to U+00C1 U+0042 U+0301, and transcoding to ISO-8859-1 will produce
> "áb́", which is normalized in both the Unicode and the extended
> XML sense, and will read back correctly.

Oh, I see. You're thinking of cases where the 
normalized character doesn't exist in the target 
character set, and therefore is output as a 
character reference and this character reference 
can be in two forms: normalized or unnormalized; 
and of course we want the normalized one. That 
makes sense.

Bonus: it's the easiest solution to implement 
since that's what the code already does. Cool.

Hmm, I do wonder if there are any legacy 
character sets where the set has the unnormalized 
form and not the normalized form. In this case, 
it might not do quite the right thing (if such a 
case exists). On the other hand, when the parser 
reads the legacy character set back in it should 
convert the legacy set to the normalized form. 
Then again, who knows what the parser really 
does. I've found it very hard to dig out any 
information about how either parsers or Java's 
InputStreamReader class actually convert 
characters when there are multiple options.

Possibly with ICU4J I could write some tests to 
figure out what's really going on here. Generate 
all possible characters in various sets, parse 
them in, normalize the strings, and compare the 
prenormalized and post-normalized strings to see 
if there's any difference. I wouldn't be 
surprised to find some flaky results. Earlier 
when I was working in the other direction (output 
in various character sets) I found a number of 
mistakes in Java's encoding classes because 
nobody had ever really exhausted all the 
possibilities before. This is important for 
canonical XML, which requires input from 
non-Unicode encodings to be in NFC. OK. Time to 
write some more tests.

--

  Elliotte Rusty Harold
  elharo AT metalab.unc.edu
  Processing XML with Java (Addison-Wesley, 2002)
  http://www.cafeconleche.org/books/xmljava
  http://www.amazon.com/exec/obidos/ISBN%3D0201771861/cafeaulaitA