Skip to Content.
Sympa Menu

xom-interest - Re: [XOM-interest] NFC serialization

xom-interest AT lists.ibiblio.org

Subject: XOM API for Processing XML with Java

List archive

Chronological Thread  
  • From: Elliotte Rusty Harold <elharo AT metalab.unc.edu>
  • To: John Cowan <cowan AT mercury.ccil.org>
  • Cc: xom-interest AT lists.ibiblio.org
  • Subject: Re: [XOM-interest] NFC serialization
  • Date: Mon, 20 Oct 2003 23:17:16 -0400

At 9:00 PM -0400 10/20/03, John Cowan wrote:


Neither. The righteous thing is to normalize first and then use your
smart transcoding algorithm. So if an attribute value or character
content contains U+0041 U+0301 U+0042 U+0301, normalization will transform
it to U+00C1 U+0042 U+0301, and transcoding to ISO-8859-1 will produce
"áb&#x0301;", which is normalized in both the Unicode and the extended
XML sense, and will read back correctly.

Oh, I see. You're thinking of cases where the normalized character doesn't exist in the target character set, and therefore is output as a character reference and this character reference can be in two forms: normalized or unnormalized; and of course we want the normalized one. That makes sense.

Bonus: it's the easiest solution to implement since that's what the code already does. Cool.

Hmm, I do wonder if there are any legacy character sets where the set has the unnormalized form and not the normalized form. In this case, it might not do quite the right thing (if such a case exists). On the other hand, when the parser reads the legacy character set back in it should convert the legacy set to the normalized form. Then again, who knows what the parser really does. I've found it very hard to dig out any information about how either parsers or Java's InputStreamReader class actually convert characters when there are multiple options.

Possibly with ICU4J I could write some tests to figure out what's really going on here. Generate all possible characters in various sets, parse them in, normalize the strings, and compare the prenormalized and post-normalized strings to see if there's any difference. I wouldn't be surprised to find some flaky results. Earlier when I was working in the other direction (output in various character sets) I found a number of mistakes in Java's encoding classes because nobody had ever really exhausted all the possibilities before. This is important for canonical XML, which requires input from non-Unicode encodings to be in NFC. OK. Time to write some more tests.

--

Elliotte Rusty Harold
elharo AT metalab.unc.edu
Processing XML with Java (Addison-Wesley, 2002)
http://www.cafeconleche.org/books/xmljava
http://www.amazon.com/exec/obidos/ISBN%3D0201771861/cafeaulaitA




Archive powered by MHonArc 2.6.24.

Top of Page