Skip to Content.
Sympa Menu

xom-interest - Re: [XOM-interest] Character conversion problem

xom-interest AT lists.ibiblio.org

Subject: XOM API for Processing XML with Java

List archive

Chronological Thread  
  • From: Wolfgang Hoschek <whoschek AT lbl.gov>
  • To: Elliotte Harold <elharo AT metalab.unc.edu>
  • Cc: xom-interest AT lists.ibiblio.org
  • Subject: Re: [XOM-interest] Character conversion problem
  • Date: Thu, 4 Nov 2004 11:01:03 -0800

Thanks for all who responded to the issue.
Turns out the problem was on my side, not in XOM or the parser.

I have test cases for weird corner cases, and was printing debug info with System.err.println(doc.toXML()).
System.err.println(str) is fragile, since the PrintStream used by System.err uses the platform's default encoding by default, which is typically not UTF-8 on MacOSX, but rather "MacRoman".

This ugly work-around retain surrogate chars correctly:

System.setErr(new PrintStream(new FileOutputStream("test.out"), true, "UTF-8"));
System.err.println(doc.toXML());

All is fine now :-)
Wolfgang.

On Nov 4, 2004, at 4:10 AM, Elliotte Harold wrote:

Wolfgang Hoschek wrote:

Hi,
Some characters such as "&#x50000;" and "'[[[&#x50000;]]]'" seem to get turned by XOM to "?" characters.
I've seen similar things before when using byte to char conversions, both with the nio converters and the java.lang.String converters. Any ideas on what could be done about it, or what I'm doing wrong here?

I've walked through this with a debugger and I'm pretty sure that what's going on here is that you're doing System.out.println or some equivalent and that is turning the surrogate pairs into question marks. Neither XOM nor the XML parser is doing this. I've only tested the toXML case but I assume the serializer is the same until proven otherwise.

If you're using an OutputStreamWriter anywhere that might be doing it. According to the JavaDoc for OutputStreamWriter, "If the character represented by a surrogate pair cannot be encoded by a given charset then a charset-dependent substitution sequence is written to the output stream." Normally this substitution char is a question mark.


--
Elliotte Rusty Harold elharo AT metalab.unc.edu
XML in a Nutshell 3rd Edition Just Published!
http://www.cafeconleche.org/books/xian3/
http://www.amazon.com/exec/obidos/ISBN=0596007647/cafeaulaitA/ref=nosim





Archive powered by MHonArc 2.6.24.

Top of Page