Skip to Content.
Sympa Menu

xom-interest - [XOM-interest] Unicode forms for internal storage

xom-interest AT lists.ibiblio.org

Subject: XOM API for Processing XML with Java

List archive

Chronological Thread  
  • From: Elliotte Rusty Harold <elharo AT metalab.unc.edu>
  • To: unicode AT unicode.org
  • Cc: xom-interest AT lists.ibiblio.org
  • Subject: [XOM-interest] Unicode forms for internal storage
  • Date: Tue, 20 Jan 2004 11:59:28 -0500

I'm currently working on a project (XOM, <http://www.cafeconleche.org/XOM/>) in which the Unicode text data is a significant portion of memory usage in many important use cases. Currently, for the major class where this is an issue in practice (as proved by profiling), I store the data as UTF-8. This means ASCII data takes half the space it would in UTF-16, and many other characters take only the same amount as they would in UTF-16. However CJK characters tend to take up 50% more space than they woudl in UTF-16.

Last night it occurred to me it might be possible to design an internal storage format for this class which had better memory usage characteristics. In particular I'd like ASCII data to occupy only a single byte, and all other BMP characters from 128 to 65535 to occupy only two bytes. Non-BMP characters could be stored in surrogate pairs.

In developing such a format I have a couple of advantages:

1. Most C0 controls are forbidden, and will not appear in the data. That's already verified. If someone tries to pass in a C0 control other than tab, linefeed, or carriage return to setValue, an exception is thrown and the data is not stored. Potentially one or more of these characters could be used as markers in the stream.

2. I do not need random access to parts of the data, only to whole strings. Unlike with UTF-8, it is not important to be able to look at a single byte in isolation and tell immediately which part of what kind of character it is.

3. This is all completely private to one class. No data in this form will be passed on the wire. None will be exposed via the public API which is completely based on Java strings (that is, UTF-16).

However, I would like the translation into and out of this format to be at least as fast as the translation between UTF-8 and UTF-16 the class is currently performing on every call to setValue and getValue, ideally faster.

Has anyone done any work on Unicode formats for this use-case? Does anyone have any references or ideas to share?
--

Elliotte Rusty Harold
elharo AT metalab.unc.edu
Effective XML (Addison-Wesley, 2003)
http://www.cafeconleche.org/books/effectivexml
http://www.amazon.com/exec/obidos/ISBN%3D0321150406/ref%3Dnosim/cafeaulaitA




Archive powered by MHonArc 2.6.24.

Top of Page