Skip to Content.
Sympa Menu

xom-interest - Re: [XOM-interest] Very high main memory consumption on compression level higher then 0

xom-interest AT lists.ibiblio.org

Subject: XOM API for Processing XML with Java

List archive

Chronological Thread  
  • From: Wolfgang Hoschek <wolfgang.hoschek AT mac.com>
  • To: Martin Schaaf <ms AT 101tec.com>
  • Cc: xom-interest AT lists.ibiblio.org
  • Subject: Re: [XOM-interest] Very high main memory consumption on compression level higher then 0
  • Date: Wed, 18 Jun 2008 12:49:42 -0700

The native java.util.zip.Deflater/Inflater are used if the compression level is non-zero. Does the high mem usage go away if you reuse the same _binaryXMLCodec instance for all documents? If so, it probably has something to do with the Deflater and/or Inflater leaking memory or their finalizer not being called in time.

Wolfgang.

On Jun 18, 2008, at 12:30 PM, Martin Schaaf wrote:

Hi,

I use the following code

//serialize
private nu.xom.Document _doc
byte[] binaryDoc = _binaryXMLCodec.serialize(_doc, 0);

//deserialize
_doc = _binaryXMLCodec.deserialize(_xmlRecord.get());

to serialize xml. I want to compress it too, but get very high memory
usage. This leads in the worst case to an OOM and in other cases it
brings the system to swap memory out. If I set the compression level
to 0 I do not get the high memory usage effects. I use version 1.1. It
does not make a difference ehether I use 1 or 9 or any othe number
between for compression. All show the same effect. I use it in a map
reduce job to compress 50.000.000 documents. One document is about
10kb in size.

Is this known? Do I smething wrong?

Thanks for all answers
martin
_______________________________________________
XOM-interest mailing list
XOM-interest AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/xom-interest





Archive powered by MHonArc 2.6.24.

Top of Page