[XOM-interest] Very high main memory consumption on compression level higher then 0
Wolfgang Hoschek
wolfgang.hoschek at mac.com
Wed Jun 18 15:49:42 EDT 2008
The native java.util.zip.Deflater/Inflater are used if the
compression level is non-zero. Does the high mem usage go away if you
reuse the same _binaryXMLCodec instance for all documents? If so, it
probably has something to do with the Deflater and/or Inflater
leaking memory or their finalizer not being called in time.
Wolfgang.
On Jun 18, 2008, at 12:30 PM, Martin Schaaf wrote:
> Hi,
>
> I use the following code
>
> //serialize
> private nu.xom.Document _doc
> byte[] binaryDoc = _binaryXMLCodec.serialize(_doc, 0);
>
> //deserialize
> _doc = _binaryXMLCodec.deserialize(_xmlRecord.get());
>
> to serialize xml. I want to compress it too, but get very high memory
> usage. This leads in the worst case to an OOM and in other cases it
> brings the system to swap memory out. If I set the compression level
> to 0 I do not get the high memory usage effects. I use version 1.1. It
> does not make a difference ehether I use 1 or 9 or any othe number
> between for compression. All show the same effect. I use it in a map
> reduce job to compress 50.000.000 documents. One document is about
> 10kb in size.
>
> Is this known? Do I smething wrong?
>
> Thanks for all answers
> martin
> _______________________________________________
> XOM-interest mailing list
> XOM-interest at lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/xom-interest
More information about the XOM-interest
mailing list