Large object serialization
This entry was posted in the Serialization forum but since no one has answered it there I will give it another shot here.
Dear All,
I am trying to serialize a HashMap containing Strings as keys and and ArrayLists as values. This works fine when the number of entries is small, but does not work when the number of entries are really large, in which case ajava.lang.OutOfMemoryError: Java heap space is thrown. This happens when the number of entries is about 50,000 and the total number of objects stored in the lists exceed 500,000.
Using the ObjectOutputStream's reset() method has been suggested elsewhere, but that would mean I would have to manually write the serialization method, and to modify that everytime my class changes (both for serialization and deserialization).
Some posts suggest that this should have been fixed in Java 1.6 but it still doesn't seem to work. Do I need to provide the "-mx" flag to get this to work, or is there another way to serialize large objects? Any thoughts welcome.
Cheers,
Fred
[1082 byte] By [
fw9189a] at [2007-11-27 4:54:08]
Fred,
Yeah, try bumping up the JVM memory for starters... failing that, rolling your own serialiser & deserialiser for specific objects isn't that hard (in fact I think you'll be pleasantly surprised)... and surely maintenance won't be that big an issue, because you'll get it right the first time.
Keith.
> Fred,
>
> Yeah, try bumping up the JVM memory for starters...
> failing that, rolling your own serialiser &
> deserialiser for specific objects isn't that hard (in
> fact I think you'll be pleasantly surprised)... and
> surely maintenance won't be that big an issue,
> because you'll get it right the first time.
>
> Keith.
Hi Keith,
Thank you for a quick reply. I'd like to clarify something: Using the reset() method simply forces the serializer to send the entire object, rather than a reference to the object? Given the structure and usage of my HashMap I am sure there are no duplicates in the entire Map, so there shouldn't be a huge overhead using reset() between the iterations. Is this a valid assumption?
Thanks!
> Using the reset() method simply forces the
> serializer to send the entire object, rather than a
> reference to the object?
The reset() method has no effect unless you've already serialized the same object. If you have, and you re-serialize it without an intervening reset(), a reference to the same object will be serialized and any updates at the sending end won't be visible at the receiving end. If you do the reset(), the entire object is re-serialized. If you aren't re-serializing, reset() is of no benefit.
> Given the structure and
> usage of my HashMap I am sure there are no duplicates
> in the entire Map, so there shouldn't be a huge
> overhead using reset() between the iterations. Is
> this a valid assumption?
reset() has minimal overhead but unless you are reserializing (e.g. your data contains loops) it also has no purpose.
ejpa at 2007-7-12 10:08:43 >
