Fast Serialization

Ok - here's the deal.

Im currently passing objects between two instances of an application. Im currently using standard Serialization to get an "on the wire" format. The reason for going the Serialization route is that new classes are added and changed frequently by a large team of varied experience levels - so an intrusive "each class has to know how to write itself out" mechanism just isn't flexible enough.

Anyway, it turns out that Serialization isn't cutting it performance wise. My objects are about 7k on the wire, Im not serializing objects I shouldn't be, and Im only getting a few hundred serializations per second (note Im talking about the serialization itself here - not sends across the network).

So I got to thinking about how I could speed it up and I'd like to run an idea past you...

What about a custom "fast" serialization mechanism? As classes are loaded, they are instrumented with the code which lets them write themselves out through the stream.

As an example, take the following class:

publicclass SomeDomainType{

privatelong someLong;

privateint someInt;

private String someString;

private SomeOtherType other;

// ...

}

At class load time, this would be added:

publicclass SomeDomainTypeimplements FastSerializable{

// .. members as before

publicvoid serialize(FastSerializeStream out){

super.serialize(out);

out.write(someLong);

out.write(someInt);

out.write(someString);

out.writeObject(other);

}

}

Obviously a "de-serialize" method would be added as-well.

So, each type gets instrumented with a non-reflection based way of (de)serializing.

We'd need to have a way of creating objects without invoking constructors for the deserialization part - but that is possible with a bit of magic.

The other thing to bear in mind is that we do not have the complex class version / class loading issues that normal java (de)serialization has to deal with (we're sending objects between two instances of the same app using the same classes).

So why am I posting this anyway?

Well, I know that serialization has been improved quite a bit in the last few java versions - and I think that some code generation for reflection access already goes on behind the scenes. So it may be that what Im proposing is already some-how catered for in a different way.

This method would certainly get rid of run-time reflective access for serialization though and probably wouldn't be too hard to implement.

Any comments would be appreciated.

[3310 byte] By [] at [2007-10-2 0:17:45]
# 1
Oh b0llocks..... Didn't see ObjectStreamClass.FieldReflector....Looks like it already sets field values directly without reflection after its built up its class descriptor.So I suppose all I'd really save is the writing out of each class descriptor.......
at 2007-7-15 16:19:13 > top of Java-index,Other Topics,Patterns & OO Design...
# 2

Just in case anyone stumbles on this thread when searcing for serialization performance:

I found a nice little trick to speed up performance by about 10% when using serialization in the context I described above:

Dont bother writing out flippin class descriptors:

private static class NoDescObjectOutputStream extends ObjectOutputStream {

NoDescObjectOutputStream(OutputStream out) throws IOException {

super(out);

}

protected void writeClassDescriptor(ObjectStreamClass desc) throws IOException {

writeUTF(desc.getName());

}

}

private static class NoDescObjectInputStream extends ObjectInputStream {

NoDescObjectInputStream(InputStream in) throws IOException {

super (in);

}

protected ObjectStreamClass readClassDescriptor() throws IOException, ClassNotFoundException {

String name = readUTF();

return ObjectStreamClass.lookup(Class.forName(name));

}

}

at 2007-7-15 16:19:13 > top of Java-index,Other Topics,Patterns & OO Design...
# 3

Look at jbossSerialization

LGPL and it is fast:

http://labs.jboss.com/portal/index.html?ctrl:id=page.default.info&project=serialization

It requires JVM 5.

And if you want to clone values only, SmartCloning is cool and much better than using ByteArrayOutputStream underneath ObjectOutputStream.

Clebert Suconic

JbossSerialization lead

JbossProfiler lead

4894114a at 2007-7-15 16:19:13 > top of Java-index,Other Topics,Patterns & OO Design...
# 4
So - there are people interested in serialization performance in the java world then :o)Thank you Clebert!You know, you could have just made my year.~D
aconst_nulla at 2007-7-15 16:19:13 > top of Java-index,Other Topics,Patterns & OO Design...
# 5

Hi,

In order to performance, I implement the externalizable interface, by example I make this with Calendar.

Original Calendar, have 900 bytes of information, I serialize only the Long value using getTime. This produce only 4 bytes, is quicker than the original.

you could implements externalizable interface when you want improve the serlization process.

Thanks

barbywarea at 2007-7-15 16:19:13 > top of Java-index,Other Topics,Patterns & OO Design...
# 6

> you could implements externalizable interface when you want improve the serlization process.

Thanks for the tip - but Im aware of Externalizable.

What Im looking for is a transparent way to improve serialization performance.

Externalizable is too intrusive to employ in my context.

Cheers,

~D

aconst_nulla at 2007-7-15 16:19:13 > top of Java-index,Other Topics,Patterns & OO Design...
# 7

> Look at jbossSerialization

> LGPL and it is fast:

Uh oh....

Gonna have to take a look at the benchmarking tests....

The initial ones I've just carried out (with the objects I'll be sending over the wire) shows standard java 1.5 serialization out-performing JBoss serialization by about 35%.

aconst_nulla at 2007-7-15 16:19:13 > top of Java-index,Other Topics,Patterns & OO Design...
# 8
P.s:Note that the objects Im testing with are fairly large in size and have a fairly deep structure.Standard Java serialization certainly seems to be consistently winning at the moment! (Which isn't a great thing for me - as its slow!).~D
aconst_nulla at 2007-7-15 16:19:13 > top of Java-index,Other Topics,Patterns & OO Design...
# 9
I need access to your testcases, or to a similar testcase showing performance problem.I'm pretty sure I can fix the problem if you provide me a testcase
4894114a at 2007-7-15 16:19:13 > top of Java-index,Other Topics,Patterns & OO Design...
# 10

Probably you are the same guy who also sent a question on jboss forums.

I couldn't duplicate the behavior you told me. JBossSerialization was always faster.

I'm trying to find a scenario where JavaSerialization is faster, so I can fix any problems.

Please look at this thread:

http://www.jboss.com/index.html?module=bb&op=viewtopic&t=71200

I have placed there some information about how to replicate my tests.

4894114a at 2007-7-15 16:19:13 > top of Java-index,Other Topics,Patterns & OO Design...
# 11
> Please look at this thread:> http://www.jboss.com/index.html?module=bb&op=viewtopic&t=71200Many thanks,~D
aconst_nulla at 2007-7-15 16:19:13 > top of Java-index,Other Topics,Patterns & OO Design...
# 12
Hi,You could change the Class that you need serialize with the externalizable class.You could make this changing the boot classloader.Thanks
barbywarea at 2007-7-15 16:19:13 > top of Java-index,Other Topics,Patterns & OO Design...