ois.readObject() is taking long
Hi,
Is it normal for ois.readObject() to take long? I have searched on Google and have found that many people have had a problem with readObject() hanging, but have not found any answer as to whether this sort of behaviour is normal.
I am using ois.readObject() to read a TreeModel from a flat file. The flat file is 1MB (and the TreeModel is the only object on it). In this case it takes 3 minutes.
When I use ois.readObject() to read a HashMap from an 8MB flat file, it times out. When it is to read a Hashtable from a 15MB flat file, it takes at least 40 minutes.
Is this normal? Is there a way to speed it up? If not, is there anything else I can use that may read these objects faster (another protocol, maybe)?
Thanks.
[762 byte] By [
RoxyLJa] at [2007-10-3 10:35:32]
A 15 MB object file? What the c r a p do you have stored in it?
Anyway, when you read an Object such as a Hashtable or TreeModel, Java has to locate the class file for every Object listed inside.
So, every object contained in your Hashtable must be fetched, read, and and so on within that class, etc.
This takes a few nanoseconds to a few miliseconds when things contained are only simple Java API classes - as they are already loaded, but when you start storing massive amounts of MANY different object types (classes), each with its own references to other classes, this starts to take a very long time.
Now, you're likely using a standard desktop computer. A single hard drive that holds the OS, Application, Data, and class definitions, in addition to disk-based memory page file. All components are trying to read their data as fast as possible. This totals up to a ton of read requests, in addition to the various write requests.
> A 15 MB object file? What the c r a p do you have
> stored in it?
A very big Hashtable.
> This takes a few nanoseconds to a few miliseconds
> when things contained are only simple Java API
> classes - as they are already loaded, but when you
> start storing massive amounts of MANY different
> object types (classes), each with its own references
> to other classes, this starts to take a very long
> time.
Ok, any suggestions on anything else I can use to read in objects faster? I have had this issue for a while now... querying the database to retrieve information every time simply takes too long, which is why I am trying to cache a lot of frequently used objects. Loading the cache on startup of the application is what I am trying to optimize now.
I need S P E E D.
Maybe your cure is worse than the disease? Get rid of the cache!
ejpa at 2007-7-15 5:59:01 >
What is stored in the hashtable?
If possible, it would be best to take any defining data or values and simply store the type and values in a file, and load them in a seperate thread.
Load the UI components first, as that's the major component of loading that a user sees, which is what they based the startup process on.
So, load the UI in one thread, and read one set of values of objects in another, or simply load the entire UI and then load the entire table.
> Get rid of the cache!Agreed.
No comment on the sensibility of the caching approach, but here is a quickie test on reading a largish file with readObject(). 12 MB = 2.5 seconds. Are you using Buffered{In,Out}putStream?
import java.util.*;
import java.io.*;
public class ObjectFile
{
static final int HASH_SIZE = 500000;
public static void main(String args[])
throws Exception
{
File file = new File("junk-objects");
System.out.println("==== writeObject() / readObject()");
writeFile(file, 0);
readFile(file, 0);
System.out.println("==== writeUnshared() / readUnshared()");
writeFile(file, 1);
readFile(file, 1);
System.out.println("==== write separately / read separately");
writeFile(file, 2);
readFile(file, 2);
}
static void writeFile(File file, int how)
throws Exception
{
HashMap hash = new HashMap();
for (int n = 0; n < HASH_SIZE; n++)
hash.put(new Integer(n), "hello " + n);
ObjectOutputStream out = new ObjectOutputStream(new BufferedOutputStream(
new FileOutputStream(file)));
long start = System.currentTimeMillis();
switch (how) {
case 0: out.writeObject(hash); break;
case 1: out.writeUnshared(hash); break;
case 2:
for (Iterator it = hash.keySet().iterator(); it.hasNext(); ) {
Object key = it.next();
out.writeUnshared(key);
out.writeUnshared(hash.get(key));
}
break;
}
out.close();
long end = System.currentTimeMillis();
System.out.println("writing took " + (end - start) + " ms");
System.out.println("file size " + file.length());
}
static void readFile(File file, int how)
throws Exception
{
ObjectInputStream in = new ObjectInputStream(new BufferedInputStream(
new FileInputStream(file)));
long start = System.currentTimeMillis();
HashMap hash = null;
switch (how) {
case 0: hash = (HashMap) in.readObject(); break;
case 1: hash = (HashMap) in.readUnshared(); break;
case 2:
hash = new HashMap();
try {
while (true) {
Object key = in.readUnshared();
Object value = in.readUnshared();
hash.put(key, value);
}
} catch (EOFException e) {
// ugly
}
break;
}
in.close();
long end = System.currentTimeMillis();
System.out.println("reading took " + (end - start) + " ms");
System.out.println("hash size " + hash.size());
}
}
Apologies for the delay in replying. I have been away.> Maybe your cure is worse than the disease? Get rid of> the cache!Again, I need speed. Querying the database is not fast enough.
> No comment on the sensibility of the caching
> approach, but here is a quickie test on reading a
> largish file with readObject(). 12 MB = 2.5 seconds.
> Are you using Buffered{In,Out}putStream?
Actually, no, I wasn't. I have stuck it in and made one or two other little adjustments, and that seemed to do the trick. Thanks!
Getting back to the sensibility of the caching approach... Yes, I am well aware of its disadvantages, but for the moment, at least, the pros outweigh the cons.
except that 40 minutes to read the hash table is even slower!
Questions:
(a) What version of the JDK are you using?
(b) How many objects are in the HashMap/Hashtable?
(c) You don't get read timeouts reading from a file. Are you really reading from a socket supplied from a file at the other end? and if so what does the writing code look like?
ejpa at 2007-7-15 5:59:01 >
> except that 40 minutes to read the hash table is even
> slower!
>
> Questions:
>
> (a) What version of the JDK are you using?
>
> (b) How many objects are in the HashMap/Hashtable?
>
> (c) You don't get read timeouts reading from a file.
> Are you really reading from a socket supplied from a
> file at the other end? and if so what does the
> writing code look like?
I think that problem has been solved at least for now. Thanks.
Thanks. Would you like to share the solution with us, or the reason for the amazingly poor performance?
ejpa at 2007-7-15 5:59:01 >
> Thanks. Would you like to share the solution with us,
> or the reason for the amazingly poor performance?
Yes. As sjasja pointed out, I needed to use the BufferedInput and OutputStreams. Once I put that in, there was no more problem. After that, I just played around with my code - I can't really think of anything specific that I did that would have made much more of an improvement. The main thing was just that I had neglected to use BufferedIO. It's amazing to see what a significant difference it made!
My next challenge is keeping the memory usage constant. The application starts to throw OutOfMemoryErrors after a couple of minutes of using it. I am still investigating why this is happening. As far as I can tell, after the initial startup of the application, nothing more should be loaded into memory. I might post another message in this regard once I have more information.
Thanks again to all of you for your help.
Can you explain about the timeout reading from a file?
ejpa at 2007-7-15 5:59:01 >
ejp,
(a) I'm running JDK1.5.0_06 on XP.
(b) There are 20,000 <Key, Value> pairs in the HashMap that times out, where each of the Value objects encapsulate between 0 and 15 objects, either as object references or in LinkedLists. The Key is a String.
(c)
> You don't get read timeouts reading from a file.
> Are you really reading from a socket supplied from a
> file at the other end? and if so what does the
> writing code look like?
I understand that timeouts can't theoretically occur when reading from a file, but this is what appears to happen when reading the HashMap. The timeout occurs when the application reads from an object file containing an 8MB HashMap without using BufferedIO. It does not occur when the equivalent Hashtable object file is read without using BufferedIO.
This is what happens on application startup:
1. The client logs in to the server.
2. The client requests the HashMap from the server.
3. The server receives the request and executes ois.readObject() to read the HashMap from file. It is inside this ois.readObject() that the timeout would appear to be occurring.
I need to do a few more tests before I am completely satisfied that I have, indeed, diagnosed the problem correctly. However, my manager wants this application finished yesterday, so those tests will have to wait for now.
What are your thoughts on all this?
> My next challenge is keeping the memory usage
> constant. The application starts to throw
> OutOfMemoryErrors after a couple of minutes of using
> it. I am still investigating why this is happening.
> As far as I can tell, after the initial startup of
> the application, nothing more should be loaded into
> memory. I might post another message in this regard
> once I have more information.
This is still happening. I've been profiling the application at runtime to identify possible memory leaks, but have not yet been able to find any.
As discussed before, the application loads all of the objects into memory on startup. It seems logical to me that this is the only place where usage of the Java heap space should increase dramatically. Once the nodes are displayed in the JTree, usage of the Java heap space climbs for every time that a node is clicked in order to display its' details. I can understand that it would increase due to the Hashtable element retrieval and GUI update, but shouldn't it come back down again after the node has been displayed? In other words, shouldn't usage of the Java heap space plateau after startup?
A quick way of finding out where your memory is: add a menu option that does this:
System.gc();
Thread.sleep(5000); // in case gc() is asynchronous
System.exit(0);
Run the program with "java -Xrunhprof:heap=sites TheProgram", use it for a while, then do the gc()/exit() thing. That produces the file java.hprof.txt which has this in there somewhere:
percent live alloc'ed stack class
rankself accumbytes objsbytes objs trace name
1 54.78% 54.78%639200 10000639200 10000 300231 char[]
2 20.57% 75.35%240000 10000240000 10000 300232 java.util.LinkedList$Entry
3 13.71% 89.06%160000 10000160000 10000 300230 java.lang.StringBuffer
4 0.73% 89.79%8520185201 300087 byte[]
5 0.05% 89.84%61646164 300038 char[]
...etc...
"Live bytes" is where your memory is. "Stack trace" numbers are earlier in the file if you need to see where the objects were allocated.
You can do this without the gc() stuff, just run with -Xrunhprof:heap=sites and exit the program. But then "live objects" will contain some gc'able objects too.
Hi sjasja,
I've taken a while to reply to your last post as I wanted to take some time to investigate the problem using your suggestion of running with Xrunhprof:heap=sites. I've done this and have also profiled the application. This helped me to identify a couple of areas in which objects were being retained unnecessarily.
I redesigned some objects, which reduced usage of the memory considerably. However, I am still having the same problem of a steady increase of use of the Java heap space over time. The application can even be open in the background without being used at all, and the amount of heap space being used still increases.
Is there any generic reason that usage of the Java heap space would increase over time simply by having an application open in the background?
Please note, I will be away again until next week, so I may not be able to reply to posts until then.
Do you perhaps mean that the client times out waiting for a response from the server while the server is still reading this large object?
> 2. The client requests the HashMap from the server.
> 3. The server receives the request and executes ois.readObject() to read the HashMap from file. It is inside this ois.readObject() that the timeout would appear to be occurring.
So why does the server have to de-serialize the HashMap at all? Why not just send the contents of the file to the client as binary, and let the client do the deserialization?
ejpa at 2007-7-21 13:21:30 >
No, I have solved the timeout that was occurring in the ois.readObject(). The problem I am having now is completely on the client side. Perhaps this post belongs in a new thread...
Basically, the client reads the object file from the server, stores the file in its own memory (i.e. a copy is kept on the client side), and then opens and displays the contents of this file in the application. Up until this point, everything is fine.
Now the application is just sitting open, waiting for the user to click something. However, while it appears just to be sitting (i.e. passively listening, not actively doing anything), waiting, the heap space usage increases gradually. It is a very gradual increase that is barely noticeable, but it is there.
I just don't see why the Java heap space usage should be increasing when the application is not actually doing anything. Surely there must be a way to prevent this from happening...
Does anyone have any ideas as to why this might be happening?
hey... sorry to interrupt your thread, however, been waiting for your reply in the following thread that you previously post http://forum.java.sun.com/thread.jspa?threadID=736662&tstart=120i have similar problem and hope u could help mi.
> I just don't see why the Java heap space usage should
> be increasing when the application is not actually
> doing anything. Surely there must be a way to
> prevent this from happening...
Well obviously something is happening. Your first task is to discover what that is. Are there timed events or other background activity in your application? What does a thread dump actually show? Have you tried using a memory profiler to see where the objects are going and who is allocating them?
ejpa at 2007-7-21 13:21:30 >
> Well obviously something is happening. Your
> first task is to discover what that is. Are there
> timed events or other background activity in your
> application? What does a thread dump actually show?
> Have you tried using a memory profiler to see where
> the objects are going and who is allocating them?
There are no timed events or background activity. Due to the fact that the increase is so slight, comparing thread dumps from different runs has not proven to be very helpful thus far. I will probably have to let the application run over quite a number of days so that the thread dump will be more useful. I will post another message if I still need help after a few more weeks of profiling. Thanks.
