SocketChannel.write() blocking application
Greets,
I'm developping a huge application and got a latency/block problem using the write(ByteBuffer) method on a Socket from a socketchannel connection.
Running java 1.5 (diablo) on Freebsd 6 servers, 4Gb ram (2.2 allocated to jvm), with dual xeon dual-core (total 4 cores)
Here is the application schema :
- A thread accepting connexion on the socketchannel
- A thread selecting keys with data to process, enqueuing it after some basic checks on a command FIFO
- A thread getting commands from the FIFO and processing 'em, generating answers on 4 answer FIFOs
- 4 threads (1 per FIFO) to get answers and send 'em back to the socket.
The application usually runs with 4500-5000 simultaneous clients.
My problem is that the only write() method sometimes takes over 20ms to write a message, with a length smaller than 50 bytes.
As I got about 25000 answers to process each second, when some of 'em decide to be slow, the 4 threads runs slowly, and all the connected clients are suffering of that latency, for the few minutes needed to empty the FIFOs.
Every client socket get about 5 answers per second.
On about 1 hour running, there are about 3 'peaks' of slowness, that I cannot explain yet. That's why I'm in need of advices !
I monitored the application when such case happens. TOP indicates 40% cpu idle, JVM memory got >500Mb free, network runs @ about 1.2Mbps, where maximal transfer rate is >20Mbps. netstat -m told me no erros, and a large amount of free buffers available.
As the only slow process is the write() method that usually runs faster than 1ms each call, but in those case I got delays over 20ms.
freebsd tcp default sendbuffer size is 64k, receive buffer is 32k
Commands average received size is below 1k, Answers average sending size below 8k.
This application is running live, and as I cannot emulate 5000+ connections with a similar beahviour to test withour being sure that won't crash all.
What points could be responsible of such slow write() calls ? Seems it's not CPU, not RAM, not network itself...
I suppose it's the network buffers that are causing problems. But I don't really know if I have to fit 'em to a lower size, fitting my requirements, or to a larger size, to be sure there won't be full buffers blocking all ?
I need advices. Thanks for your ideas !
Bill
[2452 byte] By [
Dofusiana] at [2007-11-27 9:06:24]
# 1
Is this 20ms an observed value per write, or is it calculated from grosser data?Can you also show us your writing code.
ejpa at 2007-7-12 21:41:46 >
# 2
The 20ms delay (that sometimes goes over 30) is the only write() delay. I check the sync time too but got pretty no sync delays.
Here is the writing code :
client -> SocketChannel
outputDirectBuffer -> ByteBuffer
data -> String
long synchro = System.currentTimeMillis();
synchronized (client) {
synchro = Math.max(0, System.currentTimeMillis() - synchro);
if (synchro > 5)
logger.warn(new StringBuilder().append(synchro).append("ms sync"));
if (data != null && data.length() > 0) {
outputDirectBuffer.clear();
outputDirectBuffer.put(charset.encode(data));
outputDirectBuffer.flip();
dataToSend = outputDirectBuffer.limit();
dataRemaining = dataToSend;
attempts = 0;
do {
long before = System.currentTimeMillis();
dataSent = client.write(outputDirectBuffer);
timeToSend += Math.max(System.currentTimeMillis() - before, 0);
dataRemaining -= dataSent;
attempts++;
} while(dataSent > 0 && dataRemaining > 0 && attempts < 5);
if (timeToSend > 5)
logger.warn(new StringBuilder().append(timeToSend).append("ms send"));
}
}
# 3
Hmm. So you're happy to lose data?
A few comments:
(a) SocketChannels are thread-safe. I don't think you need the synchronization at all, unless maybe multiple writing threads are possible. I would eliminate that possibility and the sync myself.
(b) If you're getting write delays of 30ms occasionally, the sync must also take 30ms at the same points if it is doing anything at all, i.e. if the possibility of multiple writing threads does exist. So maybe it doesn't?
(c) I would have a good look at this:
http://forum.java.sun.com/thread.jspa?threadID=459338
and specifically the part on how to manage a channel that presents write blocks, using OP_WRITE when it happens and turning it off when it doesn't.
(d) You seem to be using one output buffer for all channels. You might be better off using a small one per channel. Then that way you don't clear, you just do put/flip/write/compact, and if the write returned 0 just post OP_WRITE for next time around the select loop. Then you won't lose any data at all, except to a client who really isn't reading: you can detect that situation by keeping track of the last successful write time to a channel, and when there is pending data and the last write is too long ago have a think about what the block means in terms of the application. Maybe you should just disconnect the client?
(e) It would be interesting to know how many times the write loop looped when you get these large delays, and also what the data size was, and also to know that for the other cases to see if there is a difference.
(f) Generally from a fairness point of view I prefer not to have write loops, just one attempt and if it returns even a short read I post OP_WRITE as above. Otherwise you're spending too long servicing one channel.
You can contact me offline via http://www.telekinesis.com.au if you like.
ejpa at 2007-7-12 21:41:46 >
# 4
Thanks for your reply.
Let's give you some answers.
(a) There are 4 threads writing data, that's why I synchronized the SocketChannel. One month ago there was only one thread, but I've seen a large improvement of response delay using multiple threads to write answers.
(b) As told on my first post, the logger gives me less than 0.1% or sync delays noticeable. so this happens, but not frequently.
(c) I'm reading this, thanks.
(d) The thing is I'm having 5000 channels opened. As some messages need up to 250kb data to send, I can't imagine having 5000 * 256kb buffers allocated. Anyway I'll try to go this way.
About disconnecting on send failure, that's what I used to to some weeks ago, but seems on such large data transfers sometimes there are buffers problems that avoids the first write, but the next ones run correctly.
(e) I'll add a loop count feedback to get those stats. But logs shows me small packet problems, sometimes less than 10 bytes caused 30ms delay...Length of packet don't seem to be the write error.
(f) I'll try to think about it and repost the unwritten data to process it on another execution.
Thank you for answering. I'll try some improvements and feedback here. Anyway I'm still in need of advices, so if other people or other ideas come, feel free to reply !
# 5
OK. If you assign socket channels uniquely to the writing threads, such that each channel is always handled by the same thread, you can do away with the synchronization. You are likely to need this constraint anyway if the threads have selectors.
ejpa at 2007-7-12 21:41:46 >
# 6
Well, sockets are already linked to only one of these threads, the link is made on connect, socket is assigned to the thread having less sockets in use.
But another thread not shown here also sends data to the socketchannel sometimes, but with low priority, and blocking on this specific thread is not a problem since data are not mandatory (on this specific thread).
I use selectors for connect and for reading, I was not using 'em for writing. Do you think this would help reducing the delays I'm having on write() calls ?
Each Thread gets its FIFO to get aswers to process, perhaps using only one shared FIFO, with the 4 thread will pick answers into, would be more reliable ?
I'm also considering about changing the ServerSocketChannel ReceiveBufferSize (currently 256kb), and socketchannels receivebuffer/sendbuffer sizes (unmanaged values gives me a sendbuff. at 43kb and recvbuff at 71kb on connect), but I don't know exactly if the better way is to fit the size to my average minimal needs (8 or 16kb) or to increase 'em to 128kb or more to have larger buffers ?
