Performance of UDP Receiver during high load
Hello All,
I have the following problem and hopefully somebody can help me with this.
Our Application, SIP Proxy, is multi-threaded and this is the way we run it.
We have a UDPReceiver Thread that listens on a particular port. We have a ThreadPoolManager that has 50 Worker threads that are started when the SIP Proxy is brought up. These threads follow the wait-notify mechanism. When we get a packet ( in UDPReceiver),we add the packet to a Vector and notify the threads ( not notifyAll). The thread that is notified picks up the first job in the vector. If there are no jobs in the vectors, the workers all wait on the synchronized object.
So far no issues. Everything works fine when there is reasonable load. But when I reach a certain load I see UDP packets getting dropped (netstat -s -v 1 | grep udpInOverflows in T2000). The CPU utilization at this point is only 40%.)
I face the same packet drop issue in Linux as well. I analyzed the problem and this is what I found out.
Sometimes all the Workers are busy and UDP Receiver adds the jobs to the Vectors and there atleast 300 of these jobs in the queue ( Normally this would be 1 or 2). So when the Workers finish their existing job, they take the next one if there is any. Since there are 300+ jobs, the Workers (50 of them) pick their job. This is continuous till all the jobs in the queue are complete. Now I believe the UDP Receiver thread is not getting a chance to execute during this time
and hence the UDP buffer overflows and we get packet drops ( I have done all the OS tuning to increase the receive buffer etc).
I set a higher priority to the UDP Receiver Thread but still the same thing happened).. What is really puzzling me, how come this
thread is starved for CPU if it shows as only 40% in prstat?
Have any of you faced similar problems and if so how did you fix it?
I can paste code snippets if need be.
Thanks in advance
Gokul
