Weired behavior of Threadpool under Linux
I am writing a multithreading web page fetcher under windows with jdk1.5.0-09. I have a task dispatcher like this:
publicvoid run(){
while (true){
try{
if(_fetcherPool.getActiveCount()<_config.numThreads){
_fetcherPool.execute(new RobotstxtFetcher(_crawler, getURLQueue().take()));
}else{
Thread.sleep(someDelay);
}
}catch (InterruptedException e){
log.error("Dispatcher.run", e);
}
}
}
Everything works fine as expected, 500 threads are generated, and new tasks will be submitted to the pool. I got performance of finishing 20 tasks per second.
But when I moved this program to Redhat Enterprise Linux with jdk1.5.0_09, the behavior changed completely. 500 threads are still generated. But they are not processing anything for about 60 seconds, but then suddenly finished about 300 tasks in 1-2 seconds. Then the program will wait for another 60 seconds and then finish 300 tasks in 1-2 seconds again.
So I tried to test it with multithread without using threadpoolexecutor. I generated 500 threads, and each thread get tasks from the queue directly without thread pooling. It works fine just as under windows.
Can anyone tell me why?
BTW, I suspect the creation of the thread pool may be an issue. Here's my code:
ThreadPoolExecutor _fetcherPool =new ThreadPoolExecutor(
500, 500+20, 1, TimeUnit.SECONDS,new LinkedBlockingQueue<Runnable>() );
Please help. Thanks.
