[W2003]: Broker not responding [OBJECT_MESSAGE] for 120 seconds. Still tryi

I've been running MQ 3.5 SP1 for days. But yesterday I got the following message from my client:

--

[W2003]: Broker not responding [OBJECT_MESSAGE] for 120 seconds. Still trying...

[W2003]: Broker not responding [OBJECT_MESSAGE] for 360 seconds. Still trying...

[W2003]: Broker not responding [OBJECT_MESSAGE] for 840 seconds. Still trying...

[W2003]: Broker not responding [OBJECT_MESSAGE] for 1800 seconds. Still trying...

[W2003]: Broker not responding [OBJECT_MESSAGE] for 3720 seconds. Still trying...

--

My client could no longer get message from MQ, and even the command "imqcmd query dst -t q -n xx" was not responsed. The MQ process was stilling running, from it's log I got these lines:

--

[27/Jun/2004:17:05:13 CST] [B1065]: Accepting: guest@10.1.0.195:36432->jms:46522. Count=22

[27/Jun/2004:17:05:18 CST] [B1066]:Closing: guest@10.1.0.195:36432->jms:46522. Count=21

[27/Jun/2004:19:30:38 CST]

--

note: on the last line , there was no message

Under what circumstance would this happen?

Thanks in advance.

[1143 byte] By [ken-shi] at [2007-11-25 17:19:42]
# 1

The message means that the client has sent an ObjectMessage to imqbrokerd

and is waiting for a response from imqbrokerd confirming that the broker

has received and processed the message. Since it hasn't gotten a response

in over an hour something is likely wrong. Are other producers or consumers

working OK?

When a broker appears "hung" there are some things you can do to narrow

down what the problem is:

From the client:

1. To verify the network is OK try pinging the server imqbrokerd is running on.

If the ping fails you have a network problem.

2. To verify the imqbrokerd process is not completely wedged try telnet'ing

to the main port number. For example, if imqbrokerd is running on

port 7676 on myserver run: 'telnet myserver 7676'

It should respond with a list of broker services and dynamic port numbers

they are running on.

3. To verify the broker's general service code is OK run a simple admin command

like: 'imqcmd query bkr -b myserver:7676'

4. Assuming you are using the standard JMS service (not SSL) make sure

it is still running: 'imqcmd query svc -n jms -b myserver:7676'

And verify the state is "RUNNING".

If 1-4 don't turn up any network or system issues, then it is possible

some threads have deadlocked in imqbrokerd. In this case, if you could

generate a JVM thread dump of imqbrokerd and send it to us that would be

great:

1. Assuming you're on Solaris (or Linux) generate a thread dump

by doing a "kill -QUIT" on the java process spawned by imqbrokerd

(typically the java process with imqbrokerd as the parent process

and started with imqbroker.jar in the classpath).

On Windows you're kinda hosed unless imqbrokerd is running in the

foreground and you can do a Ctrl-\.

2. Send the thread dump and the broker log file to mq-feedback@sun.com

To get going for now first try resarting the client. If that doesn't

work try restarting imqbrokerd.

jfdp at 2007-7-3 12:49:28 > top of Java-index,Application & Integration Servers,Sun Java System Message Queue...
# 2

Thanks for you reply.

At that time, NO producer or consumer was working OK; and the command "imqcmd query dst -t q -n myQueue -u admin -p admin" hung there(the command issued at the same machine where imqbrokerd process resides). It seemed the broker is not accessible.

When I restarted the broker and my clients, everything was OK.

I will do as you said when it happens next time.

ken-shi at 2007-7-3 12:49:28 > top of Java-index,Application & Integration Servers,Sun Java System Message Queue...
# 3

Hello, I have the same problem..

Everything you outline to do from the client worked for me, but when I tried "kill -QUIT <imq java pid>", I get nothing back. I did try that command on a different java process, and that printed out the thread dump as expected. Does the imq server send stderr/stdout somewhere else that I should be able to look at?

ZacJacobson at 2007-7-3 12:49:28 > top of Java-index,Application & Integration Servers,Sun Java System Message Queue...
# 4
imqbrokerd doesn't do anything special with stderr/out, so where it goes depends on how you started the broker. Typically the output goes to the controlling console. If you're on Solaris and imqbrokerd is started via /etc/init.d/imq then the output goes to the system console.
jfdp at 2007-7-3 12:49:28 > top of Java-index,Application & Integration Servers,Sun Java System Message Queue...