jconsole connection trouble. Port redirection?

Hello!

I have a strange issue with jconsole and my JMX enabled daemon. It used to work fine until sysadmin in datacenter placed out machine behind a kind of firewall (cisco accesslist or something like that).

After that remote connection between jconsole and server no longer can be established.

Let's say ip of machine is:host_ip_address

Server side code to start the app:

/usr/java/bin/java -server -Xms64m -Xmx512m -Djava.util.logging.config.file=bin/../conf/logging.properties -Duser.timezone=Europe/Amsterdam -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.host=host_ip_address -Dcom.sun.management.jmxremote.port=8184 -Dcom.sun.management.jmxremote.password.file=bin/../conf/jmx.password -Dcom.sun.management.jmxremote.access.file=bin/../conf/jmx.access -Djava.naming.factory.initial=org.jnp.interfaces.NamingContextFactory -Djava.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces -Djava.naming.provider.url=jnp://localhost:1099 -Dsecurity.policy=client.policy -Djava.security.auth.login.config=/usr/local/jboss/client/auth.conf -cp bin/../lib/Daemon.jar:bin/../lib/Cashier-client.jar:bin/../lib/CoreEJB-client.jar:/usr/local/jboss/client/jbossall-client.jar daemon.Main

Sorry for long string. Just wanted to show it asis.

The daemon used to be controllable by jcosole using remote connection tohost_ip_address:8184

And one day control was lost, jconsole tells that "connecction could ont be established"

After some investigation I found out that jconsole starts connecting to 8184. At this step it works. After that jconsole decides to connect tohost_ip_address:49706. Why? I'm amazed.

I will appriciate any ideas how to get out of the problem.

Or probably explanation about that port49706 redirection.

Here is log of jconsole. Sorry it is quite long, but contains everything.

19.06.2007 18:47:18 sun.rmi.transport.tcp.TCPEndpoint <clinit>

FINE: JConsole.addHost: localHostKnown = true, localHost = 192.168.1.5

19.06.2007 18:47:18 sun.rmi.server.UnicastRef newCall

FINE: JConsole.addHost: get connection

19.06.2007 18:47:18 sun.rmi.transport.tcp.TCPTransport <init>

FINE: JConsole.addHost: Version = 2, ep = [192.168.1.5:0]

19.06.2007 18:47:18 sun.rmi.transport.tcp.TCPEndpoint getLocalEndpoint

FINE: JConsole.addHost: created local endpointfor socket factorynull on port 0

19.06.2007 18:47:18 sun.rmi.transport.tcp.TCPChannel createConnection

FINE: JConsole.addHost: create connection

19.06.2007 18:47:18 sun.rmi.transport.tcp.TCPEndpoint newSocket

FINER: JConsole.addHost: opening socket to [host_ip_address:8184,javax.rmi.ssl.SslRMIClientSocketFactory@170bea5]

19.06.2007 18:47:19 sun.rmi.server.UnicastRef newCall

FINE: JConsole.addHost: get connection

19.06.2007 18:47:19 sun.rmi.transport.tcp.TCPChannel createConnection

FINE: JConsole.addHost: create connection

19.06.2007 18:47:19 sun.rmi.transport.tcp.TCPEndpoint newSocket

FINER: JConsole.addHost: opening socket to [host_ip_address:8184]

19.06.2007 18:47:19 sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket

FINE: JConsole.addHost: host: host_ip_address, port: 8184

19.06.2007 18:47:19 sun.rmi.transport.tcp.TCPChannel createConnection

FINER: JConsole.addHost: server suggested 91.76.3.103:60229

19.06.2007 18:47:19 sun.rmi.transport.tcp.TCPChannel createConnection

FINER: JConsole.addHost: using 192.168.1.5:0

19.06.2007 18:47:19 sun.rmi.server.UnicastRef newCall

FINER: JConsole.addHost: create call context

19.06.2007 18:47:19 sun.rmi.server.UnicastRef logClientCall

FINER: JConsole.addHost: outbound call: [endpoint:[host_ip_address:8184](remote),objID:[0:0:0, 0]] : sun.rmi.registry.RegistryImpl_Stub[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)

19.06.2007 18:47:19 sun.rmi.transport.StreamRemoteCall <init>

FINER: JConsole.addHost: write remote call header...

19.06.2007 18:47:19 sun.rmi.transport.StreamRemoteCall getOutputStream

FINER: JConsole.addHost: getting output stream

19.06.2007 18:47:19 sun.rmi.server.UnicastRef invoke

FINER: JConsole.addHost: execute call

19.06.2007 18:47:19 sun.rmi.transport.StreamRemoteCall getInputStream

FINER: JConsole.addHost: getting input stream

19.06.2007 18:47:20 sun.rmi.server.LoaderHandler loadClass

FINE: JConsole.addHost: name ="javax.management.remote.rmi.RMIServerImpl_Stub", codebase ="", defaultLoader = sun.misc.Launcher$AppClassLoader@d9f9c3

19.06.2007 18:47:20 sun.rmi.server.LoaderHandler loadClass

FINER: JConsole.addHost:class"javax.management.remote.rmi.RMIServerImpl_Stub" found via defaultLoader, defined bynull

19.06.2007 18:47:20 sun.rmi.server.LoaderHandler loadClass

FINE: JConsole.addHost: name ="java.rmi.server.RemoteStub", codebase ="", defaultLoader = sun.misc.Launcher$AppClassLoader@d9f9c3

19.06.2007 18:47:20 sun.rmi.server.LoaderHandler loadClass

FINER: JConsole.addHost:class"java.rmi.server.RemoteStub" found via defaultLoader, defined bynull

19.06.2007 18:47:20 sun.rmi.server.LoaderHandler loadClass

FINE: JConsole.addHost: name ="java.rmi.server.RemoteObject", codebase ="", defaultLoader = sun.misc.Launcher$AppClassLoader@d9f9c3

19.06.2007 18:47:20 sun.rmi.server.LoaderHandler loadClass

FINER: JConsole.addHost:class"java.rmi.server.RemoteObject" found via defaultLoader, defined bynull

19.06.2007 18:47:20 sun.rmi.server.UnicastRef done

FINE: JConsole.addHost: free connection (reuse =true)

19.06.2007 18:47:20 sun.rmi.transport.tcp.TCPChannel free

FINE: JConsole.addHost: reuse connection

19.06.2007 18:47:20 sun.rmi.transport.tcp.TCPChannel free

FINE: JConsole.addHost: create reaper

19.06.2007 18:47:20 sun.rmi.server.UnicastRef newCall

FINE: JConsole.addHost: get connection

19.06.2007 18:47:20 sun.rmi.transport.tcp.TCPChannel createConnection

FINE: JConsole.addHost: create connection

19.06.2007 18:47:20 sun.rmi.transport.tcp.TCPEndpoint newSocket

FINER: JConsole.addHost: opening socket to [host_ip_address:49706]

19.06.2007 18:47:20 sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket

FINE: JConsole.addHost: host: host_ip_address, port: 49706

19.06.2007 18:47:35 sun.rmi.transport.tcp.TCPChannel$1 run

FINER: RMI Scheduler(0): wake up

19.06.2007 18:47:35 sun.rmi.transport.tcp.TCPChannel freeCachedConnections

FINER: RMI Scheduler(0): connection timeout expired

19.06.2007 18:47:35 sun.rmi.transport.tcp.TCPConnection close

FINE: RMI Scheduler(0): close connection

19.06.2007 18:47:41 sun.rmi.server.UnicastRef newCall

FINE: RMI RenewClean-[host_ip_address:49706]: get connection

19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPChannel createConnection

FINE: RMI RenewClean-[host_ip_address:49706]: create connection

19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPEndpoint newSocket

FINER: RMI RenewClean-[host_ip_address:49706]: opening socket to [host_ip_address:49706]

19.06.2007 18:47:41 sun.rmi.transport.WeakRef pin

FINER: JConsole.addHost: strongRef = sun.rmi.transport.DGCImpl@11c2b67

19.06.2007 18:47:41 sun.rmi.transport.ObjectTable putTarget

FINER: JConsole.addHost: add object [0:0:0, 2]

19.06.2007 18:47:41 sun.rmi.transport.ConnectionInputStream done

FINER: JConsole.addHost: send ack

19.06.2007 18:47:41 sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket

FINE: RMI RenewClean-[host_ip_address:49706]: host: host_ip_address, port: 49706

19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPChannel createConnection

FINE: JConsole.addHost: create connection

19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPEndpoint newSocket

FINER: JConsole.addHost: opening socket to [host_ip_address:8184]

19.06.2007 18:47:41 sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket

FINE: JConsole.addHost: host: host_ip_address, port: 8184

19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPChannel createConnection

FINER: JConsole.addHost: server suggested 91.76.3.103:60247

19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPChannel createConnection

FINER: JConsole.addHost: using 192.168.1.5:0

19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPChannel free

FINE: JConsole.addHost: reuse connection

19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPChannel free

FINE: JConsole.addHost: create reaper

19.06.2007 18:47:41 RMIConnector connect

FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[host_ip_address:49706](remote),objID:[-57f44ee5:111746985d0:-8000, 2408504052969290368]]]]] connecting...

19.06.2007 18:47:41 RMIConnector connect

FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[host_ip_address:49706](remote),objID:[-57f44ee5:111746985d0:-8000, 2408504052969290368]]]]] finding stub...

19.06.2007 18:47:41 RMIConnector connect

FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[host_ip_address:49706](remote),objID:[-57f44ee5:111746985d0:-8000, 2408504052969290368]]]]] connecting stub...

19.06.2007 18:47:41 RMIConnector connect

FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[host_ip_address:49706](remote),objID:[-57f44ee5:111746985d0:-8000, 2408504052969290368]]]]] getting connection...

19.06.2007 18:47:41 sun.rmi.server.UnicastRef invoke

FINER: VMPanel.connect: method:publicabstract javax.management.remote.rmi.RMIConnection javax.management.remote.rmi.RMIServer.newClient(java.lang.Object)throws java.io.IOException

19.06.2007 18:47:41 sun.rmi.server.UnicastRef logClientCall

FINER: VMPanel.connect: outbound call: [endpoint:[host_ip_address:49706](remote),objID:[-57f44ee5:111746985d0:-8000, 2408504052969290368]] : javax.management.remote.rmi.RMIServerImpl_Stub[-57f44ee5:111746985d0:-8000, 2408504052969290368]:publicabstract javax.management.remote.rmi.RMIConnection javax.management.remote.rmi.RMIServer.newClient(java.lang.Object)throws java.io.IOException

19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPChannel createConnection

FINE: VMPanel.connect: create connection

19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPEndpoint newSocket

FINER: VMPanel.connect: opening socket to [host_ip_address:49706]

19.06.2007 18:47:41 sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket

FINE: VMPanel.connect: host: host_ip_address, port: 49706

19.06.2007 18:47:56 sun.rmi.transport.tcp.TCPChannel$1 run

FINER: RMI Scheduler(0): wake up

19.06.2007 18:47:56 sun.rmi.transport.tcp.TCPChannel freeCachedConnections

FINER: RMI Scheduler(0): connection timeout expired

19.06.2007 18:47:56 sun.rmi.transport.tcp.TCPConnection close

FINE: RMI Scheduler(0): close connection

19.06.2007 18:48:02 RMIConnector connect

FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[host_ip_address:49706](remote),objID:[-57f44ee5:111746985d0:-8000, 2408504052969290368]]]]] failed to connect: java.rmi.ConnectException: Connection refused to host: host_ip_address; nested exception is:

java.net.ConnectException: Connection timed out: connect

19.06.2007 18:48:02 RMIConnector close

FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[host_ip_address:49706](remote),objID:[-57f44ee5:111746985d0:-8000, 2408504052969290368]]]]] closing.

19.06.2007 18:48:23 sun.rmi.server.UnicastRef newCall

FINE: RMI RenewClean-[host_ip_address:49706]: get connection

19.06.2007 18:48:23 sun.rmi.transport.tcp.TCPChannel createConnection

FINE: RMI RenewClean-[host_ip_address:49706]: create connection

19.06.2007 18:48:23 sun.rmi.transport.tcp.TCPEndpoint newSocket

FINER: RMI RenewClean-[host_ip_address:49706]: opening socket to [host_ip_address:49706]

19.06.2007 18:48:23 sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket

FINE: RMI RenewClean-[host_ip_address:49706]: host: host_ip_address, port: 49706

19.06.2007 18:49:26 sun.rmi.server.UnicastRef newCall

FINE: RMI RenewClean-[host_ip_address:49706]: get connection

19.06.2007 18:49:26 sun.rmi.transport.tcp.TCPChannel createConnection

FINE: RMI RenewClean-[host_ip_address:49706]: create connection

19.06.2007 18:49:26 sun.rmi.transport.tcp.TCPEndpoint newSocket

FINER: RMI RenewClean-[host_ip_address:49706]: opening socket to [host_ip_address:49706]

19.06.2007 18:49:26 sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket

FINE: RMI RenewClean-[host_ip_address:49706]: host: host_ip_address, port: 49706

[13983 byte] By [_lauria] at [2007-11-27 8:09:56]
# 1
Have a look at: http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html#gdjou
lmalventa at 2007-7-12 19:53:14 > top of Java-index,Core,Monitoring & Management...