jconsole connection trouble. Port redirection?
Hello!
I have a strange issue with jconsole and my JMX enabled daemon. It used to work fine until sysadmin in datacenter placed out machine behind a kind of firewall (cisco accesslist or something like that).
After that remote connection between jconsole and server no longer can be established.
Let's say ip of machine is:host_ip_address
Server side code to start the app:
/usr/java/bin/java -server -Xms64m -Xmx512m -Djava.util.logging.config.file=bin/../conf/logging.properties -Duser.timezone=Europe/Amsterdam -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.host=host_ip_address -Dcom.sun.management.jmxremote.port=8184 -Dcom.sun.management.jmxremote.password.file=bin/../conf/jmx.password -Dcom.sun.management.jmxremote.access.file=bin/../conf/jmx.access -Djava.naming.factory.initial=org.jnp.interfaces.NamingContextFactory -Djava.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces -Djava.naming.provider.url=jnp://localhost:1099 -Dsecurity.policy=client.policy -Djava.security.auth.login.config=/usr/local/jboss/client/auth.conf -cp bin/../lib/Daemon.jar:bin/../lib/Cashier-client.jar:bin/../lib/CoreEJB-client.jar:/usr/local/jboss/client/jbossall-client.jar daemon.Main
Sorry for long string. Just wanted to show it asis.
The daemon used to be controllable by jcosole using remote connection tohost_ip_address:8184
And one day control was lost, jconsole tells that "connecction could ont be established"
After some investigation I found out that jconsole starts connecting to 8184. At this step it works. After that jconsole decides to connect tohost_ip_address:49706. Why? I'm amazed.
I will appriciate any ideas how to get out of the problem.
Or probably explanation about that port49706 redirection.
Here is log of jconsole. Sorry it is quite long, but contains everything.
19.06.2007 18:47:18 sun.rmi.transport.tcp.TCPEndpoint <clinit>
FINE: JConsole.addHost: localHostKnown = true, localHost = 192.168.1.5
19.06.2007 18:47:18 sun.rmi.server.UnicastRef newCall
FINE: JConsole.addHost: get connection
19.06.2007 18:47:18 sun.rmi.transport.tcp.TCPTransport <init>
FINE: JConsole.addHost: Version = 2, ep = [192.168.1.5:0]
19.06.2007 18:47:18 sun.rmi.transport.tcp.TCPEndpoint getLocalEndpoint
FINE: JConsole.addHost: created local endpointfor socket factorynull on port 0
19.06.2007 18:47:18 sun.rmi.transport.tcp.TCPChannel createConnection
FINE: JConsole.addHost: create connection
19.06.2007 18:47:18 sun.rmi.transport.tcp.TCPEndpoint newSocket
FINER: JConsole.addHost: opening socket to [host_ip_address:8184,javax.rmi.ssl.SslRMIClientSocketFactory@170bea5]
19.06.2007 18:47:19 sun.rmi.server.UnicastRef newCall
FINE: JConsole.addHost: get connection
19.06.2007 18:47:19 sun.rmi.transport.tcp.TCPChannel createConnection
FINE: JConsole.addHost: create connection
19.06.2007 18:47:19 sun.rmi.transport.tcp.TCPEndpoint newSocket
FINER: JConsole.addHost: opening socket to [host_ip_address:8184]
19.06.2007 18:47:19 sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket
FINE: JConsole.addHost: host: host_ip_address, port: 8184
19.06.2007 18:47:19 sun.rmi.transport.tcp.TCPChannel createConnection
FINER: JConsole.addHost: server suggested 91.76.3.103:60229
19.06.2007 18:47:19 sun.rmi.transport.tcp.TCPChannel createConnection
FINER: JConsole.addHost: using 192.168.1.5:0
19.06.2007 18:47:19 sun.rmi.server.UnicastRef newCall
FINER: JConsole.addHost: create call context
19.06.2007 18:47:19 sun.rmi.server.UnicastRef logClientCall
FINER: JConsole.addHost: outbound call: [endpoint:[host_ip_address:8184](remote),objID:[0:0:0, 0]] : sun.rmi.registry.RegistryImpl_Stub[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)
19.06.2007 18:47:19 sun.rmi.transport.StreamRemoteCall <init>
FINER: JConsole.addHost: write remote call header...
19.06.2007 18:47:19 sun.rmi.transport.StreamRemoteCall getOutputStream
FINER: JConsole.addHost: getting output stream
19.06.2007 18:47:19 sun.rmi.server.UnicastRef invoke
FINER: JConsole.addHost: execute call
19.06.2007 18:47:19 sun.rmi.transport.StreamRemoteCall getInputStream
FINER: JConsole.addHost: getting input stream
19.06.2007 18:47:20 sun.rmi.server.LoaderHandler loadClass
FINE: JConsole.addHost: name ="javax.management.remote.rmi.RMIServerImpl_Stub", codebase ="", defaultLoader = sun.misc.Launcher$AppClassLoader@d9f9c3
19.06.2007 18:47:20 sun.rmi.server.LoaderHandler loadClass
FINER: JConsole.addHost:class"javax.management.remote.rmi.RMIServerImpl_Stub" found via defaultLoader, defined bynull
19.06.2007 18:47:20 sun.rmi.server.LoaderHandler loadClass
FINE: JConsole.addHost: name ="java.rmi.server.RemoteStub", codebase ="", defaultLoader = sun.misc.Launcher$AppClassLoader@d9f9c3
19.06.2007 18:47:20 sun.rmi.server.LoaderHandler loadClass
FINER: JConsole.addHost:class"java.rmi.server.RemoteStub" found via defaultLoader, defined bynull
19.06.2007 18:47:20 sun.rmi.server.LoaderHandler loadClass
FINE: JConsole.addHost: name ="java.rmi.server.RemoteObject", codebase ="", defaultLoader = sun.misc.Launcher$AppClassLoader@d9f9c3
19.06.2007 18:47:20 sun.rmi.server.LoaderHandler loadClass
FINER: JConsole.addHost:class"java.rmi.server.RemoteObject" found via defaultLoader, defined bynull
19.06.2007 18:47:20 sun.rmi.server.UnicastRef done
FINE: JConsole.addHost: free connection (reuse =true)
19.06.2007 18:47:20 sun.rmi.transport.tcp.TCPChannel free
FINE: JConsole.addHost: reuse connection
19.06.2007 18:47:20 sun.rmi.transport.tcp.TCPChannel free
FINE: JConsole.addHost: create reaper
19.06.2007 18:47:20 sun.rmi.server.UnicastRef newCall
FINE: JConsole.addHost: get connection
19.06.2007 18:47:20 sun.rmi.transport.tcp.TCPChannel createConnection
FINE: JConsole.addHost: create connection
19.06.2007 18:47:20 sun.rmi.transport.tcp.TCPEndpoint newSocket
FINER: JConsole.addHost: opening socket to [host_ip_address:49706]
19.06.2007 18:47:20 sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket
FINE: JConsole.addHost: host: host_ip_address, port: 49706
19.06.2007 18:47:35 sun.rmi.transport.tcp.TCPChannel$1 run
FINER: RMI Scheduler(0): wake up
19.06.2007 18:47:35 sun.rmi.transport.tcp.TCPChannel freeCachedConnections
FINER: RMI Scheduler(0): connection timeout expired
19.06.2007 18:47:35 sun.rmi.transport.tcp.TCPConnection close
FINE: RMI Scheduler(0): close connection
19.06.2007 18:47:41 sun.rmi.server.UnicastRef newCall
FINE: RMI RenewClean-[host_ip_address:49706]: get connection
19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPChannel createConnection
FINE: RMI RenewClean-[host_ip_address:49706]: create connection
19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPEndpoint newSocket
FINER: RMI RenewClean-[host_ip_address:49706]: opening socket to [host_ip_address:49706]
19.06.2007 18:47:41 sun.rmi.transport.WeakRef pin
FINER: JConsole.addHost: strongRef = sun.rmi.transport.DGCImpl@11c2b67
19.06.2007 18:47:41 sun.rmi.transport.ObjectTable putTarget
FINER: JConsole.addHost: add object [0:0:0, 2]
19.06.2007 18:47:41 sun.rmi.transport.ConnectionInputStream done
FINER: JConsole.addHost: send ack
19.06.2007 18:47:41 sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket
FINE: RMI RenewClean-[host_ip_address:49706]: host: host_ip_address, port: 49706
19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPChannel createConnection
FINE: JConsole.addHost: create connection
19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPEndpoint newSocket
FINER: JConsole.addHost: opening socket to [host_ip_address:8184]
19.06.2007 18:47:41 sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket
FINE: JConsole.addHost: host: host_ip_address, port: 8184
19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPChannel createConnection
FINER: JConsole.addHost: server suggested 91.76.3.103:60247
19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPChannel createConnection
FINER: JConsole.addHost: using 192.168.1.5:0
19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPChannel free
FINE: JConsole.addHost: reuse connection
19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPChannel free
FINE: JConsole.addHost: create reaper
19.06.2007 18:47:41 RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[host_ip_address:49706](remote),objID:[-57f44ee5:111746985d0:-8000, 2408504052969290368]]]]] connecting...
19.06.2007 18:47:41 RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[host_ip_address:49706](remote),objID:[-57f44ee5:111746985d0:-8000, 2408504052969290368]]]]] finding stub...
19.06.2007 18:47:41 RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[host_ip_address:49706](remote),objID:[-57f44ee5:111746985d0:-8000, 2408504052969290368]]]]] connecting stub...
19.06.2007 18:47:41 RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[host_ip_address:49706](remote),objID:[-57f44ee5:111746985d0:-8000, 2408504052969290368]]]]] getting connection...
19.06.2007 18:47:41 sun.rmi.server.UnicastRef invoke
FINER: VMPanel.connect: method:publicabstract javax.management.remote.rmi.RMIConnection javax.management.remote.rmi.RMIServer.newClient(java.lang.Object)throws java.io.IOException
19.06.2007 18:47:41 sun.rmi.server.UnicastRef logClientCall
FINER: VMPanel.connect: outbound call: [endpoint:[host_ip_address:49706](remote),objID:[-57f44ee5:111746985d0:-8000, 2408504052969290368]] : javax.management.remote.rmi.RMIServerImpl_Stub[-57f44ee5:111746985d0:-8000, 2408504052969290368]:publicabstract javax.management.remote.rmi.RMIConnection javax.management.remote.rmi.RMIServer.newClient(java.lang.Object)throws java.io.IOException
19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPChannel createConnection
FINE: VMPanel.connect: create connection
19.06.2007 18:47:41 sun.rmi.transport.tcp.TCPEndpoint newSocket
FINER: VMPanel.connect: opening socket to [host_ip_address:49706]
19.06.2007 18:47:41 sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket
FINE: VMPanel.connect: host: host_ip_address, port: 49706
19.06.2007 18:47:56 sun.rmi.transport.tcp.TCPChannel$1 run
FINER: RMI Scheduler(0): wake up
19.06.2007 18:47:56 sun.rmi.transport.tcp.TCPChannel freeCachedConnections
FINER: RMI Scheduler(0): connection timeout expired
19.06.2007 18:47:56 sun.rmi.transport.tcp.TCPConnection close
FINE: RMI Scheduler(0): close connection
19.06.2007 18:48:02 RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[host_ip_address:49706](remote),objID:[-57f44ee5:111746985d0:-8000, 2408504052969290368]]]]] failed to connect: java.rmi.ConnectException: Connection refused to host: host_ip_address; nested exception is:
java.net.ConnectException: Connection timed out: connect
19.06.2007 18:48:02 RMIConnector close
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[host_ip_address:49706](remote),objID:[-57f44ee5:111746985d0:-8000, 2408504052969290368]]]]] closing.
19.06.2007 18:48:23 sun.rmi.server.UnicastRef newCall
FINE: RMI RenewClean-[host_ip_address:49706]: get connection
19.06.2007 18:48:23 sun.rmi.transport.tcp.TCPChannel createConnection
FINE: RMI RenewClean-[host_ip_address:49706]: create connection
19.06.2007 18:48:23 sun.rmi.transport.tcp.TCPEndpoint newSocket
FINER: RMI RenewClean-[host_ip_address:49706]: opening socket to [host_ip_address:49706]
19.06.2007 18:48:23 sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket
FINE: RMI RenewClean-[host_ip_address:49706]: host: host_ip_address, port: 49706
19.06.2007 18:49:26 sun.rmi.server.UnicastRef newCall
FINE: RMI RenewClean-[host_ip_address:49706]: get connection
19.06.2007 18:49:26 sun.rmi.transport.tcp.TCPChannel createConnection
FINE: RMI RenewClean-[host_ip_address:49706]: create connection
19.06.2007 18:49:26 sun.rmi.transport.tcp.TCPEndpoint newSocket
FINER: RMI RenewClean-[host_ip_address:49706]: opening socket to [host_ip_address:49706]
19.06.2007 18:49:26 sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket
FINE: RMI RenewClean-[host_ip_address:49706]: host: host_ip_address, port: 49706

