If you have the possibility of changing your application so that it uses the RMI connector defined by the JMX Remote API, then I would strongly encourage you to do so. You can get an implementation from http://java.sun.com/javase/technologies/core/mntr-mgmt/javamanagement/download.jsp (look for JMX Remote API Reference Implementation).
If you don't have the option of doing this in the short term, then you could try turning on JDMK traces by calling com.sun.jdmk.TraceManager.parseTraceProperties() in the main method of your application, and specifying -DLEVEL_DEBUG on the command-line. You will probably see some logging information that will help you diagnose your problem.
蒩monn McManus -- JMX Spec Lead -- http://weblogs.java.net/blog/emcmanus