The out-of-the-box management agent has been designed to implement authentication based on JAAS Login Modules.
Have a look at the com.sun.management.jmxremote.login.config property in the <jre-home>/lib/management/management.properties file for more detailed info.
You should be able to plug the Kerberos JAAS Login Module:
http://java.sun.com/javase/6/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5LoginModule.html.
Regarding encryption, JSSE also allows you to supply Kerberos cipher suites:
http://java.sun.com/javase/6/docs/technotes/guides/security/jsse/JSSERefGuide.html#KRB
Regards,
Luis