If you are accessing MBeans from within the same JVM, then you don't need to use -Dcom.sun.management.jmxremote. You can simply use ManagementFactory.getPlatformMBeanServer() and access the MBeans directly in the returned MBeanServer object. There is basically zero overhead in doing this because the JVM is tracking the same information whether or not you have a handle on its MBeanServer.
If you access MBeans through -Dcom.sun.management.jmxremote, then indeed it does use RMI and serialization. The cost of providing this option is negligible if you do not connect to the JVM. If you do connect, then there is obviously a slight cost due to the extra threads and networking that RMI needs.