What you are asking doesn't make much sense because, setMaximizedBounds() only defines the bounds for the window in the maximized state, not the maximum size you can manually resize. And JDialogs don't have a maximized state. So what you are asking is not possible.
If you want to restrict manual resizes, then use a component listener and restrict the bounds as it's resized.