There is a getSelectedFrame() method in JDesktopPane. This should return null if there is no selected frame.Although, this doesn't guarantee that there are no open windows, it's a pretty good indication. Also, you can try getAllFrames() and check the length of the array. You will have to fiddle around with these to get a definitely correct answer. A window that isn't visible will still show up as being in the JDesktopPane, which may not be what you want.
Hope this helps,
m