By "open" I'm guessing you mean "the application inside is already running" as opposed to being "opened" by an archiver tool. In short, to allow exactly one instance of your app to run?
Most effective way is to have your app open up a listening socket on a known port, and if it can't, assume your app is already running, and close down. Be careful to choose a port that's unlikely to be used by anything else
You could also use lock files, but they have the disadvantage that if your app shuts down ungracefully, the lock file has to be manually deleted
Sun's Java Networking Tutorial is pretty good. To be honest, though, all you need to do is open a ServerSocket on a particular port. If that throws an exception about already being in use, your app is (probably) already running. You don't even need to bother setting up input or output streams (although you could do, to do some extra checking that it really is your app that's got the socket)