Can't be done in pure Java. Can't be done in pure C/Berkeley Sockets API either actually unless you have a hook into the IP stack. I have read that you can do this in WINSOCK but don't quote me.
Apart from any possible existing implementations as above the requirement is nonsense. TCP/IP accepts the connection independently of you calling accept(), so by the time you enter accept() the connection may already have been completely established.
What is your actual requirement?
You can't implement the 'refused' part in Java, at least according to the conventional TCP meaning of 'refused' where the client would get a ConnectException 'connection refused'.
However you could just check the address of each accepted Socket and if it isn't in the list just close the socket immediately. The client in this case would probably get a SocketException 'connection reset'.
If the list of legal addresses is static you could get the same effect via a security policy file and a set of SocketPermissions.