SSLEngine, shaky handshake
I'm a C coder who has plunged into the Java dominion and my task at hand is to set up secure communications between a server and its clients. I've found sample code using the classes ChannelIO.java and ChannelIOSecure.java. I merged these two classes into one, since I'm only ever going to use secure communication, and injected the code into my project. Then I installed stunnel to make it possible to use telnet through it to test the communication with my server.
Now to the problem; when I get to handshaking the SSLEngine throws an exception and since there is no source to look into I'm not able to find *why* SSLEngine acts up. The stacktrace looks like this:
java.lang.IllegalArgumentException: appData[0] == null
at com.sun.net.ssl.internal.ssl.EngineArgs.init(EngineArgs.java:101)
at com.sun.net.ssl.internal.ssl.EngineArgs.<init>(EngineArgs.java:64)
at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:
659)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:566)
at SecureConnection.doHandshake(SecureConnection.java:157)
at ClientListener.checkForClientData(ClientListener.java:178)
at Server.main(Server.java:121)
This happens after 100 bytes have been read from the SocketChannel just after the client, in the shape of stunnel, has connected to my server port.
If i just dump the bytes read to the terminal i get
[B@5483cd
and I guess that's the start of the client handshake (with a NULL character ending the printout prematurely).
Has anyone *any* ideas as to how I can go about to try to debug this? This has stolen two whole days and I'm close to writing a wrapper in C using OpenSSL just to get away from this black-boxed insightless hole. People complained about not having SSL and NIO in Java 1.4 but I do wonder if this is any better. This is complex and with no source to try to get to understand it, chaos is the result. To have to implement a statemachine in *every* application just to do the handshake and to manage the IO is ... wrong. *groans*
I use a homemade KeyStore as instructed in the documentation and with all javax debug enabled I see that it is used and added to the security environment. The only thing I do not supply to the SSLContext is my own SecureRandom.
