Writing an 'end of stream' to a nio socket
I am trying to wite a server and client which transmit data in xml format. On both ends i am using stax, hence data is accessed by streams.
I want to do the data transfer over the network via NIO non blocking sockets. However i cannot find any information about how to tell either the server or the client that a particular set of data has been finished i.e all xml in this present set has been written out, so start processing it. if it starts processing earlier, there might be problems of xml structure etc..
I think the problem might arrise because there is a large probability of the amount of data being very large. hence i cannot allocate buffers to hold all the data to be transferred at once. It will thus be transferred in batches and then the sending thread will block waiting for the buffer to refill.
Does NIO have the ability to do this?
# 1
Sockets don't have the ability to do this, let alone NIO, unless this is the last thing you want to transfer in that direction, in which case you can shutdown the socket for output. Otherwise you have to do something at the application protocol level of the application protocol.
ejpa at 2007-7-12 1:09:13 >
# 2
i had similar issue.
i have one socket that is used to multiple purposes.
one is sending a file as a byte stream.
the problem is that the reader never exits from the receive loop.
it keeps waiting for more data.
the sender exits its send loop once end of file is reached.
i have the sender write 3 pre-set integers once EOF is read.
the receiver has a byte[3] buffer that it checks for that sequence of
integers.
this works but seems like a bad solution.
is there a better way?
# 3
Best solution is to create a complete protocol, with application headers, checksumming and everything.
It's extremely useful if you want to tell apart the actual client from someone port-scanning your machine.
Basically, the client should produce pre-designed packets that contain a protocol/packet identifier(s), a packet length, the data, and optionally a checksum to verify the packet is valid(protocol and length might be caught by random data, but that is far less likely with a checksum).
From here there are three possibilities:
1. Have the entire transferred file put in one packet.
2. Have the file-length, either in bytes or in packets or any other deterministic one-one measure, transfered either with or as the first packet.
3. Implement an end-of-file packet, or an end-of-file marker for the last packet("last packet" identifier).
Then the server simply needs to parse the input stream into packets, and disconnect any client that doesn't provide a valid packet stream. In packet form, the file ending is obvious.
Even if you only transfer one file and can afford the "close the socket" solution, the above is recommended for security. As I've said, you want to be able to tell apart a real client from a port scanner sending random data to see what comes back.
# 4
If you want to reliably send data over the internet, I would use ServerSockets and Sockets, you grab their streams and wrap them in ObjectStreams. Don't forget to flush the ObjectOutputStreams before you create or read the corresponding ObjectInputStreams (they would block until the corresponding ObjectOutputStreams are flushed).
This way it is very easy to transfer data, you wrap all you have to send in an object of the right format and the ObjectInputStream at the receiver's end only returns when the whole Object is read.
There is a performance issue though at the server side if you use a separate thread to handle each incoming client, which is normally the case with normal io/network programming. Some 100 connections goes fine, but if it goes up much more, the performance of the server goes down dramatically.
The purpose of JAVA nio is to allow to program a server so it doesn't need 1000 threads to handle 1000 clients, in fact 3 threads may suffice to handle 10 or 100 or 1000 or 1000000 clients. However JAVA nio is low level and you program in buffers.
What I do if I have to send data of multiple clients over the network, I make two connections for every client. The one is a regular socket, of which the streams are wrapped in ObjectStreams. However, these aren't handled by their own threads, they are just stored somewhere. The other connection I make is a buffer of fixed size with JAVA nio. And every time the client is ready to send some data, I just write a byte to this buffer. The server keeps a list of buffers to which is written and you can pick the corresponding ObjectStreams and delegate them to as many threads as you want to to handle them.
It might be a bit complicated to find out how JAVA nio works. For myself, I made a summary of JAVA and I put my "chapter" of network programming online at http://www.macharis.be/toon/wiki/index.php/JAVA_Network_Programming . It might give you some idea how JAVA nio works. The code example contains the ideas, but I didn't check it, so it might contain some minor bugs. Also I didn't write the text for it to be read by other people, it is a summary for myself. However if you would have a lack of resources about the topic, you can find some inspiration there.
# 5
Two TCP connections per client is getting expensive and only half as scalable. Instead of all that elaboration it is much simpler to forget the Socket connection, the ByteBuffer, and the signal byte. Especially the signal byte - NIO is an expensive way in programming terms of sending around single bytes. Instead when the channel becomes readable, put it into non-blocking mode, get a stream for it from Channels, wrap an InputStream around that, and read the object. This would require creating a new ObjectOutputStream at the writer each time.
Your NIO server example contains several major errors. Have you tested this code? The whole point of NIO is to avoid per-client threads - this is what makes it more scalable than java.io. So you should handle all the I/O for all sockets and server sockets in the same select loop. And you should certainly never register a channel for OP_WRITE unless you actually have data for it. This example will just spin meaninglessly in the thread's select loop. Also I don't see where the Handler thread ever gets started.
See http://forum.java.sun.com/thread.jspa?threadID=459338 for a rather complete discussion.
ejpa at 2007-7-12 1:09:13 >
