Wierd behaviour of BufferedInputStream.available.
Hi All.
I've something that has me stumped.
I have the code below. It connects to a ScriptServlet, passes in some commands and expects to read back the output of the servlet. The servlet sets the content type to "text/xml" and the data is source from a DB2 database.
codeBase =new URL("http://localhost:9080/dial/softphone/");
URL url =new URL(codeBase,"../ScriptServlet?cmd=" + command);
URLConnection urlConnection = url.openConnection();
InputStream in = urlConnection.getInputStream();
BufferedInputStream bis =new BufferedInputStream(in, 32768);
int available1 = bis.available();
int available2 = bis.available();
int available3 = bis.available();
int available4 = bis.available();
AgentApplet.println("Bytes Available 1: " + available1);
AgentApplet.println("Bytes Available 2: " + available2);
AgentApplet.println("Bytes Available 3: " + available3);
AgentApplet.println("Bytes Available 4: " + available4);
Most of the time, everything works. I see the correct number of bytes output by the servlet, and I see the expected output of:
AgentApplet [31] :: Bytes Available 1: 7524
AgentApplet [32] :: Bytes Available 2: 7524
AgentApplet [33] :: Bytes Available 3: 7524
AgentApplet [34] :: Bytes Available 4: 7524
I read the 7524 bytes and I'm all happy.
However, sometimes something really silly happens, and this is the output that I do not understand:
AgentApplet [41] :: Bytes Available 1: 9936
AgentApplet [42] :: Bytes Available 2: 18121
AgentApplet [43] :: Bytes Available 3: 19209
AgentApplet [44] :: Bytes Available 4: 0
The 19209 is the correct figure, as output by the servlet.
There are no reads occurring in between the successive calls to avail().
So what is going on?
I'm stumped!
Help!
-Chris
# 1
There's nothing wierd about it, it's working as documented.
What's going on is that you can't control or rely on how a TCP message will get delivered: whether as one large segment, or N segments of one byte each, or anything in between.
So there can't be any 'expected output'.
InputStream.available() only has one useful purpose that I've ever encountered, and this isn't one of them.
Just do the read.
ejpa at 2007-7-9 20:38:40 >
# 2
With TCP you can rely on it, that was the whole point of it. If it was UDP, then yes, I'd totally agree with you.What I don't get, is why/how the bytes available suddenly drops to zero. Where do they go?I'll give your suggestion a go. Thanks.-Chris
# 3
> With TCP you can rely on it, that was the whole point
> of it. If it was UDP, then yes, I'd totally agree
> with you.
Other way around. With UDP you can rely on datagrams being sent and received the same way, with message boundaries preserved. TCP is a byte-stream protocol without message boundaries and you cannot rely on any particular received-size behaviour.
> What I don't get, is why/how the bytes available
> suddenly drops to zero. Where do they go?
You must have read the buffer before the zero reading. I can't think of any other explanation.
ejpa at 2007-7-9 20:38:40 >
# 4
> > With TCP you can rely on it, that was the whole
> point
> > of it. If it was UDP, then yes, I'd totally agree
> > with you.
>
> Other way around. With UDP you can rely on datagrams
> being sent and received the same way, with message
> boundaries preserved. TCP is a byte-stream protocol
> without message boundaries and you cannot rely on any
> particular received-size behaviour.
Ok, now I get you. You were talking "receive-size". I agree. I was talking "received at all".
> What I don't get, is why/how the bytes available
> suddenly drops to zero. Where do they go?
>
> You must have read the buffer before the zero
> reading. I can't think of any other explanation.
Nope. And that is what has me stumped.
This is the complete routine:
private String sendCommand(String command)
{
try
{
codeBase = new URL("http://localhost:9080/dial/softphone/");
AgentApplet.println("codeBase = " + codeBase);
URL url = new URL(codeBase, "../ScriptServlet?cmd=" + command);
URLConnection urlConnection = url.openConnection();
InputStream in = urlConnection.getInputStream();
BufferedInputStream bis = new BufferedInputStream(in, 32768);
int available1 = bis.available();
int available2 = bis.available();
int available3 = bis.available();
int available4 = bis.available();
AgentApplet.println("Bytes Available 1: " + available1);
AgentApplet.println("Bytes Available 2: " + available2);
AgentApplet.println("Bytes Available 3: " + available3);
AgentApplet.println("Bytes Available 4: " + available4);
byte data[] = new byte[bis.available()];
int read = bis.read(data);
AgentApplet.println("Bytes Read: " + read);
if (read == 0)
{
read = bis.read(data);
AgentApplet.println("Bytes Read (1): " + read);
}
bis.close();
in.close();
String str = new String(data);
return str;
}
catch (MalformedURLException me)
{
AgentApplet.println("MalformedURLException: " + me);
return null;
}
catch (IOException ioe)
{
AgentApplet.println("IOException: " + ioe);
return null;
}
}
# 5
I think it's a bug in sun.net.www.http.ChunkedInputStream. But don't call available(), just keep reading until you get the EOF.
ejpa at 2007-7-9 20:38:40 >
# 6
Would not surprise me.
I just checked a few things. Given that it is a URL connection of a HTTP session, I can get the content-length, cann't I?
Interestingly enough, the calls that work, have the content legth return the real and correct value.
The one that fails returns a content-length of null!
I'll continue to dig deeper.
-Chris
# 7
Look at msg: 2451469.I'm not alone, nor does it appear to have been fixed.-Chris
# 8
Msg 2451469 where?
ejpa at 2007-7-9 20:38:40 >
# 9
In these forums, ie:
http://forum.java.sun.com/thread.jspa?threadID=666843&messageID=2451469
Also look at:
http://forum.java.sun.com/thread.jspa?threadID=666843&messageID=3903775
You're definately right, the Transfer-Encoding of chunked is giving it large amounts of grief.
-Chris
# 10
The problem that I saw isn't anything to do with transfer-encoding.
The problem is that it runs its own internal buffer with an associated count, but it ignores that when computing available(). When it has received the entire last chunk, it just returns 0 rather than the count of the chunk data still to be read out.
Compare the specification of BufferedInputStream.available() which explicitly says it returns the sum of the local buffer count plus in.available().
So as long as you avoid calling available() you won't hit this particular problem.
ejpa at 2007-7-9 20:38:40 >
# 11
> The problem that I saw isn't anything to do with
> transfer-encoding.
>
> The problem is that it runs its own internal buffer
> with an associated count, but it ignores that when
> computing available(). When it has received the
> entire last chunk, it just returns 0 rather than the
> count of the chunk data still to be read out.
Not going to disagree with you there.
> Compare the specification of
> BufferedInputStream.available() which explicitly says
> it returns the sum of the local buffer count plus
> in.available().
In that respect, I do consider it a bug.
> So as long as you avoid calling available() you won't
> hit this particular problem.
I found another way around it.
Here is my solution:
private String sendCommand(String command)
{
try
{
// Next Line: Debugging only. Use to override the codeBase
// when run from a page. Uncomment to be able to debug
// this applet directly inside RSA.
codeBase = new URL("http://localhost:9080/dial/softphone/");
AgentApplet.println("codeBase = " + codeBase);
URL url = new URL(codeBase, "../ScriptServlet?cmd=" + command);
URLConnection urlConnection = url.openConnection();
InputStream in = urlConnection.getInputStream();
BufferedInputStream bis = new BufferedInputStream(in, 32768);
StringBuffer xml = new StringBuffer();
// Handles Normal or Chunked transfers
int avail = 0;
while ((avail = bis.available()) != 0)
{
byte data[] = new byte[avail];
int read = bis.read(data);
AgentApplet.println("Bytes Read: " + read);
xml.append(new String(data));
}
bis.close();
in.close();
return xml.toString();
}
catch (MalformedURLException me)
{
AgentApplet.println("MalformedURLException: " + me);
return null;
}
catch (IOException ioe)
{
AgentApplet.println("IOException: " + ioe);
return null;
}
}
Works for me.
Thank you for all of your assistance, it is appreciated.
-Chris
# 12
> // Handles Normal or Chunked transfers
> int avail = 0;
> while ((avail = bis.available()) != 0)
> {
>byte data[] = new byte[avail];
> int read = bis.read(data);
>AgentApplet.println("Bytes Read: " + read);
> xml.append(new String(data));
> }
That's still an invalid use of available(), and there's no need to use it at all:
int read;
byte[] data = new byte[8192]; // or more
while ((read = bis.read(data)) > 0)
{
AgentApplet.println("Bytes Read: " + read);
xml.append(new String(data, 0, read));
}
ejpa at 2007-7-9 20:38:40 >
# 13
> > > // Handles Normal or Chunked transfers
> > int avail = 0;
> > while ((avail = bis.available()) != 0)
> > {
> >byte data[] = new byte[avail];
> > int read = bis.read(data);
> >AgentApplet.println("Bytes Read: " + read);
> > xml.append(new String(data));
> > }
>
> That's still an invalid use of available(), and
> there's no need to use it at all:
> >int read;
> byte[] data = new byte[8192]; // or more
>while ((read = bis.read(data)) > 0)
> {
> AgentApplet.println("Bytes Read: " + read);
> xml.append(new String(data, 0, read));
>}
>
You are, of course, correct. Thank you very, very much for your help. You've saved by butt.
-Chris
