Parsing HTTP Headers

I believe that java.net.HttpURLConnection will do this for you already.

And there are probably other libraries that do this. I'd check the jakarta HttpClient library.

Apart from that...I believe the HTTP spec for this is pretty simple. A blank line separates headers from the body. Headers are key/value pairs, where multiple pairs are sometimes permitted, and the values can be lists, etc.

paulcwa at 2007-7-12 9:47:01 >

What features does HttpURLConnection lack? It might be easier to wrap one of those than to rewrite it from scratch.

All your code does is look for a newline, and append to a StringBuilder. It's not clear to me why you're doing the latter; you're trying to parse the headers, aren't you? So why undo the parsing you've done so far? Keep in mind that not only are you undoing the tokenization of lines; you're actually messing up the data because readLine strips newlines, and so the sb.append is creating a big mass of text without the line separators.

paulcwa at 2007-7-12 9:47:01 >

Anyway, some other ideas...

trim the line before you look at the length. Possibly also look for other header related info. Like after the initial HTTP message (with the response code) I believe that ever HTTP header link must have at least one colon. (I am fairly certain of this.) So you could look to see if the line has a colon.

cotton.ma at 2007-7-12 9:47:01 >

The main reason I don't want to use a HttpURLConnection is because:

1. It is slower and bloated for what I want to do, I just want a very simple way to do what I want to do and only what I want to do.

2. To my understanding it doesn't support persistent connections.

3. I like making something lower level because I can control more.

ChromoXNHa at 2007-7-12 9:47:01 >

> The main reason I don't want to use a

> HttpURLConnection is because:

> 1. It is slower and bloated for what I want to do, I

> just want a very simple way to do what I want to do

> and only what I want to do.

Yes... simple.

As far as the performance issue goes. You have some data to support this assumption?

I use HttpURLConnection all the time for application purposes and I certainly don't find it to be an application bottleneck.

cotton.ma at 2007-7-12 9:47:01 >

Ok yes I already have a class using the HttpUrlConnection that handles GZIPed pages. But I want to make my own wrapper that does the same thing just faster, and yes it might not be all that faster but every millisecond counts...

Data: Just ran a test with Socket2(The wrapper using a socket), and Socket(The wrapper using a HttpURLConnection). 10 requests to www.yahoo.com, and calculate the average of the connection time between the sockets...

Socket2: 0.23252920000000002

Socket: 0.30110169999999997

And I think that speed increase is because it doesn't have to recreate the connection every time.

ChromoXNHa at 2007-7-12 9:47:01 >

> Well basically its a sniping program

Well my ethical problem with this aside I don't think this is your performance bottleneck and really not worth worrying about.

Better to have well tested robust code provided by Sun then what you are writing.

Your bottlenecks will come in creating the content to send back but moreover there will be delays and hiccups due to vagaries in internet traffic and what is happening at the site you are "sniping" anyway so at the end of the day I really just don't think this is worth worrying about at all.

And if you really believe otherwise I would inclined to tell you to give up in Java and go right this in C++ or something where you can get lower in the TCP scheme of things and you will be able to extract better peformance then. Although again I believe the advantages would be negligable and largely irrelevent.

cotton.ma at 2007-7-21 21:10:24 >

Wow.

Not to be mean, but if you're trying to tune your online auction bidding down to the millisecond, then maybe it's time that you ask yourself whether you really need that rare collectible Pez dispenser.

Anyway...back to the issue at hand...

First, I'd advise fiddling with the parameters in the request to see if you can prevent the site from sending you gzipped data to begin with. If so that would simplify the problem significantly. Although, you (the OP) are probably worried that you'll lose precious milliseconds that way. But keep in mind that you'll gain a bit of the time, since you and the server won't have to zip/unzip.

If that's not an option, you'll need read newlines in the input stream to look for headers and the end of headers. Keep in mind that the input will be text for the headers, but binary for the body, which makes things a pain.

Etc.

Actually I still suspect that the real improvements to the program would be in making it smarter, not by shaving off milliseconds in the HTTP connections.

paulcwa at 2007-7-21 21:10:24 >

I've done this stuff before - i.e., taking only the body part of the HTTP message.

HTTP structure is very simple than I had thought. The indicator that will tell you that you're about to begin reading the body of the HTTP message is simply a CRLF [blank line].

That's clearly specified in the HTTP 1.0 Protocol Specification. I've done an application that listens on the network and capture data packets. All these are HTTP protocol and was only interested in the body of the message for further processing.

If you have an input stream, you should be reading in bytes and I suppose you'll already know the byte-equivalent of the CR+LF to test against HTTP message.

Check this website for further info on HTTP structure: http://www.jmarshall.com/easy/http/

HTH.

eadelarosaa at 2007-7-21 21:10:24 >