Networking - How can I ensure a packet leaves the interface?
Greetings,
Here's an interesting one for you:
We have a configuration that utilizes a load balancer to balance across a couple of V880's (Solaris 8 10/01). Occasionally, we need to make an http request from "inside" the balancer to the load-balancing IP address. It seems that when the balancer chooses the other server to process the packet, everything works well. However, when the balancer chooses to send the request back to the originating server, the request hangs.
Now, what I think is happening is that the balancer is manipulating the request packet by replacing the balancing IP (virtual IP) with the IP of the server that it has chosen. When that server is the opposite server to the one from which the request was made, everything is fine because the balancer is able to process the returning request, replacing the server IP in the packet header with the virtual IP, which is what the client request is expecting.
However, when the balancer sends the request back to the originating server, the IP address in the packet header is now the same for the source and destination. When the response is sent, it never leaves the interface, presumably because the packet is destined for the same server, so the balancer never gets a chance to process the response.
So, the question becomes, how can I ensure that the response packet gets back to the balancer (just a fancy switch) ; i.e., leaves the interface, even though it is destined for itself?
Any help/comments/ideas are appreciated.
Thanks in advance!
Braden Walters
[1593 byte] By [
BradenW] at [2007-11-25 23:19:04]
# 1
We had a similar issue with our load balancers. I wasn't directly involved with this but these are the basics that I remember.
If I remember correctly, the server sends the out following lets say:
source: 192.168.2.1:32222destination: 65.65.65.65:80
The LDs get a hold of this and change the destination address to 192.168.2.1:80 and load balances the packet back to the originating machine in this case. The problem is your originating machine never made a connection to a 192.168.2.1:80 destination address, so this connection is not in your kernel connection state table, and the connection gets dropped and you appear to hang or maybe get an error.
WSDs have a feature called Triangulation that deals with this issue just fine. Somewhere I saw a diagram on how they deal with this. I can't remember exactly what they do - if I though on this long enough I could probably remember. Our development environment uses these LDs and this problem was handled perfectly with these using this feature.
In our Staging/Production environment we use Cisco Intelligent Directors because that's what our hoster provides and supports. Cisco IDs can't handle this situation. At least the IOS revision we are running can't. To get around this in Staging and Production, we shortcircuited DNS by putting an entry in the /etc/hosts file for the external VIP address e.g. www.mysite.com which points to the nonroutable physical IP address of the interface on the machine making the request. So box A talk directly to box A (e.g. packet never leaves the machine) and never to any of the other machines in the LD group. Not very elegant I know but it works.
For us the problem was the app tier machines have a need to talk to the load balanced web tier but always using the www.mysite.com hostname. Same problem you have just with another tier involved. Our other thought (not implemented yet) is to put the web tier machines on a seperate VLAN. In this case, the problem goes away because the LDs basically become intellegent routers again - nothing weird going on. But it doesn't sound like this would help you though.
I think I remembered this correctly. Hope this helps.
# 2
I need to update what I remember. I thought about this again and my explaination doesn't hold water.
If I remember correctly, the server sends the out following lets say:
source: 192.168.2.1:32222 destination: 65.65.65.65:80
The LDs get a hold of this and change the destination address to 192.168.2.1:80 and load balances the packet back to the originating machine in this case. The originating machine sees this as a brand new connection and opens a connection and replies back to the source - which is 192.168.2.1:32222 e.g. it replies to itself. The problem is your originating machine isn't expecting a reply from 192.168.2.1:80, so this connection is not in your kernel connection state table, and the connection gets dropped and you appear to hang or maybe get an error.
I'm not sure why it works going to other machine in the load balance group.
Again, this is what I remember and I'm probably missing some important points but at least its in the right direction.
# 3
Thanks for replying!
I think that it all works when balanced to the other server because packets are always going through the balancers, and so replies can be processed to restablish the original destination host (the VIP, now in the source header field). It is telling to view a snoop of a successful request to the other server; the other server's IP never shows up in the snoop, only the VIP.
Of course, when the balancer decided to bounce the packet back to the originating server, the server is now replying to itself and so the packet never leaves the interface (since it is already there). The balancers cannot process the reply, and since the "client" is expecting a reply from the VIP, it ignores the response packet.
The more I think about this, the less I think that there is a solution. I think that we are just asking for something that can't be done.
BTW, we are using Alteon WebDirectors as our balancers.
# 4
> The more I think about this, the less I think that
> there is a solution. I think that we are just asking
> for something that can't be done.
>
> BTW, we are using Alteon WebDirectors as our
> balancers.
There is a solution as inelegant as it is. You put an entry on each machine in /etc/hosts so it talks to itself for www.myexternalvipaddress.com.This way when box A needs something from the site/URL, it retrieves it from itself. This will work.
So Box A /etc/hosts has:
www.myexternalvipaddress.com 192.168.1.1 # Box A physical HME0 interface IP for example
This short circuits the connection attempting to go through the load balancers.
# 5
And if you think about it ... If Box A needs to resolve something from www.myexternalvipaddress.com, it can serve that request to itself a hell of a lot faster then going through the load balancers anyway.
# 6
Hi everybody
I have a problem with Alteon and iPlanet Application Server 6.5.
We have one application developer with JAVA and JSP. We have this situation:
- Solaris 9
- iPlanet Application Server
- Oracle
- Alteon (Load Balance)
When one cliente access the site in the iPlanet, sometimes the iPlanet sends to Alteon to finish the session... so the client not open the page...
If we put the application only in the Web Server, the same client, got to access...
Have one parameter in iPlanet to solve this problem?
Thansks,
Beto
