|
Janno de Wit wrote:
>
> Joseph,
>
> Problem is we cannot reproduce the problem but there are customers who
> has this. Just 10 minutes ago I got a report of a school using
> BorderManager that gives:
>
> "504 Gateway Timeout - Lost connection to neighbor proxyserver"
> "502 Bad Gateway - Mal-formed reply from origin server"
>
> When they point their BorderManager to realserver-1 there are no
> problems.
This is no fun for you and the customers get riled.
I take it you can't get the list of what they did to get the error and
go to the same site yourself and type the same commands.
Otherwise you may just have to sit there with tcpdump and watch traffic till
you get a hit. You're going to have an enormous tcpdump file to search
through. With a live setup at the traffic flow you have, you'll probably
need a separate fast machine to be able to write all the packet info to disk.
> It is not with all sites, especially with Hotmail and searching at
> www.vikingdirect.nl and other sites not specified.
>
> I stopped all firewalls on load-balancer and realservers, but that does
> not solve the problem, so we can assume there are no packets dropped by
> iptables.
Let's assume it's either LVS or squid or an interaction between LVS and squid.
Having the customers going directly to the realservers without problems
would indicate that it's not the squid.
To test that its not the LVS, can you put the squid into pass through mode
(a null squid) for a test? I know the customers won't have any caching for
a day or so, but too bad.
If it's an interaction between LVS and squid, you'll need tcpdump again
sitting between the director and each realserver.
> As you can see above, we are already using persistant connections.
OK, missed that.
> > No there are still questions:
> > - Can this be the MTU (both on WAN and LAN: 1500 bytes, at LB and
> > Realservers)? What is MTU's impact on LVS-TUN (maybe ip-encap?)?
It's the same problem as the other encapsulated packet protocols (eg VPNs).
An MTU sized packet is encapsulated, making it bigger than the MTU, and
the packet must be fragmented. Some protocols have checksums on the packet
and these get broken. I don't know what goes wrong with LVS-Tun, but previous
e-mails with Julian indicates that he thinks there maybe something wrong there.
Whether it's with LVS or the ipip handling in Linux I don't know. The problem
is that only Linux has ipip and it doesn't get much attention. Hardly anyone
is using LVS-Tun either. It seems that Julian has things of higher priority
to deal with, but still the MTU/LVS-Tun problem is a hole which anyone can
fall into, until it's dealt with.
Horms, Julian,
Do you know what's going on with MTU and LVS-Tun?
> > MTU and LVS-Tun are written up in the HOWTO. I don't think it's a solved
> > problem.
>
> Is a better way to switch to DR-mode?
for LVS-DR you need the realservers on the same segment as the director, ie
the director and realservers must be able to exchange arp packets. LVS-Tun
was designed for situations when the realservers have to be remote.
Another possibility is to make the MTU of all packets 1492 before they hit the
director. Do you have a box infront of the director(s) that can change the MTU?
> > > - How can I see if connectiontable is full? `dmesg` gives no output.
> > hmm, don't know. probably you can get it with ipvsadm.
Horms just replied recently that you can't fill the table. The limiting
factor is memory.
> Isn't it possible that the hashtable overwrites entries?
I would assume that active entries aren't overwritten and expired ones
are removed as part of the expiration process.
I'm going to be away for 2 weeks, incase you wonder why I don't reply.
Will be back early Feb.
good luck
Joe
--
Joseph Mack PhD, High Performance Computing & Scientific Visualization
LMIT, Supporting the EPA Research Triangle Park, NC 919-541-0007
Federal Contact - John B. Smith 919-541-1087 - smith.johnb@xxxxxxx
|
