|
On Thu, Jan 06, 2005 at 03:17:11PM +0200, Johan van den Berg wrote:
> Johan van den Berg wrote:
>
> >Joseph Mack wrote:
> >
> >>Janno de Wit wrote:
> >>
> >>
> >>>- How can I see if connectiontable is full? `dmesg` gives no output.
> >>>
> >>
> >>
> >>hmm, don't know. probably you can get it with ipvsadm.
> >>
> >>
> >>
> >I would really like to find this out too! How do I know if packets or
> >connections are being lost or if something malfunctions, or if the
> >connection table is full?
> >
> >In my scenario I have LVS-NAT for incoming connections into my
> >cluster, and IPTABLES SNAT outgoing. Now, if none of my servers need
> >to initiate a connection to the outside, then the IPTABLES connection
> >table should be clear, and LVS-NAT table contain entries of all
> >incoming connections. This it actually does, except every now and
> >again, a client would try to establish a connection to the cluster,
> >and IPVS would store the SYN in the IPVS connection table, but the
> >SYN/ACK from the server would be NATted through IPTABLES, as if the
> >original SYN never existed. This would result in the wrong IP on the
> >director being used for the SYN/ACK, meaning the client would respond
> >with a RESET, and resend its SYN to the original IP, as if connection
> >was never established. This happens about once every two days, and
> >then only for a few minutes. One can actually see a SYN in the IPVS
> >connection table that just stays like that until it times out, and an
> >equivalent SYN/ACK using netstat for the response.
> >
> >I can only assume that some or other limit was reached in one of the
> >connection tables, or that something else went berzerk, but as nothing
> >is reported in syslog or klog, and I couldn't figure out how to find
> >out exactly what I should otherwise be looking at.
> >
> >Any further advice would be appreciated.
> >
> >>Joe
> >>--
> >>
> >>Joseph Mack PhD, High Performance Computing & Scientific Visualization
> >>LMIT, Supporting the EPA Research Triangle Park, NC 919-541-0007
> >>Federal Contact - John B. Smith 919-541-1087 - smith.johnb@xxxxxxx
> >>_______________________________________________
> >
> Upon setting /proc/sys/net/ipv4/vs/debug_level to 7, I noticed that
> every now and again, I get a "lookup/out" entry that states that a
> connection from the virtual ip on port 80 to a client port on the client
> IP was "not hit". This seems to confirm that the original SYN from the
> client to port 80 on the virtual ip is not stored in the IPVS connection
> table, and therefore the reply to the client IP is not handled by IPVS,
> but rather iptables, which causes havoc.
Yes, that does seem to confirm that the entry is not in the IPVS
connection table, and thys falls back to iptables.
For clafiication, this is most likely occuring in ip_vs_out()
which is a netfilter hook that reverses IPVS-NATed packets,
and thus is specific to NAT, though I don't think your problem
neccessarily is.
> If anyone can help me in finding further helpful information to debug
> this I would be thankful. The biggest issue that I have is the fact that
> this machine is production and recieving quite a lot of hits, and
> therefore debugging becomes quite difficult.
Understood.
If the problem is in LVS (rather than perhaps a race in netfilter,
or something eles that happens before it gets to lvs_in()) then it
I am guessing it is most likely to be because lvs_in() is incorrectly
returning NF_ACCEPT, for packets that it should just consume.
There are a couple of places this can occur, the most likely
one to be behaving weirldy would be the one corresponding
to the debug level 12 message.
"packet for tcp W.X.Y.Z:N continue traversal as normal."
This occurs when a packet is not sceduled to a real server
and thus not added to the connection table. Usually this
happens because the packet is not for a virtual service.
But it could also occur if the scheduling (ip_vs_schedule()),
or lookup (ip_vs_conn_in_get(), ip_vs_service_get()) failed
mysteriously. You may want to look there.
Another attack might be to add some loging or accounting rules
to iptables before and after the packet should be accepted
by LVS.
--
Horms
|
