|
On Thu, 14 Oct 2004, Horms wrote:
> You should get a "not hit" for all packets that come through the
> box that are not related to LVS, and a "hit" for all packets that are.
> If you think that packets are getting "not hit" when they should get
> "hit" then try looking at ipvsadm -Lcn, this should give you a dump
> of the connections that LVS is trying to match incoming packets against.
That might be quite difficult, given the rate of traffic through these boxes!
Details:
Fedora Core 2
kernel 2.6.8-1.521 (recompiled/repackaged to switch on IP_VS_DEBUG)
keepalived 1.1.7
ipvsadm 1.24
ip_vs version 1.2.0 (size=4096) from kernel package
LVS-NAT
persistence 120 sec
This should make it look fairly obvious. The following sanitised logfile shows
a connection from a remote NMS poller failing to get through correctly. I'll
annotate it in sections.
00:28:16 IPVS: lookup/in TCP $NMS_ADDR:52059->$VIP_ADDR:80 hit
00:28:16 IPVS: Incoming packet: TCP $NMS_ADDR:52059->$VIP_ADDR:80
00:28:16 IPVS: TCP input [S...] $RS2_ADDR:80->$NMS_ADDR:0 state:
NONE->SYN_RECV cnt:2
00:28:16 IPVS: After DNAT: TCP $NMS_ADDR:52059->$RS2_ADDR:80
OK so far, connection arrives to a previously stored connection from 60 or so
seconds ago. Packet gets sent off to the RS, but then it comes back and...
00:28:16 IPVS: lookup/out TCP $RS2_ADDR:80->$NMS_ADDR:52059 not hit
00:28:16 IPVS: packet continues traversal as normal: TCP
$RS2_ADDR:80->$NMS_ADDR:52059
00:28:16 IPVS: lookup/out TCP $NMS_ADDR:52059->$RS2_ADDR:80 not hit
00:28:16 IPVS: packet continues traversal as normal: TCP
$NMS_ADDR:52059->$RS2_ADDR:80
...the packets aren't getting rewritten by the ip_vs code, so the client isn't
seeing them. It then sends another SYN which gets sent in to the RS:
00:28:19 IPVS: lookup/in TCP $NMS_ADDR:52059->$VIP_ADDR:80 hit
00:28:19 IPVS: Incoming packet: TCP $NMS_ADDR:52059->$VIP_ADDR:80
00:28:19 IPVS: After DNAT: TCP $NMS_ADDR:52059->$RS2_ADDR:80
...but again it misses the bucket and gets passed to the netfilter code
instead of ip_vs:
00:28:19 IPVS: lookup/out TCP $RS2_ADDR:80->$NMS_ADDR:52059 not hit
00:28:19 IPVS: packet continues traversal as normal: TCP
$RS2_ADDR:80->$NMS_ADDR:52059
00:28:19 IPVS: lookup/out TCP $NMS_ADDR:52059->$RS2_ADDR:80 not hit
00:28:19 IPVS: packet continues traversal as normal: TCP
$NMS_ADDR:52059->$RS2_ADDR:80
...and so on. I have miles and miles of this happening occasionally.
I hope that I haven't missed anything obvious in my interpretation of the
logs.
Any hints or pointers greatly received!
Graeme
|
