|
Hi Mark,
Excellent problem report!
*takes a bow*
I here by dub thee once ... I dub thee twice ... I dub thee Sir LVS Bug
Reporter, you may rise and go forth. Will you accept from Us this honor,
and will you swear fealty to this, Our order of LVS?
# ipvsadm --list -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 100.1.1.2:25 wlc
-> 120.1.1.1:25 Tunnel 1 0 0
-> 120.1.1.2:25 Tunnel 1 0 0
iptables has no rules and is default-to-accept. There is no firewall
in front of the box.
Mail server 1 (120.1.1.1)
=================
relevant iptables rules:
$IPTABLES -A INPUT -i eth0 -s 100.1.1.2 -p ipencap -j ACCEPT
$IPTABLES -A INPUT -i tunl0 -p tcp --dport smtp -j ACCEPT
Why do you need those rules if you're not having any netfilter rules
and a ACCEPT policy?
The mailservers _do_ have firewall rules, its just the new load balancer
that does not. However, I don't think this is a firewall issue as
dropped packets still show up in tcpdump, and also I am able to telnet
directly to port 25 on both mailservers from the new (broken) load
balancer.
Not necessarily but this is hopefully not hitting you. Depending on the
kernel, netfilter in the PREROUTING table handling could drop the skb
before tcpdump would get a skb->clone() of it.
I'm a bit confused by your obfuscation technique :), what's the
designation for the servers regarding the obfuscated IP ranges in
100.x.x.x, the 120.x.x.x, the 130.x.x.x and the 140.x.x.x?
140: your test machine
130: working LVS tunnel
120: RS (mail server)
100: new (non-functional) LVS tunnel
Is my observation correct?
Yes, sorry for the obfuscation - I was all for just pasting the real IPs
but my manager refused to let me ;)
That's very noble of him.
So this works perfectly, as shown above, which actually indicates that
you have at one point got LVS to work. Sidenote: Your LVS seems to be
a bit out of sync regarding time; otherwise your trace looks odd.
Yes, it was actually someone else who got it working before, and he is
far too busy to assist me with the new one :)
This is the part where your manager should probably call him back :).
Now, if I try the same thing but telnet to 100.1.1.2:25 (the new load
balancer), the connection times out. tcpdumps show:
Care to show the whole ipvsadm -L -n output? Or is the one above
representative enough to display the problem?
Didn't I paste this above? --list is the same as -L I believe, at least
the output is no different..
Sure, but there was no indication to which state of your test conducts
your quoted output pertained to. When you say "the new load balancer"
above, you do not mean a physically different machine to the "old load
balancer", do you?
Mar 29 11:01:48 dev1 kernel: IPVS: lookup/in TCP
140.1.1.1:4042->100.1.1.2:25 not hit
Mar 29 11:01:48 dev1 kernel: IPVS: lookup service: fwm 0 TCP
100.1.1.2:25 hit
Now this is very very weird. The normal TCP service lookup did not
succeed, although it should have, but the FWM TCP service lookup did.
Are you sure that:
a) You have cleanly shutdown (rmmod ip_vs if necessary) IPVS between
the functional and the non-functional test conduct?
ipvs is compiled statically into the kernel, so how would I shut it
down? I had no idea it was necessary to shut it down and bring it back
up, although I have rebooted the server a couple of times which I am
sure would accomplish the same effect.
Absolutely. The point is that the template entries are not flushed when
you simply remove the destination servers from the kernel, only detached.
b) You have no iptables or iproute2 rules indicating firewall marks?
# iptables --list
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
That's not all :). You've only shown the filter table, but I'm also
interested in the mangle table.
# iproute2
bash: iproute2: command not found
It's the ip command output from the iproute2 framework I was looking for.
This is the successor to ifconfig and route and netstat and whatnot. The
Linux world decided at one point in its history (around 1999) that
ifconfig/route/other networking setup tools are not appropriate anymore
and replaced them with the iproute2 framework. Unfortunately the guy who
started all this is a bloody genius and as such did two things: a)
completely forgot to document it, b) never told anyone outside the
kernel community about this, for years. So, if you find time, invoke
"man ip" on a recent enough Linux distribution of your choice.
I built this server myself and never did anything with iproute2.. so
I'm guessing the answer is no. Although I do believe Debian is evil and
so I guess it could have possibly done this itself behind my back.
Debian people hopefully do not have evil intentions, however could pass
along the output of:
ip rule show
ip route show
ip link show
ip addr show
grep -r . /proc/sys/net/ipv4/conf/*
c) You have no port 0 service set up?
Definitely not
I see. Not! :)
Mar 29 11:01:48 dev1 kernel: IPVS: ip_vs_wlc_schedule(): Scheduling...
Mar 29 11:01:48 dev1 kernel: IPVS: WLC: server 120.1.1.1:25
activeconns 0 refcnt 1 weight 1 overhead 0
Mar 29 11:01:48 dev1 kernel: IPVS: Bind-dest TCP c:140.1.1.1:4042
v:100.1.1.2:25 d:120.1.1.1:25 fwd:T s:0 conn->flags:182
conn->refcnt:1 dest->refcnt:2
Mar 29 11:01:48 dev1 kernel: IPVS: Schedule fwd:T c:140.1.1.1:4042
v:100.1.1.2:25 d:120.1.1.1:25 conn->flags:1C2 conn->refcnt:2
This looks like it would happily send it.
Mar 29 11:01:48 dev1 kernel: IPVS: TCP input [S...]
120.1.1.1:25->140.1.1.1:4042 state: NONE->SYN_RECV conn->refcnt:2
Ok, we do the state transition indicating that we've allocated the
connection structure for the hash table entry.
Mar 29 11:01:51 dev1 kernel: IPVS: lookup/in TCP
140.1.1.1:4042->100.1.1.2:25 hit
Second SYN as seen in your non-functional tcpdump trace.
Mar 29 11:01:57 dev1 kernel: IPVS: lookup/in TCP
140.1.1.1:4042->100.1.1.2:25 hit
Third SYN as seen in your non-functional tcpdump trace.
Mar 29 11:02:04 dev1 kernel: IPVS: Unbind-dest TCP c:140.1.1.1:4039
v:100.1.1.2:25 d:120.1.1.2:25 fwd:T s:3 conn->flags:182
conn->refcnt:1 dest->refcnt:2
This is not belonging to the trace above since it's port 4039 which
must have been a test performed before you took the trace. Most likely
this one ran into the normal 60 sec timeout.
I really am at a loss as to why this doesn't work, the debug log
seems to show IPVS passing traffic to mail 1 (120.1.1.1) however the
tcpdump for that server shows absolutely nothing. If anyone can
point me in the right direction here I would be very grateful.
Can you show your routing information on your LVS? As well as the tun*
device configuration in the proc-fs?
Sure, by LVS i'm going to assume you mean the broken load balancer.
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
100.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
0.0.0.0 100.1.1.254 0.0.0.0 UG 0 0 0
eth0
Could you please send me the iproute2 related output, as indicated
above? route -n does not show all the routing entries on a Linux box.
# find /proc |grep tun
Sidenote: You might not call that command like that too often on your
productive server. I've seen nasty kernel OOPS more than once after such
a stat()-intensive command.
This is odd, tunl0 does exist:
# ifconfig tunl0
tunl0 Link encap:IPIP Tunnel HWaddr
NOARP MTU:1480 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Sure, but it's not activated. Could you by any chance call following
command on your box?
ip link set dev tunl0 up
Don't know why its absent from /proc.
Since there are no IFF_RUNNING|IFF_UP flags set, there's no point in
setting any entries for this virtual device in the proc-fs.
Thanks again for your assistance,
Always when receiving such nice bug reports,
Roberto Nibali, ratz
--
echo
'[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc
|
