|
Hi,
I've been reading for the past few hours in the archives and on Google.
I would like to ask for some advice on the list, as I'm stuck trying to
configure LVS loadbalancing for SMTP.
My goals:
- Use one VIP for all incoming SMTP traffic (just one MX to prevent SPAM
on non-primary MXes)
- Balance incoming SMTP over a number of real servers (2 for now) with
wrr and 120 seconds persistance to enable SMTP connection thresholding
(spam)
- Use the same single VIP for outgoing SMTP sessions which are initiated
on real servers (single SPF record needed, easy for clients to implement
in DNS)
- Real servers and director require a working internet connection (RBL
checks and whatnot)
I've given LVS-DR a spin and that worked fine, but the issue with that
is that outgoing SMTP messages (e.g. sendmail queue) would be sent from
the various real server public source IPs. This is not what we want
(more difficult to manage), so I moved to LVS-NAT, but I'm stuck. Here's
the setup:
- External public gateway from upstream provider, say 80.1.1.1
- Director server has two NICs. eth0: 80.1.1.2, eth0:1 80.1.1.10 (VIP).
eth1: 192.168.254.1.
- Real servers are connected to Director eth1. Real server ips:
192.168.254.2, 192.168.254.3. Default gateway 192.168.254.1 (director).
ipvsadm rules:
-A -t 80.1.1.10:25 -s wrr
-a -t 80.1.1.10:25 -r 192.168.254.2:25 -m -w 1
-a -t 80.1.1.10:25 -r 192.168.254.3:25 -m -w 1
At this point I have not applied any iptables rules yet. Nothing has
been set in /proc/sys/net/ipv4/* on any of the servers.
Now obviously I need to configure forwarding so that it'll send out the
packets that are returning from RIP to director which then should NAT
them to VIP and send the packets to client. Here's where the problems begin.
echo 1 > /proc/sys/net/ipv4/ip_forward
When I've done this, I am getting traffic all the way: client >
(internet) > gateway 80.1.1.1 > VIP on director > RIP on real server >
back to director (ipvs?) > gateway 80.1.1.1 > (internet) > client.
But telnet session initiated from the client looks like:
HELO localhost
MAIL FROM: infernix@xxxxxxxxxxxx
RCPT TO: root@localhost
No text is returned from the server. At all.
Tshark output on the client shows:
390.642177 85.86.87.88 -> 80.1.1.10 TCP 57875 > smtp [SYN] Seq=0 Len=0
MSS=1460 TSV=258379259 TSER=0 WS=6
390.665292 80.1.1.10 -> 85.86.87.88 TCP smtp > 57875 [SYN, ACK] Seq=0
Ack=1 Win=5792 Len=0 MSS=1460 TSV=647361 TSER=258379259 WS=7
390.665345 85.86.87.88 -> 80.1.1.10 TCP 57875 > smtp [ACK] Seq=1 Ack=1
Win=5888 Len=0 TSV=258379282 TSER=647361
394.437218 85.86.87.88 -> 80.1.1.10 SMTP Command: HELO localhost
394.453271 80.1.1.10 -> 85.86.87.88 TCP smtp > 57875 [ACK] Seq=1 Ack=17
Win=5888 Len=0 TSV=647740 TSER=258383056
407.188602 85.86.87.88 -> 80.1.1.10 SMTP Command: MAIL FROM:
infernix@xxxxxxxxxxxx
407.204640 80.1.1.10 -> 85.86.87.88 TCP smtp > 57875 [ACK] Seq=1 Ack=51
Win=5888 Len=0 TSV=649015 TSER=258395814
414.636360 85.86.87.88 -> 80.1.1.10 SMTP Command: RCPT TO: root@localhost
414.652519 80.1.1.10 -> 85.86.87.88 TCP smtp > 57875 [ACK] Seq=1 Ack=76
Win=5888 Len=0 TSV=649760 TSER=258403266
415.688113 80.1.1.10 -> 85.86.87.88 SMTP Response: 554
mx-node01.test.local ESMTP not accepting messages
415.906143 80.1.1.10 -> 85.86.87.88 SMTP [TCP Retransmission] Response:
554 mx-node01.test.local ESMTP not accepting messages
416.346103 80.1.1.10 -> 85.86.87.88 SMTP [TCP Retransmission] Response:
554 mx-node01.test.local ESMTP not accepting messages
417.226323 80.1.1.10 -> 85.86.87.88 SMTP [TCP Retransmission] Response:
554 mx-node01.test.local ESMTP not accepting messages
418.986255 80.1.1.10 -> 85.86.87.88 SMTP [TCP Retransmission] Response:
554 mx-node01.test.local ESMTP not accepting messages
422.506412 80.1.1.10 -> 85.86.87.88 SMTP [TCP Retransmission] Response:
554 mx-node01.test.local ESMTP not accepting messages
429.546635 80.1.1.10 -> 85.86.87.88 SMTP [TCP Retransmission] Response:
554 mx-node01.test.local ESMTP not accepting messages
Tcpdump output can be found at the end of this email.
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP lb01.xxlwebhosting.nl:smtp wrr
-> 192.168.254.3:smtp Masq 1 0 0
-> 192.168.254.2:smtp Masq 1 1 0
ActiveConn = 1 so ipvsadm does see the connection. But somehow something
goes horribly wrong with the tcp session.
When I add masquerading according to the HOWTOS:
iptables -t nat -A POSTROUTING -s 192.168.254.0/24 -j MASQUERADE
The transmission is the same. The telnet session initiated from the
client does not see any SMTP replies whatsoever.
When I use SNAT instead of MASQUERADE the problem also remains the same.
I've looked at iptables CLUSTERIP already as an alternative but it is
not really suitable for our use and our intended scalability. So if
someone knows what to do, or has nightmares from SMTP load balancing
over LVS-NAT, I'd be happy to hear about it :)
Thanks!
infernix
Tcpdump -ln on the client:
03:38:37.132268 IP 85.86.87.88.34500 > 80.1.1.10.25: S
4080210697:4080210697(0) win 5840 <mss 1460,sackOK,timestamp 260339630
0,nop,wscale 6>
03:38:37.148458 IP 80.1.1.10.25 > 85.86.87.88.34500: S
2892912467:2892912467(0) ack 4080210698 win 5792 <mss
1460,sackOK,timestamp 843287 260339630,nop,wscale 7>
03:38:37.148478 IP 85.86.87.88.34500 > 80.1.1.10.25: . ack 1 win 92
<nop,nop,timestamp 260339646 843287>
03:38:38.182905 IP 80.1.1.10.25 > 85.86.87.88.34500: P 1:188(187) ack 1
win 46 <nop,nop,timestamp 843391 260339646>
03:38:38.402698 IP 80.1.1.10.25 > 85.86.87.88.34500: P 1:188(187) ack 1
win 46 <nop,nop,timestamp 843413 260339646>
03:38:38.842779 IP 80.1.1.10.25 > 85.86.87.88.34500: P 1:188(187) ack 1
win 46 <nop,nop,timestamp 843457 260339646>
03:38:39.722834 IP 80.1.1.10.25 > 85.86.87.88.34500: P 1:188(187) ack 1
win 46 <nop,nop,timestamp 843545 260339646>
03:38:41.482947 IP 80.1.1.10.25 > 85.86.87.88.34500: P 1:188(187) ack 1
win 46 <nop,nop,timestamp 843721 260339646>
03:38:43.239380 IP 85.86.87.88.34500 > 80.1.1.10.25: P 1:17(16) ack 1
win 92 <nop,nop,timestamp 260345740 843287>
03:38:43.255383 IP 80.1.1.10.25 > 85.86.87.88.34500: . ack 17 win 46
<nop,nop,timestamp 843898 260345740>
03:38:45.002989 IP 80.1.1.10.25 > 85.86.87.88.34500: P 1:188(187) ack 17
win 46 <nop,nop,timestamp 844073 260345740>
03:38:48.999949 IP 85.86.87.88.34500 > 80.1.1.10.25: P 17:51(34) ack 1
win 92 <nop,nop,timestamp 260351504 843287>
03:38:49.015968 IP 80.1.1.10.25 > 85.86.87.88.34500: . ack 51 win 46
<nop,nop,timestamp 844474 260351504>
03:38:52.043232 IP 80.1.1.10.25 > 85.86.87.88.34500: P 1:188(187) ack 51
win 46 <nop,nop,timestamp 844777 260351504>
03:38:54.519917 IP 85.86.87.88.34500 > 80.1.1.10.25: P 51:76(25) ack 1
win 92 <nop,nop,timestamp 260357027 843287>
03:38:54.535966 IP 80.1.1.10.25 > 85.86.87.88.34500: . ack 76 win 46
<nop,nop,timestamp 845026 260357027>
03:39:06.123817 IP 80.1.1.10.25 > 85.86.87.88.34500: P 1:188(187) ack 76
win 46 <nop,nop,timestamp 846185 260357027>
Same transmission, tcpdump -ln on the real server:
03:38:37.141789 IP 85.86.87.88.34500 > 192.168.254.2.25: S
4080210697:4080210697(0) win 5840 <mss 1460,sackOK,timestamp 260339630
0,nop,wscale 6>
03:38:37.141944 IP 192.168.254.2.25 > 85.86.87.88.34500: S
2892912467:2892912467(0) ack 4080210698 win 5792 <mss
1460,sackOK,timestamp 843287 260339630,nop,wscale 7>
03:38:37.157764 IP 85.86.87.88.34500 > 192.168.254.2.25: . ack 1 win 92
<nop,nop,timestamp 260339646 843287>
03:38:38.176052 IP 192.168.254.2.25 > 85.86.87.88.34500: P 1:188(187)
ack 1 win 46 <nop,nop,timestamp 843391 260339646>
03:38:38.395957 IP 192.168.254.2.25 > 85.86.87.88.34500: P 1:188(187)
ack 1 win 46 <nop,nop,timestamp 843413 260339646>
03:38:38.835972 IP 192.168.254.2.25 > 85.86.87.88.34500: P 1:188(187)
ack 1 win 46 <nop,nop,timestamp 843457 260339646>
03:38:39.715947 IP 192.168.254.2.25 > 85.86.87.88.34500: P 1:188(187)
ack 1 win 46 <nop,nop,timestamp 843545 260339646>
03:38:41.476015 IP 192.168.254.2.25 > 85.86.87.88.34500: P 1:188(187)
ack 1 win 46 <nop,nop,timestamp 843721 260339646>
03:38:43.248477 IP 85.86.87.88.34500 > 192.168.254.2.25: P 1:17(16) ack
1 win 92 <nop,nop,timestamp 260345740 843287>
03:38:43.248495 IP 192.168.254.2.25 > 85.86.87.88.34500: . ack 17 win 46
<nop,nop,timestamp 843898 260345740>
03:38:44.995956 IP 192.168.254.2.25 > 85.86.87.88.34500: P 1:188(187)
ack 17 win 46 <nop,nop,timestamp 844073 260345740>
03:38:49.008820 IP 85.86.87.88.34500 > 192.168.254.2.25: P 17:51(34) ack
1 win 92 <nop,nop,timestamp 260351504 843287>
03:38:49.008850 IP 192.168.254.2.25 > 85.86.87.88.34500: . ack 51 win 46
<nop,nop,timestamp 844474 260351504>
03:38:52.035960 IP 192.168.254.2.25 > 85.86.87.88.34500: P 1:188(187)
ack 51 win 46 <nop,nop,timestamp 844777 260351504>
03:38:54.528630 IP 85.86.87.88.34500 > 192.168.254.2.25: P 51:76(25) ack
1 win 92 <nop,nop,timestamp 260357027 843287>
03:38:54.528692 IP 192.168.254.2.25 > 85.86.87.88.34500: . ack 76 win 46
<nop,nop,timestamp 845026 260357027>
03:39:06.115981 IP 192.168.254.2.25 > 85.86.87.88.34500: P 1:188(187)
ack 76 win 46 <nop,nop,timestamp 846185 260357027>
On the director, on the internal interface:
03:38:37.140435 IP 85.86.87.88.34500 > 192.168.254.2.25: S
4080210697:4080210697(0) win 5840 <mss 1460,sackOK,timestamp 260339630
0,nop,wscale 6>
03:38:37.140620 IP 192.168.254.2.25 > 85.86.87.88.34500: S
2892912467:2892912467(0) ack 4080210698 win 5792 <mss
1460,sackOK,timestamp 843287 260339630,nop,wscale 7>
03:38:37.156572 IP 85.86.87.88.34500 > 192.168.254.2.25: . ack 1 win 92
<nop,nop,timestamp 260339646 843287>
03:38:37.163400 IP 192.168.254.2.36614 > 85.86.87.88.113: S
2882687399:2882687399(0) win 5840 <mss 1460,sackOK,timestamp 843289
0,nop,wscale 7>
03:38:37.179277 IP 85.86.87.88.113 > 192.168.254.2.36614: R 0:0(0) ack
2882687400 win 0
03:38:38.174949 IP 192.168.254.2.25 > 85.86.87.88.34500: P 1:188(187)
ack 1 win 46 <nop,nop,timestamp 843391 260339646>
03:38:38.394842 IP 192.168.254.2.25 > 85.86.87.88.34500: P 1:188(187)
ack 1 win 46 <nop,nop,timestamp 843413 260339646>
03:38:38.834849 IP 192.168.254.2.25 > 85.86.87.88.34500: P 1:188(187)
ack 1 win 46 <nop,nop,timestamp 843457 260339646>
03:38:39.714857 IP 192.168.254.2.25 > 85.86.87.88.34500: P 1:188(187)
ack 1 win 46 <nop,nop,timestamp 843545 260339646>
03:38:41.474924 IP 192.168.254.2.25 > 85.86.87.88.34500: P 1:188(187)
ack 1 win 46 <nop,nop,timestamp 843721 260339646>
03:38:43.247261 IP 85.86.87.88.34500 > 192.168.254.2.25: P 1:17(16) ack
1 win 92 <nop,nop,timestamp 260345740 843287>
03:38:43.247369 IP 192.168.254.2.25 > 85.86.87.88.34500: . ack 17 win 46
<nop,nop,timestamp 843898 260345740>
03:38:44.994848 IP 192.168.254.2.25 > 85.86.87.88.34500: P 1:188(187)
ack 17 win 46 <nop,nop,timestamp 844073 260345740>
03:38:49.007605 IP 85.86.87.88.34500 > 192.168.254.2.25: P 17:51(34) ack
1 win 92 <nop,nop,timestamp 260351504 843287>
03:38:49.007737 IP 192.168.254.2.25 > 85.86.87.88.34500: . ack 51 win 46
<nop,nop,timestamp 844474 260351504>
03:38:52.034854 IP 192.168.254.2.25 > 85.86.87.88.34500: P 1:188(187)
ack 51 win 46 <nop,nop,timestamp 844777 260351504>
03:38:54.527393 IP 85.86.87.88.34500 > 192.168.254.2.25: P 51:76(25) ack
1 win 92 <nop,nop,timestamp 260357027 843287>
03:38:54.527553 IP 192.168.254.2.25 > 85.86.87.88.34500: . ack 76 win 46
<nop,nop,timestamp 845026 260357027>
03:39:06.114880 IP 192.168.254.2.25 > 85.86.87.88.34500: P 1:188(187)
ack 76 win 46 <nop,nop,timestamp 846185 260357027>
And finally on the director, the external interface:
03:38:37.140366 IP 85.86.87.88.34500 > 80.1.1.10.25: S
4080210697:4080210697(0) win 5840 <mss 1460,sackOK,timestamp 260339630
0,nop,wscale 6>
03:38:37.140629 IP 80.1.1.10.25 > 85.86.87.88.34500: S
2892912467:2892912467(0) ack 4080210698 win 5792 <mss
1460,sackOK,timestamp 843287 260339630,nop,wscale 7>
03:38:37.156565 IP 85.86.87.88.34500 > 80.1.1.10.25: . ack 1 win 92
<nop,nop,timestamp 260339646 843287>
03:38:37.163409 IP 83.96.151.178.36614 > 85.86.87.88.113: S
2882687399:2882687399(0) win 5840 <mss 1460,sackOK,timestamp 843289
0,nop,wscale 7>
03:38:37.179271 IP 85.86.87.88.113 > 83.96.151.178.36614: R 0:0(0) ack
2882687400 win 0
03:38:38.175004 IP 80.1.1.10.25 > 85.86.87.88.34500: P 1:188(187) ack 1
win 46 <nop,nop,timestamp 843391 260339646>
03:38:38.394851 IP 80.1.1.10.25 > 85.86.87.88.34500: P 1:188(187) ack 1
win 46 <nop,nop,timestamp 843413 260339646>
03:38:38.834858 IP 80.1.1.10.25 > 85.86.87.88.34500: P 1:188(187) ack 1
win 46 <nop,nop,timestamp 843457 260339646>
03:38:39.714920 IP 80.1.1.10.25 > 85.86.87.88.34500: P 1:188(187) ack 1
win 46 <nop,nop,timestamp 843545 260339646>
03:38:41.474970 IP 80.1.1.10.25 > 85.86.87.88.34500: P 1:188(187) ack 1
win 46 <nop,nop,timestamp 843721 260339646>
03:38:43.247243 IP 85.86.87.88.34500 > 80.1.1.10.25: P 1:17(16) ack 1
win 92 <nop,nop,timestamp 260345740 843287>
03:38:43.247379 IP 80.1.1.10.25 > 85.86.87.88.34500: . ack 17 win 46
<nop,nop,timestamp 843898 260345740>
03:38:44.994869 IP 80.1.1.10.25 > 85.86.87.88.34500: P 1:188(187) ack 17
win 46 <nop,nop,timestamp 844073 260345740>
03:38:49.007588 IP 85.86.87.88.34500 > 80.1.1.10.25: P 17:51(34) ack 1
win 92 <nop,nop,timestamp 260351504 843287>
03:38:49.007748 IP 80.1.1.10.25 > 85.86.87.88.34500: . ack 51 win 46
<nop,nop,timestamp 844474 260351504>
03:38:52.034881 IP 80.1.1.10.25 > 85.86.87.88.34500: P 1:188(187) ack 51
win 46 <nop,nop,timestamp 844777 260351504>
03:38:54.527372 IP 85.86.87.88.34500 > 80.1.1.10.25: P 51:76(25) ack 1
win 92 <nop,nop,timestamp 260357027 843287>
03:38:54.527564 IP 80.1.1.10.25 > 85.86.87.88.34500: . ack 76 win 46
<nop,nop,timestamp 845026 260357027>
03:39:06.114904 IP 80.1.1.10.25 > 85.86.87.88.34500: P 1:188(187) ack 76
win 46 <nop,nop,timestamp 846185 260357027>
|
