|
From the next version of the HOWTO. Comments gladly accepted.
Joe
7.1.2 One network VS-NAT
The disadvantage of the 2 network VS-NAT is that the real-servers
are not able to connect to machines in the network of the VIP.
You couldn't make a VS-NAT setup out of machines already on
your LAN, which were also required for other purposes to stay
on the LAN network.
Here's a 1 network VS-NAT LVS.
________
| |
| client |
|________|
CIP=192.168.1.254
|
(router)
|
__________ |
| | | VIP=192.168.1.110 (eth0:110)
| director |---| DIP=192.168.1.1 (eth0)
|__________| | DIIP=192.168.1.9 (eth0:9)
|
|
------------------------------------
| | |
| | |
RIP1=192.168.1.2 RIP2=192.168.1.3 RIP3=10.1.1.4 (all eth0)
_____________ _____________ _____________
| | | | | |
| real-server | | real-server | | real-server |
|_____________| |_____________| |_____________|
The problem:
A return packet from the real-server (with address RIP->CIP)
will be sent to the real-server's default gw (the director).
ICMP redirects will be sent from the director telling the
real-server of the better route directly to the client.
The real-server will then send the packet directly to the
client and it will not be demasqueraded by the director.
The client will get a reply from the RIP rather than the VIP
and the connection will hang.
The cure:
In the previous HOWTO I said that initial attempts to handle this
by turning off redirects had not worked. The problem appears now
to be solved.
Thanks to michael_e_brown@xxxxxxxx and Julian ja@xxxxxx
for sorting this out.
To get a VS-NAT LVS to work on one network -
1. On the director, turn off icmp redirects
director:/etc/lvs# echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
director:/etc/lvs# echo 0 > /proc/sys/net/ipv4/conf/default/send_redirects
director:/etc/lvs# echo 0 > /proc/sys/net/ipv4/conf/eth0/send_redirects
(Note: eth0 may be eth1 etc, on your machine).
2. Make the director the default and only route for outgoing packets.
You will probably have set the routing on the real-server up like this
realserver:/etc/lvs# netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 director 0.0.0.0 UG 0 0 0 eth0
Note the route to 192.168.1.0/24. This allows the real-server to send packets
to the client by just putting them out on eth0, where the client will
pick them up directly (without being demasqueraded) and the LVS will
not work.
Remove the route to 192.168.1.0/24.
realserver:/etc/lvs#route del -net 192.168.1.0 netmask 255.255.255.0 dev eth0
This will leave you with
realserver:/etc/lvs# netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 director 0.0.0.0 UG 0 0 0 eth0
The VS-NAT LVS now works. If LVS is forwarding telnet, you can
telnet from the client to the VIP and connect to the real-server.
You can ping from the client to the real-server.
You can also connect _directly_ to services on the real-server not
being forwarded by LVS (in this case e.g. ftp).
You can no longer connect directly to the real-server for services
being forwarded by the LVS. Here telnet ports are not being rewritten
by the LVS (ie telnet->telnet).
client:~# telnet realserver
Trying 192.168.1.11...
^C
(i.e. connection hangs)
Here's tcpdump on the director (which can see all packets).
The client initiates telnet. `netstat -a` on the client
shows a SYN_SENT from port 4121.
director:/etc/lvs# tcpdump
tcpdump: listening on eth0
16:37:04.655036 realserver.telnet > client.4121: S 354934654:354934654(0) ack
1183118745 win 32120 <mss 1460,sackOK,timestamp 111425176[|tcp]> (DF)
16:37:04.655284 director > realserver: icmp: client tcp port 4 121 unreachable
[tos 0xc0]
(repeats every second until I kill telnet on client)
I don't know why I don't see the connect request from client->real-server.
The first packet I see is the ack from the real-server,
which will be forwarded via the director.
The director will rewrite the ack to be from the director.
The client will not accept an ack to port 4121 from director:telnet.
Various debugging techniques (in discussion with Julian) in
case you want to know more:
The routes added with the route command go into the kernel FIB
(Forwarding information base) route table. The contents are
displayed with the route (or netstat -a) command.
Following an icmp redirect, the route updates go into the kernel's
route cache (route -C).
Here's the route cache on the real-server before any packets are sent.
(You can flush the route cache with
echo 1 > /proc/sys/net/ipv4/route/flush
or
ip route flush cache
)
realserver:/etc/rc.d# route -C
Kernel IP routing cache
Source Destination Gateway Flags Metric Ref Use Iface
realserver director director 0 1 0 eth0
director realserver realserver il 0 0 9 lo
With icmp redirects enabled on the director, repeatedly running
traceroute to the client shows the routes changing from 2 hops to 1 hop.
realserver:/etc/rc.d# traceroute client
traceroute to client (192.168.1.254), 30 hops max, 40 byte packets
1 director (192.168.1.9) 0.932 ms 0.562 ms 0.503 ms
2 client (192.168.1.254) 1.174 ms 0.597 ms 0.571 ms
realserver:/etc/rc.d# traceroute client
traceroute to client (192.168.1.254), 30 hops max, 40 byte packets
1 director (192.168.1.9) 0.72 ms 0.581 ms 0.532 ms
2 client (192.168.1.254) 0.845 ms 0.559 ms 0.5 ms
realserver:/etc/rc.d# traceroute client
traceroute to client (192.168.1.254), 30 hops max, 40 byte packets
1 client (192.168.1.254) 0.69 ms * 0.579 ms
Although the route command shows no change in the FIB, the route cache
has changed. (The new route of interest is bracketted by >< signs.)
realserver:/etc/rc.d# route -C
Kernel IP routing cache
Source Destination Gateway Flags Metric Ref Use Iface
client realserver realserver l 0 0 8 lo
realserver realserver realserver l 0 0 1038 lo
realserver director director 0 1 138 eth0
>realserver client client 0 0 6 eth0<
director realserver realserver l 0 0 9 lo
director realserver realserver l 0 0 168 lo
Packets to the client now go directly to the client instead of via
the director.
It takes about 10mins for the client's route cache to expire
(experimental result). The timeouts may be in /proc/sys/net/ipv4/route/gc_*,
but their location and values are well encrypted in the sources :).
Here's the route cache after 10mins.
realserver:/etc/rc.d# route -C
Kernel IP routing cache
Source Destination Gateway Flags Metric Ref Use Iface
realserver realserver realserver l 0 0 1049 lo
realserver director director 0 1 139 eth0
director realserver realserver l 0 0 0 lo
director realserver realserver l 0 0 236 lo
There are no routes to the client anymore. Checking with traceroute,
shows that 2 hops are initially required to get to the client (i.e.
the routing cache has reverted to using the director as the route
to the client). After 2 iterations, icmp redirects route the packets
directly to the client again.
realserver:/etc/rc.d# traceroute client
traceroute to client (192.168.1.254), 30 hops max, 40 byte packets
1 director (192.168.1.9) 0.908 ms 0.572 ms 0.537 ms
2 client (192.168.1.254) 1.179 ms 0.6 ms 0.577 ms
realserver:/etc/rc.d# traceroute client
traceroute to client (192.168.1.254), 30 hops max, 40 byte packets
1 director (192.168.1.9) 0.695 ms 0.552 ms 0.492 ms
2 client (192.168.1.254) 0.804 ms 0.55 ms 0.502 ms
realserver:/etc/rc.d# traceroute client
traceroute to client (192.168.1.254), 30 hops max, 40 byte packets
1 client (192.168.1.254) 0.686 ms 0.533 ms *
If you now turn off icmp redirects on the director.
director:/etc/lvs# echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
director:/etc/lvs# echo 0 > /proc/sys/net/ipv4/conf/default/send_redirects
director:/etc/lvs# echo 0 > /proc/sys/net/ipv4/conf/eth0/send_redirects
Checking routes on the real-server -
realserver:/etc/lvs# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 director 0.0.0.0 UG 0 0 0 eth0
nothing changed here.
Flush the kernel routing table and show the kernel routing table -
realserver:/etc/lvs# ip route flush cache
realserver:/etc/lvs# route -C
Kernel IP routing cache
Source Destination Gateway Flags Metric Ref Use Iface
realserver director director 0 1 0 eth0
director realserver realserver l 0 0 1 lo
There are no routes to the client.
Now when you send packet to the client, the route stays via the director
needing 2 hops to get to the client. There are no one hop packets
to the client.
realserver:/etc/rc.d# traceroute client
traceroute to client (192.168.1.254), 30 hops max, 40 byte packets
1 director (192.168.1.9) 0.951 ms 0.56 ms 0.491 ms
2 client (192.168.1.254) 0.76 ms 0.599 ms 0.574 ms
realserver:/etc/rc.d# traceroute client
traceroute to client (192.168.1.254), 30 hops max, 40 byte packets
1 director (192.168.1.9) 0.696 ms 0.562 ms 0.583 ms
2 client (192.168.1.254) 0.62 ms 0.603 ms 0.576 ms
realserver:/etc/rc.d# traceroute client
traceroute to client (192.168.1.254), 30 hops max, 40 byte packets
1 director (192.168.1.9) 0.692 ms * 0.599 ms
2 client (192.168.1.254) 0.667 ms 0.603 ms 0.579 ms
realserver:/etc/rc.d# traceroute client
traceroute to client (192.168.1.254), 30 hops max, 40 byte packets
1 director (192.168.1.9) 0.689 ms 0.558 ms 0.487 ms
2 client (192.168.1.254) 0.61 ms 0.63 ms 0.567 ms
realserver:/etc/rc.d# traceroute client
traceroute to client (192.168.1.254), 30 hops max, 40 byte packets
1 director (192.168.1.9) 0.705 ms 0.563 ms 0.526 ms
2 client (192.168.1.254) 0.611 ms 0.595 ms *
realserver:/etc/rc.d# traceroute client
traceroute to client (192.168.1.254), 30 hops max, 40 byte packets
1 director (192.168.1.9) 0.706 ms 0.558 ms 0.535 ms
2 client (192.168.1.254) 0.614 ms 0.593 ms 0.573 ms
The kernel route cache
realserver:/etc/rc.d# route -C
Kernel IP routing cache
Source Destination Gateway Flags Metric Ref Use Iface
client realserver realserver l 0 0 17 lo
realserver realserver realserver l 0 0 2 lo
realserver director director 0 1 0 eth0
>realserver client director 0 0 35 eth0<
director realserver realserver l 0 0 16 lo
director realserver realserver l 0 0 63 lo
shows the the only route to the client (labelled with >< ) is via the director.
#end
--
Joseph Mack mack@xxxxxxxxxxx
|
