|
The problem arose with JDFPalmer, who is using a 3-tier LVS
(realservers are squids). We've been doing most of this offline.
Below is a lengthly explanation of the problem, explaning
the logic, in case someone can spot the error. My question
is at the end.
The problem:
Part of securifying a realserver is to give it routes for only
the expected packets. All other packets will not have routes and
will be dropped. Thus in a normal (2-tier) LVS, eg serving http,
the realservers will be sending packets from the VIP to 0/0 and
packets from the RIP will be sent on the RIP network only locally.
Thus the realserver has these routes
from RIP to RIP_network
from VIP to 0/0 via SERVER_GW
There is no default gw and in particular no default gw for packets
from the RIP since there are no clients on the RIP that need
to talk to 0/0.
In a 3-tier LVS, clients on the realservers need to talk to
machines which may not be on the RIP network. If the realserver is
in an LVS serving DNS, then a DNS client will need to contact
root DNS machines on 0/0:53. If the realserver is a squid,
clients on the squid will need to contact webservers on 0/0:80.
The realservers may have clients that talk to a backend database.
If the 3rd tier servers are not on the RIP network, then
a route needs to be added for packets from the RIP to
to the chosen port on the 3rd tier server.
from RIP to 0/0:80 via SERVER_GW
Again there is no need for a default route for the realserver.
Routing is usually by src or dst address. However we can route
by ports using fwmark.
#mark the packet (here CHAIN=OUTPUT)
iptables -t mangle -A ${CHAIN} -p tcp -s ${RIP}/32 -d 0/0 --dport ${my_PORT} -j
MARK --set-mark 1
#log the packets for your sanity (thanks to Horms for syntax)
iptables -t mangle -A ${CHAIN} -m mark --mark 1 -j LOG --log-level DEBUG
--log-prefix "fwmark 1: "
#add a rule for packets with the fwmark (routes are in table 3_TIER)
ip rule add prio 99 from ${RIP} fwmark 1 table 3_TIER
#provide a route in table 3_TIER
ip route add default via ${my_GW} dev ${my_DEV} table 3_TIER
This sets up routing to 0/0:${my_PORT} for packets from the RIP
However if you test this with my_PORT == telnet
realserver:/etc/lvs# telnet outside_machine
Trying 192.168.2.254...
telnet: Unable to connect to remote host: Network is unreachable
The problem appears to be
realserver:/etc/lvs# ip route get outside_machine
RTNETLINK answers: Network is unreachable
ie with no default route, there is no route to outside_machine
available, when the host looks up its routing tables. Of course
once a packet to outside_machine:telnet is generated, there
will be a route for it, since that packet will have a fwmark on
it, but we never get that far.
Lets add a default route (the following steps work using
any machine on the RIP network as the default gw, let's us
the director_IP)
route add default gw DIP
Now we can connect
realserver:/etc/lvs# telnet outside_machine
Trying 192.168.2.254...
Connected to outside_machine
Escape character is '^]'.
and we have a route
realserver:/etc/lvs# ip route get outside_machine
outside_IP via DIP dev eth0 src RIP
tcpdump on the default gw does not see any packets.
tcpdump on the SERVER_GW sees the connection, showing
that the packets are being routed by the fwmark.
You can check the fwmark routing, by ftp'ing to the
outside_machine. The ftp packets will not be fwmarked
and by running tcpdump, these packets are seen on the
default gw and not the SERVER_GW (the opposite of
what happens for the telnet packets), confirming
that the ftp packets are not being fwmark'ed.
Here's the default route
realserver:/etc/lvs# ip route show table main
192.168.1.0/24 dev eth0 scope link
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.11
127.0.0.0/8 dev lo scope link
default via 192.168.1.9 dev eth0
However for all the tests above, all along, there was a
route to 0/0 from RIP in place, that wasn't being
used.
Here's the rules
realserver:/etc/lvs# ip rule show
0: from all lookup local
99: from 192.168.2.110 lookup VIP
100: from 192.168.1.11 to 192.168.1.0/24 lookup RIP
100: from 192.168.1.11 lookup RIP
32766: from all lookup main
32767: from all lookup 253
The 2nd rule of priority 100 should handle packets from
the RIP (x.x.x.11) and indicates that the routes are
in table RIP. Here's the routes in table RIP
realserver:/etc/rc.d# ip route show table RIP
192.168.1.0/24 dev eth0 scope link src 192.168.1.11
default via 192.168.1.9 dev eth0
There's a default route there (it seems to me)!
Question: why doesn't the realserver use this default
route for packets from the RIP? Why does it need a default
route in table main?
Joe
--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@xxxxxxx ph# 919-541-0007, RTP, NC, USA
|
