LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: [lvs-users] IPVSADM/IPTables question

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: [lvs-users] IPVSADM/IPTables question
From: "Gary W. Smith" <gary@xxxxxxxxxxxxxxx>
Date: Thu, 13 Sep 2007 08:27:39 -0700
Joe, 
 
I think I have it.  I missed the part about deleting the local route when using 
masquerade.  When I do that I can access it from both environments.
 
On both of the realservers, I ran "route del -net 10.10.17.0 netmask 
255.255.255.0" and kept my rules with masquerade.
 
I can now see traffic in ipvsadm as expected.
 
[root@hsdevfiw01 ~]# ipvsadm -l -n --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
TCP  10.0.16.45:80                     120      629      616    64004   278061
  -> 10.10.17.5:80                      64      313      293    35578    55198
  -> 10.10.17.6:80                      56      316      323    28426   222863
 
One last question though, for local traffic, will this put much more of a load 
on the firewall?  We do have other services that we will use LVS on but these 
are internal only traffic, which I assume we could do direct route but since we 
have dropped the local route, I assume these will loop back to the firewall for 
the first routing requests from the web servers, until the direct route is 
established.
 
Any ideas/thoughts/suggestions?
 
Gary

________________________________

From: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx on behalf of Gary W. Smith
Sent: Thu 9/13/2007 8:03 AM
To: LinuxVirtualServer.org users mailing list.; LinuxVirtualServer.org users 
mailing list.
Subject: Re: [lvs-users] IPVSADM/IPTables question



Joe,

I have included a configuration from my test environment below.  I've had mixes 
success and am probably missing something simple.  I can reliably hit the LVS 
machines from anything inside the test network, but anything outside the 
firewall box (which is currently wide open) doesn't work.  Below are all of th 
configuration's including actual IP's.

I'm unsure if I'm running the routes right.  One thing I read said delete the 
local routes and to force everything out the firewall.  Somewhere I read that I 
should use ipip (someones are all google notes, with no response to if it 
worked or not).

I've also tried using -m instead of -g, but that didn't seem to make a 
difference.

Can you take a look at the config and let me know if something stands out as 
broken?

Thanks,

Gary

Test configuration:

4 boxes, configured as follows:

Box 1, firewall/director.
[root@hsdevfiw01 sysconfig]# ip addr
1: lo: <LOOPBACK,UP,10000> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:40:05:86:c8:7f brd ff:ff:ff:ff:ff:ff
    inet 10.10.17.254/24 brd 10.10.17.255 scope global eth0
    inet6 fe80::240:5ff:fe86:c87f/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:02:a5:59:42:b2 brd ff:ff:ff:ff:ff:ff
    inet 10.0.16.44/24 brd 10.0.16.255 scope global eth1
    inet 10.0.16.45/24 scope global secondary eth1
    inet6 fe80::202:a5ff:fe59:42b2/64 scope link
       valid_lft forever preferred_lft forever
      
[root@hsdevfiw01 sysconfig]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.16.0       0.0.0.0         255.255.255.0   U     0      0        0 eth1
10.10.17.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth1
0.0.0.0         10.0.16.254     0.0.0.0         UG    0      0        0 eth1    
  

[root@hsdevfiw01 sysconfig]# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.16.45:http wlc
  -> 10.10.17.6:http              Route   100    0          0        
  -> 10.10.17.5:http              Route   100    0          0        

[root@hsdevfiw01 sysconfig]# iptables-save      
# Generated by iptables-save v1.3.1 on Thu Sep 13 06:29:46 2007
*nat
:PREROUTING ACCEPT [1302:307491]
:POSTROUTING ACCEPT [6:324]
:OUTPUT ACCEPT [32:1946]
-A PREROUTING -i eth1 -p tcp -m tcp --dport 20005 -j DNAT --to-destination 
10.10.17.5:22
-A PREROUTING -i eth1 -p tcp -m tcp --dport 20006 -j DNAT --to-destination 
10.10.17.6:22
-A PREROUTING -i eth1 -p tcp -m tcp --dport 20007 -j DNAT --to-destination 
10.10.17.7:22
-A POSTROUTING -o eth1 -j MASQUERADE
COMMIT
# Completed on Thu Sep 13 06:29:46 2007

[root@hsdevfiw01 sysconfig]# grep  forward /etc/sysctl.conf 
# Controls IP packet forwarding
net.ipv4.ip_forward = 1



Box 2, test web server A
[root@hsdevweb01 ~]# ip addr
1: lo: <LOOPBACK,UP,10000> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 10.0.16.45/32 scope global lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:0d:88:3d:c5:82 brd ff:ff:ff:ff:ff:ff
    inet 10.10.17.5/24 brd 10.10.17.255 scope global eth0
    inet6 fe80::20d:88ff:fe3d:c582/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 00:02:a5:03:de:d5 brd ff:ff:ff:ff:ff:ff
   
[root@hsdevweb01 ~]# grep arp /etc/sysctl.conf
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.eth0.arp_ignore = 1
net.ipv4.conf.eth0.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

[root@hsdevweb01 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.10.17.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
0.0.0.0         10.10.17.254    0.0.0.0         UG    0      0        0 eth0



Box 3, test web server B
[root@hsdevweb02 ~]# ip addr
1: lo: <LOOPBACK,UP,10000> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 10.0.16.45/32 scope global lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:40:05:7d:98:ca brd ff:ff:ff:ff:ff:ff
    inet 10.10.17.6/24 brd 10.10.17.255 scope global eth0
    inet6 fe80::240:5ff:fe7d:98ca/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 00:02:a5:0f:64:54 brd ff:ff:ff:ff:ff:ff

[root@hsdevweb02 ~]# grep arp /etc/sysctl.conf
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.eth0.arp_ignore = 1
net.ipv4.conf.eth0.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

[root@hsdevweb02 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.10.17.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
0.0.0.0         10.10.17.254    0.0.0.0         UG    0      0        0 eth0


Box 4, test box on the inside of the dev firewall
[root@hsdevmta01 ~]# ip addr
1: lo: <LOOPBACK,UP,10000> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:0d:88:46:1b:22 brd ff:ff:ff:ff:ff:ff
    inet 10.10.17.7/24 brd 10.10.17.255 scope global eth0
    inet6 fe80::20d:88ff:fe46:1b22/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 00:02:a5:1f:85:ad brd ff:ff:ff:ff:ff:ff

[root@hsdevmta01 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.10.17.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
0.0.0.0         10.10.17.254    0.0.0.0         UG    0      0        0 eth0

[root@hsdevmta01 ~]# wget 10.0.16.44
--04:02:54--  http://10.0.16.44/ <http://10.0.16.44/>  <http://10.0.16.44/ 
<http://10.0.16.44/> >
           => `index.html'
Connecting to 10.0.16.44:80... failed: Connection refused.

[root@hsdevmta01 ~]# wget 10.0.16.45 <== Expected response, HTTP answered
--04:02:56--  http://10.0.16.45/ <http://10.0.16.45/>  <http://10.0.16.45/ 
<http://10.0.16.45/> >
           => `index.html'
Connecting to 10.0.16.45:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
04:02:57 ERROR 403: Forbidden.

Box 4, text box on the outside of the dev firewall
[root@soho01 ~]# ip addr
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:16:ff:aa:00:01 brd ff:ff:ff:ff:ff:ff
    inet 10.0.16.171/24 brd 10.0.16.255 scope global eth0
    inet6 fe80::216:ffff:feaa:1/64 scope link
       valid_lft forever preferred_lft forever
3: sit0: <NOARP> mtu 1480 qdisc noop
    link/sit 0.0.0.0 brd 0.0.0.0
[root@soho01 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.16.0       0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
0.0.0.0         10.0.16.254     0.0.0.0         UG    0      0        0 eth0
[root@soho01 ~]# ping 10.0.16.45 -c 1
PING 10.0.16.45 (10.0.16.45) 56(84) bytes of data.
64 bytes from 10.0.16.45: icmp_seq=0 ttl=64 time=0.281 ms

--- 10.0.16.45 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.281/0.281/0.281/0.000 ms, pipe 2
[root@soho01 ~]# telnet 10.0.16.45 80
Trying 10.0.16.45...
telnet: connect to address 10.0.16.45: Connection timed out





<http://lists.graemef.net/mailman/listinfo/lvs-users> 






<Prev in Thread] Current Thread [Next in Thread>