LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

[lvs-users] stuck on LVS-TUN, realservers receiving ipip packet, but not

To: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Subject: [lvs-users] stuck on LVS-TUN, realservers receiving ipip packet, but not doing anything because it think's it's martian.
From: Vincent Young <nard@xxxxxxx>
Date: Wed, 14 Oct 2009 00:54:11 -0400
Hello,

This is my first time setting up LVS, and I am abit stuck.  So I was  
hoping to maybe get a little insight and advice from some of the more  
experienced members of this mailing list.

So first things first, I'm trying to get this set up on linode.com.  
and I've been in their IRC channel, and asked if this would work. and  
one of the official responses on this issue:

caker:if packets get rewritten, it's not gonna work
[
caker:we filter based on source ip and mac, and dest ip and mac
[caker:^-- for a given Linode

So i decided to use LVS-TUN. Each linode has a public IP on eth0, and  
an aliased eth0:0 private ip address with no gateway.

This is where I am not sure if it was the correct approach or not,  
please correct me.  On the director, I set the VIP to be the same as  
my eth0 public IP. and on the real servers I created a tunl0 interface  
that matched the VIP. I dont think i needed to add a route, since they  
both share a common gateway on their public IP's, and they can talk to  
each other.

all machines:
running Centos 5.3
Kernel@ 2.6.18.8-x86_64
realserver contains nginx


director setup:
sysctl.conf has this loaded:
net.ipv4.ip_forward = 1

# /sbin/ifconfig
eth0      Link encap:Ethernet  HWaddr FE:FD:61:6B:85:EA
           inet addr:97.107.133.234  Bcast:97.107.133.255  Mask: 
255.255.255.0
           inet6 addr: fe80::fcfd:61ff:fe6b:85ea/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:4440 errors:0 dropped:0 overruns:0 frame:0
           TX packets:6386 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:796449 (777.7 KiB)  TX bytes:1195747 (1.1 MiB)

eth0:0    Link encap:Ethernet  HWaddr FE:FD:61:6B:85:EA
           inet addr:192.168.134.25  Bcast:192.168.255.255  Mask: 
255.255.128.0
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

lo        Link encap:Local Loopback
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet6 addr: ::1/128 Scope:Host
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:65 errors:0 dropped:0 overruns:0 frame:0
           TX packets:65 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:5944 (5.8 KiB)  TX bytes:5944 (5.8 KiB)
# /sbin/route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref     
Use Iface
97.107.133.0    0.0.0.0         255.255.255.0   U     0      0         
0 eth0
192.168.128.0   0.0.0.0         255.255.128.0   U     0      0         
0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0         
0 eth0
0.0.0.0         97.107.133.1    0.0.0.0         UG    0      0         
0 eth0

# /sbin/ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  97.107.133.234:80 wlc
   -> 97.107.130.68:80             Tunnel  1      0          0






real server with http web server listening on port 80:
sysctl.conf already loaded with:
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.eth0.arp_announce=2
net.ipv4.conf.eth0.arp_ignore=1
net.ipv4.conf.lo.arp_announce=2
net.ipv4.conf.lo.arp_ignore=1
net.ipv4.conf.tunl0.arp_ignore = 1
net.ipv4.conf.tunl0.arp_announce = 2


# /sbin/ifconfig
eth0      Link encap:Ethernet  HWaddr FE:FD:61:6B:82:44
           inet addr:97.107.130.68  Bcast:97.107.130.255  Mask: 
255.255.255.0
           inet6 addr: fe80::fcfd:61ff:fe6b:8244/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:64369 errors:0 dropped:0 overruns:0 frame:0
           TX packets:92259 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:48183677 (45.9 MiB)  TX bytes:23467359 (22.3 MiB)

eth0:0    Link encap:Ethernet  HWaddr FE:FD:61:6B:82:44
           inet addr:192.168.134.109  Bcast:192.168.255.255  Mask: 
255.255.128.0
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

lo        Link encap:Local Loopback
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet6 addr: ::1/128 Scope:Host
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:48 errors:0 dropped:0 overruns:0 frame:0
           TX packets:48 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:6877 (6.7 KiB)  TX bytes:6877 (6.7 KiB)

tunl0     Link encap:IPIP Tunnel  HWaddr
           inet addr:97.107.133.234  Mask:255.255.255.255
           UP RUNNING NOARP  MTU:1480  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

# /sbin/route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref     
Use Iface
97.107.133.234  0.0.0.0         255.255.255.255 UH    0      0         
0 tunl0
97.107.130.0    0.0.0.0         255.255.255.0   U     0      0         
0 eth0
192.168.128.0   0.0.0.0         255.255.128.0   U     0      0         
0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0         
0 eth0
0.0.0.0         97.107.130.1    0.0.0.0         UG    0      0         
0 eth0



iptables is clear and is accepting everything on both director and  
real server.



director: cannot ping realserver or telnet port 80 into realserver  
eth0 public ip. can ping client.
realserver: can ping both realserver and client.when i telnet into VIP  
on port 80, i believe it bypasses the director, since tcpdump host  
97.107.130.68 on the director showed no activity.
client (public ip 99.247.97.70) can ping director and realserver, and  
can telnet port 80 to real server fine. when i telnet to the  
VIP,client doesnt get a response. When i run tcpdump on the director  
and realserver, this is the what happens when a client tries to telnet  
port 80 into the VIP:

director tcpdump:
# /usr/sbin/tcpdump -nn host 97.107.130.68
tcpdump: verbose output suppressed, use -v or -vv for full protocol  
decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes

04:39:47.872616 IP 97.107.133.234 > 97.107.130.68: IP  
99.247.97.70.34213 > 97.107.133.234.80: S 2271054937:2271054937(0) win  
65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 343332259 0,sackOK,[| 
tcp]> (ipip-proto-4)
04:39:51.874495 IP 97.107.133.234 > 97.107.130.68: IP  
99.247.97.70.34213 > 97.107.133.234.80: S 2271054937:2271054937(0) win  
65535 <mss 1460,sackOK,eol> (ipip-proto-4)

realserver tcp dump:
# /usr/sbin/tcpdump -nn host 97.107.133.234
tcpdump: verbose output suppressed, use -v or -vv for full protocol  
decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes

04:39:47.860998 IP 97.107.133.234 > 97.107.130.68: IP  
99.247.97.68.34213 > 97.107.133.234.80: S 2271054937:2271054937(0) win  
65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 343332259 0,sackOK,[| 
tcp]> (ipip-proto-4)
04:39:51.863289 IP 97.107.133.234 > 97.107.130.68: IP  
99.247.97.68.34213 > 97.107.133.234.80: S 2271054937:2271054937(0) win  
65535 <mss 1460,sackOK,eol> (ipip-proto-4)

realserver has an entry in /var/log/messages:
Oct 14 04:39:51 li60-68 kernel: martian source 97.107.130.68 from  
97.107.133.234, on dev eth0
Oct 14 04:39:51 li60-68 kernel: ll header: fe:fd:61:6b:82:44:00:0e: 
39:6f:48:00:08:00


conclusion so far:
it looks like the ipip packet is reaching the realserver, but want to  
find out if it's being discarded because it thinks it's a martian  
source? I thought with kernel 2.6+ all i need was the arp_ignore and  
arp_announce flags set on the real servers. do i need to do stuff with  
arptables or iptables? If any additional information is needed, let me  
know. is it possible to do LVS-DR or LVS-TUN over the eth0:0 aliased  
private ip's?

What can I try next? I've been exploring LVS for the last 2 days or  
so, and read through the documentation several times. I know i'm not  
as experienced as some people here, so I'm hoping someone can point me  
in the right direction.
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>