LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

[lvs-users] In LVS-RD , The nginx cannot receive the request from LVS, w

To: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Subject: [lvs-users] In LVS-RD , The nginx cannot receive the request from LVS, while the real server receives the correct package .
From: Allen He <allenhooo@xxxxxxxxx>
Date: Tue, 14 Dec 2010 13:36:45 +0800
Hi All,

   We have been running it with LVS-RD for a year, but when i add a
new real server , the new real server not work. while the other real
server works fine.
   The problem is The real server can receive the package from LVS,
but the nginx
installed on the real server never received the package.
   The real server:
admin@speed:~$ uname  -a
Linux speed 2.6.35-22-server #35-Ubuntu SMP Sat Oct 16 22:02:33 UTC
2010 x86_64 GNU/Linux

   It has two ports, eth0 and eth1.  The two cards talk to two
networks, one starting with 66. and going to the outside and the other
192. and going to the inside. the VIP is 66.63.253.120 and
67.249.117.56. Here is the result of ifconfig:

admin@speed:~$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:1e:c9:ba:20:41
         inet addr:66.63.253.88  Bcast:66.63.253.127  Mask:255.255.255.192
         inet6 addr: fe80::21e:c9ff:feba:2041/64 Scope:Link
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:94896 errors:0 dropped:0 overruns:0 frame:0
         TX packets:20648 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:10900353 (10.9 MB)  TX bytes:4580223 (4.5 MB)
         Interrupt:16 Memory:f8000000-f8012800

eth0:1    Link encap:Ethernet  HWaddr 00:1e:c9:ba:20:41
         inet addr:67.249.117.24  Bcast:67.249.117.63  Mask:255.255.255.192
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         Interrupt:16 Memory:f8000000-f8012800

eth1    Link encap:Ethernet  HWaddr 00:1e:c9:ba:20:43
         inet addr:192.168.16.88  Bcast:192.168.16.255  Mask:255.255.255.0
         inet6 addr: fe80::21e:c9ff:feba:2043/64 Scope:Link
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:126534 errors:0 dropped:0 overruns:0 frame:0
         TX packets:56944 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:28644545 (28.6 MB)  TX bytes:5306425 (5.3 MB)
         Interrupt:16 Memory:f4000000-f4012800

lo        Link encap:Local Loopback
         inet addr:127.0.0.1  Mask:255.0.0.0
         inet6 addr: ::1/128 Scope:Host
         UP LOOPBACK RUNNING  MTU:16436  Metric:1
         RX packets:680 errors:0 dropped:0 overruns:0 frame:0
         TX packets:680 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:74461 (74.4 KB)  TX bytes:74461 (74.4 KB)

lo:0      Link encap:Local Loopback
         inet addr:66.63.253.120  Mask:255.255.255.255
         UP LOOPBACK RUNNING  MTU:16436  Metric:1

lo:1      Link encap:Local Loopback
         inet addr:67.249.117.56  Mask:255.255.255.255
         UP LOOPBACK RUNNING  MTU:16436  Metric:1

I have dumped the package on the real server port eth1, when i access
http://66.63.253.120 :

admin@speed:~$ sudo tcpdump -i eth1 dst host 66.63.253.120 -e
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
10:37:19.438541 00:22:19:4f:64:70 (oui Unknown) > 00:1e:c9:ba:20:43
(oui Unknown), ethertype IPv4 (0x0800), length 66: 126.89.61.122.54459
> 66.63.253.120.www: Flags [S], seq 3247425561, win 8192, options [mss
1460,nop,wscale 2,nop,nop,sackOK], length 0
10:37:22.437969 00:22:19:4f:64:70 (oui Unknown) > 00:1e:c9:ba:20:43
(oui Unknown), ethertype IPv4 (0x0800), length 66: 126.89.61.122.54459
> 66.63.253.120.www: Flags [S], seq 3247425561, win 8192, options [mss
1460,nop,wscale 2,nop,nop,sackOK], length 0
10:37:28.442390 00:22:19:4f:64:70 (oui Unknown) > 00:1e:c9:ba:20:43
(oui Unknown), ethertype IPv4 (0x0800), length 62: 126.89.61.122.54459
> 66.63.253.120.www: Flags [S], seq 3247425561, win 8192, options [mss
1460,nop,nop,sackOK], length 0
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel


The following is the real server setup scritps:

VIP1=66.63.253.120
VIP2=67.249.117.56
      /sbin/ifconfig lo:0 $VIP1 broadcast $VIP1 netmask 255.255.255.255 up
      /sbin/route add -host $VIP1 dev lo:0
      /sbin/ifconfig lo:1 $VIP2 broadcast $VIP2 netmask 255.255.255.255 up
      /sbin/route add -host $VIP2 dev lo:1
      echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
      echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
      echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
      echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
      /sbin/sysctl -p >/dev/null 2>&1

AND the route table:
admin@speed:~$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
66.249.117.56   *               255.255.255.255 UH    0      0        0 lo
66.63.253.120   *               255.255.255.255 UH    0      0        0 lo
66.249.117.0    *               255.255.255.192 U     0      0        0 eth0
localnet        *               255.255.255.192 U     0      0        0 eth0
192.168.16.0    *               255.255.255.0   U     0      0        0 eth1
default         66.63.253.65    0.0.0.0         UG    100    0        0 eth0


when i access http://66.63.253.88  , it works fine.


Any ideas?

Best Regards.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>