|
Hi,
First of all, this is a very nice problem description. We will be releasing
a draft for problem description soon and it'll look like that one.
cwmin@xxxxxxxxxxxx wrote:
>
> Hello,
>
> I am setting up a LVS/DR cluster with 2 nodes(FreeBSD), but It doesn't work.
>
> Here is my network configuration;
>
> Internet (203.231.63.70 is Virtual IP)
> |
> |
> Router (203.231.63.0/24 network)
> |
> | <----- eth0 : 203.231.63.74
> LVS(Linux)
> | <----- eth1 : 203.231.63.70 (VIP)
Wow, here you might have a routing decision problem. Where should a
packet with 203.231.63.71 go for example? Take out one NIC and set
the VIP on eth0:0 and take care of the arp problem (Setup and arp
problem description is all in the HOWTO). If you really want to use
the LVS with DR and in a router like setup, which is rather a not
so intelligent design since incoming/outgoing packets must be processed
be the LVS and therefore we have LVS-NAT, you must use a private net
and do some simple routing. However I recommend you my first proposal.
> |
> ------------------
> | |
> SVR1 SVR2 <-- Real Servers are FreeBSD 4.2-RELEASE
>
> fxp0 : 203.231.63.70 203.231.63.70 (VIP)
> fxp1 : 203.231.63.71 203.231.63.72 (Real IP)
This is ok.
> ----] in Load Valancing Server(203.231.63.74);
>
> [root@ha1 log]# ifconfig -a
> eth0 Link encap:Ethernet HWaddr 00:10:5A:80:D7:FF
> inet addr:203.231.63.74 Bcast:203.231.63.255 Mask:255.255.255.0
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>
> eth1 Link encap:Ethernet HWaddr 00:10:5A:76:02:49
> inet addr:203.231.63.70 Bcast:203.231.63.70
> Mask:255.255.255.255
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Ok, now routing decision problem, but still, with LVS-DR you only need
one NIC normally. Have a look at the HOWTO under section 2.1.2.2ff the
first following picture shows you the concept of Direct Routing.
> [root@LVS /]# route -n
> Kernel IP routing table
> Destination Gateway Genmask Flags Metric Ref Use
> Iface
> 203.231.63.70 0.0.0.0 255.255.255.255 UH 0 0 0 eth1
> 203.231.63.74 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
> 203.231.63.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
Well, see, here you had to add manually a route which is not needed.
Again, please change your setup and it will work.
> 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
> 0.0.0.0 203.231.63.254 0.0.0.0 UG 0 0 0 eth0
>
> [root@LVS /]# sysctl -p
> net.ipv4.ip_forward = 1
> net.ipv4.conf.all.rp_filter = 1
> net.ipv4.ip_always_defrag = 0
> kernel.sysrq = 0
Perfect, you might even set rp_filter to 2.
> [root@LVS /]# vi /etc/ha.d/conf/ldirectord.cf
> timeout=3
> checkinterval=5
> autoreload=no
> fallback=127.0.0.1:80
> virtual=203.231.63.70:80
> real=203.231.63.71:80 gate 1
> real=203.231.63.72:80 gate 1
> service=http
> request="index.html"
> receive="Test Page"
> scheduler=rr
> protocol=tcp
I don't know ldirectord but it looks ok for me.
> ----] in Real Server(231.63.71,72);
>
> SVR1# ifconfig -a
> fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> inet 203.231.63.70 netmask 0xffffffff broadcast 203.231.63.70
You forgot to set the NOARP flag. Oh wait, again two NICs? You just need
one NIC and a lo that doesn't arp. See explaination below or read section
8.3.1.2 about Other Unices.
> fxp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> inet 203.231.63.72 netmask 0xffffff00 broadcast 203.231.63.255
>
> SVR2 in the same way..
>
> ******* Problem:
>
> 1. LVS dosen't forward http request packet to the real server..
...see description above why. If you tcpdump, remeber the signature and the
pattern of the dump to avoid future problems like this.
> 2. I don't know how FreeBSD(real server) can avoid arp request..
ifconfig fxp0 netmask 0xffffff00 broadcast 203.231.63.255 up
ifconfig lo0 alias 203.231.63.70 netmask 0xffffffff -arp up
> About first problem:
>
> Client try to connect 203.231.63.70:80, but LVS doesn't reply to that SYN
> packet..
You send the SYN, the client sends a SYN/ACK and this is not directly seen by
the LVS in DR mode.
> system(/sbin/ipvsadm -a -t 203.231.63.70:80 -R 203.231.63.72:80 -g -w 1)
> failed
> **Why this error occured?? What should I do to eliminate this error
> message??
> Second, Here's my ipvsadm output:
> [root@LVS /]# ipvsadm -L -n
> IP Virtual Server version 0.9.7 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 203.231.63.70:www rr
> -> 255.255.255.255:52199 Masq 4194304 0 0
Could you please patch your kernel with a more recent LVS-patch level?
And do compile the correct ipvsadm tool which is in the package as well.
> Last output line seems wrong,, I think It should look like this.. right?
>
> TCP 203.231.63.70:www rr
> -> 203.231.63.71:80 route 1 0 0
> -> 203.231.63.72:80 route 1 0 0
Yes, it should look like this, except the www will be 80 because you told
above that you use -n :)
> **How can I fix this thing??
See above.
> Second problem:
>
> As you know.. in LVS cluster, real servers should not reply to arp request
> that
> asks VIP's MAC address.. Only LVS should reply to that arp request..
Correct.
> I have an idea about it.. Let the real server reply to client's arp
> request(for VIP)
> with LVS's hardware address.. then all client's packet that towards VIP go
> to the LVS..
> That's a good idea..
> so I commanded like this..
>
> arp -s 203.231.63.70 00:10:5A:76:02:49 pub
See the arp section in the HOWTO about solutions like yours.
> 203.231.63.70 is VIP and 00:10:5A:76:02:49 is LVS's MAC address..
>
> I got a meessage :
> set: proxy entry exists for non 802 device
Haven't seen it yet but the 802 indicates very low level stuff.
Its CSMA/CD and you can get more infos at:
http://www.cis.ohio-state.edu/~gurari/course/cis677/ethernet.html
> without 'pub' flag :
> set: can only proxy for 203.231.63.70
>
> ***What is this message means??
> ***ARP reply with another MAC address is impossible??
IMHO yes but I don't know if I understood you here. But if so, then
you would have twice the same MAC on different nodes but in the same
physical net you end up with a mess!
> ***How can I solve this arp problem??
See the HOWTO
> Thanks in advance..
no problem, as long as you directly address the mailinglist ;)
Best regards,
Roberto Nibali, ratz
--
mailto: `echo NrOatSz@xxxxxxxxx | sed 's/[NOSPAM]//g'`
|
