|
Wha't i'm trying to do is setup a load banancer using 3 servers for a
webservice (HTTP)
1st - the main server which will redirect to the 2nd or 3rd server. I called
this LVS (and it's where ldirectord is)
2nd - runs a simple http service and when a client enters...and puts the ip,
or domain of the 1st server, he should get the contents from the 2nd server
(if up). eighterwise the content from the 3rd server
3rd - runs the same http service..
the only thing different from the ALL tutorials that i've read is that those
3 servers are on the internet , with public ips not behind some private
192.168.xx network
2 of the ip's are in the same subbet of a class C
1 is not even in the same continent :)
So a relatively simple thing to do , that ldirectord doesn't do..in this
case
----------------------
I have 3 PUBLIC ip's like (none are with 192..) , two of them are on the
same subnet,
I'm running Cent-oS 5, and here is a COMPLETE list of what i have done.
XX.XX.XX.234 (this is the lvs..)
XX.XX.XX.235 real server (web ngix)
YYY.YYY.YYY.163 real server (web apache) - I don't really care about this
one, i can move it into the
we will use XX.XX.XX.236 as virtual..(there isn't an ip on the net with that
number up). of course it will be simpler to use a 192.. but, i have tried
that also ,and no luck
same subnet as the first 2 ones, i just want to make it work from the 234 ->
235..but it gives me a timeout on the browser...
Here's what i did:
[root@linux ~]# cat /etc/ha.d/ldirectord.cf
checktimeout=3
checkinterval=10
autoreload=yes
logfile="/var/log/ldirectord.log"
quiescent=no
virtual=XX.XX.XX.236:80
fallback=127.0.0.1:80
real=XX.XX.XX.235:80 gate
real=YYY.YYY.YYY.235:80 gate
service=http
request="test.html"
receive="Still alive"
scheduler=rr #here i've tried with wlr as well
protocol=tcp
checktype=negotiate
[root@linux ~]# /usr/sbin/ldirectord -d /etc/ha.d/ldirectord.cf start
DEBUG2: Running exec(/usr/sbin/ldirectord -d /etc/ha.d/ldirectord.cf start)
Running exec(/usr/sbin/ldirectord -d /etc/ha.d/ldirectord.cf start)
DEBUG2: Invoking ldirectord invoked as: /usr/sbin/ldirectord /etc/ha.d/
ldirectord.cf start
Invoking ldirectord invoked as: /usr/sbin/ldirectord
/etc/ha.d/ldirectord.cfstart
DEBUG2: Starting Linux Director v1.186-ha-2.1.4 with pid: 18619
Starting Linux Director v1.186-ha-2.1.4 with pid: 18619
DEBUG2: Running system(/sbin/ipvsadm -A -t XX.XX.XX.236:80 -s rr )
Running system(/sbin/ipvsadm -A -t XX.XX.XX.236:80 -s rr )
DEBUG2: Added virtual server: XX.XX.XX.236:80
Added virtual server: XX.XX.XX.236:80
DEBUG2: Running system(/sbin/ipvsadm -a -t XX.XX.XX.236:80 -r
127.0.0.1:80-g -w 1)
Running system(/sbin/ipvsadm -a -t XX.XX.XX.236:80 -r 127.0.0.1:80 -g -w 1)
DEBUG2: Added fallback server: 127.0.0.1:80 (XX.XX.XX.236:80) (Weight set to
1)
Added fallback server: 127.0.0.1:80 (XX.XX.XX.236:80) (Weight set to 1)
DEBUG2: Disabled real
server=negotiate:http:tcp:XX.XX.XX.235:80:::1:gate:\/test\.html:Still\ alive
(virtual=tcp:XX.XX.XX.236:80)
DEBUG2: Checking negotiate: real
server=negotiate:http:tcp:XX.XX.XX.235:80:::1:gate:\/test\.html:Still\ alive
(virtual=tcp:XX.XX.XX.236:80)
DEBUG2: check_http: url="http://XX.XX.XX.235:80/test.html";
virtualhost="XX.XX.XX.235"
LWP::UserAgent::new: ()
DEBUG2: Starting Check
DEBUG2: Starting HTTP/HTTPS
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: GET
http://XX.XX.XX.235:80/test.html<http://xx.xx.xx.235/test.html>
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 12 bytes
LWP::UserAgent::request: Simple response: OK
DEBUG2: Finished HTTP/HTTPS
DEBUG2: Running system(/sbin/ipvsadm -a -t XX.XX.XX.236:80 -r
XX.XX.XX.235:80 -g -w 1)
Running system(/sbin/ipvsadm -a -t XX.XX.XX.236:80 -r XX.XX.XX.235:80 -g -w
1)
DEBUG2: system(/sbin/ipvsadm -a -t XX.XX.XX.236:80 -r XX.XX.XX.235:80 -g -w
1) failed: No child processes
system(/sbin/ipvsadm -a -t XX.XX.XX.236:80 -r XX.XX.XX.235:80 -g -w 1)
failed: No child processes
DEBUG2: Added real server: XX.XX.XX.235:80 (XX.XX.XX.236:80) (Weight set to
1)
Added real server: XX.XX.XX.235:80 (XX.XX.XX.236:80) (Weight set to 1)
DEBUG2: Running system(/sbin/ipvsadm -d -t XX.XX.XX.236:80 -r 127.0.0.1:80)
Running system(/sbin/ipvsadm -d -t XX.XX.XX.236:80 -r 127.0.0.1:80)
DEBUG2: system(/sbin/ipvsadm -d -t XX.XX.XX.236:80 -r 127.0.0.1:80) failed:
No child processes
system(/sbin/ipvsadm -d -t XX.XX.XX.236:80 -r 127.0.0.1:80) failed: No child
processes
DEBUG2: Deleted fallback server: 127.0.0.1:80 (XX.XX.XX.236:80)
Deleted fallback server: 127.0.0.1:80 (XX.XX.XX.236:80)
DEBUG2: Enabled real
server=negotiate:http:tcp:XX.XX.XX.235:80:::1:gate:\/test\.html:Still\ alive
(virtual=tcp:XX.XX.XX.236:80)
DEBUG2: check_http:
http://XX.XX.XX.235:80/test.html<http://xx.xx.xx.235/test.html>is up
If i go to the webserver i can see that ldirector is actually testing the
test.html..every 10 seconds like in the conf
-----------------------------------------------
I am sure that is because of the configurations of the IPs and the aditional
eth0:0 and lo:0 and that's why it doesn't work, i will paste everything that
i did, maybe, just maybe you can help me out on this one, i'm really
stuck..probably because i don't know lots of stuff on how the OSI layer is
build and how arp works
Aditional Network conf..
XX.XX.XX.234 (this is the lvs..) is spawned on eth1..
so i spawned another eth1:0
[root@linux ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1:0
DEVICE=eth1:0
IPADDR=XX.XX.XX.236 # this is from the same subnet and i doesn't conflict
with an existent one
NETMASK=255.255.255.0
NETWORK=XX.XX.XX.0
BROADCAST=XX.XX.XX.255
GATEWAY=XX.XX.XX.233 # i've put the same gateway as the default ip
XX.XX.XX.234 had
ONBOOT=yes
i also enabled port forwarding
net.ipv4.ip_forward = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
And i did the modprobe with all those modules for IPV
modprobe ip_vs_dh
modprobe ip_vs_ftp
modprobe ip_vs_dh
modprobe ip_vs_ftp
modprobe ip_vs
modprobe ip_vs_lblc
modprobe ip_vs_lblcr
modprobe ip_vs_lc
modprobe ip_vs_nq
modprobe ip_vs_rr
modprobe ip_vs_sed
modprobe ip_vs_sh
modprobe ip_vs_wlc
modprobe ip_vs_wrr
and that's all i did for the LVS server..
Now for the webserver
XX.XX.XX.235 (this is the lvs..) is spawned on eth0..
root@linux ~]# cat /etc/sysconfig/network-scripts/ifcfg-lo:0
DEVICE=lo:0
IPADDR=XX.XX.XX.236
NETMASK=255.255.255.255
NETWORK=XX.XX.XX.XX.0
BROADCAST=XX.XX.XX.255
ONBOOT=yes
NAME=loopback
and
net.ipv4.ip_forward = 0
net.ipv4.conf.lo.arp_ignore = 1 #here i have tried with eth0 instead of lo,
no luck..
net.ipv4.conf.lo.arp_announce = 2 #here i have tried with eth0 instead of
lo, no luck..
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
here i've tried with ip forward 0 and 1 , no luck the requests simply don't
reach this server only the direct ones
Now i've understand that this is an ARP problem, and as CentOS doesn't
support the arp hidden flag on sysctl , i tried with
/etc/init.d/arptables_jf
arptables -A IN -j DROP -d XX.XX.XX.236
-A OUT -j mangle -o eth0 -s XX.XX.XX.234 --mangle-ip-s XX.XX.XX.236
arptables -A OUT -j mangle -o eth0 -s XX.XX.XX.234 --mangle-ip-s
XX.XX.XX.236
/etc/init.d/arptables_jf save
/etc/init.d/arptables_jf start
/sbin/arptables -L -v -n
hain IN (policy ACCEPT 1353 packets, 37884 bytes)
pkts bytes target in out source-ip destination-ip source-hw destination-hw
hlen op hrd pro
0 0 DROP * * 0.0.0.0/0 XX.XX.XX.236 00/00 00/00 any 0000/0000 0000/0000
0000/0000
Chain OUT (policy ACCEPT 25 packets, 700 bytes)
pkts bytes target in out source-ip destination-ip source-hw destination-hw
hlen op hrd pro
0 0 mangle * eth0 XX.XX.XX.234 0.0.0.0/0 00/00 00/00 any 0000/0000 0000/0000
0000/0000 --mangle-ip-s XX.XX.XX.236
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target in out source-ip destination-ip source-hw destination-hw
hlen op hrd pro
-----------------------------------------------
i've probed with ipvsadm or something like it, to see the active
connections, and they are always to 0
i've modprobed here the same modules, no luck..
So from this point i'm really stuck and don't know what to do...
Here's the ifconfig from both servers if that helps
from the LVS (xx.234)
eth1 Link encap:Ethernet HWaddr 00:1B:21:46:3E:A9
inet addr:XX.XX.XX.234 Bcast:XX.XX.XX.239 Mask:255.255.255.248
inet6 addr: fe80::21b:21ff:fe45:3ea9/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:276338 errors:0 dropped:2 overruns:0 frame:0
TX packets:220590 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:156995116 (149.7 MiB) TX bytes:30754525 (29.3 MiB)
Base address:0xd000 Memory:e2020000-e2040000
eth1:0 Link encap:Ethernet HWaddr 00:1B:21:46:3E:A9
inet addr:XX.XX.XX.236 Bcast:XX.XX.XX.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Base address:0xd000 Memory:e2020000-e2040000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:150 errors:0 dropped:0 overruns:0 frame:0
TX packets:150 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:16592 (16.2 KiB) TX bytes:16592 (16.2 KiB)
here's the one from the webserver
eth0 Link encap:Ethernet HWaddr 00:24:1D:72:61:AB
inet addr:XX.XX.XX.235 Bcast:XX.XX.XX.239 Mask:255.255.255.248
inet6 addr: fe80::224:1dff:fe74:61ab/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:301529 errors:0 dropped:0 overruns:0 frame:0
TX packets:255827 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:149822776 (142.8 MiB) TX bytes:36393789 (34.7 MiB)
Interrupt:233 Base address:0x2000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:56 errors:0 dropped:0 overruns:0 frame:0
TX packets:56 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4448 (4.3 KiB) TX bytes:4448 (4.3 KiB)
lo:0 Link encap:Local Loopback
inet addr:XX.XX.XX.236 Mask:255.255.255.255
UP LOOPBACK RUNNING MTU:16436 Metric:1
i've checked the firewall it's disabled , iptables stoped, selinux is
disabled..
I am thanking you in advance for helping solve this problem. I think it's a
problem that many have with Fedora/Red Hat/ Cent-oS
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
