LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Other strange LB balancing issue

To: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Subject: Other strange LB balancing issue
From: Cyrille Mertes <cyrille@xxxxxxxxxxxxx>
Date: Fri, 29 Nov 2002 10:57:26 -0500
Hello,


Actual LB : redhat 8.0, kernel 2.4.18-14, IP Virtual Server version 1.0.4 (source), piranha-0.7.0-3 (source) Old configuration : redhat 7.2, kernel 2.4.18, IP Virtual Server version 1.0.1 (rpm source), piranha-0.6.1-1 (rpm source)

I see a message of Jeremy Kusnetz on 11/18/2002 that looks a lot like a
problem that we have. I don't know if he found a solution. We have been
setuping LVS on different servers and always obtaining the same issue.

Our first unsuccesfull installation was with UltraMonkey, after we decided to try with Piranha to see if our problem would go away, but the problem is still
present. We also installed it on different kernels, with RPMs, with the
source, and we always have this problem.
The problem is that the server stops balancing the requests, some times it
stops only on one virtual server and continues balancing correctly on the
others. When it stops balancing it sends all the requests to one of the
real servers (always the same) and stops showing anything in ipvsadm.
It can stop 3 times in one hour, and after it can take 4 hours before stopping again.

Here is output of ipvsadm when everything is ok :

IP Virtual Server version 1.0.4 (size=65536)
Prot LocalAddress:Port Scheduler Flags
 -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  216.187.127.124:http wrr
 -> gragra.privatedns.com:http     Route   1      170        398
 -> koon.privatedns.com:http       Route   1      122        212
 -> motti.privatedns.com:http      Route   1      121        233
TCP  216.187.127.122:http wrr
 -> motti.privatedns.com:http      Route   2      277        852
 -> gragra.privatedns.com:http     Route   1      150        433
 -> koon.privatedns.com:http       Route   2      236        837

Here is one output of ipvsadm where you can see that IP .122 stopped balancing the requests :


IP Virtual Server version 1.0.4 (size=65536)
Prot LocalAddress:Port Scheduler Flags
 -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  216.187.127.124:http wrr
 -> gragra.privatedns.com:http     Route   1      170        398
 -> koon.privatedns.com:http       Route   1      122        212
 -> motti.privatedns.com:http      Route   1      121        233
TCP  216.187.127.122:http wrr


Here is my pirnha lvs.conf :

serial_no = 344
primary = 216.187.127.77
primary_private = 192.168.6.21
service = lvs
backup_active = 0
backup = 216.187.127.117
backup_private = 192.168.6.10
heartbeat = 1
heartbeat_port = 539
keepalive = 6
deadtime = 15
network = direct
nat_nmask = 255.255.255.255
reservation_conflict_action = preempt
debug_level = MAX
virtual sebulba_http_122 {
    active = 1
    address = 216.187.127.122 eth0:1
    vip_nmask = 255.255.255.0
    port = 80
    send = "GET / HTTP/1.0\r\n\r\n"
    expect = "HTTP"
    load_monitor = none
    scheduler = wrr
    protocol = tcp
    timeout = 6
    reentry = 15
    quiesce_server = 0
    server motti {
        address = 192.168.6.12
        active = 1
        weight = 2
    }
    server gragra {
        address = 192.168.6.13
        active = 1
        weight = 1
    }
    server koon {
        address = 192.168.6.14
        active = 1
        weight = 2
    }
}
virtual sebulba_http_124 {
    active = 1
    address = 216.187.127.124 eth0:3
    vip_nmask = 255.255.255.0
    port = 80
    send = "GET / HTTP/1.0\r\n\r\n"
    expect = "HTTP"
    load_monitor = none
    scheduler = wrr
    protocol = tcp
    timeout = 6
    reentry = 15
    quiesce_server = 0
    server motti {
        address = 192.168.6.12
        active = 1
        weight = 1
    }
    server gragra {
        address = 192.168.6.13
        active = 1
        weight = 2
    }
    server koon {
        address = 192.168.6.14
        active = 1
        weight = 1
    }
}

Finally, I have a lot of connection timed out with my real servers in /var/log/messages :

Nov 29 10:34:43 lb nanny[5987]: CONNECT to 192.168.6.13:8008 timed out
Nov 29 10:34:43 lb nanny[5987]: shutting down 192.168.6.13:8008 due to connection failure
Nov 29 10:34:55 lb nanny[5987]: making 192.168.6.13:8008 available
Nov 29 10:36:14 lb nanny[5980]: CONNECT to 192.168.6.13:80 timed out
Nov 29 10:36:14 lb nanny[5980]: shutting down 192.168.6.13:80 due to connection failure
Nov 29 10:36:15 lb nanny[5996]: CONNECT to 192.168.6.13:8008 timed out
Nov 29 10:36:15 lb nanny[5996]: shutting down 192.168.6.13:8008 due to connection failure
Nov 29 10:36:26 lb nanny[5980]: making 192.168.6.13:80 available
Nov 29 10:36:27 lb nanny[5996]: making 192.168.6.13:8008 available
Nov 29 10:37:00 lb nanny[6001]: READ to 192.168.6.14:80 timed out
Nov 29 10:37:00 lb nanny[6001]: shutting down 192.168.6.14:80 due to connection failure
Nov 29 10:37:00 lb nanny[5993]: READ to 192.168.6.14:80 timed out
Nov 29 10:37:00 lb nanny[5993]: shutting down 192.168.6.14:80 due to connection failure
Nov 29 10:37:00 lb nanny[5992]: READ to 192.168.6.13:80 timed out
Nov 29 10:37:00 lb nanny[5992]: shutting down 192.168.6.13:80 due to connection failure
Nov 29 10:37:02 lb nanny[5979]: READ to 192.168.6.12:80 timed out
Nov 29 10:37:02 lb nanny[5979]: shutting down 192.168.6.12:80 due to connection failure
Nov 29 10:37:12 lb nanny[6001]: making 192.168.6.14:80 available
Nov 29 10:37:12 lb nanny[5993]: making 192.168.6.14:80 available
Nov 29 10:37:12 lb nanny[5992]: making 192.168.6.13:80 available
Nov 29 10:37:14 lb nanny[5979]: making 192.168.6.12:80 available
Nov 29 10:38:09 lb nanny[5996]: CONNECT to 192.168.6.13:8008 timed out
Nov 29 10:38:09 lb nanny[5996]: shutting down 192.168.6.13:8008 due to connection failure
Nov 29 10:38:10 lb nanny[5992]: CONNECT to 192.168.6.13:80 timed out
Nov 29 10:38:10 lb nanny[5992]: shutting down 192.168.6.13:80 due to connection failure
Nov 29 10:38:10 lb nanny[5987]: CONNECT to 192.168.6.13:8008 timed out
Nov 29 10:38:10 lb nanny[5987]: shutting down 192.168.6.13:8008 due to connection failure
Nov 29 10:38:21 lb nanny[5996]: making 192.168.6.13:8008 available
Nov 29 10:38:22 lb nanny[5992]: making 192.168.6.13:80 available
Nov 29 10:38:22 lb nanny[5987]: making 192.168.6.13:8008 available
Nov 29 10:39:09 lb nanny[6000]: CONNECT to 192.168.6.13:80 timed out
Nov 29 10:39:09 lb nanny[6000]: shutting down 192.168.6.13:80 due to connection failure
Nov 29 10:39:21 lb nanny[6000]: making 192.168.6.13:80 available
Nov 29 10:39:46 lb nanny[5992]: READ to 192.168.6.13:80 timed out
Nov 29 10:39:46 lb nanny[5992]: shutting down 192.168.6.13:80 due to connection failure Nov 29 10:39:55 lb nanny[5980]: READ returned error 104:Connection reset by peer Nov 29 10:39:55 lb nanny[5980]: shutting down 192.168.6.13:80 due to connection failure Nov 29 10:39:55 lb nanny[6000]: READ returned error 104:Connection reset by peer Nov 29 10:39:55 lb nanny[6000]: shutting down 192.168.6.13:80 due to connection failure
Nov 29 10:41:40 lb nanny[5992]: making 192.168.6.13:80 available
Nov 29 10:41:43 lb nanny[5980]: making 192.168.6.13:80 available
Nov 29 10:41:43 lb nanny[6000]: making 192.168.6.13:80 available
Nov 29 10:44:40 lb nanny[6000]: CONNECT to 192.168.6.13:80 timed out
Nov 29 10:44:40 lb nanny[6000]: shutting down 192.168.6.13:80 due to connection failure
Nov 29 10:44:52 lb nanny[6000]: making 192.168.6.13:80 available
Nov 29 10:45:45 lb nanny[5980]: CONNECT to 192.168.6.13:80 timed out
Nov 29 10:45:45 lb nanny[5980]: shutting down 192.168.6.13:80 due to connection failure
Nov 29 10:45:57 lb nanny[5980]: making 192.168.6.13:80 available
Nov 29 10:47:00 lb nanny[5992]: CONNECT to 192.168.6.13:80 timed out
Nov 29 10:47:00 lb nanny[5992]: shutting down 192.168.6.13:80 due to connection failure
Nov 29 10:47:12 lb nanny[5992]: making 192.168.6.13:80 available
Nov 29 10:50:35 lb nanny[5987]: CONNECT to 192.168.6.13:8008 timed out
Nov 29 10:50:35 lb nanny[5987]: shutting down 192.168.6.13:8008 due to connection failure
Nov 29 10:50:47 lb nanny[5987]: making 192.168.6.13:8008 available
Nov 29 10:55:03 lb nanny[5992]: CONNECT to 192.168.6.13:80 timed out
Nov 29 10:55:03 lb nanny[5992]: shutting down 192.168.6.13:80 due to connection failure
Nov 29 10:55:03 lb nanny[5980]: CONNECT to 192.168.6.13:80 timed out
Nov 29 10:55:03 lb nanny[5980]: shutting down 192.168.6.13:80 due to connection failure
Nov 29 10:55:15 lb nanny[5992]: making 192.168.6.13:80 available
Nov 29 10:55:15 lb nanny[5980]: making 192.168.6.13:80 available

Thanks for your help.

Cyrille.




<Prev in Thread] Current Thread [Next in Thread>