|
Eric,
Our usual advice is that for anything with over 200 health checks
(which your configuration sounds like it definitely is) then you
should look carefully at your options for simplifying the checks.
We do use a version of Ldirectord but each virtual service runs its
health check in a sub process, so the real limitation is number of
backend servers per virtual service.
Maybe you could send me your configuration file off list
(malcolm@xxxxxxxxxxxxxxxx) and I'll get someone to test the
performance of the health checks on our system?
It sounds like the kind of configuration that would definitely
highlight any scalability issues in our product :-).
On 19 September 2015 at 16:50, Eric Robinson <eric.robinson@xxxxxxxxx> wrote:
> I've been looking for an alternative to ldirectord because it is
> single-threaded. My config file is so long that it now takes about 3 or 4
> minutes to health-check all my servers, which means a server could be down
> for that long before the load balancer notices. Ldirectord does have the
> advantage of being free, which is nice, but I might be interested in
> something that isn't too expensive. Is loadbalancer.org what I'm looking for?
>
> --Eric
>
> -----Original Message-----
> From: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx
> [mailto:lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx] On Behalf Of Malcolm
> Turnbull
> Sent: Saturday, September 19, 2015 1:24 AM
> To: LinuxVirtualServer.org users mailing list.
> <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
> Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t
>
> Eric,
>
> That's great news, glad to hear that it is working!
>
> Fudge sounds nice :-). But maybe just spread the word that Loadbalancer.org
> are nice people?
> http://loadbalancer.org/blog/so-just-how-should-you-compare-load-balancers
>
> I remember now that we did some work with Horms to fix GRO for a customer of
> ours who needed it (but RHEL 6.3 probably doesn't have it)
>
> Dug out of my email:
> In all 2.6.39 was a rather busy release for LVS with a total of 92 patches.
> Some highlights are:
>
> Features
> * Network namespace support
> * New synchronisation protocol
> * Optimised statistics reading
>
> Bug Fixes
> * Allow transmit of GRO aggregated skbs
> * Allow fragmented packets to be handled by SIP persistence engine
> * Correct zeroing of statistics
>
>
> On 19 September 2015 at 08:28, Eric Robinson <eric.robinson@xxxxxxxxx> wrote:
>> Malcom -- My wife makes wonderful fudge at Christmas time. Where should I
>> send yours?
>>
>> Because...
>>
>> ethtool -K p2p1 gro off
>> ethtool -K p2p2 gro off
>> ethtool -K p2p3 gro off
>> ethtool -K p2p4 gro off
>>
>> ...totally fixed it! The change was instantaneous. I didn't even have to
>> down/up the interfaces.
>>
>> I'm sitting here stunned. The application isn't freezing any more. I hardly
>> know what to do with myself.
>>
>> Thanks!
>>
>> --Eric
>>
>> -----Original Message-----
>> From: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx
>> [mailto:lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx] On Behalf Of Malcolm
>> Turnbull
>> Sent: Saturday, September 19, 2015 12:05 AM
>> To: LinuxVirtualServer.org users mailing list.
>> <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
>> Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went
>> to Sh*t
>>
>> Eric,
>>
>> Ah , OK I thought that might be too simple :-).
>>
>> Some general advice from our end is:
>> We have had problems with both LRO & GRO in the past (we disable them both),
>> and we also run irqbalance (to save on CPU overhead from network traffic).
>> Also we tend to use Intel NICs with the latest drivers.
>> We do use Centos 6.3 (roughly) but with a more recent Kernel so I'm afraid I
>> can't be much help.
>>
>>
>>
>>
>>
>> On 18 September 2015 at 23:54, Eric Robinson <eric.robinson@xxxxxxxxx> wrote:
>>> That's the odd part. We don't have any WRR health checks at all. Everything
>>> is WLC. And the ldirectord.cf file did not change, not did the version of
>>> ldirectord itself. I have checked through the file, and there are no
>>> virtual servers that are not backed by realservers, and there are no
>>> realservers showing in a down state. I would post the file but it is quite
>>> long. We have approximately 1700 virtual services configured.
>>>
>>> --Eric
>>>
>>>
>>> -----Original Message-----
>>> From: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx
>>> [mailto:lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx] On Behalf Of
>>> Malcolm Turnbull
>>> Sent: Friday, September 18, 2015 2:20 PM
>>> To: LinuxVirtualServer.org users mailing list.
>>> <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
>>> Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went
>>> to Sh*t
>>>
>>> Eric,
>>>
>>> What kind of health checks are you using, it sounds like no backend servers
>>> available in the cluster?
>>>
>>> On 18 September 2015 at 17:11, Eric Robinson <eric.robinson@xxxxxxxxx>
>>> wrote:
>>>> ...oh, and one other thing. After the upgrade, I am now seeing these
>>>> syslog messages once approximately per second...
>>>>
>>>> Sep 18 09:06:26 lb01a kernel: IPVS: WRR: no destination available:
>>>> no destinations present Sep 18 09:06:28 lb01a kernel: IPVS: WRR: no
>>>> destination available: no destinations present Sep 18 09:06:29 lb01a
>>>> kernel: IPVS: WRR: no destination available: no destinations present
>>>> Sep 18 09:06:31 lb01a kernel: IPVS: WRR: no destination available:
>>>> no destinations present Sep 18 09:06:32 lb01a kernel: IPVS: WRR: no
>>>> destination available: no destinations present Sep 18 09:06:34 lb01a
>>>> kernel: IPVS: WRR: no destination available: no destinations present
>>>> Sep 18 09:06:35 lb01a kernel: IPVS: WRR: no destination available:
>>>> no destinations present Sep 18 09:06:37 lb01a kernel: IPVS: WRR: no
>>>> destination available: no destinations present Sep 18 09:06:38 lb01a
>>>> kernel: IPVS: WRR: no destination available: no destinations present
>>>> Sep 18 09:06:40 lb01a kernel: IPVS: WRR: no destination available:
>>>> no destinations present Sep 18 09:06:41 lb01a kernel: IPVS: WRR: no
>>>> destination available: no destinations present Sep 18 09:06:43 lb01a
>>>> kernel: IPVS: WRR: no destination available: no destinations present
>>>> Sep 18 09:06:44 lb01a kernel: IPVS: WRR: no destination available:
>>>> no destinations present
>>>>
>>>> _______________________________________________
>>>> Please read the documentation before posting - it's available at:
>>>> http://www.linuxvirtualserver.org/
>>>>
>>>> LinuxVirtualServer.org mailing list -
>>>> lvs-users@xxxxxxxxxxxxxxxxxxxxxx Send requests to
>>>> lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
>>>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>>
>>>
>>>
>>> --
>>> Regards,
>>>
>>> Malcolm Turnbull.
>>>
>>> Loadbalancer.org Ltd.
>>> Phone: +44 (0)330 1604540
>>> http://www.loadbalancer.org/
>>>
>>> _______________________________________________
>>> Please read the documentation before posting - it's available at:
>>> http://www.linuxvirtualserver.org/
>>>
>>> LinuxVirtualServer.org mailing list -
>>> lvs-users@xxxxxxxxxxxxxxxxxxxxxx Send requests to
>>> lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
>>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>>
>>> _______________________________________________
>>> Please read the documentation before posting - it's available at:
>>> http://www.linuxvirtualserver.org/
>>>
>>> LinuxVirtualServer.org mailing list -
>>> lvs-users@xxxxxxxxxxxxxxxxxxxxxx Send requests to
>>> lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
>>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>
>>
>>
>> --
>> Regards,
>>
>> Malcolm Turnbull.
>>
>> Loadbalancer.org Ltd.
>> Phone: +44 (0)330 1604540
>> http://www.loadbalancer.org/
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
>> Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
>> Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
>
>
> --
> Regards,
>
> Malcolm Turnbull.
>
> Loadbalancer.org Ltd.
> Phone: +44 (0)330 1604540
> http://www.loadbalancer.org/
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx Send
> requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
> Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
--
Regards,
Malcolm Turnbull.
Loadbalancer.org Ltd.
Phone: +44 (0)330 1604540
http://www.loadbalancer.org/
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
