LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t
From: Malcolm Turnbull <malcolm@xxxxxxxxxxxxxxxx>
Date: Sat, 19 Sep 2015 09:23:31 +0100
Eric,

That's great news, glad to hear that it is working!

Fudge sounds nice :-). But maybe just spread the word that
Loadbalancer.org are nice people?
http://loadbalancer.org/blog/so-just-how-should-you-compare-load-balancers

I remember now that we did some work with Horms to fix GRO for a
customer of ours who needed it (but RHEL 6.3 probably doesn't have it)

Dug out of my email:
In all 2.6.39 was a rather busy release for LVS with a total of 92 patches.
Some highlights are:

Features
* Network namespace support
* New synchronisation protocol
* Optimised statistics reading

Bug Fixes
* Allow transmit of GRO aggregated skbs
* Allow fragmented packets to be handled by SIP persistence engine
* Correct zeroing of statistics


On 19 September 2015 at 08:28, Eric Robinson <eric.robinson@xxxxxxxxx> wrote:
> Malcom -- My wife makes wonderful fudge at Christmas time. Where should I 
> send yours?
>
> Because...
>
> ethtool -K p2p1 gro off
> ethtool -K p2p2 gro off
> ethtool -K p2p3 gro off
> ethtool -K p2p4 gro off
>
> ...totally fixed it! The change was instantaneous. I didn't even have to 
> down/up the interfaces.
>
> I'm sitting here stunned. The application isn't freezing any more. I hardly 
> know what to do with myself.
>
> Thanks!
>
> --Eric
>
> -----Original Message-----
> From: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx 
> [mailto:lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx] On Behalf Of Malcolm 
> Turnbull
> Sent: Saturday, September 19, 2015 12:05 AM
> To: LinuxVirtualServer.org users mailing list. 
> <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
> Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t
>
> Eric,
>
> Ah , OK I thought that might be too simple :-).
>
> Some general advice from our end is:
> We have had problems with both LRO & GRO in the past (we disable them both), 
> and we also run irqbalance (to save on CPU overhead from network traffic).
> Also we tend to use Intel NICs with the latest drivers.
> We do use Centos 6.3 (roughly) but with a more recent Kernel so I'm afraid I 
> can't be much help.
>
>
>
>
>
> On 18 September 2015 at 23:54, Eric Robinson <eric.robinson@xxxxxxxxx> wrote:
>> That's the odd part. We don't have any WRR health checks at all. Everything 
>> is WLC. And the ldirectord.cf file did not change, not did the version of 
>> ldirectord itself. I have checked through the file, and there are no virtual 
>> servers that are not backed by realservers, and there are no realservers 
>> showing in a down state. I would post the file but it is quite long. We have 
>> approximately 1700 virtual services configured.
>>
>> --Eric
>>
>>
>> -----Original Message-----
>> From: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx
>> [mailto:lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx] On Behalf Of Malcolm
>> Turnbull
>> Sent: Friday, September 18, 2015 2:20 PM
>> To: LinuxVirtualServer.org users mailing list.
>> <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
>> Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went
>> to Sh*t
>>
>> Eric,
>>
>> What kind of health checks are you using, it sounds like no backend servers 
>> available in the cluster?
>>
>> On 18 September 2015 at 17:11, Eric Robinson <eric.robinson@xxxxxxxxx> wrote:
>>> ...oh, and one other thing. After the upgrade, I am now seeing these syslog 
>>> messages once approximately per second...
>>>
>>> Sep 18 09:06:26 lb01a kernel: IPVS: WRR: no destination available: no
>>> destinations present Sep 18 09:06:28 lb01a kernel: IPVS: WRR: no
>>> destination available: no destinations present Sep 18 09:06:29 lb01a
>>> kernel: IPVS: WRR: no destination available: no destinations present
>>> Sep 18 09:06:31 lb01a kernel: IPVS: WRR: no destination available: no
>>> destinations present Sep 18 09:06:32 lb01a kernel: IPVS: WRR: no
>>> destination available: no destinations present Sep 18 09:06:34 lb01a
>>> kernel: IPVS: WRR: no destination available: no destinations present
>>> Sep 18 09:06:35 lb01a kernel: IPVS: WRR: no destination available: no
>>> destinations present Sep 18 09:06:37 lb01a kernel: IPVS: WRR: no
>>> destination available: no destinations present Sep 18 09:06:38 lb01a
>>> kernel: IPVS: WRR: no destination available: no destinations present
>>> Sep 18 09:06:40 lb01a kernel: IPVS: WRR: no destination available: no
>>> destinations present Sep 18 09:06:41 lb01a kernel: IPVS: WRR: no
>>> destination available: no destinations present Sep 18 09:06:43 lb01a
>>> kernel: IPVS: WRR: no destination available: no destinations present
>>> Sep 18 09:06:44 lb01a kernel: IPVS: WRR: no destination available: no
>>> destinations present
>>>
>>> _______________________________________________
>>> Please read the documentation before posting - it's available at:
>>> http://www.linuxvirtualserver.org/
>>>
>>> LinuxVirtualServer.org mailing list -
>>> lvs-users@xxxxxxxxxxxxxxxxxxxxxx Send requests to
>>> lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
>>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>
>>
>>
>> --
>> Regards,
>>
>> Malcolm Turnbull.
>>
>> Loadbalancer.org Ltd.
>> Phone: +44 (0)330 1604540
>> http://www.loadbalancer.org/
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
>> Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
>> Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
>
>
> --
> Regards,
>
> Malcolm Turnbull.
>
> Loadbalancer.org Ltd.
> Phone: +44 (0)330 1604540
> http://www.loadbalancer.org/
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx Send 
> requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
> Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users



-- 
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)330 1604540
http://www.loadbalancer.org/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>