Re: [lvs-users] RX Packet drops on high traffic LVS in DR setup

[Sashi Kant <sashi.kant@xxxxxxxxxxxxx>]

Thanks Graeme,

We have bonding setup in all the servers and it works fine.

I agree with your observation about packet per second problem , rather
than MB/s issue. I was able to push > 800 Mb/s of traffic to similar
setup (bonding-rr + Link aggregation )using a single machine as traffic
generator.

We start seeing dropped packet when traffic goes above 40k pps (tx+rx). 

Situation was improved when we start using both NIC in round robin mode,
however we did not see 2X improvement as expected. Could PCI-X bus limit
be playing a role ?

  *-pci:6
             description: PCI bridge
             product: 631xESB/632xESB/3100 Chipset PCI Express Root Port
1
             vendor: Intel Corporation
             physical id: 1c
             bus info: pci@00:1c.0
             version: 09
             width: 32 bits
             clock: 33MHz
             capabilities: pci normal_decode bus_master cap_list
             configuration: driver=pcieport-driver
           *-pci
                description: PCI bridge
                product: EPB PCI-Express to PCI-X Bridge
                vendor: Broadcom
                physical id: 0
                bus info: pci@02:00.0
                version: c3
                width: 32 bits
                clock: 33MHz
                capabilities: pci normal_decode bus_master cap_list
              *-network DISABLED
                   description: Ethernet interface
                   product: NetXtreme II BCM5708 Gigabit Ethernet
                   vendor: Broadcom Corporation
                   physical id: 0
                   bus info: pci@03:00.0
                   logical name: eth3
                   version: 12
                   serial: 00:1d:09:6a:db:aa
                   width: 64 bits
                   clock: 66MHz
                   capabilities: bus_master cap_list ethernet physical
                   configuration: broadcast=yes driver=bnx2 latency=32
mingnt=64 multicast=yes
                   resources: iomemory:da000000-dbffffff irq:169


We are using bnx2 driver from dist kernel.

-  

Thanks
-Sashi


On Thu, 2008-10-16 at 09:39 +0100, Graeme Fowler wrote:
> On Wed, 2008-10-15 at 10:50 -0700, Sashi Kant wrote:
> > We experiencing lots of RX packets being dropped on LVS server. 
> > Moving some high traffic VIPs to other LVS which reduced rate of packet
> > drops however not completely eliminated. 
> > 
> > Anyone seen this issue or have insight into it ?
> 
> Nope.
> 
> I might suggest, however, that the combination of Broadcom NICs, bonded,
> and the Cisco device is the cause of the problem.
> 
> If you switch to a single NIC (for testing) with no bonding, does the
> problem go away or does it get worse?
> 
> It seems that you may need to do some system tuning to get your system
> to handle the high packet rates you have - it's likely the pps, not the
> bps, which is the source of the problem.
> 
> What Broadcom driver is being used?
> 
> Graeme
> 
> 
> _______________________________________________
> LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
> Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users