[lvs-users] LVS rejecting large packets

To: " users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: [lvs-users] LVS rejecting large packets
From: Roger Littin <roger@xxxxxxxxxxxxx>
Date: Thu, 24 Nov 2011 14:10:43 +1300
Hi Simon,

I didn't want to hijack the other thread so I have started a new one.

The issue I was seeing was with tcp packets to media servers behind LVS.
When the media streams are going out to the clients, everything is fine as the 
packets going through LVS are tiny.

When I try to send incoming streams through LVS, I start seeing issues 
sometimes where an icmp is sent back to the client stating the packet is too 
large to forward to the real server.  This causes the incoming stream to get 
further and further behind.

Originally, on all the servers, I had mtu set to 1500 and tcpdump would be 
showing some packets that were larger than this and would be rejected.  I 
increased the mtu to 9000 but then tcpdump simply reported that the same 
clients were sending packets larger than 9000 bytes. 

The last tests I did was on a server that I no longer have access to so I 
cannot be sure of the settings then.

My current servers are running Centos 5.7 2.6.18-274.7.1.el5 and have GRO off.  
LRO is not listed.

[root@laxlb01 ~]# ethtool -k eth0
Offload parameters for eth0:
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off
generic-receive-offload: off

I see that tcp segmentation offload is set to on.  Will this cause the same 

I would really like to get to the bottom of what is causing this problem so 
that I can run the incoming streams through the LVS.  At the moment, we are 
balancing these with DNS which is not ideal.



On 24/11/2011, at 1:09 PM, Simon Horman wrote:

> Hi Roger,
> if you are seeing that problem then I suggest turning off
> both GRO and LRO on the interface on the director that receives
> the packets.
> e.g.
> # ethtool -k eth0 | egrep '(receive-offload)'
> generic-receive-offload: on
> large-receive-offload: on
> # ethtool -K eth0 lro off
> # ethtool -K eth0 gro off
> # ethtool -k eth0 | egrep '(receive-offload)'
> generic-receive-offload: off
> large-receive-offload: off
> IPVS may be used in conjunction with GRO since 2.6.39,
> so it should not be necessary to disable GRO if
> a 2.6.39 is used.
> IPVS is not compatible with LRO as of 3.1 (or 3.2, when it is released)
> LRO is specific to TCP.
> GRO affects both TCP and UDP (and other protocols IIRC).

Please read the documentation before posting - it's available at: mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to

<Prev in Thread] Current Thread [Next in Thread>