LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: [lvs-users] tuning http/https uploads through ipvs

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: [lvs-users] tuning http/https uploads through ipvs
From: David Warden <warden@xxxxxxxxxxx>
Date: Fri, 24 Aug 2012 22:56:04 -0400
We actually went through something similar with lvs-frontended Roundcube 
webmail, although our upload size limits were considerably lower than yours. 
Our behavior was mostly similar to yours: fast uploads when going directly to 
the realserver, slow uploads when going through the vip/director.

In our case the fix was tuning the ethtool -k <interface> values on the actual 
interface that carries the vip aliased interface on the lvs director. We didn't 
see errors on any interfaces either which confused us at first but packet 
captures led us to suspect the ethtool -k settings. I'm pretty sure we turned 
large receive offload, tcp segmentation offload and generic segmentation off 
and that seemed to make a difference. I understand your hesitation to tweak 
those kinds of settings but we haven't seen a negative impact since making the 
change and have been running with them in their current fashion for the better 
part of this year.

-David

On Aug 24, 2012, at 5:35 PM, Tim Mooney wrote:

> 
> All-
> 
> I recently replaced our director with a newer system (newer hardware, newer
> OS & kernel, newer ipvsadm) and although nearly everything is functioning
> very well, since the start of the fall semester we've had some complaints
> about slow file uploads to our Blackboard learning management system
> (LMS).  Normally I would immediately assume the problem is Blackboard
> (because it always is), but this time it looks like the issue may at least
> relate to the load balancer.
> 
> Our director:
> 
> Hardware: Intel S5500 motherboard, dual Intel Xeon E5540 @ 2.53 GHz
> (hyperthreading is enabled), 8 GiB RAM, onboard Intel Gigabit NIC
> OS: RHEL 6.3, x86_64
> Kernel: RHEL's 2.6.32-279.2.1
> ipvsadm: 1.25-9.el6 (from EPEL)
> 
> The director fronts several services (LDAP, smtp, unrelated http and
> https), but with respect to Blackboard, we're load balancing both http
> (which is little used) and https:
> 
> 
> $sudo ipvsadm -L -t bb.ndsu.nodak.edu:http 
> Prot LocalAddress:Port Scheduler Flags
>   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
> TCP  bb.ndsu.NoDak.edu:http wlc persistent 10860
>   -> blkbrd-app1.ndsu.NoDak.edu:h Route   1      0          3
>   -> blkbrd-app2.ndsu.NoDak.edu:h Route   1      0          0
>   -> blkbrd-app3.ndsu.NoDak.edu:h Route   1      0          0
> 
> 
> $ sudo ipvsadm -L -t bb.ndsu.nodak.edu:https
> Prot LocalAddress:Port Scheduler Flags
>   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
> TCP  bb.ndsu.NoDak.edu:https wlc persistent 10860
>   -> blkbrd-app1.ndsu.NoDak.edu:h Route   1      98         910
>   -> blkbrd-app2.ndsu.NoDak.edu:h Route   1      97         580
>   -> blkbrd-app3.ndsu.NoDak.edu:h Route   1      90         550
> 
> We are using DR and the real servers are correctly configured to avoid
> the ARP problem.  This isn't our first rodeo; we've been using ipvs for
> more than a decade, though I will say that it's been so trouble-free for
> us that it's certainly possible I've forgotten something when setting up
> our recent refresh of the director.
> 
> We're not having any complaints from users about file *downloads*, where
> the return traffic would of course be direct because of DR, but our
> instructors have been complaining that file uploads are quite slow.
> Obviously with uploads, a much larger percentage of the traffic goes
> through the director.
> 
> Our blackboard admin has done repeated timings uploading a 624 MB file
> (a RHEL CD ISO) to Blackboard, first by uploading through the https
> virtual service on the director, and then by doing the upload by
> connecting directly to the each of the real servers, avoiding the vip.
> 
> When he uploads directly to a realserver, it takes about 80-90 seconds to
> upload the ISO, so about as close to line rate as we can expect for the
> 100 MB connection from his workstation.
> 
> When he uploads through the director, it reliably takes 4 minutes 
> to 4 minutes 30 seconds.  It's essentially taking three times as long
> to do the upload through the director as it is to do it directly.
> 
> We're not seeing any errors or collisions on the director's NIC.  Our
> network folks have also looked at the switch port and report no problems.
> The NIC has autonegotiated at 1 Gigabit full-duplex with the switch.  Our
> network staff also indicate that even the largest traffic spike they've
> seen on the director's port is only about 24 MiB.
> 
> At this point I'm thinking it likely has to be some TCP tuning that needs
> to be done, but I'm a little hesitant to just tune for large file
> transfers because the majority of the other services fronted by the
> director don't need that.
> 
> Can anyone provide some suggestions for things to look at or tunables I
> should consider?
> 
> Any thoughts appreciated!
> 
> Tim
> -- 
> Tim Mooney                                             Tim.Mooney@xxxxxxxx
> Enterprise Computing & Infrastructure                  701-231-1076 (Voice)
> Room 242-J6, IACC Building                             701-231-8541 (Fax)
> North Dakota State University, Fargo, ND 58105-5164
> 
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
> 
> LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
> Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>