|
All-
I recently replaced our director with a newer system (newer hardware, newer
OS & kernel, newer ipvsadm) and although nearly everything is functioning
very well, since the start of the fall semester we've had some complaints
about slow file uploads to our Blackboard learning management system
(LMS). Normally I would immediately assume the problem is Blackboard
(because it always is), but this time it looks like the issue may at least
relate to the load balancer.
Our director:
Hardware: Intel S5500 motherboard, dual Intel Xeon E5540 @ 2.53 GHz
(hyperthreading is enabled), 8 GiB RAM, onboard Intel Gigabit NIC
OS: RHEL 6.3, x86_64
Kernel: RHEL's 2.6.32-279.2.1
ipvsadm: 1.25-9.el6 (from EPEL)
The director fronts several services (LDAP, smtp, unrelated http and
https), but with respect to Blackboard, we're load balancing both http
(which is little used) and https:
$sudo ipvsadm -L -t bb.ndsu.nodak.edu:http
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP bb.ndsu.NoDak.edu:http wlc persistent 10860
-> blkbrd-app1.ndsu.NoDak.edu:h Route 1 0 3
-> blkbrd-app2.ndsu.NoDak.edu:h Route 1 0 0
-> blkbrd-app3.ndsu.NoDak.edu:h Route 1 0 0
$ sudo ipvsadm -L -t bb.ndsu.nodak.edu:https
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP bb.ndsu.NoDak.edu:https wlc persistent 10860
-> blkbrd-app1.ndsu.NoDak.edu:h Route 1 98 910
-> blkbrd-app2.ndsu.NoDak.edu:h Route 1 97 580
-> blkbrd-app3.ndsu.NoDak.edu:h Route 1 90 550
We are using DR and the real servers are correctly configured to avoid
the ARP problem. This isn't our first rodeo; we've been using ipvs for
more than a decade, though I will say that it's been so trouble-free for
us that it's certainly possible I've forgotten something when setting up
our recent refresh of the director.
We're not having any complaints from users about file *downloads*, where
the return traffic would of course be direct because of DR, but our
instructors have been complaining that file uploads are quite slow.
Obviously with uploads, a much larger percentage of the traffic goes
through the director.
Our blackboard admin has done repeated timings uploading a 624 MB file
(a RHEL CD ISO) to Blackboard, first by uploading through the https
virtual service on the director, and then by doing the upload by
connecting directly to the each of the real servers, avoiding the vip.
When he uploads directly to a realserver, it takes about 80-90 seconds to
upload the ISO, so about as close to line rate as we can expect for the
100 MB connection from his workstation.
When he uploads through the director, it reliably takes 4 minutes
to 4 minutes 30 seconds. It's essentially taking three times as long
to do the upload through the director as it is to do it directly.
We're not seeing any errors or collisions on the director's NIC. Our
network folks have also looked at the switch port and report no problems.
The NIC has autonegotiated at 1 Gigabit full-duplex with the switch. Our
network staff also indicate that even the largest traffic spike they've
seen on the director's port is only about 24 MiB.
At this point I'm thinking it likely has to be some TCP tuning that needs
to be done, but I'm a little hesitant to just tune for large file
transfers because the majority of the other services fronted by the
director don't need that.
Can anyone provide some suggestions for things to look at or tunables I
should consider?
Any thoughts appreciated!
Tim
--
Tim Mooney Tim.Mooney@xxxxxxxx
Enterprise Computing & Infrastructure 701-231-1076 (Voice)
Room 242-J6, IACC Building 701-231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
