LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

[lvs-users] IMAP slow performance

To: <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: [lvs-users] IMAP slow performance
From: "Dean Manners" <deanm@xxxxxxxxxxxxx>
Date: Mon, 4 May 2009 23:48:49 +1000
Hey guys,
                We have an existing (2yrs running) setup that hasn't
recently changed.  But are now, fairly suddenly, having problems with our
IMAP servers experiencing very slow/unusable performance when connecting via
the virtual IP.


The setup:
Two "directors" - one active, one standby.  
 - Debian Sarge, 2.6.20.1 kernels.
 - Running Ultramonkey
 - Uptimes of:
   director01:~# uptime
    22:17:17 up 510 days
   director02:~# uptime
    22:17:20 up 238 days

Two "mailget" servers - both active, direct routing with loopback addresses,
running IMAP2 & POP3 services on the same virtual IP.  
 - Running Dovecot

All behind a Cisco 6500 switch, running aggregated firewall rules (eg the
same one rule affects both real IP and virtual IP address).


The symptoms/troubleshooting so far:
~ When IMAPing via the virtual IP IMAP responsiveness is very slow and
frequently actions times out.
~ When IMAPing directly to the real servers there is no apparent performance
issue, with speedy responses from both servers.
~ POP3 performance is normal, to same real servers via the same virtual IP
as our troubled IMAP connections.
* Tried restarting dovecot on both mailget servers (tyre kick measure)
* Tried failing over the heartbeat/ultramonkey monitor on director01, so
that director02 took over the vip's/LVS. There was still noticeable/sluggish
performance when using the IMAP virtual IP from this 2nd machine though.  So
that ruled out my director01 being a problem (was thinking the long uptime
may have been at play).
~ Our webmail server however connects IMAP the same way via the load
balanced address and appears to have no problems, large'ish mailboxes load
almost instantly and all actions are responsive.  The only difference I can
note, is that this server is in the same subnet as the directors/real
servers.
~ IMAP connections represent about 15% of total mail connections, most use
POP3:

# ipvsadm --list |grep mail
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  mail.myserver.blah:pop3 wlc persistent 600
  -> mailget01.myserver.blah:pop3 Route   1      54         144
  -> mailget02.myserver.blah:pop3 Route   1      59         173
TCP  mail.myserver.blah:imap2 wlc persistent 600
  -> mailget01.myserver.blah:imap Route   1      12         1
  -> mailget02.myserver.blah:imap Route   1      14         16


The ldirectord.cf (note, has been the same for 2+ years):

   # Global Directives
   checktimeout=3
   checkinterval=15
   autoreload=yes
   quiescent=no
   logfile="/var/log/ldirectord.log"

   # Main POP
   virtual=xxx.xxx.xx.80:110
          real=xxx.xxx.xx.81:110 gate
          real=xxx.xxx.xx.82:110 gate
          service=pop
          scheduler=wlc
          login="monitoring@xxxxxxxxxxxxxx"
          passwd="blahblah"
          persistent=600
          protocol=tcp

   # Main IMAP
   virtual=xxx.xxx.xx.80:143
          real=xxx.xxx.xx.81:143 gate
          real=xxx.xxx.xx.82:143 gate
          service=imap
          scheduler=wlc
          login="monitoring@xxxxxxxxxxxxxx"     
          passwd="blahblah"
          persistent=600
          protocol=tcp


Im at a loss of what to try next, with POP3 and other services (HTTP/HTTPS)
operating as normal from the same directors, no recent changes in configs,
and two directors giving the same symptoms - I cant see where the problem
lies.  Any tips on where to start looking ?


Regards
__________________________________________ 
Dean Manners



_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>