Re: LVS and NFS

To: <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: LVS and NFS
From: "Don Hinshaw" <dwh@xxxxxxxxxxxxxxxxx>
Date: Sat, 26 May 2001 01:34:29 -0000
Shain Miley <smiley@xxxxxxxxxx> said:

> Well I am not 100% sure on the implementation..I am trying to get an
> idea in my head about the best way to go about this.  Right now we have
> an IMAP server running WU-IMAP.  This one machine is doing both IMAP and
> SMTP.  I plan to split these services up and smtp can be delt with by
> another less powerful machine.  

I use Qmail+(many patches) for SMTP, Vpopmail for a single UID mail sandbox 
(shell accounts my ass, not on this rig), and Courier-Imap. Vpopmail is 
configured to store userinfo in MySQL and Courier-Imap auths out of 
Vpopmail's tables.

Everything is installed in /var/qmail/* and that /var/qmail/ is an NFS v3 
export from a RAID server. All servers connect to a dedicated MySQL server 
that also stores it's databases on another NFS share from the RAID. Also each 
server mounts /www from the RAID.

Each realserver runs all services, smtpd, imapd, httpd and dns. I use TWIG as 
a webmail imap client, which is configured to connect to imapd on localhost 
(each server does this). Incoming smtp, httpd and dns requests are load 
balanced, but not imapd, since they are local connections on each server. 
Each server stores it's logs locally, then they are combined with a cron 
script and moved to the raid.

It's been working very well in a devel environment for over a year (round-
robin dns, not lvs). I've recently begun the project to rebuild the system 
and scale it up into a commercially viable system, which is quite a task 
since most of the software packages are at least a year old, and I'll be 
using a pair of LVS directors instead of the RRDNS.

> Users will also be using some sort of webmail (IMP/HORDE) to get their
> mail when they are off site...other than that standard Eudora/Netscape
> will be used for retrieval.

I settled on TWIG mainly because of it's vhost support. With Vpopmail, I can 

# /var/qmail/vpopmail/bin/vadddomain <postmaster passwd>

and add that domain to dns and begin adding users and serving it up. I had to 
tweak TWIG just a bit to get it to properly deal with the "user@domain" style 
login that Vpopmail requires, but otherwise it works great. Each vhost domain 
can have it's own config file, but there is only one copy of TWIG in /www. 
TWIG uses MySQL, and though it doesn't require it, I also create a seperate 
database for each vhost domain.

IMP's development runs along at about Mach 0.00000000004 and I got tired of 
waiting for them to get a working addressbook. That plus it doesn't vhost all 
that well. SquirrelMail is very nice, but again not much vhost support. Plus 
TWIG includes the kitchen sink. Email, contacts, schedule, notes, todo, 
bookmarks and even USENET (which I don't use), each module can be 
enabled/disabled in the config file for that domain, and it's got a very 
complete advanced security module (which I also don't use). It's all PHP and 
using mod_gzip is pretty fast. I tested the APC optimizer for PHP, but every 
time I made a change to a script I had to reload Apache. Not very handy for a 
devel system, but it did add some noticable speed increases, until I unloaded 

> The real servers would need access to both the users home directories as
> well as the /var/mail directory. I am not too familiar with the actual
> locking problems...I understand the basics but I also hear that NFS V3
> was supposed to fix some of the locking issues with V2...I also saw some
> links to GFS,AFS,etc not too sure how they would work...

Just a quick note: About a week ago I tried compiling a kernel that had been 
patched by SGI for XFS. The kernel (2.4.2) compiled fine, but choked once the 
LVS patches had been applied. Not having a lot of time to play around with 
it, I simply moved to 2.4.4+lvs 0.9 and decided not to bother with XFS on the 
director boxes.

> also I thought
> about samba and only found one post from last year where someone was
> going to try it but there was no more info there.  I am very willing to
> take suggestions :-)

Well, there's how I do it. I've tried damned near every combination of GPL 
software available over about the last 2 years to finally arrive at my 
current setup. Now if I could just load balance MySQL...


> Thanks,
> Shain
> Joseph Mack wrote:
> > 
> > Shain Miley wrote:
> > >
> > > Hi,
> > > I am new to the list and I was wondering if someone could either provide
> > > a link or some details about the maturity of LVS
> > 
> > LVS is in great shape :-)
> > 
> > and the available
> > > 'shared disk' options.
> > 
> > it depends a little bit on how you're going to share the files
> > (which you haven't told us about). If the
> > NFS is just sharing files like is does anywhere else, then you'll
> > only have the usual troubles (which you allude to).
> > 
> > > I am planning on setting up an LVS IMAP cluster
> > 
> > do you understand the single writer, many reader properties of LVS?
> > 
> > One of the first things LVS was used for was for a mail server
> > (by Ted Pavlic). He had to do a lot of thinking to get all the
> > locking straight and unfortunately he never wrote it up for us.
> > So unfortunately what you're (probably) about to do isn't a canned
> > setup yet. It's also probably a little tricky and you're going
> > to have to test it with single mail events and then run something
> > hard against it to see if you loose test mails. Even loosing
> > 1 mail in 1000 will annoy users (quite rightly), but will
> > require some accurate testing.
> > 
> > Can you describe more how you plan to do this (eg how the IMAP
> > servers get their mail), how the clients get the mail from the
> > IMAP servers, how the client send their mail out, where the LVS
> > fits in? We're very good with free advice :-)
> > 
> > I haven't setup an LVS mail server. How big a single machine is
> > needed to handle your 500-700 people?
> > 
> > Joe
> > 
> > --
> > Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
> > contractor to the National Environmental Supercomputer Center,
> > mailto:mack.joseph@xxxxxxx ph# 919-541-0007, RTP, NC, USA
> _______________________________________________
> mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
> Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
> or go to


________________________________________________________________ For musicians by musicians.
Now with free Web-Based email too!

<Prev in Thread] Current Thread [Next in Thread>