Re: [lvs-users] ipvs does not sync DNATted or fwmarked connection state

To: Patrick Schaaf <netdev@xxxxxx>
Subject: Re: [lvs-users] ipvs does not sync DNATted or fwmarked connection state
Cc: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
From: Simon Horman <horms@xxxxxxxxxxxx>
Date: Tue, 21 Dec 2010 07:41:06 +0900
On Mon, Dec 20, 2010 at 09:43:35PM +0100, Patrick Schaaf wrote:
> Is the following known / does a solution exist?
> I'm setting up two machines with kernel as master/backup ipvs
> directors, with keepalived checking real servers and implementing vrrp
> failover.
> Virtual service is for HTTP connections, using NAT method towards the
> real servers.
> The basic setup has been working fine, with an exemplary set of three
> virtual IPs balancing to some real servers, replicating connection state
> (ipvsadm -ln counters increasing on the backup, -lc state visible
> there).
> However, for the production setup, I have to implement roughly 200
> different virtual IP addresses, all running onto the same (rather small)
> set of real servers.
> As is well known, doing that with the corresponding number of different
> ipvs virtual services presents problems, as the real server state
> (connection count) is kept for each individual virtual service,
> resulting in suboptimal balancing.
> As a solution to that, I have been testing two different approaches:
> 1) using fwmark, with --set-mark in the mangle table to mark the
> incoming packets for the different virtual IPs, and an fwmark virtual
> service set up as usual.
>       iptables -t mangle -A PREROUTING -m ... -j MARK --set-mark 80
>         ipvsadm -A -f 80 ...
> and alternatively
> 2) using iptables DNAT in PREROUTING to rewrite the various virtual IPs
> to specific (few) virtual IPs set up as ipvs services.
>         iptables -t nat -A PREROUTING -m ... -j DNAT --to-dest
>         ipvsadm -A -t ...
> Both approaches work fine WRT balancing, reaching the real servers, and
> everything.
> BUT: no connection state is synchronized, in either of the approaches.
> The backup server does not show -ln counter increase, nor -lc
> connections, when I test it.
> I have even set up the fully working (normal) approach at the same time
> as as 1) and/or 2), for different addresses, and the sync-to-backup is
> working OK for the normal addresses, but not sending connection state
> for  stuff covered by approaches 1) or 2).
> Any suggestions as to why this happens? Patches to apply? Good chance
> 2.6.37-rcX could work? More info needed?

Hi Patrick,

while there are a number of limitations in the synchronisation code
I believe that what you are trying to do should work. On the backup
do you see connections showing up in the output of ipvsadm -Lcn ?

On the topic of connection synchronisation, a new synchronisation
protocol (v1) has been developed which addresses a number of the problems
in the existing code (v0). This includes the explicit synchronisation
of fwmarks. The code is currently available in the lvs-test-2.6 tree
on It should appear in 2.6.38-rc1.

Please read the documentation before posting - it's available at: mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to

<Prev in Thread] Current Thread [Next in Thread>