LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: [lvs-users] SYN spiraling between master and slave IPVS balancers

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: [lvs-users] SYN spiraling between master and slave IPVS balancers
Cc: Julian Anastasov <ja@xxxxxx>
From: Dmitry Akindinov <dimak@xxxxxxxxxxx>
Date: Mon, 18 Feb 2013 15:50:40 +0400
Hello,

I'd like to return to the discussion (some excerpts are below) to get 
some ideas on how we can plan development of our product in regards to 
IPVS development.

Can we expect that the synchronization protocol changes to avoid this 
SYNC spiraling?

Or, the suggested changes to the kernel - I doubt we can afford 
ourselves maintaining a custom version of the kernel for our customers, 
- so the question is whether these changes are planned for a commit to 
the main development branch of the kernel? The idea with the kernel 
variable so a control over the behavior is possible in run-time looks good.

Thank you.

On 2013-02-07 01:13, Julian Anastasov wrote:
>
>       Hello,
>
> On Tue, 5 Feb 2013, Dmitry Akindinov wrote:
>
>> Hello,
>>
>> We have met a quite troublesome situation which causes an internal SYN
>> storm.
>>

[]

>> The server A processes it again, directs it to the server B again, and
>> the loop spirals, since the server B does not have the new connection
>> table element I synced.
>
>       More likely the SYN comes short after the conn
> in server A is expired but the synced conn in server B
> is not expired yet. This can happen often because
> the sync protocol is not perfect, conns in backup
> tend to expire later.
>
[]

>       May be the problem will move from the normal conns
> to the persistent conn templates. The simplest solution is
> to use:
>
>       if (ipvs->sync_state & IP_VS_STATE_BACKUP)
>               return NF_ACCEPT;
>
>       in all hooks. This will stop all traffic.
> The problem is that we do not know if the backup
> function is used for part of the virtual services,
> other virtual services can be in normal mode,
> possibly using IP_VS_STATE_MASTER. I assume that
> was the reason the master and backup functions to
> be able to run together, with different sync_id.
>
>       I'm not sure what is more appropriate
> to apply here, some sysctl var that will stop
> the forwarding mode while backup function is
> enabled. This solution will work better if we
> don't want to change the tools that manage IPVS
> rules and it is easier to implement, eg.
> "backup_only=1" to activate such mode. The
> result would be:
>
>       if (ipvs->sync_state & IP_VS_STATE_BACKUP &&
>           ipvs->backup_only)
>               return NF_ACCEPT; /* Try local server */
>
>       Another solution would be to add
> optional syncid attribute to the virtual
> server. By this way we will know that received
> packet matches the backup_syncid (we are
> used as real server from director) or else it is
> directed from client to our virtual server. In the
> second case if the packet matches master_syncid (as
> additional check to the IP_VS_STATE_MASTER flag) a sync
> message would be sent.

[]

-- 
Best regards,
Dmitry Akindinov



_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>