LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: [lvs-users] SYN spiraling between master and slave IPVS balancers

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: [lvs-users] SYN spiraling between master and slave IPVS balancers
Cc: Julian Anastasov <ja@xxxxxx>
From: Dmitry Akindinov <dimak@xxxxxxxxxxx>
Date: Mon, 04 Mar 2013 14:00:26 +0400
Hello,

On 2013-02-21 13:10, Julian Anastasov wrote:
>
>       Hello,
>
> On Mon, 18 Feb 2013, Dmitry Akindinov wrote:
>
>> Hello,
>>
>> I'd like to return to the discussion (some excerpts are below) to get some
>> ideas on how we can plan development of our product in regards to IPVS
>> development.
>>
>> Can we expect that the synchronization protocol changes to avoid this SYNC
>> spiraling?
>>
>> Or, the suggested changes to the kernel - I doubt we can afford ourselves
>> maintaining a custom version of the kernel for our customers, - so the
>> question is whether these changes are planned for a commit to the main
>> development branch of the kernel? The idea with the kernel variable so a
>> control over the behavior is possible in run-time looks good.
>
>       That is what I have implemented. I hope you will
> find a way to test it and after your feedback and comments
> from others we will post this patch for kernel inclusion.
> This is the link of the patch, I'll post it in separate
> thread too:
>
> http://www.ssi.bg/~ja/tmp/0001-ipvs-add-backup_only-flag-to-avoid-loops.txt

Thank you very much!

It's about ten days since we started to use on our servers kernel 
3.8.0-28 (now 3.8.1-30) with your patch applied. So far, there are no 
signs of SYN spiraling or other problems with using ipvs balancer nodes 
on our multiprotocol real servers. It would be ver nice to have this 
feature in the standard kernels.

Thank you!

>> Thank you.
>>
>> On 2013-02-07 01:13, Julian Anastasov wrote:
>>>
>>>     Hello,
>>>
>>> On Tue, 5 Feb 2013, Dmitry Akindinov wrote:
>>>
>>>> Hello,
>>>>
>>>> We have met a quite troublesome situation which causes an internal SYN
>>>> storm.
>>>>
>>
>> []
>>
>>>> The server A processes it again, directs it to the server B again, and
>>>> the loop spirals, since the server B does not have the new connection
>>>> table element I synced.
>>>
>>>     More likely the SYN comes short after the conn
>>> in server A is expired but the synced conn in server B
>>> is not expired yet. This can happen often because
>>> the sync protocol is not perfect, conns in backup
>>> tend to expire later.
>>>
>> []
>>
>>>     May be the problem will move from the normal conns
>>> to the persistent conn templates. The simplest solution is
>>> to use:
>>>
>>>     if (ipvs->sync_state & IP_VS_STATE_BACKUP)
>>>             return NF_ACCEPT;
>>>
>>>     in all hooks. This will stop all traffic.
>>> The problem is that we do not know if the backup
>>> function is used for part of the virtual services,
>>> other virtual services can be in normal mode,
>>> possibly using IP_VS_STATE_MASTER. I assume that
>>> was the reason the master and backup functions to
>>> be able to run together, with different sync_id.
>>>
>>>     I'm not sure what is more appropriate
>>> to apply here, some sysctl var that will stop
>>> the forwarding mode while backup function is
>>> enabled. This solution will work better if we
>>> don't want to change the tools that manage IPVS
>>> rules and it is easier to implement, eg.
>>> "backup_only=1" to activate such mode. The
>>> result would be:
>>>
>>>     if (ipvs->sync_state & IP_VS_STATE_BACKUP &&
>>>         ipvs->backup_only)
>>>             return NF_ACCEPT; /* Try local server */
>>>
>>>     Another solution would be to add
>>> optional syncid attribute to the virtual
>>> server. By this way we will know that received
>>> packet matches the backup_syncid (we are
>>> used as real server from director) or else it is
>>> directed from client to our virtual server. In the
>>> second case if the packet matches master_syncid (as
>>> additional check to the IP_VS_STATE_MASTER flag) a sync
>>> message would be sent.
>>
>> []
>>
>> --
>> Best regards,
>> Dmitry Akindinov
>
> Regards
>
> --
> Julian Anastasov <ja@xxxxxx>
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
> Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>

-- 
Best regards,
Dmitry Akindinov



_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>