LVS
lvs-devel
Google
 
Web LinuxVirtualServer.org

Re: [PATCH] ipvs: fix race between ip_vs_conn_new() and ip_vs_del_dest()

To: Tan Hu <tan.hu@xxxxxxxxxx>
Subject: Re: [PATCH] ipvs: fix race between ip_vs_conn_new() and ip_vs_del_dest()
Cc: wensong@xxxxxxxxxxxx, horms@xxxxxxxxxxxx, pablo@xxxxxxxxxxxxx, kadlec@xxxxxxxxxxxxxxxxx, fw@xxxxxxxxx, davem@xxxxxxxxxxxxx, netdev@xxxxxxxxxxxxxxx, lvs-devel@xxxxxxxxxxxxxxx, netfilter-devel@xxxxxxxxxxxxxxx, coreteam@xxxxxxxxxxxxx, linux-kernel@xxxxxxxxxxxxxxx, zhong.weidong@xxxxxxxxxx, jiang.biao2@xxxxxxxxxx
From: Julian Anastasov <ja@xxxxxx>
Date: Tue, 24 Jul 2018 22:47:12 +0300 (EEST)
        Hello,

On Tue, 24 Jul 2018, Tan Hu wrote:

> We came across infinite loop in ipvs when using ipvs in docker
> env.
> 
> When ipvs receives new packets and cannot find an ipvs connection,
> it will create a new connection, then if the dest is unavailable
> (i.e. IP_VS_DEST_F_AVAILABLE), the packet will be dropped sliently.
> 
> But if the dropped packet is the first packet of this connection,
> the connection control timer never has a chance to start and the
> ipvs connection cannot be released. This will lead to memory leak, or
> infinite loop in cleanup_net() when net namespace is released like
> this:
> 
>     ip_vs_conn_net_cleanup at ffffffffa0a9f31a [ip_vs]
>     __ip_vs_cleanup at ffffffffa0a9f60a [ip_vs]
>     ops_exit_list at ffffffff81567a49
>     cleanup_net at ffffffff81568b40
>     process_one_work at ffffffff810a851b
>     worker_thread at ffffffff810a9356
>     kthread at ffffffff810b0b6f
>     ret_from_fork at ffffffff81697a18
> 
> race condition:
>     CPU1                           CPU2
>     ip_vs_in()
>       ip_vs_conn_new()
>                                    ip_vs_del_dest()
>                                      __ip_vs_unlink_dest()
>                                        ~IP_VS_DEST_F_AVAILABLE
>       cp->dest && !IP_VS_DEST_F_AVAILABLE
>       __ip_vs_conn_put
>       ...
>       cleanup_net  ---> infinite looping
> 
> Fix this by checking whether the timer already started.

        Looks like an old bug...

> 
> Signed-off-by: Tan Hu <tan.hu@xxxxxxxxxx>
> Reviewed-by: Jiang Biao <jiang.biao2@xxxxxxxxxx>
> ---
>  net/netfilter/ipvs/ip_vs_core.c | 10 +++++++---
>  1 file changed, 7 insertions(+), 3 deletions(-)
> 
> diff --git a/net/netfilter/ipvs/ip_vs_core.c b/net/netfilter/ipvs/ip_vs_core.c
> index 0679dd1..ca9e7cc 100644
> --- a/net/netfilter/ipvs/ip_vs_core.c
> +++ b/net/netfilter/ipvs/ip_vs_core.c
> @@ -1972,13 +1972,17 @@ static int ip_vs_in_icmp_v6(struct netns_ipvs *ipvs, 
> struct sk_buff *skb,
>       if (cp->dest && !(cp->dest->flags & IP_VS_DEST_F_AVAILABLE)) {
>               /* the destination server is not available */

                Add: u32 flags = cp->flags;

>  
> +             /* when timer already started, silently drop the packet.*/
> +             if (timer_pending(&cp->timer))
> +                     __ip_vs_conn_put(cp);
> +             else
> +                     ip_vs_conn_put(cp);

        When ip_vs_conn_put is called for IP_VS_CONN_F_ONE_PACKET
it is possible to call ip_vs_conn_expire and to free cp with
ip_vs_conn_rcu_free immediately. What we can do is to avoid the
ip_vs_conn_expire_now call in such case by reading the flags
early (as above) and adding the needed check (as below).

> +
>               if (sysctl_expire_nodest_conn(ipvs)) {

        Add !(flags & IP_VS_CONN_F_ONE_PACKET) check in above 'if'.

>                       /* try to expire the connection immediately */
>                       ip_vs_conn_expire_now(cp);
>               }
> -             /* don't restart its timer, and silently
> -                drop the packet. */
> -             __ip_vs_conn_put(cp);
> +
>               return NF_DROP;
>       }
>  

Regards

--
Julian Anastasov <ja@xxxxxx>
--
To unsubscribe from this list: send the line "unsubscribe lvs-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html

<Prev in Thread] Current Thread [Next in Thread>