LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: [lvs-users] 2.6.36.2: Kernel oops when ipvs is used with ip_queue in

To: Kumar Swamy <kswamy@xxxxxxxxxxxxx>
Subject: Re: [lvs-users] 2.6.36.2: Kernel oops when ipvs is used with ip_queue in x86_64
Cc: "lvs-users@xxxxxxxxxxxxxxxxxxxxxx" <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
From: Julian Anastasov <ja@xxxxxx>
Date: Fri, 5 Aug 2011 13:52:19 +0300 (EEST)
        Hello,

On Mon, 18 Jul 2011, Kumar Swamy wrote:

> Hello,
> 
> Please pardon me if this is a known issue and point me to the fix. I have the 
> following setup
> 
> Kernel : 2.6.36.2 arch: x86_64
> 
> 1)       ip_queue module is loaded and a iptable rule with QUEUE target is 
> added for an interface (IF1 - 192.168.132.215).
>     -A INPUT -d 192.168.132.215/32 -p tcp -m tcp --dport 80 -j QUEUE
> 2)        An ipvs rule with wrr load balancing listening on port 80 on IF1 
> and one real server.
>       ipvsadm-save
> -A -t 192.168.132.215:http -s wrr
> -a -t 192.168.132.215:http -r 192.168.132.232:http -m -w 100
> 3) A userland program which uses libipq to handle packets from QUEUE target.
> 
> Now any traffic on 192.168.132.215:http is causing kernel to panic. I 
> debugged userland program which sets the verdict NF_ACCEPT  
> (ipq_set_verdict(ipq,id, NF_ACCEPT, 0, NULL); I guess ip_queue module 
> reinjects the packet but somehow leading to kernel Oops. I don't see the 
> panic if there is no ipvs rule which I tested by listening on the socket in 
> user land without IPVS rule.
> 
> 
> Jul 14 02:43:28 2011 kumar 2011 kernel: [ 401.459462] BUG: unable to handle 
> kernel NULL pointer dereference at (null)
> Jul 14 02:43:28 2011 kumar 2011 kernel: [ 401.459776] IP: 
> [<ffffffff810a1edf>] put_page+0x1f/0x160
> Jul 14 02:43:28 2011 kumar 2011 kernel: [ 401.460016] Oops: 0000 [#1] SMP
> Jul 14 02:43:28 2011 kumar 2011 kernel: [ 401.460016] last sysfs file: 
> /sys/devices/pci0000:00/0000:00:18.3/temp1_input
> Jul 14 02:43:28 2011 kumar 2011 kernel: [ 401.460016] Stack:
> Jul 14 02:43:28 2011 kumar 2011 kernel: [ 401.460016] Call Trace:
> Jul 14 02:43:28 2011 kumar 2011 kernel: [ 401.460016] Code: fc ff ff eb e4 0f 
> 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 30 48 89 5d d8 4c 89 65 e0 48 89 fb 4c 
> 89 6d e8 4c 89 75 f0 4c 89 7d f8 <66> f7 07 00 c0 0f 85 0a 01 00 00 8b 47 08 
> f0 ff 4f 08 0f 94 c0
> Jul 14 02:43:28 2011 kumar 2011 kernel: [ 401.460016] RIP 
> [<ffffffff810a1edf>] put_page+0x1f/0x160
> Jul 14 02:43:28 2011 kumar 2011 kernel: [ 401.460016] CR2: 0000000000000000
> 
> Any help is appreciated.

        The oops does not explain much but looking for other
problems, I think, I found one issue with nf_reinject. I hope
it is the cause for your oops because IPVS returns NF_STOLEN
from its transmitters and nf_reinject tries to free the skb
that is already processed by IPVS.

        I just posted a patch that fixes this problem.
Your address is in CC:

http://marc.info/?l=linux-netdev&m=131254033002982&w=2

        This patch is for v3.0. Let me know if you can test
it somehow on 2.6.36. If you need patch for other kernel
for the test, let me know.

Regards

--
Julian Anastasov <ja@xxxxxx>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>
  • Re: [lvs-users] 2.6.36.2: Kernel oops when ipvs is used with ip_queue in x86_64, Julian Anastasov <=