LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: ip_vs_random_dropentry() oops

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>, Roger Tsang <roger.tsang@xxxxxxxxx>, ja@xxxxxx
Subject: Re: ip_vs_random_dropentry() oops
From: Roger Tsang <roger.tsang@xxxxxxxxx>
Date: Tue, 30 Aug 2005 02:12:26 -0400
I agree it would be ideal to run kernel-2.6.13 (OpenSSI modified), getting 
there...

As I recall the MySQL NDB cluster storage engines were syncing, and that's 
about it at that time. I guess we can say it's kinda like backup activity. 
So we could say running low on memory would explain the LVS anti-DoS 
activity. In this case it could be too high a value for amemthresh, but 
still it shouldn't oops.

In the meantime, I seem to have managed to keep LVS chugging along by 
disabling drop_entry and drop_packet and using amemthresh 1024.

If you are able to produce the patch you were speaking of, it will be a nice 
addition to this kernel for the other users... until there is one based on 
kernel-2.6.13. Thanks.

Roger


On 8/30/05, Horms <horms@xxxxxxxxxxxx> wrote:
> 
> On Mon, Aug 29, 2005 at 07:38:30PM -0400, Roger Tsang wrote:
> > The kernel oops on the failover (backup) director and I have the 
> following
> > on all nodes. I suppose the rest of the settings were at their defaults.
> >
> > echo "1" > /proc/sys/net/ipv4/vs/drop_entry
> > echo "1" > /proc/sys/net/ipv4/vs/drop_packet
> > echo "1" > /proc/sys/net/ipv4/vs/secure_tcp
> > echo "8192" > /proc/sys/net/ipv4/vs/amemthresh
> > /sbin/ipvsadm --set 900 120 300 # tcp tcpfin udp
> 
> First up, its pretty difficult to replicate problems
> on older (kernel development is pretty fast these days,
> so 2.6.10 is pretty old by now) and customised kernels.
> So it would be totally excellent if you could test
> this out on a new kernel, say 2.6.13 which seems fresh
> off the press.
> 
> It seems like there might be a bit of a race in the
> drop entry code. I have a vague memory of a patch floating around
> for this. I'll have a look into the netdev and lvs-users
> archves, and in git, but I have to run off to a meeting now.
> 
> Lastly, was the box doing anything particularly exciting at
> the time? Lots of load? System backup in progress? That
> kind of stuff.
> 
> 
> 
> 
> 
> 
> --
> Horms
> 
> 
> >
> > Roger
> >
> >
> > On 8/29/05, Roger Tsang <roger.tsang@xxxxxxxxx> wrote:
> > >
> > > Hey guys,
> > >
> > > Got an oops while using kernel-2.6.10-bk8 with some SSI enhancements, 
> but
> > > I think this oops is an LVS issue. Anybody know why? Thanks.
> > >
> > > Roger
> > >
> > >
> > > <1>Unable to handle kernel paging request at virtual address 00100100
> > > <1> printing eip:
> > > <4>c03a61a0
> > > <1>*pde = 00000000
> > > <1>Oops: 0000 [#1]
> > > more>
> > > Only 'q' or 'Q' are processed at more prompt, input ignored
> > > <4>Modules linked in: ipt_REJECT ipt_state ip_conntrack ipt_multiport
> > > iptable_filter ip_tables uhci_hcd ehci_hcd usbcore video button drbd 
> sk98lin
> > > r8169 8139too
> > > <4>CPU: 0
> > > <4>EIP: 0060:[<c03a61a0>] Not tainted VLI
> > > <4>EFLAGS: 00010246 (2.6.10-bk8-ssi)
> > > <4>EIP is at ip_vs_random_dropentry+0x30/0x120
> > > <4>eax: 00000000 ebx: 00100100 ecx: 002716ed edx: 00100100
> > > <4>esi: e080c000 edi: 0000ca12 ebp: c15e9f38 esp: c15e9f24
> > > <4>ds: 007b es: 007b ss: 0068
> > > <4>Process events/0 (pid: 4, threadinfo=c15e8000 task=dfda2000)
> > > <4>Stack: c4f4d2e0 000004a6 00000297 00000000 c0574ce4 c15e9f40 
> c03a7e17
> > > c15e9fbc
> > > <4> c012a696 00000000 c15e9f70 00000000 dffe7658 dffe7648 dffe7650
> > > c03a7df0
> > > <4> c15e8000 ffffffff ffffffff 00000001 00000000 c0116840 00010000
> > > 00000000
> > > <4>Call Trace:
> > > <4> [<c0104b4f>] show_stack+0x7f/0xa0
> > > <4> [<c0104ce5>] show_registers+0x155/0x220
> > > <4> [<c010506c>] die+0xcc/0x190
> > > <4> [<c0114cbd>] do_page_fault+0x48d/0x68a
> > > <4> [<c01047cb>] error_code+0x2b/0x30
> > > <4> [<c03a7e17>] defense_work_handler+0x27/0x40
> > > <4> [<c012a696>] worker_thread+0x186/0x230
> > > <4> [<c012e7be>] kthread+0x9e/0xe0
> > > <4> [<c01009f5>] kernel_thread_helper+0x5/0x10
> > > more>
> > > Only 'q' or 'Q' are processed at more prompt, input ignored
> > > <4>Code: 53 83 ec 08 c7 45 f0 ff 07 00 00 e8 eb b1 fb ff 8b 35 14 60 
> 66 c0
> > > 0f b7 f8 8b 1c fe eb 0d 90 90 90 90 90 90 90 90 90 90 90 90 90 <8b> 13 
> 8d 74
> > > 26 00 8d 04 fe 39 d8 0f 84 84 00 00 00 66 83 7b 14
> > > <4>
> > > kdb> bt
> > > Stack traceback for pid 4
> > > 0xdfda2000 4 2 1 0 R 0xdfda21c0 *events/0
> > > EBP EIP Function (args)
> > > 0xc15e9f38 0xc03a61a0 ip_vs_random_dropentry+0x30
> > > 0xc15e9f40 0xc03a7e17 defense_work_handler+0x27 (0x0, 0xc15e9f70, 0x0,
> > > 0xdffe7658, 0xdffe7648)
> > > 0xc15e9fbc 0xc012a696 worker_thread+0x186 (0xdffe7640, 0xc15e9fd8, 
> 0x0,
> > > 0xfffffffc, 0xc012a510)
> > > 0xc15e9fec 0xc012e7be kthread+0x9e
> > > 0xc01009f5 kernel_thread_helper+0x5
> > > kdb>
> > _______________________________________________
> > LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
> > Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
> > or go to http://www.in-addr.de/mailman/listinfo/lvs-users
>

<Prev in Thread] Current Thread [Next in Thread>