|
what makes you think it is a DOS attack? it's probably just Ratz & Julian
testing your system to see how much load the new setup will handle.. ;)
but seriously - do you have any logs you can post to indicate why you think
it is a DOS attack? here is where a nice dual proc box setup with Snort
comes in handy, maybe you have one already ;) ?
if this problem occurs again maybe you can get 5 minutes of tcpdump logs on
the real servers & the director?
Get some sleep!
Peter
-----Original Message-----
From: Jeremy Kusnetz
To: 'Roberto Nibali '; 'lvs-users@xxxxxxxxxxxxxxxxxxxxxx '
Sent: 10/1/2002 2:05 AM
Subject: RE: No buffer space available
Well my pager went off at 4:30am this morning, saying a few of our
services
were down for a couple of different VIPs.
I logged on, and did an ipvsadm -Ln, and saw that MON had show down a
bunch
of RIPs. I tried pinging those RIPs and I was able to, so it looks
like
whatever the problem is, it didn't last very long. Before when this
would
happen and I would try to ping the RIPs I would get the buffer space
problem.
So whatever we did made the problem partially go away.
I saw that on our other webserver that is not in the cluster was having
problems too, look like that apache work was trying to attack, so I
guess
it's some sort of denail of service attack, even though we are not
vonerable.
You know, I don't remember if it was just the web RIP that went down on
the
cluster. Maybe those are the only ones that went down, I now those are
the
only ones that paged me, but I seem to remeber when doing the ipvsadm
-Ln,
that there were other RIP services down to. I should have noted them,
but
hey it was 4:30am.
Anyway, I've rebooted director, and started up all the VIPs/32. I did
NOT
update the /proc filesystem, so now the only change to the system is /32
VIPs. Let's see how that does.
If this was an apache worm ddos attack, how can I stop it? If it
negativelly affects LVS, what can we do with LVS so it's not affected?
I ran dmesg before rebooting, and it had not changed since the we first
change the /proc filesystem yesterday.
Here is the shortened diags before and after the reboot to /32. (I also
have
the longer version if you want it)
BEFORE
root@director:~# cat diag2.log
grep cache /proc/slabinfo
kmem_cache 80 80 244 5 5 1 : 252 126
inet_peer_cache 408 1416 64 17 24 1 : 252 126
ip_dst_cache 15124 24700 192 906 1235 1 : 252 126
arp_cache 1710 1710 128 57 57 1 : 252 126
dnotify cache 0 0 20 0 0 1 : 252 126
file lock cache 126 126 92 3 3 1 : 252 126
fasync cache 0 0 16 0 0 1 : 252 126
uid_cache 226 226 32 2 2 1 : 252 126
skbuff_head_cache 582 960 192 36 48 1 : 252 126
cdev_cache 1239 1239 64 21 21 1 : 252 126
bdev_cache 118 118 64 2 2 1 : 252 126
mnt_cache 118 118 64 2 2 1 : 252 126
inode_cache 114077 114387 512 16315 16341 1 : 124 62
dentry_cache 117348 117600 128 3920 3920 1 : 252 126
names_cache 62 62 4096 62 62 1 : 60 30
fs_cache 287 413 64 7 7 1 : 252 126
files_cache 173 297 416 22 33 1 : 124 62
-------------------------------------------------
ip -o -s route show cache | wc -l
14757
AFTER
root@director:~# cat diag2.log2
grep cache /proc/slabinfo
kmem_cache 80 80 244 5 5 1 : 252 126
inet_peer_cache 885 885 64 15 15 1 : 252 126
ip_dst_cache 7360 7360 192 368 368 1 : 252 126
arp_cache 1080 1080 128 36 36 1 : 252 126
dnotify cache 0 0 20 0 0 1 : 252 126
file lock cache 84 84 92 2 2 1 : 252 126
fasync cache 0 0 16 0 0 1 : 252 126
uid_cache 226 226 32 2 2 1 : 252 126
skbuff_head_cache 560 560 192 28 28 1 : 252 126
cdev_cache 1239 1239 64 21 21 1 : 252 126
bdev_cache 118 118 64 2 2 1 : 252 126
mnt_cache 118 118 64 2 2 1 : 252 126
inode_cache 7352 7476 512 1061 1068 1 : 124 62
dentry_cache 7860 7860 128 262 262 1 : 252 126
names_cache 11 11 4096 11 11 1 : 60 30
fs_cache 295 295 64 5 5 1 : 252 126
files_cache 216 216 416 24 24 1 : 124 62
-------------------------------------------------
ip -o -s route show cache | wc -l
7168
-----Original Message-----
From: Roberto Nibali
To: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Cc: 'Peter Mueller'; alanr; Jeremy Kusnetz
Sent: 9/30/02 6:29 PM
Subject: Re: No buffer space available
> Okay, your patch works great for me, much better then my workaround.
Yes but I fear that I've broken some semantics.
> What do I want the netmask on the primary IP of eth0 on the director
to be?
> /24 or /32 There are a few services which go through this IP in
addition to
> the 54 VIPs (just added one more today on top of all this)
Make it /24.
> Unfortunitely IPaddr uses this to bring up the interface in the first
place,
> it's a chicken and egg type thing.
Yes, I knew that this would break something. Let's wait until Alan wakes
up or finds some time (I think this is rather the case) to address this
issue.
> Anyway, right now I still have the production director using /24 for
all the
> VIPs with the changes to the proc filesystem.
Maybe we can also leave that and observe it for a while. I hope you've
done the same rc.local on the second director :).
> In order to get the /32 working on all the VIPs I'll have to bring
> everything down and back up, not something good to do durning peak
usage
> time. I'll probably try it early tomorrow morning, or if I get paged
in the
> middle of the night, I'll try it then.
Ok. But do it in a way you can easily failback if it wouldn't work.
> I'll let you know how it goes.
Thanks.
Regards,
Roberto Nibali, ratz
--
echo
'[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq'|dc
_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://www.in-addr.de/mailman/listinfo/lvs-users
|
