|
It just happened again!
This time we were running with the VIPs/32, not /proc-fs tuning. I just ran
the script that does the tuning though.
We are running mrtg, monitoring the eth1 traffic. I see in the past 2 hours
8-10 HUGE spikes of traffic, the spikes keep on getting higher and higher,
starting at 6MB/second going all the way to 14MB/second.
According to MRTG the incoming traffic is running about 2-3MB/second greater
then the outgoing. Right now after running the /proc tuning, all I see is
outgoing traffic.
I'm pretty sure this is a spam attack. Unfortunately ntop died last night,
and I just noticed it died now, so I don't have a brake down of the ports in
use, but this patter is exactly what previous patterns of spam attacks.
We are currently doing a full tcpdump of traffic hitting the loadbalancer.
I caught this on the tail end, but hopefully it will have some good info in
it.
In the mean time, here is the full output of my diag script with everything
ratz wanted me to run (the long version) Notice dmesg is FULL of Neighbour
table overflows (I've contactinated the dmesg though since they are all the
same message)
ip -s -s link show
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
RX: bytes packets errors dropped overrun mcast
73600 676 0 0 0 0
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
73600 676 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:01:fa:ff:fc:45 brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
3331121836 15168843 0 0 0 0
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
513747252 16544341 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0
3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:01:fa:ff:fc:43 brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
313111415 13956179 0 0 0 0
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
3440923204 12587773 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0
4: eth2: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:01:03:e4:4b:93 brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
3061594 16358 0 0 0 0
RX errors: length crc frame fifo missed
0 0 0 26 0
TX: bytes packets errors dropped carrier collsns
3018033 16376 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0
-------------------------------------------------
ip addr show
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:01:fa:ff:fc:45 brd ff:ff:ff:ff:ff:ff
inet 216.163.120.8/24 brd 216.163.120.255 scope global eth0
inet 216.163.120.45/32 brd 216.163.120.45 scope global eth0:0
inet 216.163.120.29/32 brd 216.163.120.29 scope global eth0:1
inet 216.163.120.46/32 brd 216.163.120.46 scope global eth0:2
inet 216.163.120.62/32 brd 216.163.120.62 scope global eth0:3
inet 216.163.120.47/32 brd 216.163.120.47 scope global eth0:4
inet 216.163.120.63/32 brd 216.163.120.63 scope global eth0:5
inet 216.163.120.48/32 brd 216.163.120.48 scope global eth0:6
inet 216.163.120.119/32 brd 216.163.120.119 scope global eth0:7
inet 216.163.120.65/32 brd 216.163.120.65 scope global eth0:8
inet 216.163.120.49/32 brd 216.163.120.49 scope global eth0:9
inet 216.163.120.66/32 brd 216.163.120.66 scope global eth0:10
inet 216.163.120.67/32 brd 216.163.120.67 scope global eth0:11
inet 216.163.120.10/32 brd 216.163.120.10 scope global eth0:12
inet 216.163.120.11/32 brd 216.163.120.11 scope global eth0:13
inet 216.163.120.31/32 brd 216.163.120.31 scope global eth0:14
inet 216.163.120.15/32 brd 216.163.120.15 scope global eth0:15
inet 216.163.120.32/32 brd 216.163.120.32 scope global eth0:16
inet 216.163.120.16/32 brd 216.163.120.16 scope global eth0:17
inet 216.163.120.17/32 brd 216.163.120.17 scope global eth0:18
inet 216.163.120.33/32 brd 216.163.120.33 scope global eth0:19
inet 216.163.120.50/32 brd 216.163.120.50 scope global eth0:20
inet 216.163.120.34/32 brd 216.163.120.34 scope global eth0:21
inet 216.163.120.18/32 brd 216.163.120.18 scope global eth0:22
inet 216.163.120.35/32 brd 216.163.120.35 scope global eth0:23
inet 216.163.120.51/32 brd 216.163.120.51 scope global eth0:24
inet 216.163.120.36/32 brd 216.163.120.36 scope global eth0:25
inet 216.163.120.37/32 brd 216.163.120.37 scope global eth0:26
inet 216.163.120.53/32 brd 216.163.120.53 scope global eth0:27
inet 216.163.120.38/32 brd 216.163.120.38 scope global eth0:28
inet 216.163.120.54/32 brd 216.163.120.54 scope global eth0:29
inet 216.163.120.39/32 brd 216.163.120.39 scope global eth0:30
inet 216.163.120.55/32 brd 216.163.120.55 scope global eth0:31
inet 216.163.120.56/32 brd 216.163.120.56 scope global eth0:32
inet 216.163.120.57/32 brd 216.163.120.57 scope global eth0:33
inet 216.163.120.58/32 brd 216.163.120.58 scope global eth0:34
inet 216.163.120.59/32 brd 216.163.120.59 scope global eth0:35
inet 216.163.120.6/32 brd 216.163.120.6 scope global eth0:36
inet 216.163.120.78/32 brd 216.163.120.78 scope global eth0:37
inet 216.163.120.7/32 brd 216.163.120.7 scope global eth0:38
inet 216.163.120.95/32 brd 216.163.120.95 scope global eth0:39
inet 216.163.120.96/32 brd 216.163.120.96 scope global eth0:40
inet 216.163.120.9/32 brd 216.163.120.9 scope global eth0:41
inet 216.163.120.97/32 brd 216.163.120.97 scope global eth0:42
inet 216.163.120.98/32 brd 216.163.120.98 scope global eth0:43
inet 216.163.120.23/32 brd 216.163.120.23 scope global eth0:44
inet 216.163.120.24/32 brd 216.163.120.24 scope global eth0:45
inet 216.163.120.40/32 brd 216.163.120.40 scope global eth0:46
inet 216.163.120.41/32 brd 216.163.120.41 scope global eth0:47
inet 216.163.120.25/32 brd 216.163.120.25 scope global eth0:48
inet 216.163.120.42/32 brd 216.163.120.42 scope global eth0:49
inet 216.163.120.26/32 brd 216.163.120.26 scope global eth0:50
inet 216.163.120.27/32 brd 216.163.120.27 scope global eth0:51
inet 216.163.120.43/32 brd 216.163.120.43 scope global eth0:52
inet 216.163.120.20/32 brd 216.163.120.20 scope global eth0:53
inet 216.163.120.19/32 brd 216.163.120.19 scope global eth0:54
3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:01:fa:ff:fc:43 brd ff:ff:ff:ff:ff:ff
inet 10.75.0.1/16 brd 10.255.255.255 scope global eth1
4: eth2: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:01:03:e4:4b:93 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.128/24 brd 192.168.0.255 scope global eth2
-------------------------------------------------
ip link show dev lo
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
-------------------------------------------------
ip rule show
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
-------------------------------------------------
ip route show table main
216.163.120.29 dev eth0 scope link src 216.163.120.29
216.163.120.95 dev eth0 scope link src 216.163.120.95
216.163.120.45 dev eth0 scope link src 216.163.120.45
216.163.120.78 dev eth0 scope link src 216.163.120.78
216.163.120.15 dev eth0 scope link src 216.163.120.15
216.163.120.31 dev eth0 scope link src 216.163.120.31
216.163.120.46 dev eth0 scope link src 216.163.120.46
216.163.120.62 dev eth0 scope link src 216.163.120.62
216.163.120.47 dev eth0 scope link src 216.163.120.47
216.163.120.63 dev eth0 scope link src 216.163.120.63
216.163.120.9 dev eth0 scope link src 216.163.120.9
216.163.120.25 dev eth0 scope link src 216.163.120.25
216.163.120.40 dev eth0 scope link src 216.163.120.40
216.163.120.56 dev eth0 scope link src 216.163.120.56
216.163.120.24 dev eth0 scope link src 216.163.120.24
216.163.120.41 dev eth0 scope link src 216.163.120.41
216.163.120.57 dev eth0 scope link src 216.163.120.57
216.163.120.11 dev eth0 scope link src 216.163.120.11
216.163.120.27 dev eth0 scope link src 216.163.120.27
216.163.120.42 dev eth0 scope link src 216.163.120.42
216.163.120.58 dev eth0 scope link src 216.163.120.58
216.163.120.10 dev eth0 scope link src 216.163.120.10
216.163.120.26 dev eth0 scope link src 216.163.120.26
216.163.120.43 dev eth0 scope link src 216.163.120.43
216.163.120.59 dev eth0 scope link src 216.163.120.59
216.163.120.36 dev eth0 scope link src 216.163.120.36
216.163.120.20 dev eth0 scope link src 216.163.120.20
216.163.120.37 dev eth0 scope link src 216.163.120.37
216.163.120.53 dev eth0 scope link src 216.163.120.53
216.163.120.119 dev eth0 scope link src 216.163.120.119
10.5.0.5 via 10.75.255.254 dev eth1
216.163.120.7 dev eth0 scope link src 216.163.120.7
216.163.120.23 dev eth0 scope link src 216.163.120.23
216.163.120.38 dev eth0 scope link src 216.163.120.38
216.163.120.54 dev eth0 scope link src 216.163.120.54
216.163.120.6 dev eth0 scope link src 216.163.120.6
216.163.120.39 dev eth0 scope link src 216.163.120.39
216.163.120.55 dev eth0 scope link src 216.163.120.55
216.163.120.17 dev eth0 scope link src 216.163.120.17
216.163.120.32 dev eth0 scope link src 216.163.120.32
216.163.120.48 dev eth0 scope link src 216.163.120.48
216.163.120.67 dev eth0 scope link src 216.163.120.67
216.163.120.98 dev eth0 scope link src 216.163.120.98
216.163.120.16 dev eth0 scope link src 216.163.120.16
216.163.120.33 dev eth0 scope link src 216.163.120.33
216.163.120.49 dev eth0 scope link src 216.163.120.49
216.163.120.66 dev eth0 scope link src 216.163.120.66
216.163.120.19 dev eth0 scope link src 216.163.120.19
216.163.120.34 dev eth0 scope link src 216.163.120.34
216.163.120.50 dev eth0 scope link src 216.163.120.50
216.163.120.65 dev eth0 scope link src 216.163.120.65
216.163.120.96 dev eth0 scope link src 216.163.120.96
216.163.120.18 dev eth0 scope link src 216.163.120.18
216.163.120.35 dev eth0 scope link src 216.163.120.35
216.163.120.51 dev eth0 scope link src 216.163.120.51
216.163.120.97 dev eth0 scope link src 216.163.120.97
192.168.0.0/24 dev eth2 proto kernel scope link src 192.168.0.128
216.163.120.0/24 dev eth0 proto kernel scope link src 216.163.120.8
10.75.0.0/16 dev eth1 proto kernel scope link src 10.75.0.1
127.0.0.0/8 dev lo scope link
default via 216.163.120.1 dev eth0 metric 1
-------------------------------------------------
cat /proc/net/softnet_stat
00deba7a 00000000 00000001 00000000 00000000 00000000 00000000 00000000
00004f42
00de3fa9 00000000 00000001 00000000 00000000 00000000 00000000 00000000
0000502a
-------------------------------------------------
cat /proc/net/rt_cache_stat
00000f79 00ce49f1 000f10b1 00000000 00000000 0000007c 00000000 0000000b
00206bc7 00036d4e 00000046 00080086 0007d092 00000a57 00000000
00000f79 00cdcf7a 000f1338 00000000 00000001 00000083 00000000 0000000e
00201cd8 00036720 00000044 000803cf 0007d3c2 00000af1 00000000
-------------------------------------------------
ping -c2 10.75.0.1
cat /proc/net/raw
PING 10.75.0.1 (10.75.0.1): 56 data bytes
ping: wrote 10.75.0.1 64 chars, ret=-1
ping: wrote 10.75.0.1 64 chars, ret=-1
--- 10.75.0.1 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt
uid timeout inode
-------------------------------------------------
cat /proc/net/netlink
sk Eth Pid Groups Rmem Wmem Dump Locks
c1597040 0 0 00000000 0 0 00000000 2
c1597780 4 0 00000000 0 0 00000000 2
-------------------------------------------------
grep cache /proc/slabinfo
kmem_cache 80 80 244 5 5 1 : 252 126
inet_peer_cache 373 1003 64 17 17 1 : 252 126
ip_dst_cache 3274 13480 192 399 674 1 : 252 126
arp_cache 1290 1290 128 43 43 1 : 252 126
dnotify cache 0 0 20 0 0 1 : 252 126
file lock cache 84 84 92 2 2 1 : 252 126
fasync cache 0 0 16 0 0 1 : 252 126
uid_cache 226 226 32 2 2 1 : 252 126
skbuff_head_cache 588 840 192 33 42 1 : 252 126
cdev_cache 1239 1239 64 21 21 1 : 252 126
bdev_cache 118 118 64 2 2 1 : 252 126
mnt_cache 118 118 64 2 2 1 : 252 126
inode_cache 7568 7630 512 1090 1090 1 : 124 62
dentry_cache 8580 8580 128 286 286 1 : 252 126
names_cache 29 29 4096 29 29 1 : 60 30
fs_cache 354 354 64 6 6 1 : 252 126
files_cache 154 216 416 24 24 1 : 124 62
-------------------------------------------------
ip -o -s route show cache | wc -l
3084
-------------------------------------------------
dmesg
ressed.
Neighbour table overflow.
NET: 7 messages suppressed.
Neighbour table overflow.
NET: 25 messages suppressed.
Neighbour table overflow.
NET: 13 messages suppressed.
Neighbour table overflow.
NET: 24 messages suppressed.
Neighbour table overflow.
NET: 5 messages suppressed.
Neighbour table overflow.
NET: 8 messages suppressed.
Neighbour table overflow.
NET: 3 messages suppressed.
Neighbour table overflow.
Neighbour table overflow.
Neighbour table overflow.
Neighbour table overflow.
Neighbour table overflow.
Neighbour table overflow.
|
