|
2013/8/23 Julian Anastasov <ja@xxxxxx>:
>
> Hello,
>
> On Fri, 23 Aug 2013, Drunkard Zhang wrote:
>
>> 2013/8/22 Julian Anastasov <ja@xxxxxx>:
>> >
>> > for a very short period, they live up to 1 jiffie, eg. 10ms.
>> > Also, WRR should be reliable for OPS while other
>> > schedulers (eg. *LC) are not suitable.
>>
>> I noticed this too. While ops working, the InActConn is always
>> changing too, if it's fixed, the ops is not working.
>
> All traffic to director stops?
No,ingress traffic is always on going.
>> > Do you see the same problem with ipvsadm -Ln --stats ?
>> > ipvsadm -Z may be needed to zero the stats after restoring all
>> > rules. "Conns" counter in stats should be according to WRR
>> > weights, it shows the scheduler decisions.
>>
>> After every restore, the stats also zeroed, right? While, ops still not
>> working.
>
> Yes:
>
> 1. Configure/Restore rules
> 2. Zero stats: ./ipvsadm -Z
> 3. sleep 5
> 4. ./ipvsadm -ln --stats
>
>> vs3 ~/pkgs # ./ipvsadm -Z
>> vs3 ~/pkgs # ./ipvsadm -ln --stats -u [snipped]
>> Prot LocalAddress:Port Conns InPkts OutPkts InBytes
>> OutBytes
>> -> RemoteAddress:Port
>> UDP x.x.x.120:514 0 12497040 0 2572M 0
>> -> x.x.x.65:514 0 3975 0 394171 0
>
> It is really strange to have Conns=0 in stats.
> I just tested OPS on 32-bit x86 (virtualbox) with plain
> 3.10 kernel and there is no problem with CPS, Conns, etc,
> WRR is scheduling according to weights. Do you have a
> daemon that changes weights dynamically?
I'm running x86_64 kernel. I compared kernel config of my two servers,
a big difference between them is CONFIG_PREEMPT. While CONFIG_PREEMPT
is disabled, trying plenty times of "ipvsadm -C && ipvsadm -R <
rules-with-ops" will finally succeed, but with CONFIG_PREEMPT enabled
it's too hard to get --ops work. I will test again on my "good" server
another day to prove my guessing.
Is there any good debug method for this? Tuning
/proc/sys/net/ipv4/vs/debug_level didn't gave me much.
I use keepalived to manage the ipvs configuration, but as vrrp
heartbeat going on and no realserver up/down, it won't interact with
ipvs, right? So I can temporarily modify ipvs rule via ipvsadm after
keepalived started, and the modified rules didn't changed as time fly,
so do the --ops setting.
> More things to check:
>
> - if traffic stops check if some real server is hijacking the
> traffic from director due to ARP problem in the real server.
> Or explain how exactly OPS stops to work, do you see other
> traffic for the VIP coming to director during such problem?
>
No possibility, I configured VIP on lo of realserver.
for IP in $VIP; do
ip addr add $IP/32 dev $VIP_NIC brd $IP
done
sysctl -q -w net.ipv4.conf.lo.arp_ignore=1
sysctl -q -w net.ipv4.conf.lo.arp_announce=2
sysctl -q -w net.ipv4.conf.all.arp_ignore=1
sysctl -q -w net.ipv4.conf.all.arp_announce=2
> - Build ipvsadm with 'make HAVE_NL=0' to check if Conns=0 problem
> in --stats output is netlink related. This builds ipvsadm without
> netlink support but use this binary only to see stats, not
> for configuration.
>
> - show output from 'cat /proc/net/ip_vs_stats_percpu' to see
> the kernel's stats and rates. Note that these stats are not
> zeroed while stats in /proc/net/ip_vs_stats are zeroed.
Always changing.
vs3 ~ # cat /proc/net/ip_vs_stats_percpu
Total Incoming Outgoing Incoming Outgoing
CPU Conns Packets Packets Bytes Bytes
0 8F11751F 70455AB5 0 10AA672610D 0
1 1A780554 1A780554 0 E2AB71BCA 0
2 0 0 0 0 0
3 BF0E0B BF0E0B 0 4B7E409C 0
4 244BAF54 244BAF54 0 2224071265 0
5 2360B25C 2360B25B 0 1715A45DB3 0
6 0 0 0 0 0
7 E88FEF E88FEF 0 6ECC3067 0
8 1E2477AE 1E2477AE 0 12726CDE2E 0
9 10BD4D97 10BD4D97 0 A35650024 0
A BE81916 BE81914 0 6D9FD6CEF 0
B 4474D837 4474D836 0 3FCEC43B56 0
C 0 0 0 0 0
D 0 0 0 0 0
E 0 0 0 0 0
F 0 0 0 0 0
~ 721BAF1B 534F94AD 0 1B61556B50B 0
Conns/s Pkts/s Pkts/s Bytes/s Bytes/s
1120F 1120F 0 C1FEB1 0
--
To unsubscribe from this list: send the line "unsubscribe lvs-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
|
