|
Hello,
On Mon, 26 Aug 2013, Drunkard Zhang wrote:
> Good news, I finally found the crap source, it's keepalived. I tested
> several times without keepalived in runlevel 3, after kernel boots I
> add the ipvs service by hand:
OK, I was worried that my recent RCU changes broke
something in the WRR scheduler and the configuration process.
> ./ipvsadm -C
> # Clear previous log
> > /var/log/kern.log
> sleep 1
> # Start debug
> echo 20 > /proc/sys/net/ipv4/vs/debug_level
> ./ipvsadm -R < /etc/keepalived/rules-with-ops
> usleep 30000
> # Stop debug
> echo 0 > /proc/sys/net/ipv4/vs/debug_level
>
> Then add VIP manually, then do ARP announce manually:
> vs3 ~/pkgs # ip a add 150.164.100.120/32 dev eno1
> vs3 ~/pkgs # arp-sk -i eno1 -S 150.164.100.120:90:b1:1c:1a:59:46 -d
> 150.164.100.126
>
> After these actions, traffic starts come in. and all ipvsadm checks
> are fine, OPS is fine too. So I figured that maybe outdated libipvs in
> keepalived broke the ipvs in kernel. I'll try to report this to
> upstream.
OK, I have no more doubts. To summarize,
here is what I think happened:
- packet is scheduled while there is virtual service without
the --ops flag. The result is that an UDP connection is
created that expires after 5mins by default, if there are
no more packets.
- traffic is not stopped, it hits the connection and
restarts its timer. As result, this connection stays
forever and forwards traffic to single server.
- as single connection is used we see that the stats for
Conns and CPS rate do not move because we do not create
connections anymore, all traffic comes from single client
address and the scheduler is not called.
- there is one variation here: ipvsadm -C is called,
dests are moved to the trash list, new rules are
added but before the RCU grace period is expired.
In such case IP_VS_DEST_STATE_REMOVING is still set and
prevents the same dest to be reused when adding the
same dest parameters. In this case the connection will point
to unavailable dest for 5mins and the traffic that hits it
will not restart its timer. After 5mins the connection
will be removed and the first packet that comes
will use the --ops flag. There is a chance everything
to work. So, if new rules are added we have 2
situations:
1. rules reuse old dests and traffic goes to single server.
This happens if the new rules are added after at least
10ms (the RCU grace period, in fact), eg. with
usleep 10000 after ipvsadm -C. We have CPS=0 and
InPPS above 0 for single server.
2. rules allocate new dest and traffic is stopped
for 5mins. This will happen if rules are added
immediately after ipvsadm -C (while in RCU grace period).
After 5mins everything works.
- CPS 0 means we are reusing existing connection
- even if you replace the service or set --ops, the
existing connection is still used, even ipvsadm -C
can not remove it. There is only one chance: to set
expire_nodest_conn=1, to call ipvsadm -C and to wait
next packet to remove the connection. Then to add
all rules again but not before the connection is removed.
> On the other hand, ipvs didn't recovery from ipvsadm -C, rmmod ip_vs
> && ./ipvsadm -R < rules-with-ops is needed (I tested, reload ip_vs
> module could make OPS work). So robustness of IPVS needs improvement.
Some problem? May be you refer to the fact that
connections survive ipvsadm -C and that is what prevented
your traffic to be scheduled.
So, I see two problems here:
- tools do not set --ops, connection is created and is
reused from all packets from same client. The trick
to add --ops later can not work. Idea: drop traffic
before reaching IPVS (-j DROP) until --ops is applied,
by this way no connections should be created.
- no way to flush connections in IPVS without removing the
module because expire_nodest_conn works only when traffic is
received. I think, your above remark points here.
Regards
--
Julian Anastasov <ja@xxxxxx>
--
To unsubscribe from this list: send the line "unsubscribe lvs-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
|
