|
Hello,
On Thu, 27 Aug 2015, Phillip Moore wrote:
> I have IPVS setup with 2 VIPs talking to the same real server
> configured for direct server return (ie TUN type).
> One vip is port 80 http and one vip is 443 for https/SSL. The SSL vip
> doesn't work properly. There is initial communication that happens but
> then it appears as though IPVS stops tunneling the incoming packets to
> the real server and the connection stalls and times out. If I switch
> ports to just verify there is nothing crazy going on with filtering
> and I put SSL on port 80 (or any port) it still fails.
>
> I've put the relevant info in a gist in hope it might be helpful and
> not clutter up the email.
>
> https://gist.github.com/realpdm/2118bbaa298ff3debe52
>
> In various test scenarios we found that the client is having to
> retransmit packets after some initial successful back and forth. On
> the IPVS node a tcpdump shows that for some reason IPVS stops
> forwarding the packets onto the real server over the tunnel. You can
> see in the tcpdump IPVS is forwarding things over ipip just fine until
> it stops around line 15 in the dump
>
> http traffic doesn't do this at all only SSL.
>
> I'm really puzzled and hope i am missing something obvious. I
> appreciate any insights or suggestions.
I'm not sure what kind of fixes contain your kernel
but the problem should be related to PMTU (ICMP FRAG_NEEDED)
or GRO. In your packet trace I don't see ICMP messages,
may be there are such packets from real server to client.
For example:
17:09:40.820663 IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [.], ack 1449,
win 137, options [nop,nop,TS val 592982574 ecr 2501301618], length 0
17:09:40.820678 IP 10.65.74.77 > 10.65.74.72: IP 10.240.8.72.60642 >
10.64.96.10.443: Flags [.], ack 1449, win 137, options [nop,nop,TS val
592982574 ecr 2501301618], length 0 (ipip-proto-4)
17:09:40.820704 IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [.], ack 3580,
win 160, options [nop,nop,TS val 592982574 ecr 2501301618], length 0
See how client acks 1449 and then 3580, i.e. 2131 bytes
were sent to client.
SSL usually sends large certificates at start, so
that may explain the difference with plain HTTP. But you will
need more traces, mostly from the real server or better its
uplink router, if possible. For TUN mode the following traces
catch all traffic:
director# tcpdump -ln -i INDEV host CIP
director# tcpdump -ln -i OUTDEV host RIP -vvv
real server# tcpdump -ln -i IN_ETH host DIP -vvv
real server# tcpdump -ln -i tunl0 host CIP
real server# tcpdump -ln -i OUT_DEV host CIP
client# tcpdump -ln host VIP
You need to catch TCP, IPIP, ICMP. Restricting
by client IP may help.
In linux-stable I see that only 2.6.32.61+ contains
related changes:
9df2c9a ipvs: IPv6 MTU checking cleanup and bugfix
ec3dc8c ipvs: allow transmit of GRO aggregated skbs
But such fixes are for director and I don't see
any ICMP messages... And if there are ICMP messages, we
have more fixes to consider...
> OS Info:
>
> Linux adc-ipvs-lb2001 2.6.32-504.30.3.el6.x86_64 #1 SMP Tue Jul 14
> 11:18:03 CDT 2015 x86_64 x86_64 x86_64 GNU/Linux
>
> /sbin/modinfo ip_vs
> filename:
> /lib/modules/2.6.32-504.30.3.el6.x86_64/kernel/net/netfilter/ipvs/ip_vs.ko
> srcversion: 6C3CC9C055045FA0ECA1774
> depends: ipv6,libcrc32c
> vermagic: 2.6.32-504.30.3.el6.x86_64 SMP mod_unload modversions
> parm: conn_tab_bits:Set connections' hash size (int)
>
> /sbin/modinfo ip_vs_sh
> filename:
> /lib/modules/2.6.32-504.30.3.el6.x86_64/kernel/net/netfilter/ipvs/ip_vs_sh.ko
> srcversion: 2EAF6C9DD83264246DBA82C
> depends: ip_vs
> vermagic: 2.6.32-504.30.3.el6.x86_64 SMP mod_unload modversions
Regards
--
Julian Anastasov <ja@xxxxxx>
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
