|
I do not understand why on line 15 of the tcpdump you can see a 326
byte packet is received from the client, but isn't forwarded to the
real server. There wouldn't be any fragmentation issues with that
would there? On line 15 you can see it keeps receiving the same packet
6 times and fails to forward it on.
We do not see any ICMP on the hosts.
Thank you I appreciate the insights.
On Fri, Aug 28, 2015 at 2:42 AM, Julian Anastasov <ja@xxxxxx> wrote:
>
> Hello,
>
> On Thu, 27 Aug 2015, Phillip Moore wrote:
>
>> I have IPVS setup with 2 VIPs talking to the same real server
>> configured for direct server return (ie TUN type).
>> One vip is port 80 http and one vip is 443 for https/SSL. The SSL vip
>> doesn't work properly. There is initial communication that happens but
>> then it appears as though IPVS stops tunneling the incoming packets to
>> the real server and the connection stalls and times out. If I switch
>> ports to just verify there is nothing crazy going on with filtering
>> and I put SSL on port 80 (or any port) it still fails.
>>
>> I've put the relevant info in a gist in hope it might be helpful and
>> not clutter up the email.
>>
>> https://gist.github.com/realpdm/2118bbaa298ff3debe52
>>
>> In various test scenarios we found that the client is having to
>> retransmit packets after some initial successful back and forth. On
>> the IPVS node a tcpdump shows that for some reason IPVS stops
>> forwarding the packets onto the real server over the tunnel. You can
>> see in the tcpdump IPVS is forwarding things over ipip just fine until
>> it stops around line 15 in the dump
>>
>> http traffic doesn't do this at all only SSL.
>>
>> I'm really puzzled and hope i am missing something obvious. I
>> appreciate any insights or suggestions.
>
> I'm not sure what kind of fixes contain your kernel
> but the problem should be related to PMTU (ICMP FRAG_NEEDED)
> or GRO. In your packet trace I don't see ICMP messages,
> may be there are such packets from real server to client.
> For example:
>
> 17:09:40.820663 IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [.], ack 1449,
> win 137, options [nop,nop,TS val 592982574 ecr 2501301618], length 0
> 17:09:40.820678 IP 10.65.74.77 > 10.65.74.72: IP 10.240.8.72.60642 >
> 10.64.96.10.443: Flags [.], ack 1449, win 137, options [nop,nop,TS val
> 592982574 ecr 2501301618], length 0 (ipip-proto-4)
> 17:09:40.820704 IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [.], ack 3580,
> win 160, options [nop,nop,TS val 592982574 ecr 2501301618], length 0
>
> See how client acks 1449 and then 3580, i.e. 2131 bytes
> were sent to client.
>
> SSL usually sends large certificates at start, so
> that may explain the difference with plain HTTP. But you will
> need more traces, mostly from the real server or better its
> uplink router, if possible. For TUN mode the following traces
> catch all traffic:
>
> director# tcpdump -ln -i INDEV host CIP
> director# tcpdump -ln -i OUTDEV host RIP -vvv
> real server# tcpdump -ln -i IN_ETH host DIP -vvv
> real server# tcpdump -ln -i tunl0 host CIP
> real server# tcpdump -ln -i OUT_DEV host CIP
> client# tcpdump -ln host VIP
>
> You need to catch TCP, IPIP, ICMP. Restricting
> by client IP may help.
>
> In linux-stable I see that only 2.6.32.61+ contains
> related changes:
>
> 9df2c9a ipvs: IPv6 MTU checking cleanup and bugfix
> ec3dc8c ipvs: allow transmit of GRO aggregated skbs
>
> But such fixes are for director and I don't see
> any ICMP messages... And if there are ICMP messages, we
> have more fixes to consider...
>
>> OS Info:
>>
>> Linux adc-ipvs-lb2001 2.6.32-504.30.3.el6.x86_64 #1 SMP Tue Jul 14
>> 11:18:03 CDT 2015 x86_64 x86_64 x86_64 GNU/Linux
>>
>> /sbin/modinfo ip_vs
>> filename:
>> /lib/modules/2.6.32-504.30.3.el6.x86_64/kernel/net/netfilter/ipvs/ip_vs.ko
>> srcversion: 6C3CC9C055045FA0ECA1774
>> depends: ipv6,libcrc32c
>> vermagic: 2.6.32-504.30.3.el6.x86_64 SMP mod_unload modversions
>> parm: conn_tab_bits:Set connections' hash size (int)
>>
>> /sbin/modinfo ip_vs_sh
>> filename:
>> /lib/modules/2.6.32-504.30.3.el6.x86_64/kernel/net/netfilter/ipvs/ip_vs_sh.ko
>> srcversion: 2EAF6C9DD83264246DBA82C
>> depends: ip_vs
>> vermagic: 2.6.32-504.30.3.el6.x86_64 SMP mod_unload modversions
>
> Regards
>
> --
> Julian Anastasov <ja@xxxxxx>
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
