|
Hi LVS List!
IPVS (on XEN) with two Bridges fails.
We have the following setup:
A Xen Machine with one external and one internal physical interfaces
peth0 and peth2.
The dom0 is connected via eth0 to the internet and via eth2 to a local
subnet.
On this local subnet a machine Vzope1 acts as one realserver.
internet -> peth0 -|> Xen server -|> peth2 -> switch -> realserver
We use the bridged Xen Setup with multible Bridges (in our case two).
Incoming Requests were received from the external interface peth0 and
sent via eth2 to a realserver.
This ist setup in more detail:
World | XEN-Dom0 running ipvs |
Request -> peth0 -> xenbr0 -|> eth0 -> ipvs -> eth2 -> xenbr2 -|> peth2
-> physical switch -> realserver (Vzope1)
* First the Request-Package enters peth0 the pysical interface of the
XEN-Server.
* The Bridge xenbr0 then delivers the Package to eth0 which is in the
Scope of the Virtual-Server running ipvs.
* According to
http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/images/nf-lvs.png ipvs
takes the packet out of the LOCAL_IN Chain
Here the commented tcpdump:
peth0:
02:14:09.948636 IP dslb-088-074-161-106.pools.arcor-ip.net.35322 >
193.239.28.158.www: S 2531256350:2531256350(0) win 5840 <mss
1452,sackOK,timestamp 14032446 0,nop,wscale 7>
xenbr0:
02:14:09.948653 IP dslb-088-074-161-106.pools.arcor-ip.net.35322 >
193.239.28.158.www: S 2531256350:2531256350(0) win 5840 <mss
1452,sackOK,timestamp 14032446 0,nop,wscale 7>
eth0:
02:14:09.948653 IP dslb-088-074-161-106.pools.arcor-ip.net.35322 >
193.239.28.158.www: S 2531256350:2531256350(0) win 5840 <mss
1452,sackOK,timestamp 14032446 0,nop,wscale 7>
ipvs: DNAT performed succesfully
eth2:
02:14:09.948670 IP dslb-088-074-161-106.pools.arcor-ip.net.35322 >
10.1.0.124.www: S 2531256350:2531256350(0) win 5840 <mss
1452,sackOK,timestamp 14032446 0,nop,wscale 7>
xenbr2:
02:14:09.948670 IP dslb-088-074-161-106.pools.arcor-ip.net.35322 >
10.1.0.124.www: S 2531256350:2531256350(0) win 5840 <mss
1452,sackOK,timestamp 14032446 0,nop,wscale 7>
peth2:
02:14:09.948684 IP dslb-088-074-161-106.pools.arcor-ip.net.35322 >
10.1.0.124.www: S 2531256350:2531256350(0) win 5840 <mss
1452,sackOK,timestamp 14032446 0,nop,wscale 7>
Going out to realserver:
All ok this way. Let's have a look at the response packets:
World | Server running ipvs |
Request <- peth0 <- xenbr0 <- eth0 <- ipvs <- eth2 <- xenbr2 <- peth2 <-
realserver
We Received a Syn Ack Packet from the realserver at peth2:
02:14:09.948885 IP 10.1.0.124.www >
dslb-088-074-161-106.pools.arcor-ip.net.35322: S
1725501155:1725501155(0) ack 2531256351 win 5792 <mss
1460,sackOK,timestamp 956956465 14030196,nop,wscale 7>
Then it undergoes SNAT by ipvs to be delivered to xenbr2:
02:14:09.948898 IP 193.239.28.158.www >
dslb-088-074-161-106.pools.arcor-ip.net.35322: S
1725501155:1725501155(0) ack 2531256351 win 5792 <mss
1460,sackOK,timestamp 956956465 14030196,nop,wscale 7>
hitting eth2:
02:14:09.948898 IP 193.239.28.158.www >
dslb-088-074-161-106.pools.arcor-ip.net.35322: S
1725501155:1725501155(0) ack 2531256351 win 5792 <mss
1460,sackOK,timestamp 956956465 14030196,nop,wscale 7>
The response goes the folowing way
Realserver -> peth2 -> xenbr2 -> ipvs SNAT -> xenbr2 -> eth2 -> Martian
source
resulting in something like
Jan 22 20:46:31 charyptis kernel: martian source 88.74.158.52 from
193.239.28.158, on dev eth2
Jan 22 20:46:31 charyptis kernel: ll header:
00:e0:81:49:aa:af:00:16:3e:76:ff:ec:08:00
We do ipvs for a long time. So please expect us not to make beginners
mistakes.
We checked :
* the interfaces
* the bridges
* the routes
* the netfilter
We do not have a single problem. Every thing runs fine.
We reproduced this tests on a second XEN machine with exactily the same
results.
We read the dokumentation. We checked for
typical pitfalls like tcp checksum error under XEN.
We read the code of ipvs. We read to the discussion about
"route_me_harder()"
We suspect the problem the area of "route_me_harder()" since we stumbled
over the following comment in route_me_harder():
/* some non-standard hacks like ipt_REJECT.c:send_reset() can cause
* packets with foreign saddr to appear on the NF_IP_LOCAL_OUT hook.
*/
How can we track the problem further?
May anyone be capable to confirm our findings?
Best Regards
Volker
--
====================================================
inqbus it-consulting +49 ( 341 ) 5643800
Dr. Volker Jaenisch http://www.inqbus.de
Herloßsohnstr. 12 0 4 1 5 5 Leipzig
N O T - F Ä L L E +49 ( 170 ) 3113748
====================================================
|
