|
On 15/09/05, Horms <horms@xxxxxxxxxxxx> wrote:
> On Thu, Sep 15, 2005 at 11:39:43AM +0200, mquich wrote:
> > On 14/09/05, Ranga Nathan <kairanga@xxxxxxx> wrote:
> > > One of the things I missed out, that was not obvious in the documentation.
> > >
> > > Did you bind the virtual IP to the loop back adaptors of the real
> > > servers and the load balancers? Also, make sure that the loop back
> > > virtual IP is hidden. The commands to hide are different for 2.2x, 2.4x
> > > and 2.6x kernel.
> > >
> > > On my SuSE 9.2 (2.6x kernel) I do this on the real servers:
> > > echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
> > > echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
> > > echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
> > > echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
> > >
> > > Mind you, I am doing LVS/DR not the LVS/NAT. But I think it applies
> > > there too.
> > >
> >
> > Thanks for the response but my problem goes on. I've made several
> > tests and changed configurations and my situation is this:
> >
> > 1. LVS1 acts as a director and LVS2 acts as a real-server.
> > 2. LVS1 iptables marks packets.
> > 3. I've changed that LVS1 don't capture those packets.
> > 4. LVS2 captures packets with that dst_port.
> > 5. LVS2 doesn't have any ipvs rule
> > 6. No packets at all arrive to LVS.
> > 7. LVS1 doesn't forward packets to LVS2.
> > 8. As LVS1 is a bridge (br0), it forwards the packets through the DSL
> > router and everything works fine (but it doesn't capture the packets).
> > 9. If I make LVS capture the packets (iptables ...-j REDIRECT
> > --to-ports...), then "ipvsadm -l" shows an innactive connection
> > towards LVS.
> > 10. LVS2 sees packets with src_ip=CIP and dst_ip=LVS1.
> > 11. LVS1 sees packets like these:
> > 11:35:52.689019 IP 192.168.5.221 > 192.168.5.247: icmp 68: time
> > exceeded in-transit
> > 11:35:57.680777 arp who-has 192.168.5.221 tell 192.168.5.111
> > 11:35:57.680863 arp reply 192.168.5.221 is-at 00:0c:76:f4:d6:af
> >
> > I still don't understand what's happenning, but I think it may be
> > something related to bridge.
>
> It might be to do with bridging, it might not, I am not sure.
>
> Simplifying the configuration as you have, is a good idea.
> I would debug the problem from here by opening up connections,
> and tracking the packets across the network, and withing the kernel.
>
> The former can be done using tcpdump, I recommend using the -e option
> to show you the mac address, which is very handy when playing with
> LVS-DR.
>
> The later can be done by placing iptables rules in the various chains.
> They can just be -j ACEPT rules that you monotior
> using watch -n 1 iptables -L -v -n
> Or you can use LOG to get more information about the packets
>
>
> >
> > Any hint?
Hi again:
I've simplified the problem, here's what I've got.
My new topology is like this:
------------------
| INTERNET |
------------------
|
-------------------
| <DSL router> |
| 192.168.6.10 |
-------------------
|
-------------------
| SWITCH |
-------------------
|
|
------------------------
| eth1:192.168.6.111 |
| <LVS1> |
| eth0:192.168.5.111 |
------------------------
|
|
-------------------
| SWITCH |
-------------------
|
------------------
| <CLIENTS> |
| 192.168.5.0/24 |
------------------
And the configuration:
CLIENT:
IP:192.168.5.247
GW: 192.168.5.111
LVS1:
NO BRIDGES!!!
ETH0: 192.168.5.111
ETH1: 192.168.6.111
GW: 192.168.6.10
IPTABLES:
iptables -t nat -A POSTROUTING -s 192.168.5.0/255.255.255.0 -j MASQUERADE
iptables -t nat -A POSTROUTING -d 192.168.5.0/255.255.255.0 -j MASQUERADE
iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 25 -j MARK
--set-mark 65561
IPVS:
ipvsadm -A -f 65561 -s rr
ipvsadm -a -f 65561 -r 192.168.5.221:0 -g -w 1 (This machine doesn't
exist, I just want to see packet flow)
***********************************************************************
With such configuration, I do a telnet xxx.xxx.xxx.xxx 25
(xxx.xxx.xxx.xxx is an external Internet IP) and those packets don't
hit ipvs and I get a prompt from SMTP server.
LVS LOG:
Enter: ip_vs_out, net/ipv4/ipvs/ip_vs_core.c line 741
IPVS: lookup/out TCP 192.168.5.247:3554->xxx.xxx.xxx.xxx:25 not hit
IPVS: packet continues traversal as normal: TCP
192.168.5.247:3554->xxx.xxx.xxx.xxx:25
However, if now I do: (on LVS1)
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 25 -j REDIRECT
--to-ports 9999 -> (There's nothing listening on 9999)
I get the following from LVS log:
IPVS: lookup/in TCP 192.168.5.247:3555->192.168.5.111:9999 not hit
IPVS: lookup service: fwm 65561 TCP 192.168.5.111:9999 hit
IPVS: ip_vs_rr_schedule(): Scheduling...
IPVS: RR: server 192.168.5.221:0 activeconns 0 refcnt 1 weight 1
IPVS: Bind-dest TCP c:192.168.5.247:3555 v:192.168.5.111:9999
d:192.168.5.221:9999 fwd:R s:0 flg:183 cnt:1 destcnt:2
IPVS: Schedule fwd:R c:192.168.5.247:3555 v:192.168.5.111:9999
d:192.168.5.221:9999 flg:1C3 cnt:2
IPVS: Incoming packet: TCP 192.168.5.247:3555->192.168.5.111:9999
IPVS: TCP input [S...] 192.168.5.221:9999->192.168.5.247:3555 state:
NONE->SYN_RECV cnt:2
Enter: ip_vs_dr_xmit, net/ipv4/ipvs/ip_vs_xmit.c line 441
IPVS: new dst 192.168.5.221, refcnt=2, rtos=10
Leave: ip_vs_dr_xmit, net/ipv4/ipvs/ip_vs_xmit.c line 474
IPVS: lookup/in TCP 192.168.1.138:39447->192.168.5.111:22 not hit
IPVS: packet continues traversal as normal: TCP
192.168.1.138:39447->192.168.5.111:22
IPVS: lookup/in TCP 192.168.1.138:39447->192.168.5.111:22 not hit
IPVS: packet continues traversal as normal: TCP
192.168.1.138:39447->192.168.5.111:22
IPVS: lookup/in TCP 192.168.5.247:3555->192.168.5.111:9999 hit
IPVS: Incoming packet: TCP 192.168.5.247:3555->192.168.5.111:9999
Enter: ip_vs_dr_xmit, net/ipv4/ipvs/ip_vs_xmit.c line 441
Leave: ip_vs_dr_xmit, net/ipv4/ipvs/ip_vs_xmit.c line 474
I don't get any prompt, but ipvs can see the packet and tries to
forward it to 192.168.5.221 (which doesn't exist)
It looks like if when I send the packets again to my machine with
"REDIRECT", then ipvs can see them but if it has a default gateway, it
forwards through it and ipvs can't see them.
I'm going to make more tests ... but if anyone could help, please let me know
|
