|
Okay, here goes:
----------------------------------------------------------------------------
--
CLIENT: 216.163.120.2
LVS: 216.163.120.8
VIP: 216.163.120.6
DIRECTOR: 10.75.0.1 (Gateway for RS) (Second eth on LVS)
RIP: 10.75.64.17
----------------------------------------------------------------------------
---
lsmod on LVS - See ip_masq_ftp is loaded
# lsmod
Module Size Used by
iBCS-smp 136912 0
pcmcia_core 48416 0
ip_masq_ftp 2584 1
bsd_comp 3764 0 (unused)
ppp 20652 0 [bsd_comp]
slip 7388 0 (unused)
slhc 4504 0 [ppp slip]
----------------------------------------------------------------------------
--
Traceroute from RIP->Client
# traceroute -n -s 10.75.64.17 216.163.120.2
traceroute to 216.163.120.2 (216.163.120.2) from 10.75.64.17, 30 hops max,
40 byte packets
1 10.75.0.1 0.215 ms 0.137 ms 0.133 ms
2 216.163.120.2 0.325 ms 0.253 ms 0.25 ms
----------------------------------------------------------------------------
--
FTP reports with debug. First Active 'ls' then Passive 'ls'
# ftp 216.163.120.6
Connected to 216.163.120.6.
220 Welcome to (psln.com). Enter username to continue.
Name (216.163.120.6:root): webmaster
331 Welcome 'webmaster', enter password to login.
Password:
230 User 'webmaster' login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> debug
Debugging on (debug=1).
ftp> ls
---> PORT 216,163,120,13,9,248
200 PORT command successful. Data port is 216.163.120.13 port 2552.
---> LIST
150 Starting ASCII transfer for file listing.
total 84
drwxr-xr-x 2 web nogroup 4096 Jan 30 09:48 cgi-bin
drwxr-xr-x 999 web nogroup 73728 May 23 18:21 users
drwxr-xr-x 15 web nogroup 4096 May 22 11:38 www
226 Transfer done. 202 bytes transferred.
ftp> passive
Passive mode on.
ftp> ls
---> PASV
227 Passive mode on (10,75,32,17,18,31)
ftp: connect: No route to host
ftp>
------------------------------------------------------------------------
CLIENT TCP DUMP: ACTIVE 'ls' (okay)
# tcpdump -ln host 216.163.120.6
tcpdump: listening on eth0
12:05:03.114902 216.163.120.2.3200 > 216.163.120.6.21: P
185499542:185499569(27) ack 4101292867 win 16060 <nop,nop,timestamp 60803092
9303327> (DF) [tos 0x10]
12:05:03.115370 216.163.120.6.21 > 216.163.120.2.3200: P 1:69(68) ack 27 win
32120 <nop,nop,timestamp 9304353 60803092> (DF)
12:05:03.115644 216.163.120.2.3200 > 216.163.120.6.21: P 27:33(6) ack 69 win
16060 <nop,nop,timestamp 60803092 9304353> (DF) [tos 0x10]
12:05:03.117043 216.163.120.6.21 > 216.163.120.2.3200: P 69:116(47) ack 33
win 32120 <nop,nop,timestamp 9304354 60803092> (DF)
12:05:03.117360 216.163.120.6.21 > 216.163.120.2.3200: P 116:159(43) ack 33
win 32120 <nop,nop,timestamp 9304354 60803092> (DF)
12:05:03.127933 216.163.120.2.3200 > 216.163.120.6.21: . ack 159 win 16060
<nop,nop,timestamp 60803094 9304354> (DF) [tos 0x10]
----------------------------------------------------------------------------
----
CLIENT TCP DUMP: PASSIVE 'ls' (error)
# tcpdump -ln host 216.163.120.6
tcpdump: listening on eth0
12:06:25.338930 216.163.120.2.3208 > 216.163.120.6.21: P
268418021:268418027(6) ack 4183031124 win 16060 <nop,nop,timestamp 60811315
9311526> (DF) [tos 0x10]
12:06:25.339386 216.163.120.6.21 > 216.163.120.2.3208: P 1:42(41) ack 6 win
32120 <nop,nop,timestamp 9312575 60811315> (DF)
12:06:25.357949 216.163.120.2.3208 > 216.163.120.6.21: . ack 42 win 16060
<nop,nop,timestamp 60811317 9312575> (DF) [tos 0x10]
----------------------------------------------------------------------------
--------------
LVS TCP DUMP: ACTIVE 'ls' (okay)
# tcpdump -ln host 216.163.120.2
tcpdump: listening on eth0
15:58:44.219419 216.163.120.2.3221 > 216.163.120.6.21: P
487814514:487814541(27) ack 105745421 win 16060 <nop,nop,timestamp 60832025
9332521> (DF) [tos 0x10]
15:58:44.219780 216.163.120.6.21 > 216.163.120.2.3221: P 1:69(68) ack 27 win
32120 <nop,nop,timestamp 9333284 60832025> (DF)
15:58:44.220097 216.163.120.2.3221 > 216.163.120.6.21: P 27:33(6) ack 69 win
16060 <nop,nop,timestamp 60832025 9333284> (DF) [tos 0x10]
15:58:44.220586 216.163.120.8.63345 > 216.163.120.2.3222: S
128613619:128613619(0) win 32120 <mss 1460,sackOK,timestamp 9333284[|tcp]>
(DF)
15:58:44.220765 216.163.120.2.3222 > 216.163.120.8.63345: S
517730102:517730102(0) ack 128613620 win 16060 <mss 1460,sackOK,timestamp
60832025[|tcp]> (DF)
15:58:44.220912 216.163.120.8.63345 > 216.163.120.2.3222: . ack 1 win 32120
<nop,nop,timestamp 9333285 60832025> (DF)
15:58:44.221491 216.163.120.6.21 > 216.163.120.2.3221: P 69:116(47) ack 33
win 32120 <nop,nop,timestamp 9333285 60832025> (DF)
15:58:44.221747 216.163.120.8.63345 > 216.163.120.2.3222: P 1:203(202) ack 1
win 32120 <nop,nop,timestamp 9333285 60832025> (DF)
15:58:44.221756 216.163.120.8.63345 > 216.163.120.2.3222: F 203:203(0) ack 1
win 32120 <nop,nop,timestamp 9333285 60832025> (DF)
15:58:44.221790 216.163.120.6.21 > 216.163.120.2.3221: P 116:159(43) ack 33
win 32120 <nop,nop,timestamp 9333285 60832025> (DF)
15:58:44.221949 216.163.120.2.3222 > 216.163.120.8.63345: . ack 203 win
15858 <nop,nop,timestamp 60832025 9333285> (DF) [tos 0x8]
15:58:44.221979 216.163.120.2.3222 > 216.163.120.8.63345: . ack 204 win
15857 <nop,nop,timestamp 60832025 9333285> (DF) [tos 0x8]
15:58:44.222509 216.163.120.2.3222 > 216.163.120.8.63345: F 1:1(0) ack 204
win 16060 <nop,nop,timestamp 60832025 9333285> (DF) [tos 0x8]
15:58:44.222644 216.163.120.8.63345 > 216.163.120.2.3222: . ack 2 win 32120
<nop,nop,timestamp 9333285 60832025> (DF)
15:58:44.238497 216.163.120.2.3221 > 216.163.120.6.21: . ack 159 win 16060
<nop,nop,timestamp 60832027 9333285> (DF) [tos 0x10]
----------------------------------------------------------------------------
------------
LVS TCP DUMP: PASSIVE 'ls' (error)
# tcpdump -ln host 216.163.120.2
tcpdump: listening on eth0
15:59:53.681284 216.163.120.2.3221 > 216.163.120.6.21: P
487814547:487814553(6) ack 105745579 win 16060 <nop,nop,timestamp 60838971
9333285> (DF) [tos 0x10]
15:59:53.681643 216.163.120.6.21 > 216.163.120.2.3221: P 1:42(41) ack 6 win
32120 <nop,nop,timestamp 9340230 60838971> (DF)
15:59:53.697082 216.163.120.2.3221 > 216.163.120.6.21: . ack 42 win 16060
<nop,nop,timestamp 60838973 9340230> (DF) [tos 0x10]
15:59:58.673588 arp who-has 216.163.120.2 tell 216.163.120.8
15:59:58.673721 arp reply 216.163.120.2 is-at 0:90:27:54:8e:5e
----------------------------------------------------------------------------
----------
RS TCP DUMP: ACTIVE 'ls' (okay)
# tcpdump -ln host 216.163.120.2
tcpdump: listening on eth0
16:09:35.021918 216.163.120.2.3249 > 10.75.64.17.21: P
849928171:849928198(27) ack 454452698 win 16060 <nop,nop,timestamp 60867221
9367823> (DF) [tos 0x10]
16:09:35.022341 10.75.64.17.21 > 216.163.120.2.3249: P 1:69(68) ack 27 win
32120 <nop,nop,timestamp 9368479 60867221> (DF)
16:09:35.022811 216.163.120.2.3249 > 10.75.64.17.21: P 27:33(6) ack 69 win
16060 <nop,nop,timestamp 60867221 9368479> (DF) [tos 0x10]
16:09:35.023236 10.75.64.17.20 > 216.163.120.2.3251: S
491964086:491964086(0) win 32120 <mss 1460,sackOK,timestamp 9368479[|tcp]>
(DF)
16:09:35.023538 216.163.120.2.3251 > 10.75.64.17.20: S
888373952:888373952(0) ack 491964087 win 16060 <mss 1460,sackOK,timestamp
60867221[|tcp]> (DF)
16:09:35.023565 10.75.64.17.20 > 216.163.120.2.3251: . ack 1 win 32120
<nop,nop,timestamp 9368479 60867221> (DF)
16:09:35.023907 10.75.64.17.21 > 216.163.120.2.3249: P 69:116(47) ack 33 win
32120 <nop,nop,timestamp 9368479 60867221> (DF)
16:09:35.024157 10.75.64.17.20 > 216.163.120.2.3251: P 1:203(202) ack 1 win
32120 <nop,nop,timestamp 9368479 60867221> (DF)
16:09:35.024179 10.75.64.17.20 > 216.163.120.2.3251: F 203:203(0) ack 1 win
32120 <nop,nop,timestamp 9368479 60867221> (DF)
16:09:35.024240 10.75.64.17.21 > 216.163.120.2.3249: P 116:159(43) ack 33
win 32120 <nop,nop,timestamp 9368479 60867221> (DF)
16:09:35.024546 216.163.120.2.3251 > 10.75.64.17.20: . ack 203 win 15858
<nop,nop,timestamp 60867221 9368479> (DF) [tos 0x8]
16:09:35.024567 216.163.120.2.3251 > 10.75.64.17.20: . ack 204 win 15857
<nop,nop,timestamp 60867221 9368479> (DF) [tos 0x8]
16:09:35.025141 216.163.120.2.3251 > 10.75.64.17.20: F 1:1(0) ack 204 win
16060 <nop,nop,timestamp 60867221 9368479> (DF) [tos 0x8]
16:09:35.025157 10.75.64.17.20 > 216.163.120.2.3251: . ack 2 win 32120
<nop,nop,timestamp 9368479 60867221> (DF)
16:09:35.038181 216.163.120.2.3249 > 10.75.64.17.21: . ack 159 win 16060
<nop,nop,timestamp 60867223 9368479> (DF) [tos 0x10]
----------------------------------------------------------------------------
----------
RS TCP DUMP: PASSIVE 'ls' (error)
# tcpdump -ln host 216.163.120.2
tcpdump: listening on eth0
16:10:34.568736 216.163.120.2.3249 > 10.75.64.17.21: P
849928204:849928210(6) ack 454452856 win 16060 <nop,nop,timestamp 60873176
9368479> (DF) [tos 0x10]
16:10:34.569020 10.75.64.17.21 > 216.163.120.2.3249: P 1:42(41) ack 6 win
32120 <nop,nop,timestamp 9374434 60873176> (DF)
16:10:34.585247 216.163.120.2.3249 > 10.75.64.17.21: . ack 42 win 16060
<nop,nop,timestamp 60873178 9374434> (DF) [tos 0x10]
----------------------------------------------------------------------------
---------
-----Original Message-----
From: Julian Anastasov [mailto:ja@xxxxxx]
Sent: Wednesday, May 23, 2001 9:48 PM
To: Jeremy Kusnetz
Cc: 'lvs-users@xxxxxxxxxxxxxxxxxxxxxx'
Subject: Re: ip_masq_ftp nat passive
Hello,
On Wed, 23 May 2001, Jeremy Kusnetz wrote:
> Here are the IP chains I'm setting up:
>
> echo "1" > /proc/sys/net/ipv4/ip_forward
> ipchains -F
> ipchains -A forward -j MASQ -s 10.75.0.0/16 -d 0.0.0.0/0
right
> I tried setting up ipchains like your script does, but I got connection
> refused errors when trying to ftp, so I put it back the way I originally
had
> it. I tried this:
>
> ipchains -A forward -p tcp -j MASQ -s 10.75.0.9 ftp -d 0.0.0.0/0
> ipchains -A forward -p tcp -j MASQ -s 10.75.32.9 ftp -d 0.0.0.0/0
> ipchains -A forward -p tcp -j MASQ -s 10.75.64.9 ftp -d 0.0.0.0/0
wrong
> Can I do a global like I have above, or do I have to do each service for
> each realserver? If so, what is wrong with the above?
global
OK, follow the mini-troubleshooting-check-it-yourself Layer
4 NAT HOWTO:
Q.1 Can the real server ping client?
rs# ping -n client
A.1 Yes => good
A.2 No => bad
ipchains -A forward -s RIP -j MASQ
Q.2 Traceroute to client goes through LVS box and reaches the client?
traceroute -n -s RIP CLIENT_IP
A.1 Yes => good
A.2 No => bad
same ipchains command as in Q.1
Q.3 Is the traffic forwarded from the LVS box, in both directions?
For all interfaces on director:
tcpdump -ln host CLIENT_IP
The right sequence, i.e. the IP addresses and ports on each
step (the reversed for the in->out direction are not shown):
CLIENT
| CIP:CPORT -> VIP:VPORT
| ||
| \/
out | CIP:CPORT -> VIP:VPORT
|| LVS box
\/ | CIP:CPORT -> RIP:RPORT
in | ||
| \/
| CIP:CPORT -> RIP:RPORT
+
REAL SERVER
A.1 Yes, in both directions => good (for Layer 4, probably not for L7)
A.2 The packets from the real server are dropped => bad:
- rp_filter protection on the incoming interface, probably
hit from local client
- firewall rules drop the replies
A.3 The packets from the real servers leave the director unchanged
- missing -j MASQ ipchains rule in the LVS box
A.4 All packets from the client are dropped
- the requests are received on wrong interface with rp_filter
protection
- firewall rules drop the requests
A.5 The client connections are refused or are served from service
in the LVS box
- client and LVS are on same host => not valid
- the packets are not marked from the firewall and don't hit
firewall mark based virtual service
Q.4 Is the traffic replied from the real server?
For the outgoing interface on real server:
tcpdump -ln host CLIENT_IP
A.1 Yes, SYN+ACK => good
A.2 TCP RST => bad, No listening real service
A.3 ICMP message => bad, Blocked from Firewall/No listening service
A.4 The same request packet leaves the real server => missing accept
rules or RIP is not defined
A.5 No reply => real server problem:
- the rp_filter protection drops the packets
- the firewall drops the request packets
- the firewall drops the replies
A.6 Replies goes through another device or don't go to the LVS
box =? bad
- the route to the client is direct and so don't pass the LVS
box, for example:
- client on the LAN
- client and real server on same host
- wrong route to the LVS box is used => use another
The result: we need these outputs:
rs# tcpdump -ln host CIP
rs# traceroute -n -s RIP CIP
lvs# tcpdump -ln host CIP
client# tcpdump -ln host CIP
For more deep problems use tcpdump -len, i.e. sometimes the link layer
addresses help a bit.
For FTP:
LVS-NAT in Linux 2.2 requires:
- modprobe ip_masq_ftp (before 2.2.19)
- modprobe ip_masq_ftp in_ports=21 (2.2.19+)
LVS-NAT in Linux 2.4 requires:
- ip_vs_ftp
LVS-DR/TUN require persistent flag
FTP reports with debug mode enabled are useful:
# ftp
ftp> debug
ftp> open my.virtual.ftp.service
ftp> ...
ftp> dir
ftp> passive
ftp> dir
Regards
--
Julian Anastasov <ja@xxxxxx>
|
