|
On our load balancer not only are we running LVS-NAT, but we are running
stunnel which forwards things such as mysql connections from the outside
to a server on the same network as the realservers.
We wrote a quick script that runs on a machine on the same network as the
director which opens up sockets on the director to talk to various
services that the director is loadbalancing to time the speed of the
services. Nothing fancy. We did find a bug in the script on where it
wasn't closing out the sockets, but I assume the sockets should have been
closed when the script finished running.
Anyway, after running the script, the services running that talk through
stunnel to the director are no longer able to establish a new connection
and I get a connection refused error. If the connection is already
established it works fine until I need to reestablish the connection.
I've run tcpdumps both on the director and the machine the script ran from
and I can see packets getting to and from the director:
(host running script)
17:24:49.070487 smarthost.trueband.net.33935 > trueband.net.3309: S
2070384008:2070384008(0) win 5840 <mss 1460,sackOK,timestamp
7757892[|tcp]> (DF)
17:24:49.070696 trueband.net.3309 > smarthost.trueband.net.33935: R 0:0(0)
ack 2070384009 win 0 (DF)
and
(director)
17:24:49.183998 smarthost.trueband.net.33935 > trueband.net.3309: S
2070384008:2070384008(0) win 5840 <mss 1460,sackOK,timestamp
7757892[|tcp]> (DF)
17:24:49.184349 trueband.net.3309 > smarthost.trueband.net.33935: R 0:0(0)
ack 2070384009 win 0 (DF)
But I get this:
telnet 216.163.120.10 3309
Trying 216.163.120.10...
telnet: Unable to connect to remote host: Connection refused
There are no iptables rules to block these on either side. Actually there
are rules on the director to only allow these connections from that
particular machine and block everything else, but even after running
iptables -F I still get connection refused.
I've rebooted the machine sending the requests to no avail. The only
thing that cleared the problem was rebooting the director. After a reboot
everything works fine, but as soon as we run that script we start getting
the connection refused errors.
Even though LVS is not doing anything with these ports, I've flushed all
iptable rules, restarted stunnel on both ends. The only thing I haven't
tried yet is just stopping and starting LVS. That will be the next thing
I try when we do a scheduled maintanence.
Any ideas?
|
