|
Hello,
On Sat, 25 Aug 2012, Dmitry Akindinov wrote:
> Hello,
>
> We are currently stuck with the following ipvs problem:
>
> 1. The configuration includes a (potentially large) set of servers providing
> various services - besides HTTP (POP, IMAP, LDAP, SMTP, XMPP, etc.) The test
> setup includes just 2 servers, though.
> 2. Each server runs a stock version of CentOS 6.0
OK, I don't know what kernel and patches includes
every distribution. Can you tell at least what shows uname -a?
> 3. The application software (CommuniGate Pro) controls the ipvs kernel module
> using the ipvsadm commands.
> 4. On each server, iptables are configured to:
> a) disable connection tracking for VIP address(es)
> b) mark all packets coming to the VIP address(es) with the mark value of
> 100.
> 5. On the currently active load balancer, the ipvsadm is used to configure
> ipvs to load-balance packets with the marker 100:
> -A -f 100 -s rr -p 1
> -a -f 100 -r <server1> -g
> -a -f 100 -r <server2> -g
> ....
> where the active balancer itself is one of the <serverN>
> 6. All other servers (just 1 "other" server in our test config) are running
> ipvs, but with an empty rule set.
I think, running slaves without same rules is a mistake.
When the slave receives sync message it has to assign it
to some virtual server and even assign real server for
this connection. But if this slave is also a real server
the things are complicated. I now check the code and
do not see where we prevent backup to schedule traffic
received from current master. The master gives the traffic
to backup because considers it a real server but this
backup with rules decides to schedule it to different real server.
This problem can not happen for NAT, only for DR/TUN,
I see that you are using DR forwarding method. So,
currently, IPVS users do not add ipvsadm rules in backup
for DR/TUN for this reason?
> 7. The active load balancer runs the sync daemon started with ipvsadm
> --start-daemon master
> 7. All other servers run the sync daemon started with ipvsadm --start-daemon
> backup.
>
> As a result, all servers have the duplicated ipvs connection tables. If the
> active balancer fails, some other server assumes its role by arp-broadcasting
> VIP and loading the ipvs rule set listed above.
In initial email you said:
"Now, we initiate a failover. During the failover, the ipvs table on the
old "active" balancer is cleared,"
Why do you clear the connection table? What if you
decide after 10 seconds to return back control to the first
master?
> When a connection is being established to the VIP address, and the active load
> balancer directs it to itself, everything works fine.
I assume you are talking for box 2 (the new master)
> When a connection is being established to the VIP address, and the active load
> balancer directs it to some other server, the connection is established fine,
> and if the protocol is POP, IMAP, SMTP, the server prompt is sent to the
> client via VIP, and it is seen by client just fine.
You mean, new connection does 3-way handshake via
the new master to other real servers and succeeds, or already
established connection before the failover work after failover?
Is packet directed to old master?
> But when the client tries to send anything to the server, the packet
> (according to tcpdump) reaches the load balancer server, and from there it
> reaches the "other" server. Where the packet is dropped. The client resends
> that packet, it goes to the active balancer, then to the "other" server, and
> it is dropped again.
Why this real server drops the packet? What is
different in this packet? Are you talking about connections
created before failover, that they can not continue to work after
failover? May be problem happens for DR. Can you show
tcpdump in old master that 3-way traffic is received and also
that it is replied by it, not by some real server.
Problem can happen only if master sends new traffic to
backup (its real server). For example:
- master schedules SYN to real server which is backup with same rules
- SYNC conn is not sent before IPVS conn enters ESTABLISHED state,
so backup does not know for such connection, it looks like new one
- backup has rules, it decides to use real server 3 and
directs the SYN there. It can happen only for DR/TUN because
the daddr is VIP, that is why people overcome the problem
by checking that packet comes from some master and not
from uplink gateway MAC. For NAT there is no such double-step
scheduling because the backups' rules do not match the
internal real server IP in the daddr, they work only for VIP
- more traffic comes, backup directs it to real server 3
- the first SYNC message for this connection comes from
master but the SYNC message claims the backup is a real
server for this connection. Looking at current code,
ip_vs_proc_conn ignores the fact that master wants the
backup as real server for this connection, backup will
continue to use real server 3. For now, I don't see where
this can fail except if persistence comes in the game
or if failover happens to another backup which will use
real server 3. The result is that the backup acts as
balancer even if it is just a backup without master function.
> Observations:
> *) if ipvs is switched off on that "other" server, everything works just fine
> (service ipvsadm stop)
So, someone stops the SYN traffic in backup?
> *) if ipvs is left running on that "other" server, but syncing daemon is
> switched off, everything works just fine.
Without rules in this backup?
> We are 95% sure that the problem appears only if the "other server" ipvs
> connection table gets a copy of this
> connection from the active balancer. If the copy is not there (the sync daemon
> was stopped when the connection
> was established, and restarted immediately after), everything works just fine.
Interesting, new master forwards to old master,
so it should send SYNC containing the old master as real
server, how can there be a problem, may be your kernel does
not support properly the local server function which is
fixed 2 years ago.
> *) the problem exists for protocols like POP, IMAP, SMTP - where the server
> immediately sends some data (prompt) to the client, as soon as the connection
> is established.
The SYNC packets always go after the traffic, so
not sure why SYN will work while there will be difference for
other traffic. May be your kernel version reacts differently
when first SYNC message claims server 3 is the real server,
not backup 1 and the double-scheduling is broken after
3-way handshake.
> When the HTTP protocol is used, the problem does not exist, but only if the
> entire request is sent as one packet. If the HTTP connection is a "keep-alive"
> one, subsequent requests in the same connection do not reach the application
> either.
> I.e. it looks like the "idling" ipvs allows only one incoming data packet in,
> and only if there has been no outgoing packet on that connection yet.
May be SYNC message changes the destination in
backup as I already said above? Some tcpdump output will
be helpful in case you don't know how to dig into the
sources of your kernel.
> *) Sometimes (we still cannot reproduce this reliably) the ksoftirqd threads
> on the "other" server jump to 100% CPU
> utilization, and when it happens, it happens in reaction to one connection
> being established.
This sounds as a problem fixed 2 years ago:
http://marc.info/?t=128428786100001&r=1&w=2
At that time even fwmark was not supported for
sync purposes.
Note that many changes happened in this 2 year
period, some for fwmark support for IPVS sync, some for
the 100% loops. Without knowing the kernel version
I'm not willing to flood you with changes that you
should check if they are present in your kernel if
it contains additional patches.
> Received suggestions:
> *) it was suggested that we use iptables to filter the packets to VIP that
> come from other servers in the farm (using their MAC addresses) and direct
> them directly to the local application, bypassing ipvs processing. We cannot
> do that, as servers in the farm can be added at any moment, and updating the
> list of MACs on all servers is not trivial. It may be easier to filter the
> packets that come from the router(s), which are less numerous and do not
> change that often.
> But it does not look like a good solution. If the ipvs table on "inactive"
> balancer drops packets, why would it stop dropping them when it becomes an
> "active" balancer? Just because there will be ipvs rules present?
>
> *) The suggestion to separate load balancer(s) and real servers won't work for
> us at all.
>
> *) We tried not to empty the ipvs table on the "other" server(s). Instead, we
> left it balancing - but with only one "real server" - this server itself. Now,
> the "active" load balancer dsitributes packets to itself and other servers,
> and when the packets hit the "other" server(s), they get to the ipvs again,
> where they are balanced again, but to the local server only.
Very good, only that you need recent kernel for this,
2010-Nov +, there are fixes even after that time.
> It looks like it does solve the problem. But now the ipvs connection table on
> the "other" server(s) is filled by both that server ipvs itself and by the
> sync-daemon. While the locally-generated connection table entries should be
> the same as corresponding entries received with the sync daemon, it does not
> look good when the same table is modified from two sources.
Sync happens only in one direction at a time, from
current master to current backup (it can be more than one).
The benefit is that all servers used for sync have same
table and you can switch between them at any time. Of
course, there is some performance price for traffic that
goes to the local stack of backups but they should get from
current master only traffic for their stack.
> Any comment, please? Should we use the last suggestion?
I think, with fresh kernel your setup should be
supported. After showing the kernel version we can decide
for further steps. I'm not sure if we need to change kernel
not to schedule new connections for the BACKUP && !MASTER
configuration. By this way backup can have same rules
as backup which can work for DR/TUN. Without such change
we can not do role change without breaking connections
because the SYNC protocol declares real server 1 as
server while some backup overrides this decision and
uses real server 3, decision not known by other
potential masters.
> --
> Best regards,
> Dmitry Akindinov
> --
Regards
--
Julian Anastasov <ja@xxxxxx>
--
To unsubscribe from this list: send the line "unsubscribe lvs-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
|
