|
Hello Dan,
On Sat, 13 May 2000, Dan wrote:
> Hi Julian:
>
> I 100%, completely, fully understand the difference between LVS masquerading
> and Linux masquerading. My point is that LVS does not live in a vacuum.
> Other system parameters affect it - if one is not aware of these side
> effects than an LVS box (not just the few snippets of code that implement
> LVS but the LVS system as a *whole*) can easily be a bottleneck. That is not
> an opinion - that is a fact. I thought it would be helpful to share my
> experience with others who may run into the same problem under heavy load
> rather than letting them figure it out for themselves. I sort've thought
> that was the point in having a mailing list. Perhaps it would be better in
> the future if I just keep my experience to myself and let others blindly
> rediscover what I spent so many hours working out.
My first goal was to understand where the problem
come from. Of course, the details a meaningful because it is
possible MASQ to have bugs. This is the reason I asked for
details. And I still can't understand with what kind of
connections the problem happens. My opinion is "it is very
hard to reach a 4096 per-host limit with so small number of
4746 entries". You are the one who first switched to private
discussion, remember? My recommendation is: don't perform
masq-to-host tests with the masquerading. Of course, the
ports are limited. For me, the main limit in the
masquerading (lets tell it to the users too) is the limit
for the number of connections per protocol which is now
40960 . The limit is increased by changing PORT_MASQ_MUL.
The other parameter that can be tuned is IP_MASQ_TAB_SIZE to
lookup the masq tables faster. 256 is not enough. LVS uses
4096 by default for its table. And as a last step the
reserved masq port range: PORT_MASQ_BEGIN and PORT_MASQ_END.
But only when there are many connections from the internal
hosts to one external service.
I like the details :) Can you tell me what is the
reported message when a limit is reached but in situation
without your masq-to-remote_host floods. In normal client
load. In your first message you report for 36214 free ports
but induced from tests. This is the host-to-host masquerade
limit. Once you said that you have 10000-11000 connections.
What is the reported "free ports" value when the message is
displayed in normal client load? Are these connections
established or are connections reported by looking in the
/proc file system, i.e. in any state? You know, there are
many connections in FIN_WAIT state. It is possible the
40960 limit to be reached. It is recommended the
/proc/sys/net/ipv4/ip_always_defrag value to be inspected.
But in your configuration it is very difficult because this
number includes not only the number of the MASQ connections
but also the number of LVS connections (active and
inactive). And only the MASQ connections are limited.
Well, You know, the main goal is to tune the MASQ
for such situations, for all LVS users. And if possible to
solve any related problems in the masquerade. Because the
problem exists even without applying the LVS patch. Right?
Later we can include some changes in the LVS patch related
to the masquerade limitations. This is the reason I think
the details are useful. By this way everyone can decide if
this is a possible problem for his setup. And may be to
switch to direct route mode because the client requests and
the proxy requests can be separated. For example to use a
LVS director to balance the client requests to the proxy
servers but the real servers to use other gateway(s) for
their FTP/HTTP traffic. Of course, there are so many things
that must be considered. At least, it is not very easy.
My wish is the LVS web site to include many variants
for setups. Even now there are many examples but may be not
enough.
And please change your statement to "Masquerading is
a bottleneck", now, when all we know the difference :)
I just searched the archives for examples. You can
look in this interesting link:
http://wwwcache.ja.net/JanetService/PilotService-announcement.html
Regards
--
Julian Anastasov <uli@xxxxxxxxxxxxxxxxxxxxxx>
|
