|
I spoke to soon about this configuration working, the output of ipvsadm
lead me to believe connections and packets were being load balanced,
however they are now all coming from the real server which also is the
director. I'll call this the 'first node'.
Here is some more info in addition to the details in my previous emails:
First node routing:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
0.0.0.0 172.17.0.16 0.0.0.0 UG 0 0 0 bond0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 bond0
Second node routing:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
0.0.0.0 172.17.0.16 0.0.0.0 UG 0 0 0
bond0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0
bond0
The second node also has the VIP on the loopback device
How I setup the director
ipvsadm -C
ipvsadm -A -f 100 -s rr
ipvsadm -a -f 100 -r 172.17.0.16 -w 100
ipvsadm -a -f 100 -r 172.17.0.17 -w 100
After conducting a few wget's here are the expected results from ipvsadm
while not port mapping:
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Conns InPkts OutPkts InBytes
OutBytes
-> RemoteAddress:Port
FWM 100 12 61 0 5416
0
-> 172.17.0.16:0 6 25 0 2422
0
-> 172.17.0.17:0 6 36 0 2994
0
Actual results from ipvsadm when using port mapping:
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Conns InPkts OutPkts InBytes
OutBytes
-> RemoteAddress:Port
FWM 100 2 8 4 778
216
-> 172.17.0.16:0 1 4 4 389
216
-> 172.17.0.17:0 1 4 0 389
0
Packets are being sent from the RIP of the first node only. From my
understanding when using DR OutPkts should always be zero.
Output from tcpdump is a bit strange (first pass):
02:10:51.986719 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [S], seq
1873056683, win 14600, options [mss 1460,sackOK,TS val 3044575792 ecr
0,nop,wscale 7], length 0
02:10:51.986719 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [S], seq
1873056683, win 14600, options [mss 1460,sackOK,TS val 3044575792 ecr
0,nop,wscale 7], length 0
02:10:51.986719 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [S], seq
1873056683, win 14600, options [mss 1460,sackOK,TS val 3044575792 ecr
0,nop,wscale 7], length 0
02:10:51.986814 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [S.], seq
2970678457, ack 1873056684, win 14480, options [mss 1460,sackOK,TS val
978483 ecr 3044575792,nop,wscale 7], length 0
02:10:51.986919 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [S.], seq
2970678457, ack 1873056684, win 14480, options [mss 1460,sackOK,TS val
978483 ecr 3044575792,nop,wscale 7], length 0
02:10:51.987030 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [.], ack 1, win
115, options [nop,nop,TS val 3044575793 ecr 978483], length 0
02:10:51.987030 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [.], ack 1, win
115, options [nop,nop,TS val 3044575793 ecr 978483], length 0
*02:10:51.987030 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [.], ack
2970678458, win 115, options [nop,nop,TS val 3044575793 ecr 978483], length
0*
02:10:51.987079 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [P.], seq
1:174, ack 1, win 115, options [nop,nop,TS val 3044575793 ecr 978483],
length 173
02:10:51.987079 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [P.], seq
1:174, ack 1, win 115, options [nop,nop,TS val 3044575793 ecr 978483],
length 173
*02:10:51.987079 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [P.], seq
0:173, ack 1, win 115, options [nop,nop,TS val 3044575793 ecr 978483],
length 173*
02:10:51.987126 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [.], ack 174,
win 122, options [nop,nop,TS val 978484 ecr 3044575793], length 0
02:10:51.987128 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [.], ack 174,
win 122, options [nop,nop,TS val 978484 ecr 3044575793], length 0
02:10:51.987241 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [F.], seq 1,
ack 174, win 122, options [nop,nop,TS val 978484 ecr 3044575793], length 0
02:10:51.987245 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [F.], seq 1,
ack 174, win 122, options [nop,nop,TS val 978484 ecr 3044575793], length 0
02:10:51.987426 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [.], ack 2, win
115, options [nop,nop,TS val 3044575793 ecr 978484], length 0
02:10:51.987426 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [.], ack 2, win
115, options [nop,nop,TS val 3044575793 ecr 978484], length 0
*02:10:51.987426 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [.], ack 2,
win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0*
02:10:51.987480 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [F.], seq 174,
ack 2, win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0
02:10:51.987480 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [F.], seq 174,
ack 2, win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0
*02:10:51.987480 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [F.], seq
173, ack 2, win 115, options [nop,nop,TS val 3044575793 ecr 978484], length
0*
02:10:51.987527 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [.], ack 175,
win 122, options [nop,nop,TS val 978484 ecr 3044575793], length 0
02:10:51.987529 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [.], ack 175,
win 122, options [nop,nop,TS val 978484 ecr 3044575793], length 0
Second pass:
02:10:59.360433 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [S], seq
2134630706, win 14600, options [mss 1460,sackOK,TS val 3044583166 ecr
0,nop,wscale 7], length 0
02:10:59.360433 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [S], seq
2134630706, win 14600, options [mss 1460,sackOK,TS val 3044583166 ecr
0,nop,wscale 7], length 0
02:10:59.360519 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [S.], seq
2952275478, ack 2134630707, win 14480, options [mss 1460,sackOK,TS val
985857 ecr 3044583166,nop,wscale 7], length 0
02:10:59.360524 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [S.], seq
2952275478, ack 2134630707, win 14480, options [mss 1460,sackOK,TS val
985857 ecr 3044583166,nop,wscale 7], length 0
02:10:59.360627 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [.], ack 1, win
115, options [nop,nop,TS val 3044583166 ecr 985857], length 0
02:10:59.360627 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [.], ack 1, win
115, options [nop,nop,TS val 3044583166 ecr 985857], length 0
02:10:59.360728 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [P.], seq
1:174, ack 1, win 115, options [nop,nop,TS val 3044583166 ecr 985857],
length 173
02:10:59.360728 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [P.], seq
1:174, ack 1, win 115, options [nop,nop,TS val 3044583166 ecr 985857],
length 173
02:10:59.360751 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [.], ack 174,
win 122, options [nop,nop,TS val 985857 ecr 3044583166], length 0
02:10:59.360753 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [.], ack 174,
win 122, options [nop,nop,TS val 985857 ecr 3044583166], length 0
02:10:59.361058 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [F.], seq 1,
ack 174, win 122, options [nop,nop,TS val 985858 ecr 3044583166], length 0
02:10:59.361062 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [F.], seq 1,
ack 174, win 122, options [nop,nop,TS val 985858 ecr 3044583166], length 0
02:10:59.361270 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [F.], seq 174,
ack 2, win 115, options [nop,nop,TS val 3044583167 ecr 985858], length 0
02:10:59.361270 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [F.], seq 174,
ack 2, win 115, options [nop,nop,TS val 3044583167 ecr 985858], length 0
02:10:59.361331 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [.], ack 175,
win 122, options [nop,nop,TS val 985858 ecr 3044583167], length 0
02:10:59.361334 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [.], ack 175,
win 122, options [nop,nop,TS val 985858 ecr 3044583167], length 0
The end result is that the packets are always coming from the first node
and never balanced to the second node.
Thanks for any further help, seems the solution is really close!
Jacoby
On Sat, Jan 18, 2014 at 12:44 AM, Julian Anastasov <ja@xxxxxx> wrote:
>
> Hello,
>
> On Fri, 17 Jan 2014, Jacoby Hickerson wrote:
>
> > Thanks Julian! After enabling CONFIG_IP_VS_NFCT and setting conntrack
> to 1
> > that resolved the problem.
> > However, how leery should I be with it consuming memory? Is there a
> test to
> > monitor this consumption? Currently the nf_conntrack_max is set to the
> > default: 65536
>
> cat /proc/slabinfo | grep nf_conntrack
> or 'slabtop' can show the object size used by conntracks.
> It should be 240+ bytes. You can expect one conntrack per
> IPVS connection. You can also see conntracks with
> cat /proc/net/nf_conntrack | less
>
> cat /proc/sys/net/netfilter/nf_conntrack_count shows
> the current number of conntracks. You can look for 'count'
> at __nf_conntrack_alloc() and nf_conntrack_free() to see
> how it is implemented.
>
> Regards
>
> --
> Julian Anastasov <ja@xxxxxx>
>
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
