|
Hi,
I've now managed to to get the problematic lvs cluster working so that a client
can be a server (different networks)
accept_local is required which means it won't work on a standard redhat/centos
6 kernel. I used the elrepo kernel for testing.
It took me a while to work out so to help others here is the manual config that
worked for me:
Machine that is a client and a server
Where 192.168.3.107 = backend ip running the service
192.168.2.107 = front end ip used for client requests
192.168.2.1 = front end ip on main gateway (1st default gateway)
192.168.3.149 = backend ip on the load balancer that is used as the 2nd default
gateway (return traffic back from backend ip)
Example network config:
ip addr add 192.168.3.107/24 dev eth0; ip link set eth0 up
ip addr add 192.168.2.107/24 dev eth1; ip link set eth1 up
ip rule flush
ip rule del pref 0 lookup local
ip rule add pref 500 lookup local
#remember to readd these :
ip rule add pref 30000 lookup main
ip rule add pref 30100 lookup default
ip rule add pref 10 iif eth0 lookup local
ip rule add pref 11 iif eth1 lookup local
ip rule add pref 100 to 192.168.3.107 lookup secondary
ip rule add pref 101 from 192.168.3.107 lookup secondary
echo 1 >/proc/sys/net/ipv4/conf/eth0/accept_local
echo 1 >/proc/sys/net/ipv4/conf/eth1/accept_local
ip route add default via 192.168.2.1
ip route flush table secondary
ip route add default via 192.168.3.149 table secondary
This is based upon the following link
http://umweltsuende.wordpress.com/2011/11/13/monolog-fur-linux-server-send-to-self-patch-alternative/
(the english description is at the end)
The main fly in the ointment is the requirement a kernel > 2.6.32. I lack the
skills the back port accept_local and put in a module.
It seems that there appears to be a law that required kernel functionality is
always in the next one to that which you currently use.
This makes thing awkward for using the long term distros.
Best Regards
-----Original Message-----
From: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx
[mailto:lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx] On Behalf Of Dean Scothern
Sent: 08 March 2012 12:45
To: LinuxVirtualServer.org users mailing list.; David Coulson
Subject: Re: [lvs-users] annoying routing problem with a lvs cluster
Hi,
It seems that the solution to my problem involves the use of
/proc/sys/net/ipv4/conf/eth<n>/accept_local
For test purposes I've installed a more recent kernel that offers this
functionality and I've been able to test it with a simple Eth to eth external
test.
So far I've been unsuccessful with the more complicated policy routing
scenario.
Has anyone used accept_local with lvs nat to make something like the scenario I
outlined earlier work?
Best Regards
-----Original Message-----
From: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx
[mailto:lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx] On Behalf Of Dean Scothern
Sent: 02 March 2012 10:12
To: David Coulson
Cc: LinuxVirtualServer.org users mailing list.
Subject: Re: [lvs-users] annoying routing problem with a lvs cluster
Thanks,
It's worth a try but I think it will fall foul of the host route in the local
table, which will apply first.
Best Regards
From: David Coulson [mailto:david@xxxxxxxxxxxxxxxx]
Sent: 01 March 2012 21:36
To: Dean Scothern
Cc: LinuxVirtualServer.org users mailing list.
Subject: Re: [lvs-users] annoying routing problem with a lvs cluster
Had a thought on this - You could use iptables to mark your return packet, then
run it through a separate routing table based on a 'i ip ru add fwmark x table
y' option. Your separate routing table would just have a default gw pointing to
the VIP from your LVS cluster.
Not sure if that would work well, but it's worth a shot.
On Mar 1, 2012, at 8:59 AM, Dean Scothern wrote:
Thank you for your quick reply.
Whilst snat would work I would prefer not to use it as it hides the source ip
of the packets, making applications that use ip access lists more problem atic
to configure, eg mailservers. Eventually I would expand the clients to include
other networks (internet), and would like log analysis to work.
I would prefer not to use a proxy and pass magic headers with the remote ip
them either.
The link in question also probably cannot easily apply to redhat/centos 6 as
they are based on 2.6.32 kernel and the link mentions 2.6.35, 2.6.36.
Reading further it might be possible to apply the patch set and rebuild the
associated kernel modules.
To be honest I hoping for some route configuration magicry, I feel so close and
surely there must be a way.
Many Thanks
From: David Coulson
[mailto:david@xxxxxxxxxxxxxxxx]<mailto:[mailto:david@xxxxxxxxxxxxxxxx]>
Sent: 01 March 2012 13:04
To: LinuxVirtualServer.org<http://LinuxVirtualServer.org> users mailing list.
Cc: Dean Scothern
Subject: Re: [lvs-users] annoying routing problem with a lvs cluster
You need to SNAT real server traffic going to your real servers.
Quick google found this:
http://blog.loadbalancer.org/enabling-snat-in-lvs-xt_ipvs-and-iptables/
I'm presuming it's in mainline by now, but I know it's not in RHEL/SuSE yet.
David
On 3/1/12 7:55 AM, Dean Scothern wrote:
Hi,
I've been experimenting with a slightly non standard lvs cluster arrangement.
I have a set of combined real servers/real clients (each machine has both
services and clients) and two machines running lvs as a cluster.
All machines are connected directly to the same two networks: frontend and
backend.
The real servers/real clients connect to a service ip on the lvs machines on
the frontend network.
The lvs machines run in masq mode and connect to the real servers/real clients
on the backend network.
I've configured policy routing on the real servers/real clients backend
interfaces to return traffic via a second gateway on the lvs hosts.
This works very well except when a real server/real client connects to its own
backend interface via the lvs cluster ip.
I guessing that the local host route means that instead of returning the
traffic via the backend gateway on the lvs it tries to go directly locally.
Tcpdump appears to support this guess and if I turn on martian logging I can
see the traffic.
Initially I thought that reverse path filtering was preventing operation but
the problem remained when it was disabled.
Turning on routing had not beneficial effect either.
Ideally I would like to setup routing to override the local table when the
policy routing rules are applied, but I'm not sure how.
So far attempts to to do this have failed
Has anyone managed to do this?
Its more of a routing question so apologies for being slightly off topic.
Best Regards
Dean Scothern
Dr Dean Scothern
Infrastructure
[Description: Eduserv]
E:
dean.scothern@xxxxxxxxxxxxxx<mailto:dean.scothern@xxxxxxxxxxxxxx><mailto:forename.surname@xxxxxxxxxxxxxx><mailto:forename.surname@xxxxxxxxxxxxxx>
T: +44 (0)1225 474379
F: +44 (0)1225 474301
www.eduserv.org.uk<http://www.eduserv.org.uk><http://www.eduserv.org.uk/><http://www.eduserv.org.uk/>
Eduserv is a company limited by guarantee (registered in England & Wales,
company number: 3763109) and a charity (charity number 1079456), whose
registered office is at Royal Mead, Railway Place, Bath, BA1 1SR.
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org<http://LinuxVirtualServer.org> mailing list -
lvs-users@xxxxxxxxxxxxxxxxxxxxxx<mailto:lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Send requests to
lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx<mailto:lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx>
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx Send
requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx Send
requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
