|
Hello Alexandre, Stefan,
Sorry for the delay but I took half a day off yesterday and went to the
zoo with some friends to check if tux is still doing ok. ;). After that
I met Romeo Benzoni
In short :
the situation I describe into the PDF file on VRRP (on cheetah) explain the
"synchronization_instance" mechanism. VRRP is a protocol interface specific
and this is usefull for us for LVS because during takeover routing path
must be preserved :
Sorry for not having read your excellent 5 page documentation (how the
heck do you get such a nice PDF? Are the latex sources for the users
guide available?). I'm trying to fully understand it now. I assume you
refer to chapter 4.3 'VRRP instance synchronization' of
LVS-HA-using-VRRPv2.pdf.
In detailed :
In a LVS-NAT env using VRRP (LVS1 & LVS2), realservers default GW is on
eth1 IP of LVS1 and virtual services are exposed to the Internet on eth0
LVS1. All is working => all is inbound/outbound traffic are going threw
LVS1. But, but, but : if LVS1(eth1) fails all associated IP address will be
unavailable and taked-over on the VRRP BACKUP LVS2(eth1) => outbound
traffic will go threw LVS2(eth1) and inbound threw LVS1(eth0) => This
disymetric routing will broke the LVS env.
Yes, I see now, pretty damn hard so.
=> So in our VRRP software we must add a "sync_instance" capability to
preserve code against that (a VRRP extension for our specific needs). This
functionality function using the axiom : If LVS1(eth1) fails then
LVS2(eth1) takeover and LVS2(eth0) become MASTER (for owning IP address).
Now I get it, I hope. This is a 4-bit state diagram with the bits being:
LVS1(eth0), LVS2(eth0), LVS1(eth1) and LVS2(eth1). And FAULT_STATE is a
result of a test, either MII beat failure or IFF_DOWN or routing changes
or fwrules. According to the state transition table which I haven't seen
yet (but I will draw) you know what happens. Thank you Alexandre, my
slow brain starts working now.
=> This is the need : If now we use this VRRP extension and we are not able
to detect link state, we introduce the "noisy loop" => this mean :
In init state : LVS1(eth1) = MASTER, LVS1(eth0) = MASTER
LVS2(eth1) = BACKUP, LVS2(eth0) = BACKUP
LVS1 & LVS2 are using our axiom "sync_instance"
Now, for some reasons, someone unplug the wire on LVS2(eth1). So VRRP
instance will timeout and become MASTER. But axiom will force LVS2(eth0) to
transit to MASTER, so LVS1(eth0) will be BACKUP. But axiom say will force
the symetric on LVS1(eth1) (because all is going nice on LVS1(eth1)) => so
LVS1(eth1) will become BACKUP. Hear is the loop => LVS1(eth1) will timeout
receiving remote VRRP adverts since wire still unplugged ! so transit to
MASTER, then force LVS1(eth0) to MASTER and so LVS2(eth0) to BACKUP, finaly
LVS2(eth1) to BACKUP....... and infinite protocol loop :).... grrr...
For future references:
MASTER = M
BLASTER = B
LVS1(eth0) = L10
LVS1(eth1) = L11
LVS2(eth0) = L20
LVS2(eth1) = L21
LVS1_Heartbeat1 = HB11
LVS1_Heartbeat2 = HB12
LVS2_Heartbeat1 = HB21
LVS2_Heartbeat2 = HB22
FAULT_STATE(x) = FS(x)
Well that's why the HA folks invented the concept of a heartbeat. And
that's what you need to implement too in your framework. I think I
understand your approach now and I have to tell you that the heartbeat
is crucial. Your software needs to be capable of sending adverts through
physically independant heartbeat VRRP instances two each for L1x and
L2x. They do the STOMITH monitoring to avoid such protocol loops you're
mentioning. I would go as far and exchange our HA solution with your
framework, if you can provide FS interaction from HBs to the L1x/L2x
transition, user defined healthchecks and service reload on demand.
With your first approach, FS(x) is a function which has following set of
interactions to derive it's new state:
o MII beat state changes
o routing changes
o IFF_UP & IFF_RUNNING
Normal desired path:
[L1x=M|L2x=B] ---> FS(L1x,link failure) ---> [L1x=B|L2x=M]
Unwanted path:
[L1x=M|L2x=B] ---> FS(L2x,cable cut) ---> [proto loop noise, L2x=M]
With my approach you have interaction of the above mentioned 3 plus the
status information of the HBs which are physically separated. This cuts
out the unwanted path and leaves you with the desired failover/failback
path or state transition.
The only way to break this noisy loop is to introduce a low-level MII
checker for probing physical state of the NIC.
Yes and the other/additional (for me preferred way) is to have HB VRRP
instances with crosscables. The probability of having FS(HB1x) &&
FS(HB2x) && someone disconnects the cable at L20 if we had been in
initial state is so close to zero that you can safely assume it is zero.
If this situation encounters you're in the wrong job or should exchange
the responsible project leader.
Ugh, how is this possible? Do I understand you correctly that you would
like to put in a policy for handling FAULT state that every NIC driver
then must be able to handle?
no no :) my poor english again :) => This is just an extension of the VRRP
code to add a new state into the state machine drived with the NIC link
beat state => VRRP instance in MASTER or BACKUP will transit to FAULT_STATE
if MII reports bad things... And stay sticked until MII is OK.
I'd rather have HB instances as a pool of resource. This is NIC
independant and easy to implement.
Yes we are ok :) Two event condition our VRRP FAULT_STATE :
1. IFF_UP & IFF_RUNNING
2. MII registers values.
Well routing changes or other stupid user related interaction such as
packet filter rules to deny advert messages. Been there, done that.
Especially people with little knowledge about packet filtering and HA
tend to deploy such funny setups and blame the author of the software
afterwards that it's not working.
Well, patching the kernel is not as bad as it sounds, LVS has done it
with success for years now. Noone is complaining, except maybe Joe and
me ;). That's why I would like to see Stefan's patch clean and
completely independant.
Yes, but can handle userspace in our VRRP framework until it is added into
the stable kernel banches to keep compatibility.
What do you think about the HB approach?
Yes, have tested Garzik ethtool and it is not working properly on most of
the MII enabled NICs :/
He's working hard on getting them all to do what they are capable of. I
remember a recent discussion between him and DaveM about the PITA to
support all kinds of different NIC driver implementations regarding the
ioctl support state.
So I am starting with the donald becker code wich is generic and working.
Ok.
But, the use of a MII enabled NIC can be a requierment for runing VRRP. If
people is warned it can be acceptable.
It might.
Well, you've got Easter time now. Send your wife to some nice holiday
trip and start coding.
:) I really do not know if she will agree :))) : "Wait darling this is for
MII register....blabla :))))"
Oh, I thought she'd understand that :). Well I hope you have managed to
work around that problem. My girlfriend went to Montreux with her friend
to have some fun and explore Switzerland which gives me 2 free days to
finally do all the boring non-IT related tasks (taxes, government and
military stuff).
Give me his phone number and I talk to him.
:)
Seriously, I can also write him a letter. OLS is about 'what you see and
hear is what you will get in the next 2 years in the Linux corner'. And
if he doesn't agree tell him I'll never watch Canal+ anymore (not that
I'd ever watch TV, but you know ...)
Yes :) not really easy for me. And if jerome etienne has not reply to my
email I assume he will drop me... And really don't like that kind of
situation...
I don't think this would happen. He's a serious programmer and an
engineer. Engineers almost never work against each other but together to
achive a goal. The 1% of them which don't get that show no impact on the
inventions of the rest of the 99% related to direct productivity gain.
I seriously hope Stefan doesn't mind we're cc'ing him. I apologize for
any inconvenience if so. We just wanted to know his opinion on the link
beat status information change and my comments to his patch.
Happy Easter time,
Roberto Nibali, ratz
|
