|This document is available in: English Castellano Deutsch Francais Nederlands Turkce|
by Atif Ghaffar
About the author:
Atif is a chameleon. He changes his roles, from System
Administrator, to programmer, to teacher, to project manager, to
whatever is required to get the job done.
When designing a mission critical systems, either during flowcharting or when building it physically with boxes, cables etc, one has to ask the following questions :
Now for some serious discussion.
Even though I trust Linux blindly, I don't trust the companies that make the machines, power supply, network cards, motherboards etc, and I am always afraid that if one of these fail, my system will be unusable. Hence the service will be unavailable, further more I will be taking down all the company services even though they are not directly related to me. For example
High Availability is what it says it is.
Something that is Highly Available.
Some service that is really important to keep your company functional.
In this example we will theoretically create an Active/Passive
cluster running an apache server, serving the intranet.
To create this small cluster, we will use one good machine with lots of RAM, and many CPUs and another one with just enough RAM/CPU to run the service.
The fist machine will be the master node while the second will be backup node.
The job of the backup node is to take over the services from the master node if it thinks that the master node is not responding.
Lets just think, how our users access the intranet.
They type http://intranet/ in their browser and the DNS server redirects them to 10.0.0.100 (example ip)
How about if we put two servers running this intranet service which different ip address, and just ask the DNS server to redirect to the second one if the master node comes down.
Sure, thats one possibility, but there are issues about DNS caching on the clients etc and perhaps you want to run the DNS server on a HA cluster itself.
Another possibility, if master node fails, then the slave node may take over its ip address and start serving the requests.
This method is called IP takeover, and is the method that we will be using in our examples. Now all browsers will still be accessing http://intranet/ which will translate to 10.0.0.100 even if the master node fails without making any changes to the DNS.
How would the master/slave know that the other node in the
cluster has failed?
They will talk to each other over a serial cable and over a cross link Ethernet cable (for redundancy, serial cable or Ethernet cable may fail) and check each others heartbeat (yes like the heartbeat you have) If your heartbeat stops, then you are probably dead
The program to monitor the heartbeats of the cluster nodes is called... guess...heartbeat.
heartbeat is available at http://www.linux-ha.org/download/
The program for ip address take over is called fake and is integrated in heartbeat.
If you do not have an extra network card to put in two machines you may run heartbeat over a serial cable (null modem) only.
On the other hand network cards are cheap, so add another one for redundancy.
As previously mentioned, we will use one cool machine and
another not so cool machine.
Both machines will be equipped with 2 network cards each and at least one serial port.
We will need one cross link cat 5 RJ45 (Ethernet) cable and a null modem (cross link serial cable)
We will use the first network card on both machines for their Internet ip addresses (eth0)
We will use the second network card on both machine for a private network to talk udp heartbeat (eth1)
We will give both machines their Internet ip addresses and names.
For example to eth0 of both nodes
clustnode1 with ip address 10.0.0.1
clustnode2 with ip address 10.0.0.2
Now we will reserve a floating ip address (this is the service ip address that I highlighted earlier)
10.0.0.100 (intranet). We don't need to assign it to any machine at the moment
Next we configure the machines for their second network card and give them any ip addresses from a range that is not used.
for example to eth1 of both nodes an ip address with netmask 255.255.255.0
clustnode1 ip address 192.168.1.1
clustnode2 ip address 192.168.1.2
Next we connect the serial cables to Serial port 1 or 2 of the machines and make sure that they are working/talking with each other.
(Make sure that you connect to the same port of each machine, its easier that way)
Installing the software is straight forward, heart beat is
available in rpm and tar.gz both binary and source packages.
If you have problem installing the software, then you probably should not be taking the responsibility to install a HA system (it won't be HA, perhaps it will be NA)
There is an excellent Getting Started with Linux-HA guide so I wont replicate the information here.
configure the hearbeat
example if heartbeat configuration files are in /etc/ha.d
edit file /etc/ha.d/authkeys with your favourite editor
#/etc/ha.d/authkeys auth 1 1 crc #end /etc/ha.d/authkeys
debugfile /var/log/ha-debug logfile /var/log/ha-log logfacility local0 deadtime 10 serial /dev/ttyS3 #change this to appropriate port and remove this comment udp eth1 #remove this line if you are not using a second network card. node clustnode1 node clustnode2
#masternode ip-address service-name clustnode1 10.0.0.100 httpdthis defines that the masternode is clustnode1, for example when the clustnode1 goes down then clustnode2 will take over the service, but when clustnode1 comes backup again, it will reclaim its service. That is why we are using a good and not so good machine (clustnode1 is the good machine)
/etc/ha.d/httpd startif it does not find the file then it will try
/etc/ha.d/httpd stopif it does not find the file then it will try
/etc/rc.d/init.d/httpd stopWhen you are finished with the configuration on clustnode1, you can copy the files to node2.
When service httpd moves from node1 to node2 it does not see the
same data. I loose all the files that I was creating with my httpd
1. You should never write to file from your CGI's. (use a network database instead.. MySQL is pretty good)
2. You can attach the two nodes to a central external SCSI storage, and make sure that only one is talking to it at one time, and also make sure that you change the SCSI id of the host card on machine a to 6 and leave on machine b 7 or vice -versa.
I have tried this with Adaptec 2940 SCSI cards, and they let me change the SCSI id. Most cheap cards will not let you do that.
Some Raid controllers are sold as cluster-aware controllers but make sure that the vendor will allow you to change the HOST ID of the card without buying Microsoft cluster kit.
I had to NetRaid adapters from HP and they definitely do not support Linux. I had to break them to have a good feeling about the money spent.
Next step will be to buy Fibrechannel cards, a fibrechannel hub and a Fibrecahnnel storage to create a small SAN, they are definitely more costly than using shared SCSI but they are a good investment.
You can run GFS (Global File System, see below in resources) over FC which allows you to have transparent access to the storage from all machines as if they were local storage.
We are using GFS in production environment over 8 machines where 2 of them are in a similar HA configuration as I have described above.
You can easily build an Active/Active server if you have a good
storage system that allows concurrent access. Examples are Fibrechannel
If you are content with Network filesystems such as NFS, you may use that, but I would not suggest that.
Anyway, you can map serviceA to clustnode1 and serviceB to clustnode2 example of my haresource file
clustnode2 172.23.2.13 mysql clustnode1 172.23.2.14 ldap clustnode2 172.23.2.15 cyrusI use GFS for storage so I don't have a problem with concurrent access to data and can run as many services as is manageable by these machines.
Webpages maintained by the LinuxFocus Editor team
© Atif Ghaffar, FDL
Click here to report a fault or send a comment to LinuxFocus
2002-05-22, generated by lfparser version 2.28