CentOS 5 GFS Install

I have decided to include the steps I felt I needed to take to make a test install of GFS before I decided not to use it. It doesn't cover clvm configuration but it's very similar to LVM2 which is well documented on the net so I decided not to include it here.

#NTP is needed on every node in the cluster sync'd to the same place of course
yum -y install ntp
#Needed for gfs clustering
yum -y groupinstall Clustering
#GFS kernel module and FS utils
yum -y install gfs-kmod
yum -y install gfs-utils

#Needed to get the stupid system-config-cluster running for configuration of the cluster
yum -y install xorg-x11-xauth xorg-x11-fonts-base xorg-x11-fonts-Type1

#NTP is needed on every node in the cluster sync'd to the same place of course
chkconfig ntpd on
service ntpd start

#Stop updatedb trawling our gfs mounts
echo 'PRUNEFS = "auto afs iso9660 sfs udf"
PRUNEPATHS = "/afs /media /net /sfs /tmp /udev /var/spool/cups /var/spool/squid /var/tmp /cvp /mnt/cvp /media"' > /etc/updatedb.conf

# Create the filesystem with 125 journals(nodes) clustername coull and fs name cvp on /dev/hdb
gfs_mkfs -p lock_dlm -t coull:cvp -j 125 /dev/hdb

#config /etc/cluster/cluster.conf using system-config-cluster by throwing the x connection back to your machine via SSH

#add cluster hosts to /etc/hosts not DNS as this introduces a point of failure and some slowdown.

#Start clustering and GFS services
service cman start
service clvmd start
service gfs start

#Mount SAN device which should be a clv (centralised logical volume)
mount -t gfs /dev/san1/lvol0 /san -o noatime

# Memcache anyone?  Not sure what the options are for yet as I've never set it up before
/usr/bin/memcached -d -m 512 -l 192.168.2.100 -p 11211 -u nobody

GFS - Global File System

Thought I'd add my thoughts on GFS since the documentation on the net currently seems very fragmented and incomplete:

Large files can cause slowdown issues.

Machines really need 2 nics one for client access and one for access to the SAN, cluster communications and communication with the fence device.

The fence device is in most cases a APC power strip so when a node fails the other nodes can reboot that machine as part of the failover process, the other option is to close the switch port of that machine so it cant access the SAN data which IMO is worse than a reboot since recovery needs manual intervention.

When creating the GFS you need to have an idea how many nodes you are going to need as you cannot add more. I have read places that there are limits on the amount of nodes, it used to be very small(16) but now its over 100 for sure might even be 300 odd but each place I look I get a different value. We need to decide on how many we may eventually have I'm thinking just set it to 125 as I doubt we would ever have 125 web servers but then if things go well this statement could come back to haunt me in my sleep ;)

If we need to give access to the data to machines not in the cluster perhaps ones that don't use the data much we can use GNDB on each of the servers in the GFS cluster so they can be connected to as a network block device and data read and written to(albeit in a slower manner than if just gfs was used and creating more traffic on the client side of the network)

I can see no other real alternatives to GFS other than Veritas' expensive offering.

I can see now though after some heavy googling it should be possible to install gfs+clvm on both NAS servers to share the space they have in them and then anything linux can use gfs to talk to these but windows clients would be forced to use samba running on the NAS servers.