Red Hat 6.x GFS2 Cluster Build Guide

— ny_wk

Building a Red Hat 6.x GFS2 cluster means joining two RHEL 6 virtual machines into a high-availability cluster that share the same SAN-backed filesystem at the same time, using CMAN, rgmanager, CLVM and the GFS2 clustered filesystem, with a quorum disk and VMware SOAP fencing to keep data safe. This guide walks through the full two-node build on VMware end to end, then shows the modern Pacemaker/pcs equivalent you should use on RHEL 7 and later.

What a Red Hat 6.x GFS2 cluster actually does

A standard filesystem such as ext4 or XFS assumes exactly one server owns the disk. If two machines mount the same block device read-write, they corrupt it within seconds because neither knows what the other is writing. GFS2 (Global File System 2) solves this with a Distributed Lock Manager (DLM): every node coordinates locks over the network so all nodes can read and write the same shared LUN simultaneously without corruption.

On RHEL 6 the cluster stack that drives this is the older CMAN + rgmanager + Conga suite, not Pacemaker. The pieces you will configure are:

• cman — the cluster manager and membership/quorum layer.
• rgmanager — the resource group (service) manager for failover.
• clvmd — clustered LVM, so volume-group metadata stays consistent across nodes.
• ricci — the per-node agent that applies configuration pushed from the web UI.
• luci — the Conga web interface used to build and manage the cluster.
• A quorum disk (qdiskd) — a shared 10–100 MB LUN that acts as a tie-breaker vote, essential in a two-node cluster.
• Fencing (STONITH) — here, fence_vmware_soap, which reboots a misbehaving guest through vCenter so a frozen node cannot corrupt the shared filesystem.

Important: RHEL 6 is end of life — note the modern equivalent

Red Hat Enterprise Linux 6 reached end of Extended Life-cycle Support on 30 June 2024. The CMAN/rgmanager/Conga (luci/ricci) stack described here was removed in RHEL 7, which replaced it with the Pacemaker + Corosync high-availability stack managed by the pcs command. Build a new Red Hat GFS2 cluster on a supported release. This article documents the RHEL 6 procedure faithfully (you may still meet it on legacy systems and in interviews), and the final section maps every legacy step to its modern pcs counterpart.

Prerequisites and lab topology

The reference build uses two RHEL 6.5 guests on VMware, each subscribed (via Red Hat Satellite or RHSM) to the RHEL Server High Availability and RHEL Server Resilient Storage channels — both are required, or yum cannot find the cluster packages.

Each node has two NICs: eth0 for public/data traffic and eth1 for a dedicated cluster heartbeat. Keeping heartbeat on its own network prevents a busy public link from triggering false fencing.

Step 1 — Resolve names in /etc/hosts

Cluster traffic must work even if DNS is down, so add every node data IP, heartbeat IP and any service VIPs to /etc/hosts on both nodes. A clean entry set looks like this:

1. Add the public/data addresses with FQDN and short name.
2. Add the private heartbeat addresses (for example 10.10.10.11 node01-hb and 10.10.10.12 node02-hb).
3. If you run failover service IPs, add those VIPs too. Note: with GFS2 actively mounted on both nodes you often do not need a floating VIP, because the data is already live everywhere — confirm with the application owner.

Installing the cluster packages on a Red Hat 6.x GFS2 cluster

On all nodes, install the three package groups that pull in cman, rgmanager, luci, ricci, clvmd and the GFS2 utilities:

1. Install the groups:
   • yum groupinstall "High Availability"
   • yum groupinstall "High Availability Management"
   • yum groupinstall "Resilient Storage"
2. Accept the transaction summary. If yum reports packages are unavailable, the node is missing the High Availability or Resilient Storage channel — fix the subscription before continuing.

Step 2 — Disable SELinux and unsupported services

The RHEL 6 cluster stack does not run reliably with SELinux enforcing or with the local firewall and ACPI daemon active, because ACPI can intercept the power events fencing relies on. On all nodes:

1. Confirm SELinux state with getenforce. If it is not Disabled, set SELINUX=disabled in /etc/selinux/config and reboot.
2. Disable the firewall and ACPI across all runlevels:
   • chkconfig --level 0123456 iptables off
   • chkconfig --level 0123456 ip6tables off
   • chkconfig --level 0123456 acpid off

Modern note: rather than disabling the firewall outright, the supported approach is to open the cluster ports. The classic ports are 5404–5405/UDP (corosync), 11111/TCP (ricci), 21064/TCP (dlm), 16851/TCP (modclusterd) and 8084/TCP (luci).

Step 3 — Set cluster services to start at runlevel 3 and 5

Cluster daemons should run in multi-user mode. On all nodes enable them for runlevels 3 and 5 and turn them off everywhere else:

1. chkconfig --level 35 cman on
2. chkconfig --level 35 clvmd on
3. chkconfig --level 35 rgmanager on
4. chkconfig --level 35 ricci on
5. chkconfig --level 35 luci on (only on the node hosting the web UI)

Reboot both nodes with shutdown -r now, then confirm with service cman status, service ricci status and so on. Note that cman and clvmd will not fully start until the cluster is actually built, and service gfs2 status reporting no entries found in /etc/fstab is normal at this stage.

Creating the cluster with the LUCI web interface

The Conga luci web UI generates /etc/cluster/cluster.conf and pushes it to every node through ricci. First, set the ricci password on both nodes — luci authenticates to ricci with it:

1. On each node run passwd ricci and set the same password.
2. Browse to the luci UI at https://node01:8084 and log in with the node root credentials.
3. Go to Manage Clusters → Create and enter the cluster name (apm-clus1), the first node name and ricci hostname, and the ricci password.
4. Tick Use Locally Installed Packages (packages are already installed) and Enable Shared Storage Support (this turns on clustered LVM and GFS2 support).
5. Click Create Cluster. luci will reboot and join the node into the new cluster.
6. Open Nodes → Add and add the second node the same way (name, ricci hostname, password, both checkboxes).

Verify membership from the command line on either node with clustat. A healthy two-node cluster shows both members Online and Member Status: Quorate:

• node01   1   Online, Local
• node02   2   Online

Reboot both nodes once more so they rejoin cleanly, then move on to storage.

Configuring shared SAN storage and CLVM

Run the discovery and verification steps on all nodes, but create partitions, volume groups and filesystems on one node only — clustered LVM replicates the metadata to the other node automatically.

Step 4 — Identify the shared LUNs

Identify each shared device and its size so you pick the right disks for data versus quorum. With multipath, use the /dev/mapper/mpathX names rather than raw /dev/sdX:

1. fdisk -l /dev/mapper/mpathb — the 100 GB data LUN.
2. fdisk -l /dev/mapper/mpathc — the ~30 MB quorum LUN.

Create the mount points on both nodes, for example mkdir -p /opt/app/data.

Step 5 — Enable clustered locking in LVM

Clustered LVM must use lock type 3 (DLM) so volume-group changes are coordinated. On all nodes:

1. lvmconf --enable-cluster
2. Verify with grep locking_type /etc/lvm/lvm.conf — it must report locking_type = 3.

Step 6 — Partition, then build the clustered volume group (one node only)

Partition the data LUN as a Linux LVM (type 8e) partition, then build the clustered VG:

1. Run fdisk /dev/mapper/mpathb and create one primary partition spanning the whole disk: n → p → 1 → accept defaults → t → 8e → w. The Re-reading the partition table failed warning is harmless; run partprobe or kpartx -a /dev/mapper/mpathb to refresh the kernel view.
2. Initialise the physical volume: pvcreate /dev/mapper/mpathbp1.
3. Create the clustered volume group (the -c y flag marks it clustered): vgcreate -c y appvg /dev/mapper/mpathbp1.

Step 7 — Create the logical volumes (one node only)

Carve the volume group into logical volumes sized for your application:

1. lvcreate -L 130G -n applv appvg
2. lvcreate -L 17G -n traceslv appvg
3. lvcreate -L 33G -n datalv appvg

If lvcreate refuses because the VG is not active on that node, activate it first with vgchange -ay appvg and retry. Confirm everything with pvs, vgs and lvs.

Formatting the GFS2 filesystems

This is the heart of a Red Hat 6.x GFS2 cluster. Format each logical volume with mkfs.gfs2 (one node only). Two flags matter most:

• -t cluster:fsname — the lock table name. The part before the colon must exactly match your cluster name (apm-clus1), or no node will be allowed to mount it.
• -j N — the number of journals, one per node that will mount the filesystem. For two nodes use -j 2; add more if you grow the cluster (you can add journals later with gfs2_jadd).

1. mkfs.gfs2 -t apm-clus1:applv -j 2 -p lock_dlm /dev/appvg/applv
2. mkfs.gfs2 -t apm-clus1:traceslv -j 2 -p lock_dlm /dev/appvg/traceslv
3. mkfs.gfs2 -t apm-clus1:datalv -j 2 -p lock_dlm /dev/appvg/datalv

Answer y to the destroy-data prompt. -p lock_dlm selects the distributed lock manager (the correct protocol for a real cluster; lock_nolock is single-node only).

Step 8 — Add the GFS2 mounts to /etc/fstab (all nodes)

Add the filesystems to /etc/fstab on every node so the gfs2 init script mounts them at boot. Use noatime,nodiratime to cut needless lock traffic:

• /dev/mapper/appvg-applv /opt/app/EM gfs2 defaults,noatime,nodiratime 0 0
• /dev/mapper/appvg-traceslv /opt/app/EM/traces gfs2 defaults,noatime,nodiratime 0 0
• /dev/mapper/appvg-datalv /opt/app/data gfs2 defaults,noatime,nodiratime 0 0

Watch mount ordering: if traces sits inside EM, the parent must mount first. Mount the parent, run mkdir -p /opt/app/EM/traces, then mount the child. Mount everything with mount -a and confirm with mount on both nodes.

Adding the quorum disk and fencing

In a two-node cluster, if the heartbeat breaks each node thinks the other is dead and both try to take over — a split brain. A quorum disk plus fencing prevents this.

Step 9 — Create and register the quorum disk

1. On one node, label the quorum LUN: mkqdisk -c /dev/mapper/mpathc -l apm-quorum.
2. On both nodes, confirm they can read it: mkqdisk -L.
3. In luci open Configure → QDisk, choose Use a Quorum Disk, set By Device Label to apm-quorum, and click Apply.

clustat should now list the quorum device as Online, Quorum Disk alongside the two members.

Step 10 — Configure VMware SOAP fencing

Fencing is the single most important part of a GFS2 cluster: without it, a hung node can still hold stale locks and corrupt data. Since RHEL 5.7, Red Hat fully supports fencing VMware guests with the fence_vmware_soap agent, which tells vCenter to power-cycle a node.

1. Ask your VMware admin to create a service account in vCenter with permission to power off and reboot the cluster VMs, and grant it a fencing role on each node VM.
2. Collect each VM UUID from vCenter — the fence agent identifies the target by UUID, not by name. You can also test from a node with fence_vmware_soap -a vcenter_host -l user -p pass -z -o list.
3. In luci open Fence Devices → Add, choose VMware Fencing (SOAP Interface), and enter a name, the vCenter IP/hostname, the service-account login and password.
4. Under Nodes, select each node, click Add Fence Method, then Add Fence Instance, binding it to the fence device and that node UUID. Repeat for the second node.

Reboot both nodes from luci so the full configuration is loaded. Optionally add the cluster to luci on the second node (Manage Clusters → Add) so you have a management UI on either box.

Verification: testing cluster failover

Never trust a cluster you have not tried to break. Test one node at a time, and before each test confirm both members are Online and the cluster is Quorate with clustat. In a second window on the surviving node, watch events live with tail -f /var/log/messages.

1. fence_node test: from node01 run fence_node node02. node02 should be fenced (rebooted) and you should see fence node02 success. Confirm the GFS2 volumes stay mounted and writable on node01.
2. Power Off test: power off a node from the vSphere console. Fencing should automatically power it back on; the survivor keeps serving GFS2.
3. Shut Down Guest test: issue a graceful guest shutdown from vSphere and confirm the survivor is unaffected.
4. Reboot test: run reboot on one node; the cluster should ride through the orderly shutdown and rejoin.
5. Network Down test: disconnect the public NIC, then separately the heartbeat NIC, in vSphere. The isolated node should be fenced and the survivor should keep all GFS2 filesystems available.

After each test, wait until both nodes are Online again before running the next, then repeat the whole sequence with the roles reversed.

Common pitfalls when building a Red Hat 6.x GFS2 cluster

• Lock-table name mismatch: the cluster name in mkfs.gfs2 -t name:fs must match cluster.conf exactly, or mounts silently fail.
• Wrong journal count: -j 2 only supports two simultaneous mounters. Add a third node without adding a journal and it cannot mount.
• Forgetting clustered locking: if locking_type is not 3 on every node, LVM metadata diverges and the second node never sees the volumes.
• No fencing configured: the most common cause of GFS2 corruption. A cluster without working fencing is not safe for shared data.
• ACPI left enabled: ACPI can soak up power signals so fencing takes minutes instead of seconds, stalling failover.
• Quorum disk on the wrong LUN: mkqdisk destroys the target — double-check the device path before labelling.
• Heartbeat over the public network: a traffic spike can trigger false fencing; always give heartbeat its own NIC or VLAN.

The modern equivalent: Pacemaker and pcs on RHEL 7+

On RHEL 7, 8 and 9 the legacy stack is gone. Build the equivalent Red Hat GFS2 cluster with Pacemaker/Corosync, driven by pcs:

A condensed modern build looks like: install pcs pacemaker fence-agents-all gfs2-utils lvm2-lockd, set a password for the hacluster user, then pcs host auth node01 node02 and pcs cluster setup mycluster node01 node02. Add STONITH with pcs stonith create, enable the locking and DLM resources, build the volumes on a shared LVM volume group, and format with the same mkfs.gfs2 -t mycluster:fs -j 2 command — GFS2 itself works the same way; only the cluster control plane changed.

Key Takeaways

• GFS2 lets multiple RHEL nodes mount the same SAN LUN read-write at once by coordinating locks through the DLM.
• On RHEL 6 the control plane is CMAN + rgmanager + Conga (luci/ricci); on RHEL 7+ it is Pacemaker + Corosync driven by pcs.
• A two-node cluster needs a quorum disk as a tie-breaker, and working fencing (here fence_vmware_soap) to prevent split-brain data corruption.
• Match the cluster name in mkfs.gfs2 -t name:fs exactly, and provision one journal (-j) per mounting node.
• RHEL 6 is end of life as of 30 June 2024 — use this only for legacy systems or interview prep, and build new clusters on a supported release.

Frequently Asked Questions

What is the difference between GFS2 and a regular filesystem?

A regular filesystem such as ext4 or XFS assumes a single host owns the disk, so two servers mounting it read-write will corrupt it. GFS2 is a shared-disk cluster filesystem that uses a Distributed Lock Manager so every node can read and write the same LUN at the same time safely.

Do I still need a quorum disk in a two-node cluster?

In a strict two-node cluster yes, because losing one node leaves only one vote and risks split brain. The quorum disk (or, on RHEL 7+, a qdevice) provides an extra tie-breaker vote. With three or more nodes the quorum disk can be omitted.

Why is fencing so important for GFS2?

If a node hangs but is not truly down, it can still hold filesystem locks and write stale data, corrupting the shared volume. Fencing forcibly powers off or reboots the suspect node so the survivors can safely take over. A GFS2 cluster without working fencing is not supported for production data.

Can I build this exact cluster on RHEL 8 or 9?

Not with luci/ricci and cman — those were removed after RHEL 6. On RHEL 7, 8 and 9 you build the same shared-storage GFS2 cluster with Pacemaker, Corosync and the pcs command, plus the LVM lock daemon. The GFS2 format and mount steps are nearly identical; only the cluster management tooling differs.

Found this build guide useful? Subscribe to @explorenystream on YouTube for more Linux, DevOps and system administration walkthroughs.