Red Hat 6.x NFS Cluster Build Guide

— ny_wk

Building a Red Hat 6 two-node NFS cluster means stacking shared SAN storage, GFS2, the CMAN/rgmanager high-availability suite, a quorum disk and reliable fencing into one resilient, active/passive NFS service. This guide walks the full build end to end, fixes the rough edges in older runbooks, and shows the verification tests that prove failover actually works before you hand the cluster to production.

Important context first: Red Hat Enterprise Linux 6 reached end of maintenance in November 2020 and end of Extended Life-cycle Support in 2024. The CMAN + rgmanager + Conga (luci/ricci) stack described here is the correct, real way to build an HA NFS cluster on RHEL 6, but it is legacy. On RHEL 7, 8, and 9 the same outcome is delivered by Pacemaker + Corosync driven through the pcs command. Use this guide to operate or migrate an existing RHEL 6 cluster; for anything new, build on RHEL 8/9 with Pacemaker (a mapping is included near the end).

What a Red Hat 6 NFS cluster actually is

The goal is a highly available NFS export: clients mount a single virtual IP (VIP), and if the node currently serving NFS dies, the service relocates to the surviving node with the same IP, the same filesystems, and minimal disruption. In this design the cluster is two physical nodes (HP BL465c G7 blades in the reference build) sharing EMC SAN LUNs over dual Fibre Channel HBAs.

The moving parts:

• CMAN — the cluster manager (membership and messaging).
• rgmanager — the resource group manager that starts, stops and relocates the NFS service.
• clvmd — clustered LVM so both nodes see consistent volume-group metadata.
• GFS2 — a cluster-aware filesystem that more than one node can safely mount.
• Conga (luci + ricci) — the web UI (luci) and per-node agent (ricci) used to configure the cluster.
• A quorum disk (qdiskd) — a tie-breaker vote that is essential in a two-node cluster.
• Fencing — the ability to forcibly power-cycle a misbehaving node (via HP iLO / IPMI) so it can never corrupt shared storage.

Assumptions and prerequisites

Before you touch a single command, confirm the environment is ready. The build assumes:

• The base OS (RHEL 6.2/6.3) is installed, patched to current, and the data and management NICs are configured.
• Both servers are registered to your Satellite/subscription and subscribed to the RHEL Server High Availability and RHEL Server Resilient Storage channels — without these, yum cannot install the cluster packages.
• Shared SAN storage is presented to both nodes; the LUNs are masked to the HBAs on both servers; the array is EMC with PowerPath correctly installed.
• Exactly two nodes — so a quorum disk is required as the third vote. (For three or more nodes you can omit qdisk.)
• A dedicated fencing account exists on each node's iLO.
• All commands run as root; root login is enabled during the build (you can disable it afterwards).

The reference IP plan keeps networks separated: a public/data network, a private heartbeat network, the iLO management network, and a VIP that floats with the NFS service. In this build only one VIP, 10.192.45.85 (the NFS service address), matters; the others belong to separate application services on the same hardware.

Step 1: Install the clustering and GFS2 packages

The cleanest way to pull the whole stack and its dependencies is to install the package groups rather than chasing individual RPMs. On all nodes:

1. Install the High Availability and Resilient Storage groups plus the IPMI tools used for iLO fencing:
   1. yum groupinstall "High Availability"
   2. yum groupinstall "High Availability Management"
   3. yum groupinstall "Resilient Storage"
   4. yum install OpenIPMI OpenIPMI-tools
2. Review the transaction summary and accept it. If yum reports that packages such as cman, rgmanager, gfs2-utils, lvm2-cluster, luci or ricci are unavailable, the node is not subscribed to the HA / Resilient Storage channels — fix the subscription first.

Key packages you should end up with: ccs, cman, clusterlib, rgmanager, lvm2-cluster, gfs2-utils, luci, ricci, and OpenIPMI-tools for IPMI/iLO fencing.

Step 2: Prepare the OS for clustering

A few OS-level settings will make or break the cluster. Do these on every node.

Disable SELinux (this RHEL 6 cluster stack is not supported with SELinux enforcing):

1. Check current state: getenforce (you want Disabled or at least Permissive).
2. Edit /etc/selinux/config and set SELINUX=disabled, then reboot to apply.

Set services to start at the right runlevels. Cluster daemons should run in multi-user-with-networking (runlevels 3 and 5) and be off everywhere else:

1. for s in cman clvmd rgmanager luci ricci gfs2 qdiskd; do chkconfig --level 01246 $s off; done
2. for s in cman clvmd rgmanager luci ricci gfs2 qdiskd; do chkconfig --level 35 $s on; done

Disable services that conflict with the cluster. The local firewall and ACPI power management interfere with fencing and heartbeat. Either turn them off, or — the better practice — open the required cluster ports. The simplest path used in many builds:

1. chkconfig --level 0123456 iptables off
2. chkconfig --level 0123456 ip6tables off
3. chkconfig acpid off

Why disable acpid? With ACPI soft-power-off enabled, a fence "power off" can trigger a slow, graceful shutdown instead of an instant cut, which delays failover. Disabling acpid makes the power button (and the fence agent) cut power immediately.

Security note: rather than disabling iptables entirely, production clusters should instead allow the cluster traffic — CMAN/totem on udp 5404-5405, ricci on tcp 11111, luci on tcp 8084, rgmanager on tcp 21064, qdiskd, and DLM. Disabling the firewall is the quick lab approach; an allow-list is the correct one.

Reboot both nodes (shutdown -r now) and confirm: cluster services should be enabled (some, like cman, won't fully run until the cluster is defined), and iptables, ip6tables and acpid should be stopped.

Step 3: Define the cluster with Conga (luci)

You configure the cluster through luci, the web UI. Everything it does is written to /etc/cluster/cluster.conf on both nodes and propagated by ricci.

1. On both nodes, set the ricci account password (luci authenticates to ricci with it): passwd ricci. Record this password.
2. Make sure ricci and luci are running: service ricci start and service luci start.
3. Browse to luci on the first node: https://10.192.45.80:8084 and log in with root credentials.
4. Click Manage Clusters → Create. Enter the cluster name cipglu1-clus1, then add the first node by its heartbeat name (cipglu1mqs01-hb) with the ricci password.
5. Add the second node the same way (cipglu1mqs02-hb). Let luci install/enable the cluster packages and create the cluster.

Use the heartbeat hostnames as the cluster node names. This pins cluster membership traffic to the private heartbeat network and keeps it off the public NIC. After the cluster is created, reboot both nodes once so they come up cleanly as members.

Step 4: Configure shared SAN storage

Now build the storage that the NFS service will export. With EMC PowerPath, each LUN appears as a single pseudo device /dev/emcpowerX that aggregates all four FC paths.

Identify the LUNs (run on every node so you know the device names on each):

1. powermt display dev=all — confirm every path shows state=alive and look for the four Pseudo name=emcpowerX devices.
2. Match sizes to roles with fdisk -l /dev/emcpowerX. In the reference build: a 20 MB LUN becomes the quorum disk, and three ~25 GB LUNs become the GFS2 data volumes (cipg, ewp, wrap).

Create the mount points on every node (these are just directories; they must exist on both):

1. mkdir -p /mqdata/cipg /mqdata/ewp /mqdata/wrap

Enable cluster locking in LVM on every node so volume-group metadata changes are coordinated:

1. lvmconf --enable-cluster
2. Verify: grep locking_type /etc/lvm/lvm.conf | grep -v '#' should return locking_type = 3.
3. Restart clustered LVM: service clvmd restart.

Partition, then build the volumes — on ONE node only. Clustered LVM replicates the metadata to the other node automatically; doing it twice corrupts it.

1. Create a single Linux LVM (type 8e) partition on each data LUN with fdisk /dev/emcpowerb (n, p, 1, accept defaults, t, 8e, w). Repeat for emcpowerc and emcpowerd.
2. Initialize the physical volumes: pvcreate /dev/emcpowerb1 /dev/emcpowerc1 /dev/emcpowerd1.
3. Create clustered volume groups (note the -c y flag): vgcreate -c y cipgvg /dev/emcpowerb1, then ewpvg and wrapvg the same way.
4. Create a logical volume that fills each VG. Read the extent count first — vgdisplay cipgvg | grep 'Total PE' — then: lvcreate -l 100%FREE -n cipgvglv cipgvg (using -l 100%FREE is cleaner and less error-prone than copying a raw PE number). Repeat for ewpvglv and wrapvglv.

Verify the layout with pvs, vgs and lvs. The clustered VGs should show the c attribute (e.g. wz--nc), confirming cluster awareness.

Make the GFS2 filesystems

GFS2 is created on one node only. The lock table name must be clustername:fsname, and the journal count (-j) must be at least the number of nodes that will mount it — 2 for a two-node cluster.

1. mkfs.gfs2 -t cipglu1-clus1:cipgvglv -j 2 -p lock_dlm /dev/cipgvg/cipgvglv
2. mkfs.gfs2 -t cipglu1-clus1:ewpvglv -j 2 -p lock_dlm /dev/ewpvg/ewpvglv
3. mkfs.gfs2 -t cipglu1-clus1:wrapvglv -j 2 -p lock_dlm /dev/wrapvg/wrapvglv

Do not add these GFS2 volumes to /etc/fstab for cluster-managed mounts. rgmanager mounts and unmounts them as part of the NFS service group so it can move them between nodes — an fstab entry would fight the cluster.

Create the quorum disk

The quorum disk gives a two-node cluster its critical third vote, preventing a split-brain where both nodes think they are the survivor. Create it on the dedicated small LUN, on one node only:

1. mkqdisk -c /dev/emcpowera -l CIPGmqu1-quorum
2. Verify from both nodes: mkqdisk -L — both should see the same label and creation timestamp.

On the second node, if it doesn't immediately see the new PVs/VGs/LVs, rescan with pvscan, vgscan and lvscan (or reboot). Confirm both nodes show identical pvs/vgs/lvs output before continuing.

Step 5: Add fencing — the part you must not skip

Fencing is what makes a cluster safe. When a node stops responding, the cluster power-cycles it via iLO/IPMI so it cannot keep writing to shared GFS2 storage and corrupt data. A cluster without working fencing is a data-loss incident waiting to happen.

1. On each node's iLO web console, create a dedicated fence user (fence1 on node 1's iLO, fence2 on node 2's iLO). Grant it all privileges it needs — at minimum Virtual Power and Reset, plus Remote Console, Virtual Media and Configure iLO Settings.
2. In luci, go to Fence Devices → Add, choose IPMI LAN, and create one device per node:
   1. IPMI_fence1 → IP 10.194.2.128, login fence1, its password.
   2. IPMI_fence2 → IP 10.194.4.252, login fence2, its password.
3. Under Nodes, open each node, click Add Fence Method (e.g. name it IPMI), then add the matching fence instance: IPMI_fence1 to node 1, IPMI_fence2 to node 2.

Tip: for iLO3/iLO4 you can also use the dedicated fence_ilo-family agents, but IPMI LAN is the most portable. Set lanplus and the correct privilege level if the device test fails.

Step 6: Add the quorum disk to the cluster config

In luci, open Configure → QDisk, select Use a Quorum Disk, and reference it by label: CIPGmqu1-quorum. Referencing by label rather than by device path is deliberate — device names like /dev/emcpowera can differ between nodes, but the label is stable everywhere. Apply, and qdiskd will start contributing its vote.

Step 7: Build the NFS service group

This is where the NFS export becomes highly available. First, make sure NFS itself is current — older nfs-utils had real NFSv4 bugs.

1. Check the version on both nodes: rpm -qa | grep -i nfs-utils. Update to a current nfs-utils-1.2.3-x.el6 (or later) via yum update nfs-utils so both nodes match.

Create the resources in luci (Resources → Add). Build them once, then nest them into the service group:

• IP Address — the VIP 10.192.45.85 with the correct netmask. Express it in CIDR (the network here is a /23, i.e. 255.255.254.0 = 23 mask bits). Be precise here: a /23 is 23 mask bits, not 24.
• GFS2 — one per filesystem: name CIPG → mount /mqdata/cipg → device /dev/cipgvg/cipgvglv; name EWP → /mqdata/ewp → /dev/ewpvg/ewpvglv; name WRAP → /mqdata/wrap → /dev/wrapvg/wrapvglv, options rw. (Give each its own unique name and its own device path — a common copy-paste error is naming two resources "EWP" or pointing WRAP at the EWP volume.)
• NFS Export — one per share: CIPG-export, EWP-export, WRAP-export.
• NFS Client — one per share defining who may mount and how: CIPG-NFS-clients, EWP-NFS-clients, WRAP-NFS-clients, each with target 10.192.45.0/23 and options rw,no_root_squash. (no_root_squash lets remote root keep root privileges — convenient but a security trade-off; prefer root_squash unless a specific app requires otherwise.)

Create a failover domain so the service prefers a primary node and behaves predictably (Failover Domains → Add):

• Name DM-nfs; Prioritized, Restricted, and No Failback all checked.
• Members: cipglu1mqs01-hb priority 1, cipglu1mqs02-hb priority 2.

No Failback is important: it prevents the service from automatically bouncing back to node 1 the moment it recovers, which would cause a second, unnecessary outage.

Assemble the service group (Service Groups → Add):

1. Name it U1-NFSsvc, set Failover Domain to DM-nfs, and Recovery Policy to Relocate. Tick Automatically start this service.
2. Add the IP resource first.
3. For each filesystem, add the GFS2 resource, then add the matching NFS Export as a child of that GFS2 resource, then add the matching NFS Client as a child of the export. Resource nesting is what controls start order: filesystem mounts, then export is published, then clients are allowed.
4. Repeat the GFS2 → Export → Client nesting for EWP and WRAP.

Submit, then start the service. Add the existing cluster to luci on the second node too (https://10.192.45.81:8084 → Manage Clusters → Add) so you can administer it from either box.

Step 8: Verify the NFS cluster works

Configuration is not done until failover is proven. The primary tool is clustat; watch live behaviour with tail -f /var/log/messages on both nodes in separate windows.

1. Confirm baseline health: clustat should show Member Status: Quorate, both nodes Online, rgmanager, the Quorum Disk Online, and service:U1-NFSsvc started on its owner.
2. From a client, mount the export over the VIP and write a test file: mount -t nfs 10.192.45.85:/mqdata/cipg /mnt && touch /mnt/clustertest.

Then run each failure test below; after every one, both nodes must return to Online before you start the next.

• Manual fence: from the passive node, fence_node cipglu1mqs02-hb should reboot the active node and relocate U1-NFSsvc to the survivor. This is the single most important test — it proves fencing works.
• Hard power off: use iLO → Power Management → Press and Hold on the active node. The service must fail over.
• Graceful reboot: shutdown -r now on the active node; the service relocates and the node rejoins automatically on boot.
• Public NIC down: ifconfig eth0 down on the active node; the service should relocate to the other node.
• Heartbeat NIC down: ifconfig eth1 down on the active node. Expect the cluster to fence the other node; the active node keeps serving. After the fenced node reboots it may need to be re-joined manually.
• Fibre-channel failure: find the HBAs with powermt display hba_mode, then powermt disable hba=4 and powermt disable hba=5 on the active node. Losing all storage paths should fence that node; it rejoins after reboot. Re-enable with powermt restore.

From the client, repeat the write test (touch /mnt/clustertest2) after a failover to confirm the export survived the move with the same VIP.

Common pitfalls (and how to avoid them)

• No quorum disk on a two-node cluster. Without it (or the two_node="1" expected_votes="1" trick), a single node failure can leave the cluster inquorate. The qdisk gives the safe third vote.
• Fencing not configured or untested. rgmanager will refuse to recover a service if it cannot fence the failed node — the service hangs. Always test fencing with fence_node first.
• Doing one-node-only steps on both nodes. Partitioning, pvcreate, vgcreate, lvcreate, mkfs.gfs2 and mkqdisk are once-only operations; running them twice corrupts metadata or destroys the filesystem you just made.
• Forgetting locking_type = 3. Without clustered LVM locking, the second node won't see VG changes and clvmd activation fails.
• Wrong GFS2 lock table or too few journals. The -t value must be clustername:fsname exactly, and -j must cover every node that mounts it.
• Adding cluster GFS2 to fstab. Let rgmanager own the mounts; fstab entries cause mount races and block clean failover.
• Mismatched nfs-utils or NFS/portmap firewall rules. Both nodes must run the same patched version, and NFS/rpcbind ports must be reachable across the VIP.
• Referencing storage by device path instead of label/UUID. /dev/emcpowerX names can differ per node; the quorum label and GFS2 UUIDs are stable.

The modern equivalent on RHEL 7/8/9

Because RHEL 6 is end-of-life, any new HA NFS cluster should be built with Pacemaker + Corosync, managed by pcs. The concepts map almost one-to-one:

On modern releases GFS2 is often unnecessary for a simple active/passive NFS service — a single XFS/ext4 LVM volume that fails over between nodes is simpler. Reserve GFS2 for genuine concurrent multi-node access. A typical modern build groups IPaddr2, an LVM-activate resource, a Filesystem resource, and the nfsserver/exportfs resources into one pcs resource group with STONITH enabled.

Key Takeaways

• A Red Hat 6 two-node NFS cluster stacks CMAN, rgmanager, clustered LVM, GFS2, a quorum disk and iLO/IPMI fencing into one active/passive service behind a floating VIP.
• The quorum disk and working fencing are mandatory on two nodes — they prevent split-brain and data corruption, and rgmanager won't fail a service over without successful fencing.
• Run storage creation steps (partition, PV/VG/LV, mkfs.gfs2, mkqdisk) on exactly one node; let clustered LVM and the cluster replicate the rest.
• Don't trust the build until you've run the full failover test matrix — fence, power-off, reboot, NIC-down and HBA-down — verifying with clustat each time.
• RHEL 6 is EOL: operate or migrate existing clusters with this guide, but build anything new on RHEL 8/9 with Pacemaker + Corosync and pcs.

Frequently Asked Questions

Why does a two-node Red Hat cluster need a quorum disk?

With only two votes, losing one node leaves a single vote — not a majority — so the cluster can become inquorate or, worse, both nodes can each believe they are the survivor (split-brain) and both write to shared storage. The quorum disk adds a third, independent vote that breaks the tie and keeps exactly one node authoritative during fencing.

What is fencing and why is it required for HA NFS?

Fencing (STONITH — "shoot the other node in the head") forcibly powers off or resets an unresponsive node, usually via its iLO/IPMI management interface. It guarantees a failed node can't keep writing to the shared GFS2 filesystem while the service starts elsewhere. Red Hat's resource manager will not relocate a service until it has confirmed the old owner is fenced, so without it failover stalls.

Do I have to use GFS2 for an NFS cluster?

No. GFS2 lets multiple nodes mount the same volume at once, which matters only for active/active access. For a straightforward active/passive NFS export, a standard ext4 or XFS volume on shared LVM that fails over with the service is simpler and faster. This build uses GFS2 mainly to leave the door open for future concurrent mounts.

Can I still build this on RHEL 7, 8, or 9?

Not with CMAN/rgmanager/Conga — that stack was replaced. On RHEL 7 and later you build the same HA NFS service with Pacemaker and Corosync driven by the pcs command, using IPaddr2, LVM-activate, Filesystem and nfsserver/exportfs resources, a fence (STONITH) device, and corosync-qdevice as the modern quorum tie-breaker.

If this walkthrough helped, subscribe to @explorenystream on YouTube for more hands-on Linux and system administration guides.