How to Document for Rebuild and reinstall and Upgrade lin_tape to for LanFree backup on Linux 6.x

— ny_wk

Upgrading the IBM lin_tape driver on RHEL 6.x is the most reliable way to keep TSM LanFree backup working after a kernel patch or a driver version change. This guide walks through documenting the system, rebuilding the lin_tape kernel module from source, reinstalling the lin_taped daemon, re-discovering tape drives, and verifying that LAN-free data still flows to your tape library.

Important context: Red Hat Enterprise Linux 6 reached End of Maintenance (Extended Life-cycle Support aside) on 30 November 2020, and IBM Spectrum Protect (formerly Tivoli Storage Manager, TSM) clients for RHEL 6 are long out of standard support. If you are building new infrastructure, move to RHEL 8/9 with a current IBM Spectrum Protect client and the matching lin_tape release. The procedure below remains valuable for legacy estates that must be maintained, migrated, or documented for an audit before decommissioning.

Why a lin_tape Driver Upgrade Breaks LanFree Backup

The lin_tape driver is a kernel module (lin_tape.ko) that gives Linux direct device access to IBM TS1100/35xx and LTO (Ultrium) tape drives. The companion lin_taped daemon manages device error handling and the special device files (/dev/IBMtape0, /dev/IBMchanger0). TSM/Spectrum Protect LanFree backup uses the storage agent (dsmsta) to write client data straight to those tape devices over the SAN, bypassing the LAN and the TSM server's own data path.

Because lin_tape is a kernel module, it is compiled against a specific kernel version. The instant you run yum update and pick up a newer kernel, the existing lin_tape.ko no longer matches uname -r, the module fails to load, the tape devices disappear, and LanFree backups fall back to LAN or fail outright. That is exactly why this is shipped as a source RPM that you rebuild against the running kernel with rpmbuild --rebuild.

The Real Problem

• Symptom: After a kernel patch or driver swap, /dev/IBMtape* devices vanish and dsmc backups stop reporting LanFree data bytes.
• Root cause: The compiled lin_tape.ko is bound to the old kernel; the daemon cannot find drives.
• Fix: Rebuild and reinstall lin_tape for the current kernel, reinstall lin_taped, regenerate the drive map, then test a real backup and restore.

Step 1: Document and Protect the System First

Never touch a working backup server without a rollback path. Before any change, capture a full picture of the current state so you can restore it.

1. Confirm a good OS backup exists. Check the log of your scheduled system backup, for example tail -n 50 /var/log/systembackup.log, and confirm the most recent run completed.
2. Snapshot the VM. If the host is virtual, take a hypervisor snapshot (or an image-level copy from your backup product) so you can revert the kernel and driver together.
3. Record the running kernel: uname -r
4. Record the installed driver versions: rpm -qa | grep lin_tape
5. Save the device map: copy /etc/lin_taped.conf and any ERMM config under /opt/IBM/ermm/conf/ to a safe location.

A clean baseline looks like this:

Step 2: Install the Build Prerequisites

Rebuilding a source RPM needs a compiler toolchain and the kernel development headers that match your running kernel. Install them with yum:

1. Install the RPM build tools and compiler:
   yum install rpm-build gcc gcc-c++ make
2. Install SCSI generic utilities (used to enumerate tape devices):
   yum install sg3_utils
3. Install kernel headers and the kernel source tree for the running kernel:
   yum install kernel-headers kernel-devel

Critical check: the kernel-devel version must equal uname -r. Compare them:

1. uname -r
2. rpm -q kernel-devel kernel-headers

If kernel-devel is newer than the booted kernel (a common trap right after a yum update), reboot into the new kernel before building, or install the exact matching version. Mismatched headers are the number-one cause of a build that compiles but produces a module that will not load.

About the Kernel Update

If you are also patching the OS, do the kernel update and reboot before rebuilding the driver, so the module is compiled against the kernel you will actually run:

1. yum update (exclude any agent you manage separately, e.g. yum update -x CentrifyDC)
2. shutdown -r now
3. After reboot, confirm with uname -r that the intended kernel is active.

Step 3: Stop the lin_taped Daemon and Unload the Module

You cannot replace a module that is in use. Quiesce backups first, then stop the daemon and remove the old module from the running kernel.

1. Make sure no TSM/Spectrum Protect backup or restore is running and the storage agent is idle.
2. Stop the service: /etc/init.d/lin_tape stop
3. Unload the kernel module: rmmod lin_tape
4. Confirm it is gone: lsmod | grep lin_tape (should return nothing).

If rmmod reports the module is in use, find the holder with lsof /dev/IBMtape* or stop the storage agent (dsmsta) and the ERMM daemon first.

Step 4: Stage the New lin_tape Source Package

Obtain the correct package for your platform - for RHEL 6 x86_64 that is the lin_tape 3.0.10 source RPM plus the prebuilt lin_taped daemon RPM. IBM publishes tape device drivers on IBM Fix Central (search for "IBM Tape Device Drivers", platform Linux); pull the version validated for your RHEL release.

The 3.0.10 package directory contains these files:

Stage the files in a working directory and copy them to the server:

1. Create and open the staging folder:
   mkdir -p /systemfiles/lanfree/ibmtape/lin_tape/lin_tape-3.0.10-rhel6.x86_64
cd /systemfiles/lanfree/ibmtape/lin_tape/lin_tape-3.0.10-rhel6.x86_64
2. Copy the package files in (for example with scp from your software repository host):
   scp user@repohost:/path/to/lin_tape-3.0.10-rhel6.x86_64/* .
3. Verify the files arrived: ls -l

Step 5: Rebuild lin_tape from the Source RPM

This is the heart of the procedure: compiling lin_tape.ko against the running kernel. Run the rebuild from the staging directory:

1. cd /systemfiles/lanfree/ibmtape/lin_tape/lin_tape-3.0.10-rhel6.x86_64
2. rpmbuild --rebuild lin_tape-3.0.10-1.src.rpm

The build prints progress to the screen. Near the end, look for the Wrote: line - it tells you the exact path of the binary RPM you just built:

Wrote: /root/rpmbuild/RPMS/x86_64/lin_tape-3.0.10-1.x86_64.rpm

followed by a clean + exit 0. Note that path - that is the package you install next. (On older RPM setups the output may land under /usr/src/redhat/RPMS/x86_64/ instead; trust the Wrote: line.)

If the build fails, the message almost always points at the kernel source. Re-check that kernel-devel matches uname -r, that gcc and make are installed, and that /lib/modules/$(uname -r)/build exists.

Step 6: Remove the Old Driver and Install the New One

Uninstall both old packages together so the daemon and module versions never get out of sync:

1. Remove the old packages: rpm -ev lin_taped lin_tape
   (Harmless warnings such as "lin_taped is not running" or "Module lin_tape does not exist in /proc/modules" just mean it was already stopped/unloaded.)
2. Confirm both are gone: rpm -qa | grep lin_tape (should return nothing).
3. Install the freshly built driver:
   rpm -ivh /root/rpmbuild/RPMS/x86_64/lin_tape-3.0.10-1.x86_64.rpm
4. Install the matching daemon:
   rpm -ivh /systemfiles/lanfree/ibmtape/lin_tape/lin_tape-3.0.10-rhel6.x86_64/lin_taped-3.0.10-rhel6.x86_64.rpm

A successful install ends with Starting lin_tape: lin_tape loaded. Confirm the module path now references the current kernel:

rpm -ql lin_tape | grep lin_tape.ko

You should see something like /lib/modules/2.6.32-431.20.5.el6.x86_64/kernel/drivers/scsi/lin_tape.ko - note the kernel version in the path matches uname -r.

Step 7: Re-discover TLSM / ERMM Tape Drives

If your environment uses the IBM TS Library Manager / ERMM (Enterprise Removable Media Manager) layer, the device map can still point at the old generic SCSI nodes (/dev/sg1) instead of the persistent /dev/IBMtape* names. Regenerate it.

1. Inspect the current map: cat /opt/IBM/ermm/conf/hdmdrive.cfg
   (If entries read like drive /dev/sg1 ,IBM,03592E06,... the map is stale.)
2. Back up and clear the old config:
   cd /opt/IBM/ermm/conf
mv hdmdrive.cfg hdmdrive.cfg.old
3. Restart the host device manager so it rebuilds the map:
   cd /opt/IBM/ermm/bin
./ermmHDMRestart
4. Verify the new map now uses persistent device names:
   cat /opt/IBM/ermm/conf/hdmdrive.cfg
   (Entries should now read drive /dev/IBMtape0 ,IBM,03592E06,....)

If you start ERMM manually, launch the generator and confirm it is running:

1. cd /opt/IBM/ermm/bin && nohup ./ermmGenHDM &
2. ps -ef | grep ermm

Step 8: Verify the Driver, Daemon, and Devices

Before testing a backup, confirm every layer is healthy.

1. Confirm both packages are at the new version:
   rpm -qa lin_tape lin_taped -> expect lin_tape-3.0.10-1 and lin_taped-3.0.10-1.
2. Confirm the daemon is running:
   /etc/init.d/lin_tape status -> "lin_taped is running, pid ..."
3. Confirm the process and the module are present:
   ps -ef | grep lin_taped and lsmod | grep lin_tape
4. Confirm the tape device nodes exist:
   ls -l /dev/IBMtape* /dev/IBMchanger*
5. Confirm the TSM storage agent and scheduler are up (for LanFree):
   ps -ef | grep dsmsta and ps -ef | grep "dsmc sched"

Step 9: Test a Real LanFree Backup and Restore

The only proof that matters is data moving over the SAN, not the LAN. Run an incremental backup of a test path and confirm the summary reports LanFree data bytes equal to (or close to) the bytes transferred.

1. Confirm the include/exclude management class points at your LanFree-eligible storage:
   cat /opt/tivoli/tsm/client/ba/bin/include.file
2. Run an incremental backup:
   dsmc incr "/data/*" -subdir=yes
3. In the summary, check that the session established with the server and that:
   • Total number of objects failed: 0
   • LanFree data bytes is non-zero and matches the bytes transferred (proves SAN/tape path was used).
4. Validate recovery with a restore:
   dsmc restore /data/dsmerror.log
   and confirm Total number of objects restored: 1 with 0 failures and non-zero LanFree data bytes.

A healthy result shows the LanFree byte count tracking the transferred bytes - for example, 9.93 MB transferred with 9.93 MB of LanFree data - which confirms the rebuilt driver, daemon, and drive map are all working end to end.

Common Pitfalls When You Upgrade lin_tape on RHEL 6.x

• Kernel/header mismatch. Building against kernel-devel that does not equal uname -r yields a module that will not load. Always reboot into the target kernel first, then build.
• Forgetting to rebuild after a kernel patch. Any yum update that bumps the kernel requires a rpmbuild --rebuild of lin_tape. Bake this into your patch runbook.
• Daemon and driver version drift. Always remove and install lin_tape and lin_taped as a matched pair. Mixed versions cause silent device-handling failures.
• Stale ERMM/HDM map. If drives still show as /dev/sg*, regenerate hdmdrive.cfg - otherwise the library manager cannot address the drives.
• Removing a module that is in use. Stop dsmsta, ERMM, and any running backup before rmmod; otherwise it fails with "module in use".
• No rollback. Skipping the VM snapshot/OS backup leaves you stranded if the new kernel or driver misbehaves.
• Persistent binding / multipath. On SAN-attached libraries, confirm device discovery after reboot is stable; rescan with sg_scan / sg_map from sg3_utils if a drive is missing.

Verification Checklist

Key Takeaways

• lin_tape is a kernel module - any kernel patch on RHEL 6.x means you must rebuild it with rpmbuild --rebuild against the running kernel.
• Document and snapshot first: verify an OS backup, take a VM snapshot, and record kernel and driver versions before changing anything.
• Match kernel-devel to uname -r - header mismatch is the leading cause of a driver that builds but never loads.
• Upgrade lin_tape and lin_taped as a matched pair and regenerate the ERMM/HDM drive map so devices map to persistent /dev/IBMtape* names.
• Prove success with a real backup and restore - non-zero LanFree data bytes confirm data is flowing over the SAN to tape, not the LAN.

Frequently Asked Questions

Do I have to rebuild lin_tape every time I patch the kernel on RHEL 6?

Yes. Because lin_tape.ko is compiled for one specific kernel version, a kernel update breaks it. Reboot into the new kernel, then run rpmbuild --rebuild lin_tape-<ver>.src.rpm and reinstall. Add this step to your standard patching runbook so LanFree backups never silently fall back to the LAN.

How do I confirm a backup actually used the LanFree (SAN) path?

Run dsmc incr and read the summary. If the LanFree data bytes value is non-zero and roughly equals Total number of bytes transferred, the data went straight to tape over the SAN. If LanFree bytes are zero, the client used the LAN/server path - check the storage agent (dsmsta), the drive map, and zoning.

Where do I get the correct lin_tape version for my RHEL release?

Download IBM Tape Device Drivers from IBM Fix Central (filter for product "Tape device drivers", platform Linux) and pick the release validated for your RHEL version - for RHEL 6.x that was lin_tape 3.0.10. Always grab both the driver source RPM and the matching lin_taped daemon RPM.

Should I still run RHEL 6 and TSM for new tape backup systems?

No. RHEL 6 reached end of maintenance in November 2020 and IBM rebranded TSM as IBM Spectrum Protect. For anything new, deploy RHEL 8/9 with a current Spectrum Protect client and the matching lin_tape driver. Keep this guide for maintaining or migrating legacy estates until they are decommissioned.

Found this useful? Subscribe to @explorenystream on YouTube for more hands-on Linux, backup, and systems administration walkthroughs.