Log transfer shell script for Linux

— ny_wk

A reliable Linux log transfer shell script automatically collects rotated logs from a server, compresses them, and ships them to a central log server over SSH on a cron schedule. This guide shows how to build a production-grade script that gathers files from a manifest, zips them, transfers them with scp or rsync, prevents overlapping runs with a lock file, and alerts you when something breaks.

The problem this Linux log transfer script solves

On busy application and web servers, logs pile up fast and rotate often. If you need those logs on a central log server for analysis, compliance, or troubleshooting, manual copying does not scale. You need an unattended, repeatable job that knows exactly which files to send, sends only what is new, and never trips over itself when the previous run is still working.

The common pain points are predictable. Overlapping cron runs corrupt archives or saturate the network. A single failed SSH transfer silently drops a batch of logs. Hard-coded file paths break the moment a new application directory appears. The pattern below addresses each of these directly.

Solution overview: how the log transfer works

The design separates what to send from how to send it. You maintain one or more plain-text filelist manifests that list directories and files to collect. The script reads each manifest, resolves the entries into a concrete file list, packs them into a single ZIP archive, and pushes that archive to the remote log server.

Each logical group of logs (often a line of business, application, or environment) gets its own manifest, for example filelist.web or filelist.app.payments. The matching archive is delivered into a corresponding sub-directory on the log server, so logs stay organized by source.

• Manifest-driven: add a path to a filelist file, not to the script.
• Single-instance: a PID lock file blocks concurrent runs.
• Incremental option: a checkpoint mode sends only files newer than the last successful transfer.
• Alerting: a stuck job triggers an email so failures are visible.

Prerequisites

• SSH key-based authentication set up from the source host to the log server for a dedicated transfer user (for example logship). Never embed passwords.
• zip installed (sudo apt install zip on Debian/Ubuntu, sudo dnf install zip on RHEL/Fedora).
• A working directory such as /usr/local/scripts/logtransfer that holds the script and the manifests.
• A mail command (mailx / s-nail) if you want failure alerts, or replace it with a webhook.

Step-by-step: building the log transfer shell script

The script below is written for modern Bash and is portable across current Linux distributions. The original idea comes from older AIX/Solaris-style ksh scripts; the version here fixes the fragile parts and uses safer, well-supported syntax.

Step 1: Safe header and configuration

Start with a strict shell mode so undefined variables and pipe failures stop the script instead of silently producing a bad archive.

1. Set the interpreter and strict flags.
2. Define paths and the remote target in one place so they are easy to audit.

#!/usr/bin/env bash
set -Eeuo pipefail
SCRIPT_NAME=$(basename "$0")
WORKDIR="/usr/local/scripts/logtransfer"
ZIP="/usr/bin/zip"
HOSTNAME=$(hostname -s)
REMOTE_USER="logship"
REMOTE_HOST="10.0.3.73"
REMOTE_BASE="/srv/logserver/logfiles"
ALERT_EMAIL="ops@example.com"
LOCK="/tmp/${SCRIPT_NAME}.running"

Step 2: Show usage when no argument is given

The script expects a target group name (or ALL). If it is missing, print clear help and exit cleanly rather than running against nothing.

if [ "$#" -eq 0 ]; then
  echo "Usage: $SCRIPT_NAME ALL"
  echo "   or: $SCRIPT_NAME <group>"
  echo "   or: $SCRIPT_NAME <group> checkpoint"
  exit 0
fi

Step 3: Prevent overlapping runs with a PID lock file

This is the single most important reliability feature. Before doing any work, the script checks for a lock file containing the PID of a previous run. If that process is still alive, the new run aborts and alerts. If the lock is stale (the process is gone), it removes it and continues.

if [ -f "$LOCK" ]; then
  old_pid=$(cat "$LOCK")
  if kill -0 "$old_pid" 2>/dev/null; then
    echo "$SCRIPT_NAME: already running as PID $old_pid. Abort."
    mailx -s "Log transfer stuck on $HOSTNAME" "$ALERT_EMAIL" <<EOT
The log transfer on $HOSTNAME did not finish; PID $old_pid still running.
EOT
    exit 0
  else
    echo "$SCRIPT_NAME: stale lock found, removing."
    rm -f "$LOCK"
  fi
fi
echo $$ > "$LOCK"
trap 'rm -f "$LOCK"' EXIT

The kill -0 "$old_pid" test is the correct, portable way to ask "is this PID alive?" without actually sending a signal. The trap ... EXIT guarantees the lock is removed even if the script errors out, which is far safer than deleting it only at the very end.

Step 4: Resolve the manifest into a real file list

Each manifest line names a file or directory to collect. Reading it line by line lets you expand directories, optionally limit to recent files, and skip anything that no longer exists.

prepare() {
  local manifest="$1"; local mode="${2:-}"
  local filelist="/tmp/$$.$(basename "$manifest")"
  local recent=()
  [ "$mode" = "checkpoint" ] && recent=(-mtime -1)
  : > "$filelist"
  while IFS= read -r entry || [ -n "$entry" ]; do
    [ -z "$entry" ] && continue
    if [ -d "$entry" ]; then
      find "$entry" -type f "${recent[@]}" >> "$filelist"
    elif [ -f "$entry" ]; then
      echo "$entry" >> "$filelist"
    fi
  done < "$manifest"
  echo "$filelist"
}

Two corrections worth noting against the older style. First, always use IFS= read -r so backslashes and leading or trailing whitespace in paths are preserved exactly. Second, never run eval on lines read from a file, as the original did; that turns your log list into a remote-code-execution risk if any path contains shell metacharacters.

Step 5: Zip the files and transfer to the log server

With a concrete file list in hand, pack it into one archive and ship it. Using zip -@ reads the file names from standard input, which avoids hitting command-line length limits when there are thousands of files.

cd "$WORKDIR"
[ "$1" = "ALL" ] && group_glob="filelist.*" || group_glob="filelist.$1"
for manifest in $group_glob; do
  [ -e "$manifest" ] || continue
  group="${manifest#filelist.}"
  subdir="${group//./\/}"
  archive="${HOSTNAME}.${group}.logs.zip"
  rm -f "$archive"
  list=$(prepare "$manifest" "${2:-}")
  if [ -s "$list" ]; then
    "$ZIP" -q -r -y "$archive" -@ < "$list"
    scp -p "$archive" "${REMOTE_USER}@${REMOTE_HOST}:${REMOTE_BASE}/${subdir}/"
  fi
  rm -f "$list" "$archive"
done

The -y flag tells zip to store symbolic links as links instead of following them, which keeps archives small and avoids duplicating large rotated files. The ${group//./\/} parameter expansion turns a dotted group name like app.payments into a nested remote path app/payments.

Step 6: Add the cron schedule

Run the job on a fixed cadence and capture output to a log so you can audit every run. Edit the crontab with crontab -e and add:

*/15 * * * * /usr/local/scripts/logtransfer/logtransfer.sh web >> /var/log/logtransfer.log 2>&1

This runs every 15 minutes for the web group, appending both standard output and errors to a single log. Because the lock file guarantees single-instance execution, it is safe even if one run occasionally takes longer than the interval.

A faster alternative: rsync over SSH

If you do not need a single zipped archive, rsync is often the better tool for ongoing log transfer. It copies only the changed bytes of changed files, resumes cleanly after interruptions, and preserves timestamps and permissions in one command.

rsync -az --partial --append-verify \
  --files-from=/usr/local/scripts/logtransfer/filelist.web \
  -e "ssh -i /home/logship/.ssh/id_ed25519" \
  / "${REMOTE_USER}@${REMOTE_HOST}:${REMOTE_BASE}/web/"

Here -a preserves metadata, -z compresses in transit, and --partial --append-verify make large transfers resumable. Use zip + scp when the destination must receive one self-contained bundle (for example a compliance archive); use rsync when you want continuous, efficient mirroring.

Comparison at a glance

Common pitfalls in a log transfer shell script

• No lock file. Without single-instance protection, overlapping cron runs race on the same archive and corrupt it. The PID lock plus kill -0 check is essential.
• Deleting the lock only at the end. If the script crashes midway, the stale lock blocks all future runs. Use trap 'rm -f "$LOCK"' EXIT so cleanup always happens.
• Running eval on file paths. Never eval lines read from a manifest; a crafted path could execute arbitrary commands. Read with IFS= read -r and reference paths in quotes.
• Unquoted variables. Paths with spaces silently break find, scp, and zip. Quote every expansion: "$archive", "$entry".
• Password-based SSH in cron. Cron has no terminal, so prompts hang forever. Always use SSH keys for a dedicated, least-privilege transfer user.
• Following symlinks in zip. Without -y, a symlink to a huge log directory can balloon the archive. Store links as links.
• Ignoring scp exit codes. A failed transfer should be retried or alerted, not silently dropped. Wrap the transfer in a retry loop and check $?.

Verification: confirm the transfer actually worked

Never assume a cron job succeeded. Verify end to end after the first run and periodically thereafter.

1. Check the run log: tail -n 50 /var/log/logtransfer.log should show the date stamp and no errors.
2. Confirm the lock cleared: ls -l /tmp/logtransfer.sh.running should report "No such file" after a completed run.
3. List the remote archive: ssh logship@10.0.3.73 "ls -lh /srv/logserver/logfiles/web/" and confirm a fresh timestamp and non-zero size.
4. Validate the archive integrity: on the log server run unzip -t hostname.web.logs.zip; it should report "No errors detected."
5. Spot-check contents: unzip -l hostname.web.logs.zip | tail to confirm the expected files are present.
6. Test the alert path: create a fake stale lock with a live PID and confirm the email or webhook fires.

For a hands-off audit, schedule a tiny watchdog that emails you if the newest file in the remote target directory is older than the expected interval. That catches silent failures faster than any single log line.

Hardening for production

• Least privilege: give the transfer user write access only to its destination directory; restrict the SSH key with command= in authorized_keys if you can.
• Retries: loop the scp/rsync up to three times with a short sleep before alerting.
• Atomic delivery: upload to a .tmp name and rename on the server so consumers never read a half-written archive.
• Rotation on the server: let the receiving side prune old archives so disks do not fill.
• Observability: emit a one-line success record (host, group, bytes, duration) you can graph or alert on.

Key Takeaways

• Drive the Linux log transfer shell script from plain-text manifests so you add log paths without editing code.
• A PID lock file with a kill -0 liveness check plus a trap ... EXIT cleanup is the core reliability pattern.
• Use zip -@ with scp for one bundle per run, or rsync over SSH for efficient incremental sync.
• Quote every variable, read manifests with IFS= read -r, and never eval file paths.
• Always verify on the remote side: list the file, test the archive with unzip -t, and alert on stale or missing transfers.

Frequently Asked Questions

How do I prevent two log transfer cron jobs from running at once?

Write the current process ID to a lock file at the start and check it on the next run. If kill -0 "$old_pid" succeeds, the previous job is still alive, so exit and alert. If it fails, the lock is stale and safe to remove. Add trap 'rm -f "$LOCK"' EXIT so the lock is always cleaned up, even after a crash.

Should I use scp or rsync to transfer logs to a central server?

Use scp when you want a single self-contained archive delivered per run, such as a daily compliance bundle. Use rsync over SSH for ongoing mirroring, since it transfers only changed data, resumes interrupted copies with --partial, and preserves timestamps and permissions automatically.

How do I send only the newest log files instead of everything?

Filter the file list with find <dir> -type f -mtime -1 to grab files modified in the last day, or compare against the last successful run using find <dir> -newer /path/to/last_marker. Touch a marker file after each successful transfer so the next run only picks up what changed.

Why does my log transfer script hang under cron but work manually?

The usual cause is an interactive SSH prompt. Cron has no terminal, so password or host-key prompts block forever. Fix it by using key-based authentication for a dedicated transfer user and pre-populating known_hosts, or pass -o BatchMode=yes to ssh/scp so it fails fast instead of waiting for input.

If this helped, subscribe to @explorenystream on YouTube for more Linux and sysadmin walkthroughs.