Monitor & Record all Shell Commands & Send Logs to Centralized RSyslog Server

— ny_wk

To log all shell commands on Linux and ship them to a central server, you combine a per-command capture method (auditd execve rules, the Snoopy logger, or a hardened PROMPT_COMMAND) with rsyslog forwarding over TLS to a hardened log host. This gives you a tamper-resistant, searchable record of exactly what every user typed, where, and when.

Capturing user activity is one of the most requested controls in any audit, incident response, or compliance program. When a server is compromised, misconfigured, or simply changed by the wrong person, the first question is always the same: who ran what command, and when? A scattered local ~/.bash_history file will not answer that question reliably, because it is trivial to disable, edit, or delete. The durable answer is centralized command logging: capture every command on each host, then forward those events to a remote, write-restricted log server that ordinary users cannot reach. The sections below walk through three capture methods, how to forward them securely, and how to keep the logs honest.

Why Centralized Shell Command Logging Matters

The core problem is trust. Logs that live only on the same machine where the action happened are logs an attacker (or a careless admin) can rewrite. Centralized shell command logging solves three problems at once:

• Accountability. Every command is tied to a real user, a PID, a timestamp, and the source IP of the session.
• Tamper resistance. Once an event leaves the host and lands on a dedicated log server, deleting it on the origin machine no longer hides it.
• Compliance. Frameworks like PCI-DSS, HIPAA, SOC 2, and CIS benchmarks all expect privileged-command auditing with off-host retention.

There is no single perfect tool. Each approach trades completeness, performance, and resistance to evasion. A strong setup usually layers auditd (kernel-level, hard to bypass) with rsyslog forwarding, and optionally adds Snoopy or a PROMPT_COMMAND hook for human-readable shell history.

The three capture approaches at a glance

Approach 1: PROMPT_COMMAND Logging via syslog

The simplest method hooks Bash's PROMPT_COMMAND so that every time the prompt is redrawn, the last command is sent to syslog with the logger utility. It is fast to deploy and produces clean, human-readable lines, which is why it remains popular. Treat it as visibility, not security, because any user with a non-Bash shell or the ability to unset the variable can sidestep it.

Edit the system-wide Bash configuration so the hook applies to all interactive shells:

1. Open the global Bash file. On RHEL/CentOS/Rocky/Alma this is /etc/bashrc; on Debian/Ubuntu use /etc/bash.bashrc:
   sudo vi /etc/bashrc
2. Append a capture hook to the end of the file:
   declare -rx PROMPT_COMMAND='RETRN_VAL=$?; if [ "$(id -u)" -ne 0 ] || [ -n "$BASH_COMMAND" ]; then logger -p local6.notice -t bash_audit -- "user=$(whoami) pid=$$ from=$(who am i | awk '\''{print $NF}'\'' | tr -d "()") cmd=$(history 1 | sed '\''s/^[ ]*[0-9]\+[ ]*//'\'') rc=$RETRN_VAL"; fi'
3. Force history to record every command with a timestamp by adding these lines too:
   export HISTTIMEFORMAT="%F %T "
export HISTSIZE=100000
export HISTFILESIZE=100000
shopt -s histappend
4. Route the local6 facility to a dedicated file by editing /etc/rsyslog.conf (or a drop-in in /etc/rsyslog.d/):
   local6.* /var/log/user-activity.log
5. Restart logging and re-login to test:
   sudo systemctl restart rsyslog

After logging out and back in, run a few commands and inspect the file:

sudo tail -f /var/log/user-activity.log

You will see lines such as bash_audit: user=alice pid=4991 from=192.168.124.1 cmd=systemctl status named rc=0. The declare -rx makes PROMPT_COMMAND read-only, raising the bar against casual tampering, though a determined user can still launch /bin/sh, python -c, or a sub-shell that never triggers it. That limitation is exactly why the next two methods exist.

Approach 2: Snoopy Command Logger

Snoopy is a small shared library loaded through the dynamic linker (via /etc/ld.so.preload) that intercepts every execv() and execve() call on the system and writes a syslog line. Because it sits below the shell, it captures commands run by any program, including cron jobs, scripts, and non-interactive shells, not just typed Bash input.

1. Install build prerequisites and Snoopy. On Debian/Ubuntu it is packaged:
   sudo apt-get install -y snoopy
   On RHEL-family systems, build from the official release tarball:
   sudo dnf install -y gcc make && ./configure && make && sudo make install && sudo snoopy-enable
2. Confirm Snoopy registered itself with the dynamic linker:
   cat /etc/ld.so.preload
   It should list /lib/security/libsnoopy.so (path varies by distro).
3. Tune what Snoopy records in /etc/snoopy.ini — the format string can include %username, %cwd, %tty, and the full %cmdline:
   message = "snoopy: user=%{username} cwd=%{cwd} tty=%{tty} cmd=%{cmdline}"
4. Snoopy logs to the authpriv facility by default. Route it to its own file in /etc/rsyslog.d/snoopy.conf:
   authpriv.* /var/log/snoopy.log
5. Restart rsyslog and verify with any command:
   sudo systemctl restart rsyslog && ls -la / && sudo tail /var/log/snoopy.log

Snoopy's strength is that it sees the literal argv of every executed binary, giving you a complete forensic trail that PROMPT_COMMAND cannot match. Its weaknesses: it relies on dynamic linking (statically linked binaries bypass it), it does not capture shell built-ins like cd that never call exec(), and a root user can edit /etc/ld.so.preload to remove it. For genuinely hard-to-evade auditing, use auditd.

Approach 3: auditd with execve Rules (the Robust Choice)

The Linux Audit daemon, auditd, hooks the kernel audit subsystem and records events before user space can interfere. By auditing the execve syscall you capture every program execution by every user, regardless of shell, and the records are written by a privileged daemon that ordinary users cannot touch. This is the method security teams rely on.

1. Install and enable auditd:
   sudo dnf install -y audit || sudo apt-get install -y auditd
sudo systemctl enable --now auditd
2. Add execve rules in /etc/audit/rules.d/commands.rules. Tag each rule with a key so you can search it later:
   -a always,exit -F arch=b64 -S execve -F auid>=1000 -F auid!=4294967295 -k cmd_audit
-a always,exit -F arch=b32 -S execve -F auid>=1000 -F auid!=4294967295 -k cmd_audit
3. To also capture root and service-account commands, add unfiltered rules (higher volume):
   -a always,exit -F arch=b64 -S execve -F euid=0 -k root_cmd
4. Load the rules and confirm they are active:
   sudo augenrules --load && sudo auditctl -l
5. Make the audit log append-only and immutable so even root cannot quietly edit it. Add -e 2 as the last line of your rules to lock the configuration until reboot:
   -e 2

The auid (audit/login UID) field is the key advantage: it follows a user through su and sudo, so even after privilege escalation the audit trail still names the human who logged in. To search the resulting logs, use ausearch and aureport (covered in the verification section). The trade-off is volume and noise: execve auditing can generate large logs on busy hosts, so forward it to a central server and rotate aggressively.

Forwarding Logs to a Central Rsyslog Server

Capturing locally is only half the job. The events must leave the host and land somewhere users cannot reach. Configure one hardened central rsyslog server and point every client at it. Always prefer TCP with TLS over plain UDP so logs are encrypted and reliably delivered.

Step 1: Configure the central log server

1. Install the TLS-capable rsyslog modules:
   sudo dnf install -y rsyslog rsyslog-gnutls || sudo apt-get install -y rsyslog rsyslog-gnutls
2. Generate or install a CA, a server certificate, and matching key, then place them under /etc/rsyslog.d/certs/ (ca.pem, server-cert.pem, server-key.pem).
3. Create /etc/rsyslog.d/00-server.conf to load the TCP input with TLS and sort each client into its own file:
   module(load="imtcp" StreamDriver.Name="gtls" StreamDriver.Mode="1" StreamDriver.AuthMode="x509/name")
global(DefaultNetstreamDriver="gtls" DefaultNetstreamDriverCAFile="/etc/rsyslog.d/certs/ca.pem" DefaultNetstreamDriverCertFile="/etc/rsyslog.d/certs/server-cert.pem" DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/certs/server-key.pem")
input(type="imtcp" port="6514")
template(name="PerHost" type="string" string="/var/log/remote/%HOSTNAME%/messages.log")
if $fromhost-ip != '127.0.0.1' then { action(type="omfile" dynaFile="PerHost") stop }
4. Open the firewall for the TLS syslog port (6514) and restart:
   sudo firewall-cmd --add-port=6514/tcp --permanent && sudo firewall-cmd --reload
sudo systemctl restart rsyslog

On older systems still using iptables, the equivalent is iptables -A INPUT -p tcp --dport 6514 -m state --state NEW -j ACCEPT followed by service iptables save.

Step 2: Configure each client to forward

1. Install the same TLS module on the client and copy over ca.pem plus a client cert/key.
2. Create /etc/rsyslog.d/90-forward.conf pointing at the server. The @@ prefix means TCP; a single @ means UDP:
   global(DefaultNetstreamDriver="gtls" DefaultNetstreamDriverCAFile="/etc/rsyslog.d/certs/ca.pem")
action(type="omfwd" target="logserver.example.com" port="6514" protocol="tcp" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" queue.type="LinkedList" queue.fileName="fwd_q" queue.saveOnShutdown="on" action.resumeRetryCount="-1")
3. The disk-assisted queue above means that if the log server is unreachable, events are spooled locally and delivered when it returns, so nothing is lost during an outage.
4. Restart rsyslog on the client:
   sudo systemctl restart rsyslog

For auditd events specifically, the cleanest pipeline is the audisp-syslog plugin, which feeds audit records into syslog so the same rsyslog forwarder ships them centrally. Enable it by setting active = yes in /etc/audit/plugins.d/syslog.conf and restarting auditd.

Pitfalls: How Users Evade Logging (and How to Stop Them)

Anyone serious about auditing must assume the logged user will try to escape it. Common evasion tricks and their countermeasures:

• Starting a different shell. A user runs sh, zsh, or python -c 'import os; os.system(...)' to dodge a Bash-only hook. Fix: use auditd execve rules, which catch every exec regardless of shell.
• Unsetting the hook. unset PROMPT_COMMAND kills the Bash method. Fix: declare it -r (read-only) and rely on kernel-level auditing as the real control.
• Disabling history. unset HISTFILE or export HISTSIZE=0 erases local history. Fix: never depend on ~/.bash_history for security; it is user-writable by design.
• Editing or deleting local logs. A root user can rm /var/log/user-activity.log. Fix: forward off-host immediately and make the central store append-only.
• Removing the audit rules. auditctl -D wipes loaded rules. Fix: set -e 2 to make audit configuration immutable until reboot, and alert on any reboot.
• Stripping LD_PRELOAD. Static binaries and /etc/ld.so.preload edits bypass Snoopy. Fix: treat Snoopy as a supplement to auditd, not a replacement.

The unifying principle: capture at the kernel and store off the box. Any control that lives entirely in the user's shell or on the user's machine is, at best, a speed bump.

Protecting log integrity

Make the central server boring and locked down. Restrict SSH to a small admin group, mount /var/log/remote on its own partition, and apply the immutable attribute to rotated archives with chattr +a (append-only) or chattr +i (immutable). For high-assurance environments, hash each rotated log and store the hashes on a separate system, or stream into a write-once object store. The goal is simple: a person who controls the origin host should never be able to alter the record of what they did.

Verification: Confirm Logging Actually Works

Never assume a logging pipeline works. Test it end to end after every change.

• Generate a marker command on a client, e.g. id; whoami; ls /etc.
• Search auditd locally for execve events tied to your key:
  sudo ausearch -k cmd_audit -i | tail -20
  Summarize per-user activity with sudo aureport -u -i and per-command counts with sudo aureport -x --summary.
• Check the central server received the events:
  sudo tail -f /var/log/remote/<clienthostname>/messages.log
• Confirm the TLS connection is live on the server:
  sudo ss -tlnp | grep 6514 and look for the client's IP in ss -tnp.
• Test outage recovery by stopping rsyslog on the server, running commands on the client, restarting the server, and confirming the spooled events arrive. This proves the disk-assisted queue is doing its job.

If events appear locally but not centrally, suspect the firewall, a certificate mismatch (x509/name requires the cert CN/SAN to match the target hostname), or a missing rsyslog-gnutls package. Check journalctl -u rsyslog on both ends for handshake errors.

Key Takeaways

• Use auditd execve rules as the authoritative control — it captures every command at the kernel and follows users through su/sudo via auid.
• PROMPT_COMMAND and Snoopy add readable shell history, but treat them as visibility layers, not security boundaries.
• Forward everything to a central rsyslog server over TCP+TLS (port 6514) with a disk-assisted queue so no events are lost during outages.
• Assume evasion: make hooks read-only, set audit config immutable with -e 2, and store logs off-host and append-only.
• Verify end to end with ausearch, aureport, and by tailing the per-host file on the central server after every change.

Frequently Asked Questions

Does auditd slow down a busy server?

Execve auditing adds measurable overhead on exec-heavy workloads and produces large logs. Limit rules to auid>=1000 to focus on real users, exclude noisy service accounts, forward off-host, and rotate aggressively. On most general-purpose servers the impact is small and well worth the visibility.

Can I just rely on bash_history for auditing?

No. ~/.bash_history is owned and writable by the user, only records interactive Bash, updates on logout (so a killed session may lose entries), and is trivially cleared. It is a convenience feature, never a security control. Use auditd plus centralized forwarding instead.

Should I forward over UDP or TCP?

Use TCP, ideally with TLS on port 6514. UDP (single @) silently drops packets under load and offers no encryption or delivery guarantee, which is unacceptable for an audit trail. TCP (@@) with a disk-assisted queue guarantees delivery and lets you encrypt the stream.

How do I tie a command back to the actual person after sudo?

auditd records the auid (login UID), which is set at login and never changes even after su or sudo. Search with ausearch -ua <auid> or map names with aureport -u, and you will see the original human behind every escalated command.

For more Linux, DevOps, and system administration walkthroughs, subscribe to @explorenystream on YouTube.