— LiveStream

To enable SMTP services on an IBM AIX p595 server you configure sendmail by editing /etc/mail/sendmail.cf, defining the local hostname and domain, pointing outbound mail at a smart-host relay, and then starting and enabling the sendmail subsystem with the AIX startsrc and chrssys commands. This guide walks through enabling SMTP on AIX the right way, fixes several common mistakes in legacy sendmail configs, and shows you how to verify that mail actually flows.

The IBM eServer pSeries 595 (p595) is a POWER5 enterprise system that typically runs AIX 5.3, 6.1, or 7.1. Those releases ship the classic Sendmail MTA as the default SMTP transport. The hardware is long past end of service and AIX 5.3/6.1 are end of life, so treat this as a maintenance/legacy guide. If you are building anything new, see the modern-equivalent note near the end before you invest time here.

The problem: SMTP on AIX is installed but not delivering

On a stock AIX p595, sendmail is present but frequently mis-tuned. The two symptoms admins hit most are mail sitting forever in the queue, and mail that leaves the box but never reaches the destination. Both come down to three things being wrong or undefined in the sendmail configuration:

• The local hostname and domain ($w, $m, $j macros) are not set the way you intend, so sendmail rewrites addresses incorrectly.
• No smart relay host is defined, so AIX tries to deliver every message directly via DNS MX lookups, which most enterprise firewalls block outbound on port 25.
• The subsystem is not started or not set to start at boot, so nothing is listening on TCP 25 after a reboot.

The source many people copy from edits the wrong file (/etc/sendmail.cf) and shows contradictory hostname examples. The correct file on AIX is /etc/mail/sendmail.cf (with /etc/sendmail.cf usually a symlink to it), and you must keep your macro values consistent. Let's do it properly.

The solution: configure and enable SMTP on AIX step by step

The high-level approach to enable SMTP on AIX is: back up the config, set identity, set the relay, refresh the aliases and config, start the daemon, and make it persistent. Each step below is a real command you can run as root.

Step 1 — Confirm sendmail is installed and back up the config

1. Check the fileset is present:
   lslpp -l bos.net.tcp.smit and which sendmail (the binary lives at /usr/sbin/sendmail).
2. Back up the active configuration before touching it:
   cp -p /etc/mail/sendmail.cf /etc/mail/sendmail.cf.$(date +%Y%m%d)
3. Confirm which file is live. On AIX, /etc/sendmail.cf is typically a symlink:
   ls -l /etc/sendmail.cf

Step 2 — Define the local hostname and domain

In /etc/mail/sendmail.cf, find the local info section. By default sendmail derives the w, j, and D values from the output of the hostname command, so you only set these macros if you want to override the defaults. If you do override, set both the hostname and the domain and keep them consistent. Uncomment and edit (replace the example values with your real ones):

• Dwdmspar02 — the $w macro: this host's short name.
• Dmexample.net — the $m macro: your DNS domain.
• Cwlocalhost dmspar02 dmspar02.example.net — the w class: every name this server should accept mail for (always include localhost).

Important correction: a fully qualified name like dmspar02.example.net belongs in the w class or in /etc/mail/local-host-names, not crammed into the $w short-name macro. Mixing a short name and a domain that don't match (a common copy-paste error) is exactly what breaks address rewriting.

Step 3 — List the names you receive mail for

Rather than hand-editing the Cw class, the cleaner method is the file referenced by Fw. Uncomment this line in sendmail.cf:

• Fw-o/etc/mail/local-host-names

Then create /etc/mail/local-host-names with one accepted name per line, for example:

1. dmspar02
2. dmspar02.example.net
3. mail.example.net

Step 4 — Set the smart relay host (DS macro)

This is the single most important step for outbound SMTP on an enterprise AIX server. Instead of delivering directly, hand all non-local mail to a central relay/smart host. Find the "Smart" relay host comment and set the DS macro:

• DSsmtprelay.example.net — forward to a host found by DNS, or
• DS[smtprelay.example.net] — the square brackets force a direct A-record lookup and skip MX resolution (use this when the relay has no MX or you want to bypass MX), or
• DSsmtp:[10.20.30.40] — pin to a specific mailer and IP literal.

The DS macro alone tells sendmail where the relay is. To actually force all outbound mail through it, you also uncomment the corresponding rule in ruleset 0 (it is clearly marked in the file with a comment such as "uncomment to forward all mail to the relay host"). Setting DS without enabling that rule means only mail it can't otherwise route gets relayed.

Step 5 — Rebuild aliases and refresh the configuration

1. Make sure /etc/mail/aliases has a valid postmaster and MAILER-DAEMON entry, then rebuild the alias database:
   /usr/sbin/sendmail -bi (equivalent to newaliases)
2. If you edited sendmail.cf directly, that is fine on AIX — there is no .mc/m4 recompile step required by default. Just reload the running daemon:
   refresh -s sendmail

Step 6 — Start the sendmail subsystem and make it persistent

AIX manages sendmail through the System Resource Controller (SRC), not init.d scripts. Start it and set the boot-time arguments:

1. Start now (the -bd flag = daemon mode listening on port 25; -q30m = process the queue every 30 minutes):
   startsrc -s sendmail -a "-bd -q30m"
2. If it is already running, restart cleanly:
   stopsrc -s sendmail then startsrc -s sendmail -a "-bd -q30m"
3. Persist the daemon arguments so they survive a reboot (this updates the SRC object so /etc/rc.tcpip brings sendmail up correctly):
   chssys -s sendmail -a "-bd -q30m"
4. Confirm the start sendmail line is uncommented in /etc/rc.tcpip so the subsystem launches at boot.

Quick reference: AIX sendmail commands

Common pitfalls when you enable SMTP on AIX

Most failed SMTP-on-AIX deployments trace back to a short list of avoidable mistakes. Watch for these:

• Editing the wrong file. Edit /etc/mail/sendmail.cf, not a stray /etc/sendmail.cf copy. Verify with ls -l that the symlink points where you think.
• Forgetting to refresh. Editing sendmail.cf has no effect until you refresh -s sendmail (or stop/start). The running daemon keeps the old config in memory.
• Relay set but rule 0 not enabled. A DS entry without uncommenting the ruleset-0 forwarding rule means general outbound mail still tries direct delivery and gets blocked at the firewall.
• Inconsistent identity macros. A short $w that doesn't agree with $m or with local-host-names causes loops ("local configuration error" / "MX points to me") and bounced mail.
• Not persisting with chssys. Using only startsrc works until the next reboot, then SMTP silently disappears. Always run chssys too.
• Tab vs spaces. Sendmail rule lines (R lines) are tab-delimited. If you reformat them with spaces, rules break in confusing ways. Touch only the macro lines unless you know the ruleset syntax.
• Port 25 blocked or already bound. Another MTA (or a stale sendmail) may hold the port. Check before starting.
• Open relay. Never relay for the whole internet. Restrict who can send through the box via /etc/mail/relay-domains and access controls; an open relay gets your IP blacklisted fast.

Verification: prove that SMTP is actually working

Don't assume — test. Run these checks after enabling SMTP on the p595:

1. Subsystem is active: lssrc -s sendmail should show the subsystem in the active state with a PID.
2. Listening on port 25: netstat -an | grep '\.25 ' should show a LISTEN entry on *.25.
3. Config parses cleanly: sendmail -bv postmaster resolves an address without errors, and sendmail -bt opens address-test mode (type 3,0 user@example.net to watch rewriting).
4. Send a test message:
   echo "AIX SMTP test body" | mail -s "p595 smtp test" you@example.net
5. Watch it leave verbosely: sendmail -v you@example.net then type the message and end with a single . on its own line. The verbose SMTP dialogue tells you exactly where delivery succeeds or stalls.
6. Inspect the queue and logs: mailq should be empty after delivery; tail the mail log (often /var/spool/mqueue activity and the syslog facility you configured for mail.debug in /etc/syslog.conf — remember to refresh -s syslogd after editing it).

If a message stays queued, run sendmail -q -v to force a queue run and read the diagnostic output; it will name the relay it tried and the SMTP response code it got back.

Legacy note and the modern equivalent

The p595 and AIX 5.3/6.1 are end of life and no longer receive security fixes, so this configuration should only be applied to keep an existing legacy estate running while you migrate. On supported AIX 7.2/7.3, sendmail still ships and the commands above are unchanged. If you are standing up new mail infrastructure, prefer a maintained MTA such as Postfix (simpler, safer relay configuration via main.cf), and use authenticated SMTP submission on port 587 with TLS rather than plaintext port 25 between sites. The smart-host pattern shown here maps directly to Postfix's relayhost = [smtprelay.example.net]:587.

Key Takeaways

• Edit the right file: configure SMTP on AIX in /etc/mail/sendmail.cf, and always back it up first with cp -p.
• Set identity and relay: keep $w/$m consistent, list accepted names in local-host-names, and point outbound mail at a smart host with the DS macro plus the ruleset-0 forwarding rule.
• Use SRC, not init scripts: startsrc -s sendmail -a "-bd -q30m" to run it and chssys to make it survive a reboot.
• Reload after every change: refresh -s sendmail and rebuild aliases with sendmail -bi.
• Verify, don't assume: confirm with lssrc, netstat on port 25, a real test mail, and mailq; treat the platform as legacy and plan a move to a supported MTA.

Frequently Asked Questions

How do I start and stop sendmail on AIX?

AIX uses the System Resource Controller. Start with startsrc -s sendmail -a "-bd -q30m", stop with stopsrc -s sendmail, and check status with lssrc -s sendmail. To make the daemon launch automatically at boot, run chssys -s sendmail -a "-bd -q30m" and ensure the start line is uncommented in /etc/rc.tcpip.

Which file do I edit to configure SMTP on AIX?

The active sendmail configuration is /etc/mail/sendmail.cf; /etc/sendmail.cf is normally just a symlink to it. Edit the local info section for hostname/domain macros and the smart relay host section for the DS macro, then reload with refresh -s sendmail.

Why is mail stuck in the queue on my p595?

The usual cause is no smart relay (so direct port-25 delivery is firewalled) or an inconsistent hostname/domain that triggers a routing loop. Set the DS relay macro, enable the ruleset-0 forwarding rule, then force a queue run with sendmail -q -v and read the SMTP response codes it prints to find the exact failure.

Is sendmail on AIX an open relay risk?

It can be if you over-permit relaying. Restrict which networks may send through the server using /etc/mail/relay-domains and the access database, never relay for arbitrary external senders, and verify with an external relay-test before going live to avoid getting blacklisted.

For more sysadmin walkthroughs and hands-on infrastructure videos, subscribe on YouTube @explorenystream.