Creating a Linux service with systemd

— ny_wk

Ever found yourself needing a script or application to run reliably in the background on your Linux server, always available, always restarting itself if something goes wrong, and surviving reboots like a champ? That's where creating a Linux service with systemd comes into play. Forget hacky cron jobs for continuous processes or manual restarts after a crash. Learning to create a custom systemd service is a fundamental skill for any DevOps engineer or system administrator, transforming how you manage long-running applications on your server.

This comprehensive guide will walk you through the process, from understanding systemd's core principles to defining, managing, and troubleshooting your own services. We'll explore how to build a robust service definition that ensures your application stays up and running, no matter what.

Understanding systemd: The Heartbeat of Modern Linux

Chances are, if you're working with any modern Linux distribution – be it Ubuntu, Debian, CentOS, Fedora, or even Arch – you're interacting with systemd every single day, often without even realizing it. Systemd is the default initialization system (or "init system") on most contemporary Linux systems, a powerful suite of tools designed to manage system processes after the kernel has booted up. Think of it as the maestro orchestrating all the services and applications that make your server functional.

Originally developed by the Red Hat team, systemd replaced the older, more complex `SysVinit` system. It brought a fresh approach, focusing on parallelism, dependency management, and a unified control interface. Before systemd, managing services could be a bit of a scattered affair, with different scripts and conventions across various distributions. Systemd standardized this, providing a consistent way to start, stop, enable, disable, and monitor services across virtually all major Linux flavors.

Beyond simply starting processes, systemd offers a wealth of features:

• Bootstrapping System: It's responsible for bringing your system up from the kernel to a fully operational state, launching all necessary background services like networking, logging, and more.
• Service Management: This is our focus today. It allows you to define, control, and monitor individual services (also known as "unit files").
• Dependency Handling: Services can be configured to start only after specific prerequisites (like network connectivity or another database service) are met.
• Process Supervision: It can monitor your running processes and automatically restart them if they crash or stop unexpectedly, ensuring high availability.
• Resource Control: You can set limits on CPU, memory, and I/O for individual services.
• Logging: `journald`, a core component of systemd, centralizes logs from all services, making troubleshooting much easier.
• Timers: It provides a powerful alternative to `cron` for scheduling tasks.

Many commonly used software tools, such as SSH, Apache, Nginx, and databases like MySQL or PostgreSQL, ship with their own systemd service files. This consistency means that once you understand how to create your own custom systemd service, you've unlocked a powerful skill applicable across your entire Linux environment.

Why Create a Custom systemd Service? The Power of Persistence

Imagine you have a crucial PHP script, a Python daemon, or a custom application that needs to be running 24/7 on your server. Initially, you might just run it directly from the terminal:

$ php server.php

Works fine, right? But what happens if your SSH session disconnects? What if the script crashes due to an unexpected error? And most importantly, what happens when your server reboots? Your script will be gone, lost in the ether, and your application will be down until you manually intervene. This is definitely not the "always-on" behavior we want for production-grade services, yaar.

This is precisely where systemd shines. A custom systemd service provides a robust, standardized, and automated solution for managing any long-running script or process. It allows you to:

• Start Automatically on Boot: No more manual intervention after a server restart.
• Ensure Continuous Operation: Configure it to automatically restart if the process fails, crashes, or exits unexpectedly.
• Easily Manage State: Use simple commands to start, stop, restart, and check the status of your script.
• Standardized Interface: Interact with your custom service using the same `systemctl` commands you use for SSH or Apache.
• Dependency Management: Ensure your script only starts when its prerequisites (like the network or a database) are ready.

Let's illustrate this with a simple, yet practical, example: a small PHP server that listens on a UDP port, receives messages, and sends back a ROT13-encoded reply. This is a perfect candidate for a systemd service because it's a daemon-like process that needs to be constantly available.

Our Example Application: A Simple PHP UDP Server

Let's create a minimal PHP server that will listen to UDP port 10000. It's a fun little example that clearly demonstrates a long-running process.

Create a file named server.php with the following content:

<?php
$sock = socket_create(AF_INET, SOCK_DGRAM, SOL_UDP);
socket_bind($sock, '0.0.0.0', 10000);

echo "UDP server listening on 0.0.0.0:10000\n"; // Added for clarity

for (;;) {
    socket_recvfrom($sock, $message, 1024, 0, $ip, $port);
    $reply = str_rot13($message);
    socket_sendto($sock, $reply, strlen($reply), 0, $ip, $port);
    echo "Received from $ip:$port: '$message' - Replied with '$reply'\n"; // Added for logging
}
?>

Before we turn this into a service, let's test it. First, make sure you have PHP installed:

$ php -v

Then, run your server script:

$ php server.php

You should see "UDP server listening on 0.0.0.0:10000". Now, open *another* terminal and send a UDP message using `netcat` (nc):

$ nc -u 127.0.0.1 10000
Hello, world!

Press Enter. In the server terminal, you'll see the message and reply. In your `nc` terminal, you should get:

Uryyb, jbeyq!

Cool, it works! Now, if you close the terminal where `php server.php` is running, the server stops. This is exactly the problem we're solving with systemd. Let's make this script resilient.

Crafting Your systemd Unit File: The Blueprint for Your Service

The core of any systemd service is its "unit file." This is a plain text configuration file that tells systemd everything it needs to know about how to manage your application. These files typically reside in /etc/systemd/system/ or /usr/lib/systemd/system/. For custom services, /etc/systemd/system/ is the preferred location as it takes precedence over the system-provided units and won't be overwritten by package updates.

Let's create our service definition file for the ROT13 server. We'll name it rot13.service (the `.service` extension is crucial) and place it in /etc/systemd/system/:

$ sudo nano /etc/systemd/system/rot13.service

Now, paste the following content, paying close attention to the placeholders:

[Unit]
Description=ROT13 UDP Demo Service
After=network.target

[Service]
Type=simple
Restart=always
RestartSec=5
User=your_username_here
Group=your_group_here
ExecStart=/usr/bin/env php /path/to/server.php
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Chalo, let's break down each section and directive like a senior engineer explaining the blueprint of a complex system.

[Unit] Section: Defining Your Service's Identity and Dependencies

This section provides metadata about your service and defines its relationship with other systemd units.

• Description=ROT13 UDP Demo Service: This is a human-readable description of your service. It appears when you run systemctl status rot13, so make it clear and informative.
• After=network.target: This is a crucial dependency directive. It tells systemd that our rot13.service should only attempt to start *after* the network.target unit has been fully initialized. In simple terms, don't start our UDP server until the network stack is up and running. If your application needs a database, you might add After=mysql.service or After=postgresql.service. For services that require *active* network connections (not just the interface being up), consider After=network-online.target.

[Service] Section: How to Run and Manage Your Application

This is where you define the core behavior of your service, including how it starts, how it's restarted, and under what user it runs.

• Type=simple: This is the most common and straightforward type for services that run a single process in the foreground. The process specified by ExecStart is the main process of the service. Other types include:
  • forking: For traditional UNIX daemons that fork a child process and the parent exits. Systemd then monitors the child.
  • oneshot: For scripts that run once and exit successfully, like a cleanup script.
  • notify: For services that explicitly notify systemd when they are ready using `sd_notify()`.
  • idle: Similar to `simple`, but execution is delayed until all jobs are dispatched. Useful for starting services with low priority.
• Restart=always: This is super important for daemon-like services. It instructs systemd to *always* restart your service if it exits for any reason (crashes, killed, or exits cleanly). This ensures high availability. Other useful values:
  • on-failure: Restarts only if the service exits with a non-zero status (indicating an error).
  • on-success: Restarts only if the service exits cleanly (exit status 0).
  • on-abnormal: Restarts if killed by a signal (e.g., `SIGTERM`, `SIGKILL`).
  • no: (Default) Never restarts the service.
• RestartSec=5: When `Restart=` is set, this directive specifies the number of seconds systemd should wait before attempting a restart. A value of `1` or `5` seconds is usually good; avoid `0` to prevent thrashing your system with rapid restart attempts during a persistent failure.
• User=your_username_here and Group=your_group_here: This is critical for security! You should *never* run services as `root` unless absolutely necessary (e.g., a service that needs to bind to a port below 1024). Always create a dedicated, unprivileged user and group for your services (e.g., `www-data` for web servers, or a custom user like `rot13user`). This principle of "least privilege" is fundamental in secure system design.
  To create a new user: $ sudo useradd -r -s /bin/nologin rot13user (-r creates a system user, -s /bin/nologin prevents login).
  Remember to change ownership of your script if needed: $ sudo chown rot13user:rot13user /path/to/server.php.
• ExecStart=/usr/bin/env php /path/to/server.php: This is the command systemd executes to start your service.
  • Always use the full path to your executable (e.g., `/usr/bin/php` or `/usr/bin/env php`). Using `env` is generally preferred as it uses the PATH variable to find the executable, making it more flexible.
  • The script path (`/path/to/server.php`) must also be the absolute path. Don't use relative paths here.
  • You can also define ExecStop= for commands to gracefully stop the service, and ExecReload= for reloading its configuration without a full restart. If not specified, systemd sends `SIGTERM` to `ExecStart` processes to stop them.
• StandardOutput=journal and StandardError=journal: These directives are important for logging. By default, `stdout` and `stderr` go to `/dev/null` for `simple` type services. Setting them to `journal` ensures that all output from your script (e.g., our `echo` statements in `server.php`) is directed to systemd's journal, which you can then inspect with `journalctl`. This is a much better practice than writing logs to arbitrary files, as it centralizes all logs.

[Install] Section: Enabling Your Service for Automatic Boot

This section defines how your service should be integrated into systemd's target system, primarily for enabling it to start at boot.

• WantedBy=multi-user.target: This tells systemd that our service is "wanted by" the `multi-user.target`. The `multi-user.target` represents a state where the system is fully booted and ready for multiple users, but without a graphical interface (think your typical server environment). When you `enable` your service, systemd creates a symbolic link from a subdirectory of `multi-user.target.wants/` to your service file, ensuring it's started when the system reaches this target.
  • graphical.target: If your service requires a graphical environment, you might use this.
  • basic.target: A minimal system state.

Avoiding the Restart Trap: `StartLimitIntervalSec` and `StartLimitBurst`

Here's a common pitfall that can catch even experienced folks, including myself, many times. By default, when you configure Restart=always, systemd has a built-in "start limit" mechanism to prevent a constantly crashing service from consuming all system resources or endlessly restarting. This means systemd will *give up* trying to restart your service if it fails to start more than 5 times within a 10-second interval. And once it gives up, it won't try again until you manually intervene.

These limits are controlled by two directives in the [Unit] section:

• StartLimitBurst=5 (default value)
• StartLimitIntervalSec=10 (default value)

If your service crashes quickly 5 times within 10 seconds, systemd marks it as "failed" and stops trying. This is usually not what you want for critical services that must *always* be available. To ensure systemd attempts to restart your service indefinitely, even if it crashes repeatedly, you can simply set:

[Unit]
# ... other directives ...
StartLimitIntervalSec=0

Setting StartLimitIntervalSec=0 effectively disables the start limit, allowing systemd to attempt restarts forever. It's a good idea to combine this with a reasonable RestartSec= (like 1 to 5 seconds) to avoid immediate, continuous restarts that could stress your server or fill up logs during a prolonged failure. We've already included RestartSec=5 in our example, which is a sensible delay.

Alternatively, if you *do* want to acknowledge a persistent failure but don't want systemd to give up forever, you can use StartLimitAction=reboot. This would instruct systemd to reboot the entire system if the start limit is reached, which might be desirable for certain critical infrastructure components. However, for most application services, `StartLimitIntervalSec=0` is the more common and practical solution.

Managing Your Service: The `systemctl` Command

Once you've created your .service file, you need to tell systemd about it and then manage its lifecycle. The primary command for interacting with systemd services is `systemctl`.

First, after creating or modifying any unit file, you *must* reload the systemd manager configuration so it becomes aware of your changes:

$ sudo systemctl daemon-reload

Forgetting this step is another common source of "yaar, why isn't my service working?" moments.

Now, let's bring our `rot13.service` to life:

Starting Your Service

To start your service immediately:

$ sudo systemctl start rot13.service

You can omit the `.service` extension if it's clear from the context.

Checking Service Status

To see if your service is running, its process ID (PID), recent logs, and any errors:

$ systemctl status rot13.service

This command is your best friend for quickly diagnosing issues.

Enabling Auto-Start on Boot

To configure your service to automatically start every time your server boots up:

$ sudo systemctl enable rot13.service

This creates the necessary symbolic link in `multi-user.target.wants/`. You'll see output confirming the symlink creation.

Stopping Your Service

To stop your running service:

$ sudo systemctl stop rot13.service

Disabling Auto-Start

To prevent your service from starting on boot (but keep it available for manual starting):

$ sudo systemctl disable rot13.service

Restarting Your Service

To stop and then start your service again:

$ sudo systemctl restart rot13.service

Viewing Logs with `journalctl`

Since we set `StandardOutput=journal` and `StandardError=journal`, all the output from our PHP script (e.g., "Received from...", "Replied with...") goes directly into systemd's journal. To view logs specifically for your service:

$ journalctl -u rot13.service

To see the latest logs and follow new entries in real-time (like `tail -f`):

$ journalctl -f -u rot13.service

This is immensely powerful for debugging!

Beyond the Basics: Advanced systemd Concepts and Best Practices

While the `rot13.service` example covers the fundamental aspects of creating a Linux service with systemd, systemd is a powerhouse with many advanced features. Here are a few to keep in mind for more complex or production-critical services:

Environment Variables

If your script needs specific environment variables, you can define them in the `[Service]` section:

[Service]
# ...
Environment="APP_ENV=production" "DATABASE_URL=mysql://user:pass@host/db"
EnvironmentFile=/etc/sysconfig/my-app # Or a file containing KEY=VALUE pairs

Using `EnvironmentFile` is often preferred for sensitive or extensive environment configurations, as it keeps your main unit file cleaner.

Resource Control

For services that might be resource hogs, systemd integrates with Control Groups (cgroups) to limit resource usage:

[Service]
# ...
CPUShares=512 # Default is 1024, lower means less CPU
MemoryLimit=256M # Limit memory usage to 256MB
IOReadBandwidthMax=/dev/sda 10M # Limit disk read bandwidth

Security and Sandboxing

Systemd offers robust sandboxing capabilities to isolate services and enhance security:

• PrivateTmp=true: Provides a private `/tmp` and `/var/tmp` directory for the service, preventing it from interacting with other services' temporary files.
• ProtectSystem=full: Makes `/usr`, `/boot`, and `/etc` read-only for the service.
• ProtectHome=true: Makes `/home`, `/root`, and `/run/user` inaccessible to the service.
• NoNewPrivileges=true: Prevents the service from gaining new privileges.
• CapabilityBoundingSet=CAP_NET_BIND_SERVICE: Explicitly defines kernel capabilities the service is allowed to use.

For production services, always consider applying appropriate sandboxing to minimize the blast radius of a compromised application. You can explore a comprehensive list of these directives in the `systemd.exec` man page.

The `test_service.sh` Example and `systemd-cat`

The source also mentions a `test_service.sh` script using `systemd-cat`. This is another excellent way to log output to the journal, especially from simple shell scripts. `systemd-cat` acts as a pipe, routing `stdin` to the systemd journal.

Consider this script:

#!/bin/bash
DATE=`date '+%Y-%m-%d %H:%M:%S'`
echo "Example service started at ${DATE}" | systemd-cat -p info # Logs to journal with 'info' priority

while :
do
    echo "Looping..." | systemd-cat -p debug # Logs "Looping..." with 'debug' priority
    sleep 30
done

If you were to use this script in an `ExecStart`, the `systemd-cat` commands within it would ensure that "Example service started..." and "Looping..." messages appear in your `journalctl -u mytestservice.service` output, categorized by priority (`info`, `debug`). This gives you fine-grained control over what gets logged and at what severity level, helping you keep your logs clean and useful for troubleshooting Linux performance and application issues.

Conclusion: Your Services, Managed with Confidence

That's all it takes, folks! By understanding the structure of a systemd unit file and the power of `systemctl` commands, you can reliably run and manage any custom application or script on your Linux servers. Whether it's a critical microservice, a background processing daemon, or a simple utility script, systemd provides the robust framework to ensure high availability and effortless management.

This skill is not just about making your life easier; it's about building more resilient, self-healing, and professionally managed systems. So, go forth and turn those ad-hoc scripts into proper, dependable Linux services. Your future self (and your fellow DevOps engineers) will thank you!

Key Takeaways

• systemd is the default init system on most modern Linux distributions, managing system processes and services.
• A custom systemd service ensures your scripts/applications start automatically on boot, restart on failure, and are easily managed via `systemctl`.
• Unit files (`.service`) define your service's behavior, residing typically in /etc/systemd/system/ and comprising `[Unit]`, `[Service]`, and `[Install]` sections.
• Key directives include `Description`, `After` (for dependencies), `Type` (e.g., `simple`), `Restart` (e.g., `always`), `RestartSec`, `User`/`Group` (for security), and `ExecStart` (the command to run).
• Avoid the restart trap by setting `StartLimitIntervalSec=0` in the `[Unit]` section, combined with a sensible `RestartSec`, to ensure indefinite restart attempts.
• `systemctl` commands (`daemon-reload`, `start`, `stop`, `restart`, `enable`, `disable`, `status`) are your primary tools for managing services.
• `journalctl -u your-service.service` is essential for viewing centralized logs from your running service, especially when `StandardOutput=journal` is configured.
• Prioritize security by running services under dedicated, unprivileged users and groups, and explore advanced sandboxing options like `PrivateTmp=true`.

Frequently Asked Questions

What is a systemd unit file and where should I put it?

A systemd unit file is a plain text configuration file that describes a systemd unit, such as a service, socket, or mount point. For custom services, you should place your `.service` file in /etc/systemd/system/. This location is designed for administrator-created or custom unit files, and it takes precedence over unit files installed by packages in /usr/lib/systemd/system/, ensuring your custom configurations are not overwritten during system updates.

How do I make my systemd service start automatically on boot?

To configure your systemd service to start automatically on system boot, you need to use the `systemctl enable` command. First, ensure your service unit file (e.g., `my-app.service`) has the `[Install]` section with `WantedBy=multi-user.target` (or `graphical.target` if needed). Then, run: $ sudo systemctl enable my-app.service. This creates a symbolic link that systemd uses during the boot process to start your service when the system reaches the specified target state.

My systemd service isn't starting/restarting, what should I check?

If your systemd service isn't behaving as expected, start by checking its status and logs. Use $ systemctl status your-service.service to get a quick overview of its current state, recent error messages, and PID. Immediately follow up with $ journalctl -xeu your-service.service to view detailed logs, including verbose error messages. Common issues include incorrect paths in `ExecStart`, missing dependencies (check `After=` directives), incorrect file permissions for the script, the service user lacking necessary privileges, or forgetting to run `sudo systemctl daemon-reload` after editing the unit file. Also, verify if the `StartLimitIntervalSec` trap might be preventing restarts.

Can systemd manage one-off tasks or scheduled jobs like cron?

Yes, systemd can manage both one-off tasks and scheduled jobs. For one-off tasks that run a script and then exit, you can set `Type=oneshot` in your `[Service]` unit. For scheduled jobs, systemd offers `systemd.timer` units as a powerful alternative to `cron`. Timer units allow you to define schedules (e.g., `OnCalendar=`, `OnBootSec=`, `OnUnitActiveSec=`) that trigger other systemd units (like a service unit running a script). This offers better integration with the rest of systemd's logging, dependencies, and resource control features compared to traditional cron jobs.

We've covered a lot today, from the foundational understanding of systemd to the nitty-gritty of crafting resilient services. To see some of these concepts in action and get further insights, make sure to watch the full video on this topic. It’s an excellent visual companion to this guide!

Watch the original video on YouTube: @explorenystream