Kubernetes - Low disk space alerts

— ny_wk

Disclosure: some links above are affiliate links — if you buy through them I may earn a small commission at no extra cost to you. Thanks for supporting the channel!

Running into **Kubernetes low disk space alerts** can be a real headache, often leading to service disruptions if not addressed swiftly. This comprehensive guide will walk you through the essential steps to diagnose, troubleshoot, and effectively resolve disk space issues within your Kubernetes pods, particularly focusing on common culprits like growing log files and the tricky `df` vs `du` discrepancies.

Understanding Kubernetes Low Disk Space Alerts: Why They Matter

Imagine you're running a critical application in a Kubernetes pod. Suddenly, your monitoring system starts screaming about low disk space. What happens next? Potentially, the pod might get evicted, the application could crash, or worse, data corruption might occur. This isn't just an inconvenience; it's a direct threat to your application's availability and stability. Kubernetes pods, by default, often utilize ephemeral storage (local disk on the node) for things like application binaries, temporary files, and crucially, logs. While convenient, this ephemeral nature means that if an application generates a lot of data, that storage can quickly fill up. Alerts are usually triggered by metrics collected from tools like Prometheus and visualized in Grafana, warning you when disk usage crosses predefined thresholds (e.g., 80% or 90%). Ignoring these alerts is like ignoring a ticking time bomb. High disk utilization can lead to: * **Pod Eviction:** The Kubernetes scheduler might evict pods from a node if that node's ephemeral storage reaches capacity, leading to downtime for your application. * **Application Crashes:** Many applications need scratch space or rely on writing logs to function correctly. If disk space runs out, they'll fail. * **Performance Degradation:** Disk-intensive operations can slow down significantly on a full disk, impacting user experience. * **Unreliable Operations:** Databases, message queues, and other stateful applications are particularly sensitive to disk space and can become unstable or corrupt data. Common culprits for disk space hogging include: * **Log files:** Especially verbose applications or those with misconfigured log rotation can fill up disks quickly. * **Temporary files:** Applications might create temporary files that aren't properly cleaned up. * **Old application versions/artifacts:** Sometimes, deployments might leave old code or build artifacts behind. * **Cache data:** Caches that grow unbounded can also be a significant issue. The key is to act fast and systematically. Don't just delete files blindly; understand *what* is consuming space and *why*, then apply a sustainable solution.

Diagnosing Disk Usage in a Kubernetes Pod: The Initial Investigation

When a low disk space alert hits, your first step is to get inside the affected pod and see what's going on. This is where your kubectl skills come in handy.

Getting Started: Finding Your Pod

Before you can troubleshoot, you need to identify the specific pod causing the issue. You'll typically get the pod name or deployment name from your alert, but if not, you can list pods in the relevant namespace. ```bash kubectl get pods -n ``` Replace `` with the actual namespace your application resides in. For example, if your Vault application is in the `vault` namespace, you'd run: ```bash kubectl get pods -n vault ``` This command will list all pods in that namespace, showing their status, age, and ready state. Find the pod that's likely causing the issue – often, it's one belonging to a deployment that's logging excessively or has been flagged by your monitoring. In our example, let's assume it's `vault-0`.

Executing into the Pod: Your Entry Point

Once you have the pod name, you need to "exec" into it. This allows you to run commands inside the container as if you were SSHed into a traditional server. ```bash kubectl exec -it -n

-- /bin/sh ``` Let's break this down: * `kubectl exec`: The command to execute a command in a container. * `-it`: This combines `-i` (interactive) and `-t` (TTY – pseudo-terminal). It's crucial for getting an interactive shell experience. Without it, your commands won't execute interactively, and you won't see output or be able to type. * `-n `: Specifies the namespace where the pod is running. * ``: The name of your target pod, e.g., `vault-0`. * `--`: This is a separator. Any arguments after `--` are passed directly to the command being executed inside the container. * `/bin/sh`: The shell you want to open inside the container. Sometimes `/bin/bash` might be available, but `/bin/sh` is generally more common and guaranteed to be present in most minimal container images. So, for our Vault example, you'd run: ```bash kubectl exec -it -n vault vault-0 -- /bin/sh ``` Now you're inside the pod, ready to investigate!

The Go-To Commands: `df` and `du`

Inside the pod, the first two commands you'll reach for are `df` and `du`. They both deal with disk space, but they tell you different things, and understanding their distinction is absolutely crucial.

`df -Th`: Filesystem Disk Space Report

The `df` command (disk filesystem) reports disk space usage for filesystems mounted on the system. The options here are important: * `-T`: Displays the filesystem type (e.g., `ext4`, `xfs`). * `-h`: Shows sizes in human-readable format (e.g., `G` for gigabytes, `M` for megabytes). Run this command first: ```bash df -Th ``` You'll see output similar to this: ``` Filesystem Type Size Used Avail Use% Mounted on overlay overlay 9.8G 9.0G 312.4M 97% / tmpfs tmpfs 64M 0 64M 0% /dev tmpfs tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup /dev/vdb ext4 9.8G 9.0G 312.4M 97% /vault/audit shm tmpfs 64M 0 64M 0% /dev/shm tmpfs tmpfs 3.9G 12K 3.9G 1% /run/secrets/kubernetes.io/serviceaccount ``` **Interpreting `df` output:** * **`Filesystem`**: The name of the filesystem. * **`Type`**: The type of filesystem (e.g., `ext4`). * **`Size`**: Total size of the filesystem. * **`Used`**: Space currently used. * **`Avail`**: Space available for use. * **`Use%`**: Percentage of the filesystem used. This is often the metric that triggers your alerts. * **`Mounted on`**: The directory where this filesystem is mounted. From the example, we can see that `/dev/vdb` mounted on `/vault/audit` is at 97% usage! This immediately tells us where the problem area is – the `/vault/audit` directory.

`du -shc /*`: Directory Disk Usage Report

The `du` command (disk usage) estimates file space usage. Unlike `df`, `du` checks the size of files and directories from the perspective of what's *actually on disk* by traversing the file tree. * `-s`: Summarizes usage for each argument. * `-h`: Human-readable format. * `-c`: Displays a grand total. * `/*`: Check all top-level directories. Now, run this to find out which specific directory within the problem mount point is consuming space: ```bash du -shc /* ``` The output might look something like this (simplified): ``` 20.0K /bin 4.0K /dev ... 9.0G /vault ... 9.1G total ``` This tells us that `/vault` is consuming almost all the space. Since `df` pointed to `/vault/audit`, let's `cd` into `/vault` and then drill down further. ```bash cd /vault du -shc * ``` This will give you a more granular view: ``` 20.0K audit 12K bin ... 9.0G audit/vault_audit.log ... ``` Ah, `vault_audit.log`! Now we have our culprit. The `df` command told us *where* the filesystem was full (`/vault/audit`), and `du` helped us pinpoint the *exact file* within that directory (`vault_audit.log`).

Crucial Insight: When `df` and `du` Disagree

Sometimes, you might find that `df -Th` shows high disk usage (e.g., 97%), but `du -shc /*` reports significantly less (e.g., only 5GB used on a 10GB partition). This discrepancy is a critical diagnostic indicator, often pointing to a specific issue: **deleted files that are still held open by a running process.** When a file is deleted, its entry is removed from the directory structure, but the disk space isn't freed until all processes that have an open file handle to it release that handle. We'll cover how to handle this in a later section. For now, if `df` and `du` agree, you're on the right track!

Tackling the Root Cause: Managing Large Log Files (Vault Audit Log Example)

Now that you've identified a large log file, like `vault_audit.log`, consuming excessive disk space, it's time to resolve the issue. Our strategy will involve archiving the log, moving it off the pod, and then safely deleting it.

Identifying the Culprit (Revisited)

We've already established that `vault_audit.log` in `/vault/audit` is the problem. Ensure you are in the correct directory within the pod: ```bash cd /vault/audit ``` You can run `ls -lah` here to confirm the file's size and existence. ```bash ls -lah ```

Solution Strategy: Archiving and Deleting Log Files

This process assumes you need to retain the log file for compliance, auditing, or debugging purposes. If it's truly ephemeral and not needed, you could skip the transfer steps.

Step 1: Renaming for Archiving

To prevent the application from continuing to write to the massive log file while you're dealing with it, and to prepare it for archiving, rename it. Use a date-stamped format so you know exactly when it was archived. ```bash mv vault_audit.log vault_audit_MM-DD-YYYY.log ``` Replace `MM-DD-YYYY` with the current date. For example: ```bash mv vault_audit.log vault_audit_10-21-2021.log ``` **Why rename?** 1. The application will typically create a *new* `vault_audit.log` file, allowing it to continue logging without interruption to the old, large file. This ensures your service keeps running while you manage the old log. 2. The timestamp makes it clear which archived log you are dealing with. After renaming, you can confirm with `ls -lah` that the new name is visible and potentially a new (small) `vault_audit.log` has been created by the application.

Step 2: Securely Transferring the File Out of the Pod

Now, you need to get this renamed log file from the Kubernetes pod to your local machine (or an intermediate jump host) before it can be uploaded to long-term storage. You'll need a new terminal window for this, as your current terminal is "exec'd" into the pod. ```bash kubectl cp /:/path/to/pod/file /local/path/on/your/machine ``` Using our example: ```bash kubectl cp vault/vault-0:/vault/audit/vault_audit_10-21-2021.log /home/joe/vault_audit_10-21-2021.log ``` * **`vault/vault-0`**: Combines the namespace (`vault`) and pod name (`vault-0`). * **`/vault/audit/vault_audit_10-21-2021.log`**: The full path to the renamed log file *inside the pod*. * **`/home/joe/vault_audit_10-21-2021.log`**: The destination path and filename on your *local machine*. **Monitoring the transfer:** For large files, `kubectl cp` can take a while. To see its progress in real-time, open yet another terminal tab and navigate to your local destination folder. ```bash cd /home/joe watch ls -lah ``` The `watch` command will continuously execute `ls -lah` every 2 seconds by default, showing you the growing size of the file as it transfers. Once the file size stabilizes and matches the size reported by `ls -lah` inside the pod (before renaming), the transfer is complete.

Step 3: Offloading to S3 (or Similar Cloud Storage)

Once the file is safely on your local machine, it's time to upload it to long-term, cost-effective cloud storage like AWS S3. This step typically involves logging into your cloud provider's console or using their CLI tools. For AWS S3, you would: 1. Log in to the AWS Management Console. 2. Navigate to the S3 service. 3. Locate the appropriate S3 bucket and desired prefix (folder structure), e.g., `metricgaming-eu-west-2-prod-logs > topics/ > vault-audit-log/`. 4. Upload the `vault_audit_10-21-2021.log` file from your local machine to this location. If you prefer command-line tools for automation or larger files, you could use the AWS CLI: ```bash aws s3 cp /home/joe/vault_audit_10-21-2021.log s3://metricgaming-eu-west-2-prod-logs/topics/vault-audit-log/vault_audit_10-21-2021.log ``` This ensures the file is preserved for future needs and frees up your local machine's space.

Step 4: Deleting the Archived File from the Pod

With the log file safely offloaded to S3, you can now delete the original archived file from the pod. Switch back to your terminal where you are "exec'd" into the pod. ```bash rm vault_audit_10-21-2021.log ``` **Verification is key:** After deletion, immediately run `df -Th` to confirm the disk space has been freed up. You should see a significant drop in `Use%` for the `/vault/audit` mount point. ``` Filesystem Type Size Used Avail Use% Mounted on /dev/vdb ext4 9.8G 36.0M 9.2G 0% /vault/audit ``` This looks much better! The disk space has been reclaimed. Finally, ensure your application is happily writing to a new, smaller `vault_audit.log` by checking the directory again: ```bash watch ls -lah ``` You should see a `vault_audit.log` file present, and its size should be growing steadily but slowly, indicating normal operation.

When `df` and `du` Disagree: Handling Deleted Files Held Open by Processes

This is where things get a bit more advanced, but it's a very common scenario in containerized environments. As we discussed earlier, if `df -Th` shows high disk usage but `du -shc /*` (or `du -shc *` after `cd`ing into problem directories) reports significantly less, it's almost certainly because files have been deleted from the filesystem but are still being held open by a running process. This prevents the operating system from actually freeing up the disk blocks.

The Problem Explained

When a process opens a file, it gets a file descriptor. Even if another process (or you, using `rm`) deletes that file, the actual data blocks on the disk won't be released until *all* processes holding an open descriptor to that file close it. In Kubernetes, this often happens with logging. A log file might be periodically rotated and deleted by a log rotation tool, but if the application process continues to write to the *old, deleted inode* (the file descriptor it initially opened), the space remains consumed.

Identifying Open Deleted Files with `lsof`

To find these phantom files, we use the `lsof` (list open files) command. ```bash lsof 2>/dev/null | grep deleted ``` Let's break this down: * `lsof`: Lists all open files and the processes that own them. * `2>/dev/null`: Redirects standard error (usually permission denied errors) to `/dev/null` to keep the output clean. * `grep deleted`: Filters the output to show only lines containing the string "deleted". The output will look something like this: ``` COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME vault 12 vault 3w REG 259,1 912345678 12345 /vault/audit/vault_audit.log.old (deleted) ``` **Interpreting `lsof` output:** * **`COMMAND`**: The name of the process holding the file. * **`PID`**: The Process ID of the holding process. This is what we need! * **`USER`**: The user running the process. * **`FD`**: File Descriptor. `3w` means file descriptor 3, opened for writing. * **`TYPE`**: Type of file (REG for regular file). * **`DEVICE`, `SIZE/OFF`, `NODE`**: Technical details about the file. * **`NAME`**: The name and path of the file, crucially followed by `(deleted)`. In our example, the `vault` process with `PID 12` is holding open a `vault_audit.log.old` file that has been deleted. This is the reason `df` still reports high usage!

The Solution: Restarting/Killing the Holding Process

The only way to free up the disk space held by these deleted files is for the process holding them to release its file descriptor. This usually means restarting the application or, as a last resort, killing the process. **Caution is paramount here.** Killing a process can lead to application downtime. Always explore graceful restart options first, if available for your application or Kubernetes deployment. For example, if it's a deployment, you might consider `kubectl rollout restart deployment/` which gracefully terminates and restarts pods. However, if you're in an emergency situation and a graceful restart isn't immediately possible or effective, or if it's a standalone process, you might need to kill it. Identify the PID from the `lsof` output (e.g., `12` for the `vault` process). ```bash kill -9 12 ``` * `kill`: The command to send signals to processes. * `-9`: The `SIGKILL` signal. This is a "non-catchable, non-ignorable kill." It forcefully terminates the process. Use it with care! A more robust way to kill all processes holding deleted files (if you're sure you want to do this for all of them in the container): ```bash lsof | grep "deleted" | awk '{print $2}' | xargs kill -9 ``` Let's break this powerful one-liner: * `lsof | grep "deleted"`: Finds all lines with "deleted" files. * `awk '{print $2}'`: Extracts the second column from each line, which is the PID. * `xargs kill -9`: Takes the PIDs from `awk` and passes them as arguments to `kill -9`. After killing the process, it's highly likely your application in that pod will restart (if managed by a Deployment, StatefulSet, etc.) or simply stop. If it's a stateful application, be prepared for recovery procedures.

Verification

Once the process holding the deleted file is terminated, the disk space should be freed. Verify this immediately with `df -Th`: ```bash df -Th ``` You should now see the `Use%` drop significantly for the affected filesystem. ``` Filesystem Type Size Used Avail Use% Mounted on /dev/vdb ext4 9.8G 36.0M 9.2G 0% /vault/audit ``` This confirms the disk space has been reclaimed.

Proactive Measures and Best Practices to Prevent Low Disk Space

While reactive troubleshooting is essential, implementing proactive measures is key to preventing these incidents in the first place. * **Log Rotation Policies:** This is probably the most crucial step. Configure your applications or use tools like `logrotate` within containers to automatically rotate, compress, and delete old log files. For Kubernetes, consider configuring your container runtime's logging driver (e.g., Docker's `json-file` driver has `max-size` and `max-file` options) or integrating with a sidecar pattern for log management. * **Centralized Logging Solutions:** Instead of storing logs directly in pods, push them to a centralized logging system like Elasticsearch, Splunk, Loki, or Graylog. This decouples log storage from pod ephemeral storage, making pods leaner and more resilient. Tools like Fluentd or Filebeat can be deployed as DaemonSets to collect and forward logs efficiently. * **Monitoring and Alerting:** Ensure you have robust monitoring in place (e.g., Prometheus with node-exporter) to collect disk usage metrics and configure alerts in Grafana or Alertmanager to notify you well before critical thresholds are reached. Monitor both `Use%` and inode usage. * **Resource Requests and Limits (Ephemeral Storage):** For Kubernetes 1.8+, you can set `ephemeral-storage` requests and limits in your pod definitions. This allows the scheduler to ensure nodes have enough local storage for pods and prevents a single rogue pod from consuming all node storage. ```yaml resources: limits: cpu: "1" memory: "1Gi" ephemeral-storage: "5Gi" # Limit ephemeral storage to 5GB requests: cpu: "500m" memory: "512Mi" ephemeral-storage: "2Gi" # Request 2GB of ephemeral storage ``` * **Persistent Volumes for Stateful Data:** For applications requiring persistent storage (databases, message queues, etc.), always use Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) rather than relying on ephemeral storage. This ensures data survives pod restarts and provides dedicated, expandable storage. * **Application-Level Cleanup:** Implement garbage collection or cleanup routines within your applications for temporary files, cache data, or old build artifacts that might accumulate over time. * **Regular Audits:** Periodically audit your deployments for applications known to be verbose or those that might have misconfigured logging. By adopting these best practices, you can significantly reduce the frequency and impact of Kubernetes low disk space alerts, ensuring your clusters run smoothly and reliably.

Key Takeaways

* **Act Quickly:** Low disk space can lead to pod eviction and service disruption; immediate diagnosis is crucial. * **`df` vs. `du`:** Use `df -Th` to see filesystem usage and identify the mounted point, then `du -shc /*` (and subsequent `cd` and `du`) to pinpoint the exact file or directory consuming space. * **Log Management:** For large log files, rename, transfer out of the pod (e.g., `kubectl cp` to local, then upload to S3), and then safely delete. * **`df`/`du` Discrepancy:** If `df` reports high usage but `du` doesn't, deleted files held open by processes are the likely cause. * **`lsof` and `kill -9`:** Use `lsof | grep deleted` to find processes holding open deleted files, and `kill -9 ` (or the `awk` one-liner) to release the disk space (with caution). * **Proactive Prevention:** Implement log rotation, centralized logging, ephemeral storage limits, and robust monitoring to prevent future disk space issues.

Frequently Asked Questions

What is ephemeral storage in Kubernetes?

Ephemeral storage refers to the temporary, non-persistent local disk space available to a pod. It's tied to the pod's lifecycle and typically resides on the node where the pod runs. If a pod restarts or is rescheduled to another node, any data stored in its ephemeral storage is lost. This storage is commonly used for things like container images, application binaries, and, crucially, logs and temporary files.

Why is `df` showing high disk usage while `du` shows less?

This discrepancy almost always indicates that files have been deleted from the filesystem, but one or more processes are still holding open file descriptors to those deleted files. Until all processes release their hold, the operating system cannot free the disk blocks associated with those files, and `df` will continue to report them as "used" space. `lsof` is the command to diagnose this.

How can I prevent logs from filling up my Kubernetes pods?

The best approaches include implementing log rotation policies within your containers, configuring your container runtime's logging drivers (e.g., `max-size`, `max-file` for Docker), and migrating to a centralized logging solution (like ELK stack, Loki, Splunk) where logs are streamed off the pods to dedicated storage. You can also set ephemeral storage limits in your pod specifications.

Is it safe to `kill -9` a process in a Kubernetes pod?

Executing `kill -9` is a forceful termination and should be considered a last resort. It does not allow the process to shut down gracefully, potentially leading to data corruption or incomplete operations. If the pod is managed by a Deployment or StatefulSet, Kubernetes will likely restart it, but there might be a brief service interruption. Always try to identify the cause, gracefully restart the application or the pod (`kubectl rollout restart deployment/`), and only use `kill -9` if other options are exhausted and you understand the potential impact. Facing disk space issues in Kubernetes can be daunting, but with the right tools and a systematic approach, you can quickly identify and resolve them. The key is to understand your environment, know your commands, and always prioritize proactive measures. For a visual walkthrough of these steps and more practical insights, make sure to check out the original video on the @explorenystream channel. Don't forget to subscribe for more expert DevOps content!