Kubernetes - Low disk space alerts

July 09, 2026 — LiveStream

Running into Kubernetes low disk space alerts can be a real headache, often leading to service disruptions if not addressed swiftly. This comprehensive guide will walk you through the essential steps to diagnose, troubleshoot, and effectively resolve disk space issues within your Kubernetes pods, particularly focusing on common culprits like growing log files and the tricky df vs du discrepancies.

Understanding Kubernetes Low Disk Space Alerts: Why They Matter

Imagine you're running a critical application in a Kubernetes pod. Suddenly, your monitoring system starts screaming about low disk space. What happens next? Potentially, the pod might get evicted, the application could crash, or worse, data corruption might occur. This isn't just an inconvenience; it's a direct threat to your application's availability and stability.

Kubernetes pods, by default, often utilize ephemeral storage (local disk on the node) for things like application binaries, temporary files, and crucially, logs. While convenient, this ephemeral nature means that if an application generates a lot of data, that storage can quickly fill up. Alerts are usually triggered by metrics collected from tools like Prometheus and visualized in Grafana, warning you when disk usage crosses predefined thresholds (e.g., 80% or 90%). Ignoring these alerts is like ignoring a ticking time bomb. High disk utilization can lead to:

• Pod Eviction: The Kubernetes scheduler might evict pods from a node if that node's ephemeral storage reaches capacity, leading to downtime for your application.
• Application Crashes: Many applications need scratch space or rely on writing logs to function correctly. If disk space runs out, they'll fail.
• Performance Degradation: Disk-intensive operations can slow down significantly on a full disk, impacting user experience.
• Unreliable Operations: Databases, message queues, and other stateful applications are particularly sensitive to disk space and can become unstable or corrupt data.

Common culprits for disk space hogging include:

• Log files: Especially verbose applications or those with misconfigured log rotation can fill up disks quickly.
• Temporary files: Applications might create temporary files that aren't properly cleaned up.
• Old application versions/artifacts: Sometimes, deployments might leave old code or build artifacts behind.
• Cache data: Caches that grow unbounded can also be a significant issue.

The key is to act fast and systematically. Don't just delete files blindly; understand what is consuming space and why, then apply a sustainable solution.

Diagnosing Disk Usage in a Kubernetes Pod: The Initial Investigation

When a low disk space alert hits, your first step is to get inside the affected pod and see what's going on. This is where your kubectl skills come in handy.

Getting Started: Finding Your Pod

Before you can troubleshoot, you need to identify the specific pod causing the issue. You'll typically get the pod name or deployment name from your alert, but if not, you can list pods in the relevant namespace.

kubectl get pods -n <namespacename>

Replace <namespacename> with the actual namespace your application resides in. For example, if your Vault application is in the vault namespace, you'd run:

kubectl get pods -n vault

This command will list all pods in that namespace, showing their status, age, and ready state. Find the pod that's likely causing the issue – often, it's one belonging to a deployment that's logging excessively or has been flagged by your monitoring. In our example, let's assume it's vault-0.

Executing into the Pod: Your Entry Point

Once you have the pod name, you need to "exec" into it. This allows you to run commands inside the container as if you were SSHed into a traditional server.

kubectl exec -it -n <namespacename> <pod-name> -- /bin/sh

Let's break this down:

• kubectl exec: The command to execute a command in a container.
• -it: This combines -i (interactive) and -t (TTY – pseudo-terminal). It's crucial for getting an interactive shell experience. Without it, your commands won't execute interactively, and you won't see output or be able to type.
• -n <namespacename>: Specifies the namespace where the pod is running.
• <pod-name>: The name of your target pod, e.g., vault-0.
• --: This is a separator. Any arguments after -- are passed directly to the command being executed inside the container.
• /bin/sh: The shell you want to open inside the container. Sometimes /bin/bash might be available, but /bin/sh is generally more common and guaranteed to be present in most minimal container images.

So, for our Vault example, you'd run:

kubectl exec -it -n vault vault-0 -- /bin/sh

Now you're inside the pod, ready to investigate!

The Go-To Commands: df and du

Inside the pod, the first two commands you'll reach for are df and du. They both deal with disk space, but they tell you different things, and understanding their distinction is absolutely crucial.

df -Th: Filesystem Disk Space Report

The df command (disk filesystem) reports disk space usage for filesystems mounted on the system. The options here are important:

• -T: Displays the filesystem type (e.g., ext4, xfs).
• -h: Shows sizes in human-readable format (e.g., G for gigabytes, M for megabytes).

Run this command first:

df -Th

You'll see output similar to this:

Filesystem     Type   Size  Used Avail Use% Mounted on
overlay        overlay 9.8G  9.0G  312.4M   97% /
tmpfs          tmpfs   64M     0   64M    0% /dev
tmpfs          tmpfs  3.9G     0  3.9G    0% /sys/fs/cgroup
/dev/vdb       ext4   9.8G  9.0G  312.4M   97% /vault/audit
shm            tmpfs   64M     0   64M    0% /dev/shm
tmpfs          tmpfs  3.9G   12K  3.9G    1% /run/secrets/kubernetes.io/serviceaccount

Interpreting df output:

• Filesystem: The name of the filesystem.
• Type: The type of filesystem (e.g., ext4).
• Size: Total size of the filesystem.
• Used: Space currently used.
• Avail: Space available for use.
• Use%: Percentage of the filesystem used. This is often the metric that triggers your alerts.
• Mounted on: The directory where this filesystem is mounted.

From the example, we can see that /dev/vdb mounted on /vault/audit is at 97% usage! This immediately tells us where the problem area is – the /vault/audit directory.

du -shc /*: Directory Disk Usage Report

The du command (disk usage) estimates file space usage. Unlike df, du checks the size of files and directories from the perspective of what's actually on disk by traversing the file tree.

• -s: Summarizes usage for each argument.
• -h: Human-readable format.
• -c: Displays a grand total.
• /*: Check all top-level directories.

Now, run this to find out which specific directory within the problem mount point is consuming space:

du -shc /*

The output might look something like this (simplified):

20.0K	/bin
4.0K	/dev
...
9.0G	/vault
...
9.1G	total

This tells us that /vault is consuming almost all the space. Since df pointed to /vault/audit, let's cd into /vault and then drill down further.

cd /vault
du -shc *

This will give you a more granular view:

20.0K	audit
12K	bin
...
9.0G	audit/vault_audit.log
...

Ah, vault_audit.log! Now we have our culprit. The df command told us where the filesystem was full (/vault/audit), and du helped us pinpoint the exact file within that directory (vault_audit.log).

Crucial Insight: When df and du Disagree

Sometimes, you might find that df -Th shows high disk usage (e.g., 97%), but du -shc /* reports significantly less (e.g., only 5GB used on a 10GB partition). This discrepancy is a critical diagnostic indicator, often pointing to a specific issue: deleted files that are still held open by a running process. When a file is deleted, its entry is removed from the directory structure, but the disk space isn't freed until all processes that have an open file handle to it release that handle. We'll cover how to handle this in a later section. For now, if df and du agree, you're on the right track!

Tackling the Root Cause: Managing Large Log Files (Vault Audit Log Example)

Now that you've identified a large log file, like vault_audit.log, consuming excessive disk space, it's time to resolve the issue. Our strategy will involve archiving the log, moving it off the pod, and then safely deleting it.

Identifying the Culprit (Revisited)

We've already established that vault_audit.log in /vault/audit is the problem. Ensure you are in the correct directory within the pod:

cd /vault/audit

You can run ls -lah here to confirm the file's size and existence.

ls -lah

Solution Strategy: Archiving and Deleting Log Files

This process assumes you need to retain the log file for compliance, auditing, or debugging purposes. If it's truly ephemeral and not needed, you could skip the transfer steps.

Step 1: Renaming for Archiving

To prevent the application from continuing to write to the massive log file while you're dealing with it, and to prepare it for archiving, rename it. Use a date-stamped format so you know exactly when it was archived.

mv vault_audit.log vault_audit_MM-DD-YYYY.log

Replace MM-DD-YYYY with the current date. For example:

mv vault_audit.log vault_audit_10-21-2021.log

Why rename?

1. The application will typically create a new vault_audit.log file, allowing it to continue logging without interruption to the old, large file. This ensures your service keeps running while you manage the old log.
2. The timestamp makes it clear which archived log you are dealing with.

After renaming, you can confirm with ls -lah that the new name is visible and potentially a new (small) vault_audit.log has been created by the application.

Step 2: Securely Transferring the File Out of the Pod

Now, you need to get this renamed log file from the Kubernetes pod to your local machine (or an intermediate jump host) before it can be uploaded to long-term storage. You'll need a new terminal window for this, as your current terminal is "exec'd" into the pod.

kubectl cp <namespace>/<pod-name>:/path/to/pod/file /local/path/on/your/machine

Using our example:

kubectl cp vault/vault-0:/vault/audit/vault_audit_10-21-2021.log /home/joe/vault_audit_10-21-2021.log

• vault/vault-0: Combines the namespace (vault) and pod name (vault-0).
• /vault/audit/vault_audit_10-21-2021.log: The full path to the renamed log file inside the pod.
• /home/joe/vault_audit_10-21-2021.log: The destination path and filename on your local machine.

Monitoring the transfer: For large files, kubectl cp can take a while. To see its progress in real-time, open yet another terminal tab and navigate to your local destination folder.

cd /home/joe
watch ls -lah

The watch command will continuously execute ls -lah every 2 seconds by default, showing you the growing size of the file as it transfers. Once the file size stabilizes and matches the size reported by ls -lah inside the pod (before renaming), the transfer is complete.

Step 3: Offloading to S3 (or Similar Cloud Storage)

Once the file is safely on your local machine, it's time to upload it to long-term, cost-effective cloud storage like AWS S3. This step typically involves logging into your cloud provider's console or using their CLI tools.

For AWS S3, you would:

1. Log in to the AWS Management Console.
2. Navigate to the S3 service.
3. Locate the appropriate S3 bucket and desired prefix (folder structure), e.g., metricgaming-eu-west-2-prod-logs > topics/ > vault-audit-log/.
4. Upload the vault_audit_10-21-2021.log file from your local machine to this location.

If you prefer command-line tools for automation or larger files, you could use the AWS CLI:

aws s3 cp /home/joe/vault_audit_10-21-2021.log s3://metricgaming-eu-west-2-prod-logs/topics/vault-audit-log/vault_audit_10-21-2021.log

This ensures the file is preserved for future needs and frees up your local machine's space.

Step 4: Deleting the Archived File from the Pod

With the log file safely offloaded to S3, you can now delete the original archived file from the pod. Switch back to your terminal where you are "exec'd" into the pod.

rm vault_audit_10-21-2021.log

Verification is key: After deletion, immediately run df -Th to confirm the disk space has been freed up. You should see a significant drop in Use% for the /vault/audit mount point.

Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/vdb       ext4   9.8G 36.0M  9.2G    0% /vault/audit

This looks much better! The disk space has been reclaimed.

Finally, ensure your application is happily writing to a new, smaller vault_audit.log by checking the directory again:

watch ls -lah

You should see a vault_audit.log file present, and its size should be growing steadily but slowly, indicating normal operation.

When df and du Disagree: Handling Deleted Files Held Open by Processes

This is where things get a bit more advanced, but it's a very common scenario in containerized environments. As we discussed earlier, if df -Th shows high disk usage but du -shc /* (or du -shc * after cding into problem directories) reports significantly less, it's almost certainly because files have been deleted from the filesystem but are still being held open by a running process. This prevents the operating system from actually freeing up the disk blocks.

The Problem Explained

When a process opens a file, it gets a file descriptor. Even if another process (or you, using rm) deletes that file, the actual data blocks on the disk won't be released until all processes holding an open descriptor to that file close it. In Kubernetes, this often happens with logging. A log file might be periodically rotated and deleted by a log rotation tool, but if the application process continues to write to the old, deleted inode (the file descriptor it initially opened), the space remains consumed.

Identifying Open Deleted Files with lsof

To find these phantom files, we use the lsof (list open files) command.

lsof 2>/dev/null | grep deleted

Let's break this down:

• lsof: Lists all open files and the processes that own them.
• 2>/dev/null: Redirects standard error (usually permission denied errors) to /dev/null to keep the output clean.
• grep deleted: Filters the output to show only lines containing the string "deleted".

The output will look something like this:

COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
vault      12 vault    3w   REG 259,1  912345678 12345 /vault/audit/vault_audit.log.old (deleted)

Interpreting lsof output:

• COMMAND: The name of the process holding the file.
• PID: The Process ID of the holding process. This is what we need!
• USER: The user running the process.
• FD: File Descriptor. 3w means file descriptor 3, opened for writing.
• TYPE: Type of file (REG for regular file).
• DEVICE, SIZE/OFF, NODE: Technical details about the file.
• NAME: The name and path of the file, crucially followed by (deleted).

In our example, the vault process with PID 12 is holding open a vault_audit.log.old file that has been deleted. This is the reason df still reports high usage!

The Solution: Restarting/Killing the Holding Process

The only way to free up the disk space held by these deleted files is for the process holding them to release its file descriptor. This usually means restarting the application or, as a last resort, killing the process.

Caution is paramount here. Killing a process can lead to application downtime. Always explore graceful restart options first, if available for your application or Kubernetes deployment. For example, if it's a deployment, you might consider kubectl rollout restart deployment/<deployment-name> which gracefully terminates and restarts pods.

However, if you're in an emergency situation and a graceful restart isn't immediately possible or effective, or if it's a standalone process, you might need to kill it.

Identify the PID from the lsof output (e.g., 12 for the vault process).

kill -9 12

• kill: The command to send signals to processes.
• -9: The SIGKILL signal. This is a "non-catchable, non-ignorable kill." It forcefully terminates the process. Use it with care!

A more robust way to kill all processes holding deleted files (if you're sure you want to do this for all of them in the container):

lsof | grep "deleted" | awk '{print $2}' | xargs kill -9

Let's break this powerful one-liner:

• lsof | grep "deleted": Finds all lines with "deleted" files.
• awk '{print $2}': Extracts the second column from each line, which is the PID.
• xargs kill -9: Takes the PIDs from awk and passes them as arguments to kill -9.

After killing the process, it's highly likely your application in that pod will restart (if managed by a Deployment, StatefulSet, etc.) or simply stop. If it's a stateful application, be prepared for recovery procedures.

Verification

Once the process holding the deleted file is terminated, the disk space should be freed. Verify this immediately with df -Th:

df -Th

You should now see the Use% drop significantly for the affected filesystem.

Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/vdb       ext4   9.8G 36.0M  9.2G    0% /vault/audit

This confirms the disk space has been reclaimed.

Proactive Measures and Best Practices to Prevent Low Disk Space

While reactive troubleshooting is essential, implementing proactive measures is key to preventing these incidents in the first place.

• Log Rotation Policies: This is probably the most crucial step. Configure your applications or use tools like logrotate within containers to automatically rotate, compress, and delete old log files. For Kubernetes, consider configuring your container runtime's logging driver (e.g., Docker's json-file driver has max-size and max-file options) or integrating with a sidecar pattern for log management.
• Centralized Logging Solutions: Instead of storing logs directly in pods, push them to a centralized logging system like Elasticsearch, Splunk, Loki, or Graylog. This decouples log storage from pod ephemeral storage, making pods leaner and more resilient. Tools like Fluentd or Filebeat can be deployed as DaemonSets to collect and forward logs efficiently.
• Monitoring and Alerting: Ensure you have robust monitoring in place (e.g., Prometheus with node-exporter) to collect disk usage metrics and configure alerts in Grafana or Alertmanager to notify you well before critical thresholds are reached. Monitor both Use% and inode usage.
• Resource Requests and Limits (Ephemeral Storage): For Kubernetes 1.8+, you can set ephemeral-storage requests and limits in your pod definitions. This allows the scheduler to ensure nodes have enough local storage for pods and prevents a single rogue pod from consuming all node storage.

  resources:
    limits:
      cpu: "1"
      memory: "1Gi"
      ephemeral-storage: "5Gi" # Limit ephemeral storage to 5GB
    requests:
      cpu: "500m"
      memory: "512Mi"
      ephemeral-storage: "2Gi" # Request 2GB of ephemeral storage

• Persistent Volumes for Stateful Data: For applications requiring persistent storage (databases, message queues, etc.), always use Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) rather than relying on ephemeral storage. This ensures data survives pod restarts and provides dedicated, expandable storage.
• Application-Level Cleanup: Implement garbage collection or cleanup routines within your applications for temporary files, cache data, or old build artifacts that might accumulate over time.
• Regular Audits: Periodically audit your deployments for applications known to be verbose or those that might have misconfigured logging.

By adopting these best practices, you can significantly reduce the frequency and impact of Kubernetes low disk space alerts, ensuring your clusters run smoothly and reliably.

Key Takeaways

• Act Quickly: Low disk space can lead to pod eviction and service disruption; immediate diagnosis is crucial.
• df vs. du: Use df -Th to see filesystem usage and identify the mounted point, then du -shc /* (and subsequent cd and du) to pinpoint the exact file or directory consuming space.
• Log Management: For large log files, rename, transfer out of the pod (e.g., kubectl cp to local, then upload to S3), and then safely delete.
• df/du Discrepancy: If df reports high usage but du doesn't, deleted files held open by processes are the likely cause.
• lsof and kill -9: Use lsof | grep deleted to find processes holding open deleted files, and kill -9 <PID> (or the awk one-liner) to release the disk space (with caution).
• Proactive Prevention: Implement log rotation, centralized logging, ephemeral storage limits, and robust monitoring to prevent future disk space issues.

Frequently Asked Questions

What is ephemeral storage in Kubernetes?

Ephemeral storage refers to the temporary, non-persistent local disk space available to a pod. It's tied to the pod's lifecycle and typically resides on the node where the pod runs. If a pod restarts or is rescheduled to another node, any data stored in its ephemeral storage is lost. This storage is commonly used for things like container images, application binaries, and, crucially, logs and temporary files.

Why is df showing high disk usage while du shows less?

This discrepancy almost always indicates that files have been deleted from the filesystem, but one or more processes are still holding open file descriptors to those deleted files. Until all processes release their hold, the operating system cannot free the disk blocks associated with those files, and df will continue to report them as "used" space. lsof is the command to diagnose this.

How can I prevent logs from filling up my Kubernetes pods?

The best approaches include implementing log rotation policies within your containers, configuring your container runtime's logging drivers (e.g., max-size, max-file for Docker), and migrating to a centralized logging solution (like ELK stack, Loki, Splunk) where logs are streamed off the pods to dedicated storage. You can also set ephemeral storage limits in your pod specifications.

Is it safe to kill -9 a process in a Kubernetes pod?

Executing kill -9 is a forceful termination and should be considered a last resort. It does not allow the process to shut down gracefully, potentially leading to data corruption or incomplete operations. If the pod is managed by a Deployment or StatefulSet, Kubernetes will likely restart it, but there might be a brief service interruption. Always try to identify the cause, gracefully restart the application or the pod (kubectl rollout restart deployment/<name>), and only use kill -9 if other options are exhausted and you understand the potential impact.

Facing disk space issues in Kubernetes can be daunting, but with the right tools and a systematic approach, you can quickly identify and resolve them. The key is to understand your environment, know your commands, and always prioritize proactive measures. For a visual walkthrough of these steps and more practical insights, make sure to check out the original video on the @explorenystream channel. Don't forget to subscribe for more expert DevOps content!