Why does nvidia-smi show no processes?
Orphaned CUDA contexts, colloquially known as Zombie VRAM, severely degrade GPU memory on Linux AI servers. This memory leak triggers when a Docker container crashes unexpectedly, but the host process remains alive. Because the NVIDIA driver loses its PID mapping, the stranded allocation permanently locks the GPU memory. System administrators must clear this state by interrogating device files. The fuser command directly identifies the hidden threads causing the CUDA out of memory error. By forcefully terminating these processes, administrators release the trapped resources. ServerMO Bare Metal infrastructure eliminates hypervisor restrictions during this reset process, allowing instant memory recovery.
You are training a heavy LLM or running a ComfyUI workflow. Suddenly, the script crashes. You attempt to restart the model, but you are hit with a fatal RuntimeError: CUDA out of memory. You run nvidia-smi, and the output is baffling: your 80GB of VRAM is completely full, yet the processes table explicitly states "No running processes found."
The Real Root Causes
While developers call it "Zombie VRAM," the actual technical causes are usually:
- Docker Desync: The AI container dies, but the NVIDIA Container Toolkit fails to kill the underlying Python process on the Host OS.
- CUDA Context Crashes: The script terminates abruptly without safely deallocating memory via the NVIDIA driver.
- Persistence Mode Bugs: The driver gets stuck maintaining a state for a ghost PID.
Step 1: The Device File Interrogation
If nvidia-smi is blind, we must bypass the driver interface and interrogate the Linux kernel directly. We do this by checking which lingering processes are holding file locks on the physical GPU device pathways.
# Expose all processes accessing GPU 0
sudo fuser -v /dev/nvidia0
# Alternative 1: Using lsof to list open files
sudo lsof /dev/nvidia*
# Alternative 2: Brute force search for hidden Python scripts
ps aux | grep python
These commands bypass the abstraction layer. You will immediately see a list of hidden threads (e.g., root 14763 F...m python) that survived the initial crash and are hoarding your tensors.
Step 2: The Docker & SIGKILL Sweep
Before using direct kernel commands, if you are running your AI models inside a Docker container (like vLLM or Ollama), the cleanest approach is to simply restart the container. Docker will attempt to clean up its own orphaned processes.
# Attempt Docker-level cleanup first
docker restart <container_name>
If restarting Docker fails, or if you are running scripts natively on the Host OS, Python-level commands like torch.cuda.empty_cache() are useless because the interpreter has already died. We must issue a direct OS-level SIGKILL (Signal 9).
# Forcefully terminate all hidden processes holding VRAM on all GPUs
sudo fuser -k -9 /dev/nvidia*
# Alternative: Kill all Python processes globally (Use with caution!)
sudo pkill -9 python
Run nvidia-smi again. In 95% of scenarios, your VRAM will instantly drop back to 0MiB.
Step 3: The Hardware State Reset
Occasionally, the CUDA context itself becomes corrupted at the hardware level. The memory is free, but the GPU refuses to accept new workloads. We can force a soft-reset of the GPU architecture.
# Reset the internal state of GPU 0
sudo nvidia-smi --gpu-reset -i 0
Important Constraints (When Reset Fails)
The --gpu-reset command is powerful, but it will fail under three specific conditions:
- Display GPUs: If Xorg or Wayland is using the GPU for a desktop display, resetting will crash the UI. (Note: ServerMO AI servers are Headless, so this is rarely an issue).
- MIG Enabled: If NVIDIA Multi-Instance GPU (MIG) is active (common on H100s), standard resets are blocked.
- Active Processes: If you did not successfully execute Step 2, the driver will throw a "cannot reset while processes exist" error.
Avoid Hypervisor Overhead
Debugging CUDA states on a shared virtual machine is a nightmare. Virtualization layers block low-level hardware resets. For pure tensor parallelism and unthrottled hardware access, you need true Bare Metal infrastructure.
Explore Single-Tenant Dedicated GPU Servers.
