The backup is stuck and has been running for almost a day. How can I resolve this?
The backup is stuck and has been running for almost a day. How can I resolve this?
Hello, I've looked into your situation:
1. Why it's stuck on RUNNING: The K8s Job for the backup on 5/5 at 09:36 (UTC) actually OOM-killed during execution. However, the container didn't report the status in time, so the record remained stuck on RUNNING. The Job itself was cleaned up after the 6-hour ActiveDeadlineSeconds. The lack of error messages on the Dashboard is a bug on our end, which we are currently following up on internally.
2. Why it OOMed: The plugin-daemon PVC currently occupies 17GB (with cwd/langgenius/ taking up 13GB, which is the Dify official plugin package you installed). Your Hostinger instance has 8GB of RAM, with only about 1.6GB free. The backup Job requires a 2Gi memory limit to run pigz compression, and there simply isn't enough space on this server, so it crashed. I noticed you paused MinIO before the backup, likely to free up memory, but it's still not enough for a 17GB volume.
How to handle it:
Step 1 — Clear the stuck record: Dashboard → plugin-daemon → Backup History, find the entry from 5/5, and click Cancel. The backend will mark it as FAILED.
Step 2 — Since you are using your own VPS (Hostinger), the most stable approach is to SSH in and tar the volume directly, bypassing the Zeabur backup pipeline to avoid memory limitations:
ssh root@<your server IP>
# Find the actual path of the plugin-daemon PVC
PVC_DIR=$(find /var/lib/rancher/k3s/storage -maxdepth 1 -name '*data-service-67d394ddfee8bb9733b7f3b3*' -type d)
echo "$PVC_DIR"
# Stream compression directly to external storage without saving locally (very low memory usage)
tar -cf - -C "$PVC_DIR" . | pigz -p 2 > /path/to/external/plugin-daemon-backup.tar.gz
# 17GB is expected to compress to 5–8GB
You can also use rsync -avz to sync directly to your own cloud storage.
The standard Zeabur backup pipeline is not really feasible for 17GB on an 8GB RAM machine (the backup pod itself requires 2Gi of memory). I recommend periodically using SSH to tar this volume directly for peace of mind.
How can I check that the PVC for plugin-daemon is currently using 17GB in the current interface?
You can view the current disk usage in the disk tab of individual services.
This post has been inactive for a while. We will be closing it in 2 days if there is no new activity.
New replies are disabled for resolved issues.