FailedCreatePodSandBox - Node cannot pull rancher/mirrored-pause:3.6 from Docker Hub

My deployment keeps failing with FailedCreatePodSandBox.
Node cannot pull rancher/mirrored-pause:3.6 from Docker Hub (DNS lookup fails).
Please reassign my service to a different node.

Service ID: s-69d5fa88fc27d935dbf378a3

Error log:
[Zeabur] Pod/service-69d5fa88fc27d935dbf378a3-65f7b8db44-xsq7l - FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox "7775269d9c96bfab30f24c838ad3388dae3bf35927dbe1377d905bb150a97a43": failed to get sandbox image "rancher/mirrored-pause:3.6": failed to pull image "rancher/mirrored-pause:3.6": failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.6": failed to resolve reference "docker.io/rancher/mirrored-pause:3.6": failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6": dial tcp: lookup registry-1.docker.io: Try again

[Zeabur] Pod/service-69d5fa88fc27d935dbf378a3-65f7b8db44-xsq7l - FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox "ea95165c3f8c1285975b7e882d4b013c2fd8bd4f62433fbe94daec09139ae453": failed to get sandbox image "rancher/mirrored-pause:3.6": failed to pull image "rancher/mirrored-pause:3.6": failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.6": failed to resolve reference "docker.io/rancher/mirrored-pause:3.6": failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6": dial tcp: lookup registry-1.docker.io: Try again

[Zeabur] Pod/service-69d5fa88fc27d935dbf378a3-65f7b8db44-xsq7l - FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox "065b178c96d818bd4431358d8371deee2be9aa76882207750f3e81d21c19a34b": failed to get sandbox image "rancher/mirrored-pause:3.6": failed to pull image "rancher/mirrored-pause:3.6": failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.6": failed to resolve reference "docker.io/rancher/mirrored-pause:3.6": failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6": dial tcp: lookup registry-1.docker.io: Try again

[Zeabur] Pod/service-69d5fa88fc27d935dbf378a3-65f7b8db44-xsq7l - FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox "059c018f6ed945c0a63a1328cdcd7870ed01616eeaf3a6f0f2b3f46d26ebb4b4": failed to get sandbox image "rancher/mirrored-pause:3.6": failed to pull image "rancher/mirrored-pause:3.6": failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.6": failed to resolve reference "docker.io/rancher/mirrored-pause:3.6": failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6": dial tcp: lookup registry-1.docker.io: Try again

Fixed. Posting what happened for your records.

Root cause: after your recent spec-upgrade reboot, systemd-resolved on the server got stuck unable to reach Aliyun's internal DNS endpoints (100.100.2.x). The resolver had been looping in a UDP/TCP "degraded feature set" state for several days — every DNS lookup from the host was timing out.

Why the symptom looked like a node-pull issue:

Your old pod kept Running because the pause:3.6 sandbox image was already cached locally — no DNS needed
Your new pod (from the latest deployment) couldn't start because creating a new sandbox forces containerd to re-resolve registry-1.docker.io for the manifest check, and the lookup timed out

Note: this is a single-node server, so "reassign to another node" doesn't apply here — there's only one node in this project.

Fix applied:

Added a netplan drop-in pinning the eth0 resolver to Aliyun public DNS (223.5.5.5 / 223.6.6.6) and disabling DHCP-pushed DNS
Restarted systemd-resolved
The stuck pod immediately moved past sandbox creation and is now 1/1 Running

Change is persistent across reboots. If you see anything weird, reply here.

Forum

1 Reply