Your Proxmox backups aren't tested until you've restored one

Proxmox Backup Server makes backing up easy enough that you stop thinking about it. A nightly job runs, the dashboard goes green, and the question “could I actually get this container back” quietly stops being asked.

I run PBS on a three-node Proxmox cluster, snapshotting every LXC nightly. The first time I did a real restore I learned two things the docs mention only in passing: the restore command needs a parameter nobody tells you about until it fails, and a restore that completes successfully can still leave you with a service that doesn’t work.

This is the shape of a restore test that actually proves something.

The gotcha before you start: unprivileged containers and the datastore

If you’re running PBS inside an unprivileged LXC with the datastore on a bind-mounted disk — which is the sensible layout, because it keeps chunks off your root NVMe — datastore creation will fail on permissions.

An unprivileged container maps its root user to UID 100000 on the host. The bind-mount arrives owned by the host’s root, which the container cannot write to. So before you create the datastore, chown the directory on the host:

chown -R 100000:100000 /mnt/local-sata/pbs-datastore

Then create the datastore pointing at the in-container path. Skip this and you get a permissions error at chunkstore initialisation that reads like a PBS bug and isn’t.

Finding the snapshot

Restores start from the storage, not the PBS UI. From any node in the cluster:

pvesm list pbs-store --vmid 100

That lists the snapshots for one container. If it returns nothing, your storage isn’t attached the way you think it is — check pvesm status | grep pbs-store shows active before you go any further.

The restore itself

pct stop 100
pct restore 100 pbs-store:backup/ct/100/<snapshot-id> --force --rootfs local-lvm:4
pct start 100

Three things worth knowing:

--force is required to restore over an existing container. Without it the command refuses because the VMID is taken. With it, the running container is overwritten — so make sure the cluster knows it’s stopped first.

--rootfs is not optional and not remembered. You have to state the target storage and the disk size in GB. That 4 is specific to that container; a reverse proxy holding certificates might be 50. Get it wrong and you either fail to restore or silently provision a disk that’s too small for the container to be healthy later. Write the correct size down per container before you need it, because you will be looking it up during an outage otherwise.

Snapshot age is a data-loss decision. More on this below, because it’s the part that bites.

Why “restore succeeded” isn’t the finish line

Here’s the failure mode that matters. The restore exits 0. The container starts. pct status says running. Everything looks recovered.

And the service is still broken, because the snapshot predates configuration you made after it was taken.

Concretely: my DNS container holds hostname rewrites that get edited as services move around. Restoring a snapshot from before those edits gives you a container that runs perfectly and resolves nothing correctly. Nothing errors. The backup did its job. You just rolled back a week of configuration along with it.

This is why a restore test has to end with a check on the service, not the restore command:

# Not "did the container start" — "does it still do its job"
dig @192.168.10.20 grafana.lan +short

If that returns the address you expect, the restore is verified. If it returns nothing, you’ve learned something genuinely valuable in a drill instead of during an incident.

Pick one command per container that proves the service works, and put it at the bottom of that container’s recovery notes. For a reverse proxy it’s a curl against a route that should return 200. For a database it’s a query. The principle is the same: exercise the thing the container exists to do.

What a reasonable nightly job looks like

For context, the job behind all this: every LXC, 03:00, snapshot mode, zstd compression, keep-daily=7 and keep-weekly=4.

Snapshot mode matters — it’s why a container backs up without meaningful downtime. One 785 MB container completes in about six seconds. That speed is the reason nightly-everything is realistic on a homelab rather than something you schedule weekly and hope about.

Retention is the other half of the earlier point. Seven daily snapshots means your worst-case rollback is a week of config drift. If a container holds state you change often, either back it up more aggressively or — better — get that state into version control so the container is rebuildable rather than only restorable.

The checklist

1. Datastore on a non-root disk, chowned to 100000:100000 if the PBS container is unprivileged.
2. pvesm status | grep pbs-store shows active.
3. Know the correct --rootfs size for each container, written down in advance.
4. Restore into a stopped container with --force.
5. Finish with a command that proves the service works, not one that proves the container started.
6. After the drill, note what the snapshot lost — that gap is your real recovery point objective, whatever the retention policy says.

Where to go next

Restore testing is one piece of a backup design. The parts this article doesn’t cover — datastore sizing and pruning, verify jobs, what to do when the PBS container itself is the thing that died, and per-container recovery runbooks — are in PBS Backups That Actually Restore, which is written from the same cluster these commands come from.

If you’d rather just read configuration, the Ansible roles that build this lab are MIT on GitHub and don’t require the guide.