PBS Recovery

PBS Recovery rebuilds a Proxmox Backup Server datastore that was deleted, formatted over, or lost with the pool it lived on, and writes it back out as a clean datastore directory that PBS can verify. It reads the source directly and read-only: one disk or disk image that held the datastore, or the member disks of the ZFS RAID-Z pool that held it, with as many members missing as the parity allows. Every chunk, index, and manifest is proven by the checksums PBS wrote into it before it is placed, so the result contains no guessed data.

What it does

• Finds every PBS object on the source by its signature: chunk blobs, fixed and dynamic indexes (.fidx and .didx), and the index.json manifests that describe each snapshot. Nothing has to be mounted, and no file system has to be intact.
• Proves each object before it is written. A chunk’s CRC32 recovers its exact length, and the SHA-256 digest of its payload gives it its name in the chunk store. Index files carry a SHA-256 checksum that proves both content and length. Manifests are decoded and parsed, and a manifest is matched to its indexes only when their checksums agree.
• Writes a clean datastore tree: a .chunks store laid out exactly as PBS expects, each snapshot under its backup group and timestamp with the manifest, indexes, and blobs under their original names, an owner file per group, and a separate folder for valid indexes whose manifest was not found.
• Reports coverage honestly. RECOVERY-REPORT.txt lists every snapshot with the chunks found and the chunks missing, so you know what will verify before you hand the datastore to PBS.
• Runs as a window or from the command line. The window is for picking the source disks and an output folder and watching the live log. Every operation in the window is also a command, so a recovery can be scripted.
• Resumes after an interruption. Ctrl-C, a crash, or a reboot leaves a checkpoint, and the next run continues from it after confirming it is the same source disk or image.

Two kinds of source, detected automatically

Direct disk. One physical drive, volume, or dd image that held the datastore on an ordinary file system such as ext4 or XFS, or an image that has already been reassembled from an array. The program sweeps it for PBS objects by signature. This mode assumes each file’s bytes are contiguous on disk, which is how ext4 and XFS store chunk files in practice.

ZFS pool. The member disks, or images of them, of a ZFS pool whose RAID-Z vdev held the datastore. The program reads the ZFS labels, works out the pool geometry, rebuilds members that are missing or dead from parity, sweeps the reconstructed pool for the datastore’s files, and follows each file’s real block pointers so the bytes come from where ZFS actually put them. As many members as the parity level may be missing: one for RAID-Z1, two for RAID-Z2, three for RAID-Z3. Single-disk and mirror pools work the same way, and LZ4, zstd, LZJB, and ZLE compression are decoded on the fly.

You select every disk you have and the program picks the mode. A checksum-valid ZFS label, or several labelled members, means pool mode. A lone disk that carries only a stale ZFS label, for example one that was later reused for a plain PBS datastore, is carved directly. And if a pool sweep turns up nothing, the program falls back to a direct carve of the disk on its own.

It cannot invent data

PBS names every chunk by the SHA-256 of its contents, and every index and manifest carries its own checksum. PBS Recovery turns that into a safety property: a chunk, index, or manifest is placed in the output only when the format’s own integrity data proves it is correct. A wrong guess about pool geometry or a bad parity reconstruction can therefore only cause a miss, never a fabricated chunk. Every chunk in the output is bit for bit the chunk PBS wrote, and the report’s missing column is the honest measure of what came back.

What you need

• A 64-bit Windows computer. Disk images open without special rights. Reading a physical drive directly requires an Administrator command prompt or PowerShell session, or running the program as administrator. Imaging a failing disk first is what our own lab does.
• The source: the disk or image that held the datastore, or every member of the ZFS pool you still have. For a pool, bring every surviving member. The parity count may be missing, but a pool with all members present sweeps completely; see the limits below.
• Output on a different physical disk, with free space roughly equal to the used size of the original datastore. The program refuses to start otherwise.
• A Linux machine running Proxmox Backup Server for the last step, where the rebuilt tree is copied onto ext4, XFS, or ZFS and added as a datastore.

How to use it

1. Download PBS Recovery and run it. The program is a single file and does not require installation. For a physical drive, start it from an Administrator prompt.
2. Pick the source: one disk or image, or every member of the ZFS pool. The program detects the pool from the labels.
3. Pick an output folder on a different disk and click Start. Progress and a live log are shown while the scan runs, and an interrupted scan picks up where it left off.
4. Click Open report when the scan finishes. RECOVERY-REPORT.txt lists each snapshot with its chunk coverage.
5. Copy the whole output folder to a Linux machine, onto ext4, XFS, or ZFS. In the copied folder, run sh rename-snapshots.sh (NTFS cannot store the colons in PBS snapshot names; the script restores the exact names and is safe to re-run), then chown -R backup:backup .
6. Register it with proxmox-backup-manager datastore create recovered /path/to/dir and run proxmox-backup-manager verify recovered.

Command line

Everything in the window is also a command, so a recovery can be scripted or run over a remote session.

PBS-Recovery.exe                        open the window (pick disks and output)
PBS-Recovery.exe list                   show physical drives
PBS-Recovery.exe scan --input 2 --output E:\rebuilt                 one disk
PBS-Recovery.exe scan --input D:\images\disk.img --output E:\rebuilt
PBS-Recovery.exe scan --input "2;3;4;5;6" --output E:\rebuilt       ZFS pool
PBS-Recovery.exe scan --input "d0.img;d1.img;d2.img" --output E:\rebuilt
PBS-Recovery.exe finalize --output E:\rebuilt     re-run assembly and report
PBS-Recovery.exe report --output E:\rebuilt       reprint the report
PBS-Recovery.exe selftest                         end-to-end self check, about 5 s

For a ZFS pool, list the member disks or images you have, separated by semicolons, as a single input. Direct-disk scans also accept options for alignment, buffer size, thread count, a start and end offset, a forced restart, skipping the final assembly, and the owner written into each backup group. Unknown options and malformed numbers are rejected rather than ignored.

What comes out

• .chunks\ the carved and verified chunks, in the two-level layout PBS uses.
• vm\<id>\<time>\ each rebuilt snapshot with its manifest, indexes, and blobs under their original names, plus an owner file per group (root@pam unless you choose otherwise).
• orphan-indexes\ valid indexes whose manifest was not found.
• RECOVERY-REPORT.txt per-snapshot chunk coverage, unreadable sectors, and anything set aside.
• rename-snapshots.sh the script that restores the exact snapshot names on Linux.
• .pbsrebuild\ staging: the scan log, the checkpoint, the report in JSON, encrypted blobs, and the carved indexes and manifests.

Limits, stated plainly

• Encrypted datastores. Encrypted chunks are carved and CRC-checked, but their digests are keyed, so without the key they cannot be given their names in the chunk store. They are kept in the staging folder and counted in the report. Encrypted configuration blobs listed in a manifest are matched and placed, because their checksums are not keyed.
• Overwritten chunks. A chunk whose bytes were partly overwritten fails its CRC or digest and is skipped. It appears in the missing column.
• All-zero chunks referenced by an index are generated automatically, since they hash to a known digest for each length.
• Namespaces. A manifest does not record its namespace, so namespaced snapshots are rebuilt into the root namespace.
• Degraded pools. Once a file’s dnode is found, its blocks are read through the real block pointers and reconstructed exactly, even with a dead member. Finding the dnodes, however, is a sweep of the logical space in fixed windows, and with a member rebuilt from parity a window’s assumed layout may not match the real allocation, so some dnodes in rebuilt regions may not be found. The result is a miss, never wrong data. Run with all members present whenever you can.
• Pool layouts. One top-level vdev: a single disk, a mirror, or RAID-Z1, RAID-Z2, or RAID-Z3. dRAID and pools with RAID-Z expansion in their history are refused with a clear message. For a pool with several top-level vdevs, give the members of the vdev that held the datastore. Natively encrypted datasets are ciphertext without the key and are reported as missing.
• Compression. Uncompressed, LZ4, zstd, LZJB, and ZLE blocks are decoded, including the compressed dnode array ZFS keeps by default. gzip-compressed blocks are not; gzip is rare under PBS because chunk payloads are already zstd-compressed. A block that cannot be decoded is a miss.

Built from the formats’ own definitions

The PBS constants and layouts are taken verbatim from the proxmox-backup source (checkout 3.2.4), and the ZFS constants and layouts from the OpenZFS source. Before release, the direct-carve layer was checked against a synthetic deleted datastore and an independently written validator, and it recovered 453 of 453 chunks from a ground-truth image while rejecting the 3 corrupt ones. The ZFS layer recovered complete synthetic RAID-Z1 and RAID-Z2 pools with members missing at the parity limit, including blocks stored with ZFS zstd compression. PBS-Recovery.exe selftest runs the same end-to-end check on your own machine in about five seconds.

Your disks are never modified

PBS Recovery opens the source read-only, and that rule is enforced in the engine rather than left to a setting. Nothing is written to the source disk or the pool members. Recovered data goes only to the output folder you choose.

The output must be on a different physical disk, and the program checks that for itself rather than trusting the path. When the source is a raw device, the scan refuses to start unless it can prove the output lives elsewhere: every spelling of a device path, folder mount points, volume GUIDs, and volumes on VHD, VHDX, or Storage Spaces are checked, because their backing file could sit on the source. Network paths that point back at this machine, by loopback address or by its own name, are refused as local volumes in disguise, and so are shares addressed by IP address or by a dotted name, because they cannot be proven remote. Only a bare server name such as \\nas\share is accepted. An override exists for people who know exactly what they are doing.

Unreadable sectors never abort a scan. They are zero-filled, counted, logged, and listed in the report under unreadable media. If a whole read window comes back unreadable, the program treats it as a dropped device and stops at the checkpoint instead of zero-filling the rest of the disk.

If the drives themselves are failing, or more members of a pool are gone than the parity can cover, our engineers can help. Call 800.237.4200 or start a recovery ticket.