borg2: Repo corruption

[someone-somenet-org]

This is mostly a "please document repo repair a little better" - ticket.

Well, I did something stupid and corrupted some chunks of one of my borg2 repos. (dont dd if=/dev/zero of=/dev/sd... the wrong device 🙈)

Anyway.

After running borg2 check --repair -r /bkp/... it deleted the corrupted chunks.
During the next backup it re-created most of the corrupted chunks, so now I am at 2 corrupted chunks, one of which I definitely can never get back. (As it was an encrypted SeedVault-Android backup chunk)

With subsequent runs of borg2 check -r /bkp/... it will always print the Files affected:

[...]: srv/mattermost/server/prepackaged_plugins/mattermost-plugin-msteams-v2.0.3-linux-amd64.tar.gz: Missing file chunk detected (Byte 13617517-15441730, Chunk 160e59edd9a5aeafae625ee9a65139544005c036be61b18136f913ece155dc3a).
[...]: srv/mattermost/server/prepackaged_plugins/mattermost-plugin-msteams-v2.0.3-linux-amd64.tar.gz: Missing file chunk detected (Byte 13617517-15441730, Chunk 160e59edd9a5aeafae625ee9a65139544005c036be61b18136f913ece155dc3a).
[...]: srv/mattermost/server/prepackaged_plugins/mattermost-plugin-msteams-v2.0.3-linux-amd64.tar.gz: Missing file chunk detected (Byte 13617517-15441730, Chunk 160e59edd9a5aeafae625ee9a65139544005c036be61b18136f913ece155dc3a).
[...]: srv/mattermost/server/prepackaged_plugins/mattermost-plugin-msteams-v2.0.3-linux-amd64.tar.gz: Missing file chunk detected (Byte 13617517-15441730, Chunk 160e59edd9a5aeafae625ee9a65139544005c036be61b18136f913ece155dc3a).
[...]: srv/nextcloud/data/[...]/files/.SeedVaultAndroidBackup/3634a43e813679618678283efc753829f884907d24bf8a944d94b463f1e33e79/15/15fc8d419949452ed06b382cda2e09074526ce1c3fbfe47ee43f21e23013ab94: Missing file chunk detected (Byte 0-786493, Chunk 15fc8d419949452ed06b382cda2e09074526ce1c3fbfe47ee43f21e23013ab94).

I dont particulaly care about these files and would like to acknowledge their corruption and ignore them in future.

Currently borg2 compact exits with exitcode=2 and the message Repository has 2 missing objects! after successfully completing its run. This causes me to get an error-eMail.

So... what would be the next steps for me to take?

• I dont want to ignore exitcode=2 as that could also mean ssh connection issues or other important things.
• I dont want to wait for the archives to rotate away ... this would be more than a year of daily error eMails.
• I dont want to delete the whole archives ... the other files are seemingly fine.

Can I somehow mark these files as unimporatant?
Can I delete these files from the corresponding archives? (its a quite big repo; I fear recreate would take forever; or is --exclude faster than recompress?)

Also please document that in the help section for borg check. It currently doesnt even mention that corrupted chunks will get recreated during the next borg2 create, if the original files still exist. What is also not clear to me: Did I actually need a second borg2 check --repair after borg2 create?
And please add something like "considerations to fix your repo corruption", with ideas that people might want to apply to reach some post-corruption state.

[ThomasWaldmann]

borg2 deals with missing chunks differently than borg 1.x.

borg 1.x

if borg check --repair finds and patches missing chunks:

• it remembers the "healthy" (how it should be) chunks id list separately
• it creates all-zero, same-size replacement chunks and references them from the chunks id list
• thus, the reference structure is ok again, chunks id list pointing to existing chunks
• the item still "knows" that it is damaged (because the healthy list differs)
• the item still "knows" how it ideally should be

That seemed a good idea back then, but has some quite annoying consequences:

• if borg create reproduces a missing chunk, that does not automatically fix archives from the past
• after borg create has reproduced missing chunks, the admin has to run a time-consuming borg check --repair again to fix the old archives.
• if there are multiple missing chunks and they are not all reproduced at once, maybe the admin even needs *multiple borg check --repair runs to repair the past archives.
• if the admin does not do these repair runs, old archives stay in a patched (consistent, but defective) state.

borg2

• there is only one chunks id list in an item, it always refers to the healthy, correct chunks by their correct ID
• obviously, this is simpler than having to deal with 2 lists
• if a new borg create run reproduces missing chunks, all past archives are immediately "repaired"
• no need for one or multiple borg check --repair runs
• no need to create all-zero same-size patching chunks
• missing chunks are dealt with dynamically at borg extract / borg mount time (raise error or read zeros)

But, as you found, it currently keeps complaining about missing chunks.

[ThomasWaldmann]

Workaround:

Yes, you can use borg recreate to remove files with missing chunks from archives by excluding them.

Be super careful with this and always use --dry-run --list first and make sure it does only remove what you really want to remove.

recreate with only excluding some stuff and not changing other parameters (like chunker params or compression) should be relatively fast as it only operates on the metadata.

[ThomasWaldmann]

Currently, @mr-raj12 is working on packs as his Google Summer of Code project. Because that is quite a fundamental change in how a borg repository works, it will also need changes in borg check [--repair].