KVM snapshots are not being removed

[bigsys-it]

Hi!

My KVM snapshots are not being removed, neither through the "keep" parameter of the automatic scheduling nor even when I manually select the snapshot and delete it.

In the snapshot_store_ref table, the state column is only changed to Hidden, and therefore the physical files continue to occupy space in the secondary storage.

I am using version 4.21.0.0, snapshot.backup.to.secondary = true and kvm.incremental.snapshot = true.

Does anyone know what I need to do to actually remove the snapshots from the secondary storage?

[DaanHoogland]

@bigsys-it , this depends a bit, the wnapshots are “hidden” because they may still be parents to other snapshots. Untill coalesced these intermediate shapshots are still needed. Is it that there are newer snapshots after the ones you try to delete?

[bigsys-it]

Perfect! That was it!

I manually deleted the entire chain of snapshots and then they were all physically removed from the secondary storage.

Now another question has arisen... if the "keep" parameter removes the oldest backups, then there will always be a more recent file that will prevent the others (older ones) from being removed?

[DaanHoogland]

and with “backups” you mean copies of snapshots? I think this is true but will be addressed on coalesce. I am not sure under which circumstances this happens but at least a storage migration that I can think of. It might be different accorrding to hypervisor/storage type as well.

[JoaoJandre]

Hello @bigsys-it,

The KVM incremental snapshots will be kept in storage until no new snapshot depends on them, as @DaanHoogland explained.
By default, a snapshot chain will last 16 snapshots, then a new chain will be created; meaning that after you create 16 snapshots, on the 17th, it will be a new full snapshot and will not depend on the previous 16; thus, when the 16th snapshot is deleted, its whole chain will also be gone.

To configure this behavior you may change the snapshot.delta.max parameter to a different snapshot chain size.

[JoaoJandre]

Also @DaanHoogland, the KVM incremental snapshots are never coalesced, as it is expected that the chain will end and eventually be removed from storage. The coalesce operation is costly in terms of CPU and time; thus, it would bring more trouble than benefits; at least that was the decision process during implementation. It would be possible to implement coalesce in the future if it was deemed a good option.

Just for comparison's sake, Veeam does the same process of keeping old backup deltas.