Give my Storage Block back !
We had some strange issue during restoration attempt with Veeam Backup from a reversed incremental backup, that we wanted to share with you.
We received some emergency restoration request for a virtual machine (named CADILLAC) from one of the past incremental recovery point (ie not from the last FULL recovery point), and here it is, the scary error message box :
We had the same issue with vmdk file restore, full VM restore, in-guest FLR restore or even with Instant Recovery !
The backup process was errorless, the checksum integrity return no error and backup job completed successfully.
What was weird is that restoring the VM from the last recovery point (the FULL backup with the .vbk file) completed successfully, but each time we tried to restore from previous recovery point (the incremental backup with the .vrb files), the error comes back with the ugly error-message-box face, and same thing occurs for all VM in this job ! Even with the extract.exe tool, we got the same error message :
Looking in the restore job log file (VmFilesRestoreJob_Restore_CADILLAC_Files.log located, for Windows 2008 R2, in C:\Users<service account>\AppData\Local\Veeam\Backup), we isolated the error message:
Info [AP] (Client) error: The sizes of the standard snapshot block [524288] and standard storage block are different [1048576].
The sizes of the standard snapshot block is indeed 512 KB (quicly searching with Notepad++ in all the backup log files, we found the same snapshot block size of 512KB) but we cannot be sure anymore what the first configuration was:
A quick reminder, the storage block size is defined at the backup job level with the following parameter:
As pointed in this whitepaper http://www.veeam.com/hp-protection-vmware-infrastructure.html
Setting the storage target to a WAN target results in a block size of 256 KB, 512 KB for LAN targets and 1 MB for locally attached targets.
After opened a case with Veeam, they asked us to confirm this issues by exporting the log files with VeeamAgent.exe shell.
In order to do this, you have to run VeeamAgent.exe (which is located in C:\Program Files\Veeam\Backup and Replication) and then enter the following commands:
dumpStg
backup file path
file path for saving log file with metadata
Note: this command works fine with .vbk and .vrb files as well It was the first time we used VeeamAgent.exe from command line and we never heard about the dumpStg command (which basically extract metadata from backup job files in order to save them in a new log file), but after a quick search on Google and in the Veeam forum, the dumpStg command isn’t well know ^^
However, you’ll need to have enough space as the generated files are quite large (the generated metadata log files are about 400 MB for a 1 TB backup file).
The metadata created by dumpStg analysis shows clearly the block size mismatch:
Storage: e:\XXX_CHASSIS_20\XXX_CHASSIS_20.vbk
Storage format
Standard block size: 1048576
Storage: e:\XXX_CHASSIS_20\XXX_CHASSIS_202012-07-06T210152.vrb
Storage format
Standard block size: 524288
That’s why we got an error during restore job: the block size from the VBK backup file and the VRB backup one aren’t equal, so we’re able to restore from the VBK backup file, but when we tried to restore using VBK+VRB chain, we got the error.
So we might have change backup job configuration from Local Target (1MB) to LAN Target (512KB) during some tests, but we didn’t realize it could be problematic as the backup jobs kept on succedeed! Veeam confirmed it was a bug from version 5, which seems to be fixed in the 6.1 version (but the job have to be created in Veeam 6.1, importing job from version 5 to 6.1 will leave the problem as is).
Be careful not to change the block size from a Veeam B&R 5 reverse incremental backup job if you have existing backup otherwise it’ll break the VBK/VRB chain and restore will no longer succeed.