The snapshot creation on CBT enabled VM fails with error Failed in vmreparent vmkfstools clone1

I came across an interesting issue the other day after deploying some infrastructure at a new site. I had deployed a new Hyperflex Cluster and was doing some testing with HX version 3.5(2e). After migrating some test VMs over to this new cluster I configured the backups to use HX Snapshots and kicked off a test job.

The job started as per normal but when it attempted to create the HX Snapshot it failed and then switched over to creating a VM Snapshot instead.

The backup still completed fine but not using the method in which I had intended.

After looking into the issue from the vSphere Web Client I noticed the following error,

A snapshot operation cannot be performed.

Result: Failed in vm_reparent vmkfstools clone1 – out_vmkfstools_clone1: 255 clone_vmdk_path1: It then referenced the vmdk location and server name.

I then tried to just use the Hyperflex Data Platform Plugin from the vSphere Web Client to manually create a snapshot. Same issue which ruled out it being related to the backup software in any way.

So even though I knew it would also fail I tested it from the HX Connect Web Client and sure enough same issue.

Turns out there is a bug currently affecting releases – 3.5(2c), 3.5(2e), 4.0(1a), 4.0(2a). If you try to take a HX Snapshot of any CBT enabled VM it will fail. At the moment there is a workaround which will help resolve the issue until Cisco release a proper fix for this problem.

Follow the steps below,

First you need to shutdown the VM. I know this is less than ideal but stick with me and we will get it sorted.

You can disable CBT on the VM and Cisco mentions to do so but in this case I left it enabled because I wanted to see how it would go.

Check if you have any snapshots present – if so delete all existing snapshots.

If you did disable CBT you can enable it again now. As I mentioned before I skipped this step and just left it enabled.

Then from the vSphere Web Client right click the VM and select Cisco HX Data Platform from the menu, then Snapshot Now

Then enter a name for the snapshot, I was creative here and just entered test

The HX Native Snapshot will then be created successfully. If you now go to manage snapshots for the VM you will see the snapshot “test” present.

You can now go ahead and power on the VM

Then once the VM has powered on go back into your snapshot manager and select the test snapshot that you just created and click Delete.

Once you have done that you can add the VM to any backup job which leverages HX Native Snapshots and the backup will complete successfully from now on. You can also go and create them manually when needed.

Here is an example the job being run using Veeam