SQL VDI backup fails with 0x80770007

While taking a SQL Server database non-native backup using an application that calls SQLVDI.DLL, you find that the backup fails following HEX code: 0x080770007.

0x80770007 (VD_E_INSTANCE_NAME VD_ERROR) translates to: Failed to recognize the SQL Server instance name.

Then check if the following condition holds true:
There is no DEFAULT instance of SQL Server on the machine where you are trying to take a VDI backup and the SQL instance that you are connecting to perform a backup is a named instance.

If above condition is true, then the issue is with CreateEx function of the interface IClientVirtualDeviceSet2. The CreateEx function is used to create the virtual device set and has the following syntax:

HRESULT IClientVirtualDeviceSet2::CreateEx (
LPCWSTR lpInstanceName,
LPCWSTR lpName,
VDConfig* pCfg
);

The "lpInstanceName" parameter identifies the SQL Server instance to which the SQL command needs to be sent to. If the CreateEx method has NULL as the first parameter, then it would always connect to the Default instance. If the server doesn’t have a default SQL instance, then the first parameter needs to be provided with the instance name:
Eg. If you have a named instance on the server as "SERVER1\SQLINST1”, then the first parameter for CreateEx should be "SQLINST1". A way to workaround the same for simple.cpp is mentioned here.