Recover a missing virtual machine disk (VMDK) descriptor file

Details


Each disk drive for a virtual machine consists of a pair of .vmdk files. One is a text file containing descriptive data about the virtual hard disk, and the second is the actual content of that disk. For example, a virtual machine named examplevm has one disk attached to it. This disk is comprised of a examplevm.vmdk descriptor file of under 1 KB, and a 10 GB examplevm-flat.vmdk flat file which contains virtual machine content.

This article helps you to recreate a lost virtual disk descriptor file. You may need to recreate missing header/descriptor files if:

  • The virtual machine disk file listed in the Datastore Browser is your virtual machine's flat file, and does not have an icon.
  • When powering on a virtual machine, you see a File not found error.
  • The flat file exists when viewing the virtual machine's directory via the terminal, VMware vSphere Management Assistant (vMA), or VMware Command-Line Interface (vCLI).
  • The disk descriptor file for the virtual machine's disk does not exist or is corrupted.
  • In the virtual machine's vmware.log file, you see messages similar to:

    Jul 22 09:22:32.141: vmx| DISKLIB-LINK : "myvm.vmdk" : failed to open (The system cannot find the file specified).
    Jul 22 09:22:32.141: vmx| DISKLIB-CHAIN : "myvm.vmdk" : failed to open (The system cannot find the file specified).
    Jul 22 09:22:32.141: vmx| DISKLIB-LIB : Failed to open 'myvm.vmdk' with flags 0xa (The system cannot find the file specified).
    Jul 22 09:22:32.142: vmx| Msg_Post: Error
    Jul 22 09:22:32.142: vmx| [msg.disk.fileNotFound] VMware ESX Server cannot find the virtual disk "myvm.vmdk". Please verify the path is valid and try again.
    Jul 22 09:22:32.142: vmx| [msg.disk.noBackEnd] Cannot open the disk 'myvm.vmdk' or one of the snapshot disks it depends on.
    Jul 22 09:22:32.142: vmx| [msg.disk.configureDiskError] Reason: The system cannot find the file specified.

  • In the /var/log/hostd.log file of an ESXi 5.0 host, you see messages similar to:

    2011-07-13T17:59:48.704Z [74258B90 info 'Libs'] FoundryVMDBPowerOpCallback: VMDB reports power op failed for VM /vmfs/volumes/4e1
    c0986-14d88a26-416a-000c2988e4dd/myvm/myvm.vmx with error msg = "VMware ESX cannot find the virtual disk "myvm_2.vmdk". Verify the path is valid and try again.
    --> Cannot open the disk 'myvm_2.vmdk' or one of the snapshot disks it depends on.
    --> Reason: The system cannot find the file specified." and error code -57.
    2011-07-13T17:59:48.705Z [74258B90 info 'Libs'] Vix: [3057 foundryVMMsgPost.c:1354]: Error VIX_E_FAIL in FoundryVMGetMsgPostError
    (): Unknown error
    ...
    2011-07-13T17:59:48.775Z [743C0B90 info 'vm:/vmfs/volumes/4e1c0986-14d88a26-416a-000c2988e4dd/myvm/myvm.vmx' opID=841A7BF1-00000188] Failed to do Power Op: Error: (4) A file was not found


Solution

Overview steps

Note: It would be advisable to attempt to restore the missing descriptor file from backups if possible. If this is not possible, proceed with recreating the virtual machine disk descriptor file.

To create a virtual machine disk descriptor file:

  1. Identify the size of the flat file in bytes.
  2. Create a new blank virtual disk that is the same size as the original. This serves as a baseline example that is modified in later steps.

    Note: This step is critical to assure proper disk geometry.
  3. Rename the descriptor file (also referred to as a header file) of the newly-created disk to match the name of the original virtual disk.
  4. Modify the contents of the renamed descriptor file to reference the flat file.
  5. Remove the leftover temporary flat file of the newly-created disk, as it is not required.

Note: This procedure will not work on virtual disks configured with a Para-virtualized SCSI controller in the virtual machine.

Detailed steps

To create a virtual machine disk:

  1. Log into the terminal of the ESXi/ESX host:

  2. Navigate to the directory that contains the virtual machine disk with the missing descriptor file using the command:

    #
    Code:
    cd /vmfs/volumes/myvmfsvolume/mydir
    Note:
    • If you are using a version of ESXi, you can access and modify files and directories using the vSphere Client Datastore Browser or the vifs utility included with the vSphere CLI. For more information, see the section Performing File System Operations in the vSphere Command-Line Interface Documentation.
    • If you are using VMware Fusion, the default location for the virtual machine files is the home/Documents/Virtual Machines.localized/virtual_machine/ folder, where home is your home folder, and virtual_machine is the name of the virtual machine.


  3. Identify the type of SCSI controller the virtual disk is using. You can do this by examining the virtual machine configuration file (.vmx ). The controller is identified by the line scsi#.virtualDev , where # is the controller number. There may be more than one controller and controller type attached to the virtual machine, such as lsisas1068 (which is the LSILogic SAS controller),lsilogic , or buslogic . This example uses lsilogic :

    Code:
    scsi0.present = "true"
    scsi0.sharedBus = "none"
    scsi1.present = "true"
    scsi1.sharedBus = "virtual"
    scsi1.virtualDev = "lsilogic" 
  4. Identify and record the exact size of the -flat file using a command similar to:

    Code:
    # ls -l vmdisk0-flat.vmdk
    
    -rw------- 1 root root 4294967296 Oct 11 12:30 vmdisk0-flat.vmdk 
  5. Use the vmkfstools command to create a new virtual disk:

    Code:
    # vmkfstools -c 4294967296 -a lsilogic -d thin temp.vmdk 
    The command uses these flags:
    • -c size

      This is the size of the virtual disk.
    • -a virtual_controller

      Whether the virtual disk was configured to work with BusLogic, LSILogic (for both lsilogic and lsilogic SAS) or IDE.
    • -d thin

      This creates the disk in thin-provisioned format.


    Note: To save disk space, we create the disk in thin-provisioned format using the type thin . The resulting flat file then consumes minimal amounts of space (1 MB) instead of immediately assuming the capacity specified with the -c switch. The only consequence, however, is the descriptor file contains an extra line that must be manually removed in a later step.

    The temp.vmdk and temp-flat.vmdk files are created as a result.

  6. Delete temp-flat.vmdk , as it is not needed. Run the command:

    Code:
    # rm temp-flat.vmdk 
  7. Rename temp.vmdk to the name that is required to match the orphaned .flat file (or vmdisk0.vmdk , in this example):

    Code:
    # mv temp.vmdk vmdisk0.vmdk 
  8. Edit the descriptor file with a text editor:
    1. Under the Extent Description section, change the name of the .flat file to match the orphaned .flat file you have.
    2. Find and remove the line ddb.thinProvisioned = "1" if the original .vmdk was not a thin disk. If it was, retain this line.

      Code:
      # Disk DescriptorFile
      version=1
      CID=fb183c20
      parentCID=ffffffff
      createType="vmfs"
      
      # Extent description
      RW 8388608 VMFS "vmdisk0-flat.vmdk" 
      
      # The Disk Data Base
      #DDB
      
      ddb.virtualHWVersion = "4"
      ddb.geometry.cylinders = "522"
      ddb.geometry.heads = "255"
      ddb.geometry.sectors = "63"
      ddb.adapterType = "lsilogic"
      ddb.thinProvisioned = "1" 
      The virtual machine is now ready to power on. Verify your changes before starting the virtual machine.

      If powering on the virtual machine is not successful, see Troubleshooting a virtual machine that is unable to power on (2001005).

  9. To check the disk chain for consistency, run this command against the disk descriptor file:

    For ESXi 5.x:
    # vmkfstools -e filename.vmdk

    For a complete chain, you see output similar to:
    Disk chain is consistent

    For a broken chain, you will see a summary of the snapshot chain and then an output similar to:
    Disk chain is not consistent : The parent virtual disk has been modified since the child was created. The content ID of the parent virtual disk does not match the corresponding parent content ID in the child (18)

    For ESXi 3.5/4.x:
    # vmkfstools -q filename.vmdk

    For a complete chain, you see output similar to:
    filename.vmdk is not an rdm

    For a broken chain, you see output similar to:
    Failed to open 'test-000001.vmdk' : The parent virtual disk has been modified since the child was created (18)

    Note: The primary purpose of the vmkfstools -q command is(from the vmkfstools help page: vmkfstools -h ): -q --queryrdm .
    This command will identify any issues with the snapshot chain, not its intended purpose, which is to identify if the vmdk disk is a raw device mapping.

    For more information on the vmkfstools command see: vmkfstools - vSphere CLI for managing VMFS volumes


Notes:

  • Although these steps are specific to the ESX host's Service Console (otherwise referred to as the Console OS), they may also be applied to platforms using VMware desktop products such as VMware Server or VMware Workstation. Instead of using thevmkfstools command, use the vmware-vdiskmanager utility. For more information on the Virtual Disk Manager, see theVirtual Disk Manager User's Guide.
  • When performing these operations on a different platform, you must use the equivalent commands for your platform. For example,move , copy , and del for Microsoft Windows platforms, and mv , cp , and rm for Unix, Linux, and Mac OS X platforms.
  • For more information about platform-specific commands, see DOS versus Linux Commands from the Linux Documentation Project.

    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Recreating a missing virtual disk (VMDK) descriptor file for delta disks (1026353)

Purpose

This article provides steps to recreate a delta virtual disk's descriptor file, based on the vmfsSparse disk format. This disk format is often facilitated by virtual machine snapshot functionality, live virtual machine clone operations, and Storage vMotion.

THE CONTENT OF THIS ARTICLE IS PROVIDED "AS-IS," AND TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, VMWARE DISCLAIMS ALL OTHER REPRESENTATIONS AND WARRANTIES, EXPRESS OR IMPLIED, REGARDING THIS CONTENT, INCLUDING THEIR FITNESS FOR A PARTICULAR PURPOSE, THEIR MERCHANTABILITY, OR THEIR NONINFRINGEMENT. VMWARE SHALL NOT BE LIABLE FOR ANY DAMAGES ARISING OUT OF OR IN CONNECTION WITH THE USE OF THIS CONTENT, INCLUDING DIRECT, INDIRECT, CONSEQUENTIAL DAMAGES, LOSS OF BUSINESS PROFITS OR SPECIAL DAMAGES, EVEN IF VMWARE HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Resolution

A disk file of this type consists of a virtual disk descriptor file and paired delta extent file. These data files are based on the vmfsSparseformat, intended to store delta content or "change data" for a period of time, otherwise known as a snapshot.

Notes:


To recreate a missing virtual disk (VMDK) descriptor file for delta disks:

  1. Confirm if the virtual machine directory contains either another delta disk descriptor file, or a base disk's descriptor, for use as a basis or template. For example, you will see some or all of these files in the virtual machine directory:

    Code:
    drwxr-xr-x 1 root root        1400 Aug 16 09:39 .
    drwxr-xr-t 1 root root        2520 Aug 16 09:32 ..
    -rw------- 1 root root       32768 Aug 17 19:11 examplevm-000002-delta.vmdk
    -rw------- 1 root root       32768 Aug 17 19:11 examplevm-000002.vmdk
    -rw------- 1 root root       32768 Aug 16 14:39 examplevm-000001-delta.vmdk
    -rw------- 1 root root       32768 Aug 16 14:39 examplevm-000001.vmdk
    -rw------- 1 root root 16106127360 Aug 16 09:32 examplevm-flat.vmdk
    -rw------- 1 root root         469 Aug 16 09:32 examplevm.vmdk
    -rw------- 1 root root       18396 Aug 16 14:39 examplevm-Snapshot1.vmsn
    -rw------- 1 root root       18396 Aug 17 19:11 examplevm-Snapshot2.vmsn
    -rw------- 1 root root         397 Aug 16 09:39 examplevm.vmsn
    -rwxr-xr-x 1 root root        1626 Aug 16 09:39 examplevm.vmx
    -rw------- 1 root root         259 Aug 16 09:36 examplevm.vmxf
    Notes:
    • Items highlighted in red are descriptor files which are ideally used as a template or base, if they are present.
    • If you are missing the base disk's descriptor file (or examplevm.vmdk), see Recreating a missing virtual machine disk (VMDK) descriptor file (1002511).

      If you have additional delta files with intact descriptor files, you may copy one of them for use as a template. Alternatively, copy the base disk descriptor file, or use the right-most example in the table below and customize the copied content.

      Warning: Ensure that character encoding for any files modified outside of an ESXi/ESX host console or SSH session conform to UTF-8. For more information, see the Preferred Editors section in Editing configuration files in VMware ESXi and ESX (1017022). Unsupported encoding results in failure to start the virtual machine.
    • For ESXi 5.0 and later, virtual machines have their delta files in the same directory as the disk they relate to. For more information on this change, see Changing the location of snapshot delta files for virtual machines in ESXi 5.0 (2007563)

      For example, a virtual machine with two disks each on different datastores:
      • The first disk (disk.vmdk) is located on datastore1
      • The second disk (disk_1.vmdk) is located on datastore2


      After taking a snapshot, the delta file for the first disk (disk-000001.vmdk) will be stored on datastore1, and the delta file for the second disk (disk_1-000001.vmdk) will be stored on datastore2.

  2. Assume that the descriptor file for the delta disk examplevm-000001.vmdk is missing and that there are no issues with its associated delta file examplevm-000001-delta.vmdk.

    To recreate the delta descriptor file, copy the current base disk descriptor file examplevm.vmdk with the name of the missing descriptor file examplevm-000001.vmdk:

    Code:
    # cp examplevm.vmdk examplevm-000001.vmdk
    We can simply edit this new snapshot delta descriptor file to change its format from a base disk descriptor to a snapshot delta disk descriptor.

    Modify the new examplevm-000001.vmdk delta descriptor file to contain this information:

    Base disk descriptor file examplevm.vmdk Recreated delta descriptor examplevm-000001.vmdk(originally copied from examplevm.vmdk)
    # Disk DescriptorFile
    version=1
    encoding="UTF-8"
    CID=19741890
    parentCID=ffffffff
    createType="vmfs"

    # Extent description
    RW 31457280 VMFS "examplevm-flat.vmdk"

    # The Disk Data Base
    #DDB

    ddb.virtualHWVersion = "7"
    ddb.longContentID = "5fd87dda1dc77cafd5be
    881a19741890"
    ddb.uuid = "60 00 C2 9e 3d 8d 45 82-dd 1f e4 93 22 da 9c 61"
    ddb.geometry.cylinders = "1958"
    ddb.geometry.heads = "255"
    ddb.geometry.sectors = "63"
    ddb.adapterType = "lsilogic"
    # Disk DescriptorFile
    version=1
    encoding="UTF-8"
    CID=7f3a1e17
    parentCID=19741890
    createType="vmfsSparse"
    parentFileNameHint="examplevm.vmdk"

    # Extent description
    RW 31457280 VMFSSPARSE "examplevm-000001-delta.vmdk"

    # The Disk Data Base
    #DDB

    ddb.longContentID = "5fd87dda1dc77cafd5be
    881a19741890"

    Notes:
    • If you are creating the snapshot descriptor file from the the base disk description make sure that you change the entry fromVMFS to VMFSSPARSE.
    • If the base disk is greater than 2 TB (2 TB+ only supported in ESXi 5.5) in size, these changes are required:
      • createType="vmfsSparse" replaced with createType="seSparse"
      • VMFSSPARSE under Extent Description replaced with SESPARSE
      • examplevm-000001-delta.vmdk replaced with examplevm-000001-sesparse.vmdk


    • CID and parentCID values are Content Identifiers (CID). For more information on the CID and how to correct CID mismatch errors, see Resolving the CID mismatch error: The parent virtual disk has been modified since the child was created (1007969).

      The recreated file requires use of the CID value from its parent. In the example above, examplevm.vmdk was the parent. Thus, the CID from examplevm.vmdk (the parent) matches the parentCID value from examplevm-000001.vmdk(the child).

      If there are other child delta disks present, such as examplevm-000002.vmdk (not shown in the example above), review the parentCID for the child and ensure it matches the CID parameter of your recreated descriptor file.
    • Delta disks are in the vmfsSparse format; specify this for the createType parameter.
      • Set the parentFileNameHint to the parent virtual disk file. This contains a virtual machine base disk or a deltadescriptor's file name.

        Determining this may not be an immediately clear or simple process, depending on the information available.

        Refer to the relevant vmware.log file(s) for the virtual machine when it last successfully started to determine the opening sequence of individual disk files:

        vmx| DISK: OPEN scsi0:0 '/vmfs/volumes/4b96afb0-b2474ede-fc0b-001aa004abc2/examplevm/examplevm-000003.vmdk' persistent R[]
        vmx| DISKLIB-VMFS : "/vmfs/volumes/4b96afb0-b2474ede-fc0b-001aa004abc2/examplevm/examplevm-000003-delta.vmdk" : open successful (10) size = 32768, hd = 65613. Type 8
        vmx| DISKLIB-DSCPTR: Opened [0]: "examplevm-000003-delta.vmdk" (0xa)
        vmx| DISKLIB-LINK : Opened '/vmfs/volumes/4b96afb0-b2474ede-fc0b-001aa004abc2/examplevm/examplevm-000003.vmdk' (0xa): vmfsSparse, 31457280 sectors / 15 GB.
        vmx| DISKLIB-VMFS : "/vmfs/volumes/4b96afb0-b2474ede-fc0b-001aa004abc2/examplevm/examplevm-000002-delta.vmdk" : open successful (14) size = 32768, hd = 65614. Type 8
        vmx| DISKLIB-DSCPTR: Opened [0]: "examplevm-000002-delta.vmdk" (0xe)
        vmx| DISKLIB-LINK : Opened '/vmfs/volumes/4b96afb0-b2474ede-fc0b-001aa004abc2/examplevm/examplevm-000002.vmdk' (0xe): vmfsSparse, 31457280 sectors / 15 GB.
        vmx| DISKLIB-VMFS : "/vmfs/volumes/4b96afb0-b2474ede-fc0b-001aa004abc2/examplevm/examplevm-000001-delta.vmdk" : open successful (14) size = 32768, hd = 65615. Type 8
        vmx| DISKLIB-DSCPTR: Opened [0]: "examplevm-000001-delta.vmdk" (0xe)
        vmx| DISKLIB-LINK : Opened '/vmfs/volumes/4b96afb0-b2474ede-fc0b-001aa004abc2/examplevm/examplevm-000001.vmdk' (0xe): vmfsSparse, 31457280 sectors / 15 GB.
        vmx| DISKLIB-VMFS : "/vmfs/volumes/4b96afb0-b2474ede-fc0b-001aa004abc2/examplevm/examplevm-flat.vmdk" : open successful (14) size = 16106127360, hd = 65616. Type 3
        vmx| DISKLIB-DSCPTR: Opened [0]: "examplevm.vmdk" (0xe)
        vmx| DISKLIB-LINK : Opened '/vmfs/volumes/4b96afb0-b2474ede-fc0b-001aa004abc2/examplevm/examplevm.vmdk' (0xe): vmfs, 31457280 sectors / 15 GB.
        vmx| DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 4, numSubChains = 1
        vmx| DISKLIB-LIB : Opened "/vmfs/volumes/4b96afb0-b2474ede-fc0b-001aa004abc2/examplevm/examplevm-000003.vmdk" (flags 0xa).
        vmx| DISK: OPEN '/vmfs/volumes/4b96afb0-b2474ede-fc0b-001aa004abc2/examplevm/examplevm-000003.vmdk' Geo (1958/255/63) BIOS Geo (0/0/0)


        The parts highlighted in blue are the fundamental configuration parameters used in the virtual machine's virtual disk descriptor files. The layout of a virtual disk tree can be determined using the same log file, based on the sequence and timeline of files opened. In this example:
        • One disk for the virtual machine, SCSI 0:0, was opened. The file specified was examplevm-000003.vmdk.
        • This disk had a parent, and referenced examplevm-000002.vmdk. This also had a parent and referencedexamplevm-000001.vmdk.
        • An intermediary disk file itself, it referenced examplevm.vmdk as its parent. This was the base disk for the virtual machine.


      • If there are no log files available, the creation or modification dates for the files in the directory may be considered when establishing delta disk ordering.
      • Reviewing the descriptor files for each of the remaining virtual machine delta disks will also shed light on the virtual machine's existing layout.

        Warning: These files are not always ordered numerically. Incorrectly ordering the disk files can cause severe file system corruption in the guest operating system and/or trigger CID mismatch conditions when the virtual machine is powered on. If in doubt, stop and contact VMware Technical Support to file a support request. For more information, see Filing a Support Request in My VMware (2006985).

    • The Extent description specifies the corresponding data file containing the virtual machine disk's data:

      Code:
      RW 31457280 VMFS "examplevm-flat.vmdk"
      The RW number is the number of 512-Byte sectors (or "sector count") for the extent or disk file. A base disk's RW value is used throughout the chain of delta disks attached to it; no child virtual disk files should deviate from the defined number and will contain the same value.
    • The DDB section, unlike base disk descriptor files, does not contain geometry information. Only a long content ID is used (which pertains to ESX 4.x virtual machines and newer):

      ddb.longContentID = "5fd87dda1dc77cafd5be881a19741890"

      This value also matches the parent disks (and base) associated to the delta virtual disk. Do not include geometry information for the delta disk descriptor files.