Migrating virtual machines from VMware ESXi to Proxmox VE

July 25, 2021 virtualization linux

I recently migrated virtual machines from VMware ESXi to Proxmox VE, and hit a few quirks in the process. Here are my notes in case that may help.

Both VMware ESXi and Proxmox VE support OVF. The goal is to export a VM in OVF format from VMware ESXi and then import the OVF on Proxmox VE.

Prerequisites

Assuming VMware ESXi is accessible from the Proxmox VE, we can use VMware OVF Tool directly from Proxmox VE in order to export and copy the OVF in one pass.

Install VMware OVF Tool on Proxmox VE:
- Download .bundle file (requires a VMware account, can create one for free).
- chmod +x and run it.
Shutdown the VM on VMware ESXi.

Export VM from VMware ESXi

Usage:

# ovftool vi://<username>@<host>:<port>/

If using SSL with a self-signed certificate, add --noSSLVerify.

OVF Tool will prompt for the password and list objects available:

# ovftool --noSSLVerify vi://root@XXX.XXX.XXX.XXX:443/
Enter login information for source vi://XXX.XXX.XXX.XXX/
Username: root
Password: ***************
Error: Found wrong kind of object (ResourcePool). Possible completions are:
  vm1
  vm2
  dir1/
  dir2/

We can recursively list objects, e.g. here for a directory:

# ovftool --noSSLVerify vi://root@XXX.XXX.XXX.XXX:443/dir1/
Enter login information for source vi://XXX.XXX.XXX.XXX/
Username: root
Password: ***************
Error: Found wrong kind of object (ResourcePool). Possible completions are:
  vm3

When accessing a VM, we are displayed the VM details to be exported via OVF:

# ovftool --noSSLVerify vi://root@XXX.XXX.XXX.XXX:443/dir1/vm3
Enter login information for source vi://XXX.XXX.XXX.XXX/
Username: root
Password: ***************
OVF version:   1.0
VirtualApp:    false
Name:          vm3

Download Size:  Unknown

Deployment Sizes:
  Flat disks:   100.00 GB
  Sparse disks: Unknown

Networks:
  Name:        Foo
  Description: Bar

Virtual Machines:
  Name:               vm3
  Operating System:   debian10_64guest
  Virtual Hardware:
    Families:         vmx-13
    Number of CPUs:   4
    Cores per socket: 4
    Memory:           6.00 GB

    Disks:
      Index:          0
      Instance ID:    9
      Capacity:       50.00 GB
      Disk Types:     SCSI-VirtualSCSI

      Index:          1
      Instance ID:    10
      Capacity:       50.00 GB
      Disk Types:     SCSI-VirtualSCSI

    NICs:
      Adapter Type:   VmxNet3
      Connection:     Foo

References:
  File:  /12/ParaVirtualSCSIController0:0
  File:  /12/ParaVirtualSCSIController0:1

Once we have identified the VM to export, we can write the OVF to a local file on Proxmox VE:

# ovftool --noSSLVerify vi://root@XXX.XXX.XXX.XXX:443/dir2/vm4 .
Enter login information for source vi://XXX.XXX.XXX.XXX/
Username: root
Password: ***************
Opening VI source: vi://root@XXX.XXX.XXX.XXX:443/dir2/vm4
Opening VI source: vi://root@XXX.XXX.XXX.XXX:443/dir2/vm4
Opening OVF target: .
Writing OVF package: ./vm4/vm4.ovf
Disk progress: 12%
[...]
Transfer Completed
Completed successfully

Issues

For some VMs, I hit this error:

# ovftool --noSSLVerify vi://root@XXX.XXX.XXX.XXX:443/dir2/vm4
Enter login information for source vi://XXX.XXX.XXX.XXX/
Username: root
Password: ***************
Error: Fault cause: vim.fault.FileNotFound

Diagnosis: CD/ROM device attached to the VM references a Datastore ISO File that OVF Tool cannot access.
Solution: detach the CD/ROM device from the VM.

Import VM to Proxmox VE

With the OVF available locally, we can import the VM using the qm command line tool.

Usage:

qm importovf <id> path/to/file.ovf <proxmox_storage>

We may also specify --format <qcow2 | raw | vmdk> to use a specific disk image format on import. This will have no effect when importing into ZFS storage since it uses block device emulation and only allows raw.

Example using the OVF from above:

qm importovf 400 vm4/vm4.ovf local-zfs
transferred 0.0 B of 50.0 GiB (0.00%)
[...]
transferred 50.0 GiB of 50.0 GiB (100.00%)

The VM should now be available in Proxmox VE web GUI.

Post-migration checkup

Check the imported configuration. In particular:

Review VM hardware:
- Consider switching CPU type (ideally host for better performance).
- Consider switching SCSI controller type (ideally VirtIO for better performance).
- Add network(s) device(s), since these are not imported and must be recreated (ideally VirtIO for better performance).
Review VM options:
- Consider starting VM at boot.
- Adjust OS type to match guest OS.

Once satisfied, try to start the VM. With the VM up and running, we can:

Update network configuration as the interface name and/or gateway might have changed.
Install QEMU Guest Agent:
- On Debian-based systems, this is as simple as:
```
# apt install qemu-guest-agent
```
- Once installed, stop VM and enable QEMU Guest Agent in the VM options.
- Start VM again and check that the QEMU Guest Agent service is running:
```
# systemctl status qemu-guest-agent
```

Remove VMware guest additions:

Some pointers to check if they were installed:

# ls /usr/lib/vmware-tools
# systemctl status vmware
# which vmware-toolbox
# which vmware-toolbox-cmd
# dpkg -l | grep -I vm
# apt search open-vm-tools

And then:

# vmware-uninstall-tools.pl
# apt purge open-vm-tools

Issues

Some VMs failed to boot after the bootloader had successfully passed, not finding the disk partitions.

Diagnosis: missing VirtIO modules.
Solution: temporarily switch the SCSI controller type to VMware PVSCSI, start the VM, and load missing VirtIO modules in initramfs.

On Debian-based systems:

Add the following to /etc/initramfs-tools/modules:

virtio_blk
virtio_scsi

Update initramfs and shutdown VM:

# update-initramfs -u
# shutdown now

Switch back to VirtIO SCSI controller.
VM should now boot properly.

I suspect a similar issue might happen with the network device if the virtio_net module is also missing. To check the list of loaded modules:

# lsmod | grep virtio
virtio_net             53248  0
net_failover           20480  1 virtio_net
virtio_console         32768  1
virtio_balloon         20480  0
virtio_blk             20480  0
virtio_scsi            20480  2
scsi_mod              249856  4 virtio_scsi,sd_mod,libata,sg
virtio_pci             28672  0
virtio_ring            28672  6 virtio_console,virtio_balloon,virtio_scsi,virtio_pci,virtio_blk,virtio_net
virtio                 16384  6 virtio_console,virtio_balloon,virtio_scsi,virtio_pci,virtio_blk,virtio_net

Previous Post Next Post