Rebooting a hung HVM in XenServer
Posted by admin in 
Citrix XenServer has a problem that seems to have become significantly more prominent in the latest 5.6 FP1 release.
HVM (Hardware Virtual Machine) is a true virtualized server. It uses the Intel VT-x and AMD-V extensions to provide virtualization.. XenServer provides a QEMU based emulation layer for devices.
PVM (Para-Virtualized Machine) is a virtualized server that requires modified kernel that is ‘Xen aware’ to run. This kernel has special ties into Xen’s ABI, and typically runs faster than an HVM.
Windows based VM’s are para-virtualized by default. Since Citrix/Xen cannot modify the Windows kernel, they use specific device drivers to provide the para-virtualization layer. Linux based VM’s can be PVM’s or HVM’s, based on their support for Xen. Since Citrix has access to the Linux kernel source, they can directly modify (or distributions can modify) the kernel to talk to the Xen ABI. What this basically ends up doing, is XenServer only supports para-virtualization with specific Linux distributions. If you want to run a distribution that is not supported by Citrix, it by default becomes an HVM.
The ‘hung HVM’ issue I’ve seen only affects Linux HVM’s. The Linux HVM essentially locks, becoming non-responsive to pings, local console, basically everything. Using the tools provided by XenServer (XenCenter or command line) have no effect. The VM cannot be shutdown or rebooted. If this was a real machine, you could pull the power cord and deal with the results later. At least you could get the machine up and running again.
My past efforts to reboot a hung HVM was to migrate or shutdown all the VM’s running on the XenServer, and rebooting the XenServer itself. Then bringing the VM’s back. This was a task that would usually have to wait until after business hours. The hit on server downtime and impact on users was usually too great to do the reboot during the day.
A discussion on the Citrix forums finally brought a solution to light. It’s entirely command line based, can be done during business hours, and doesn’t affect any other running VM. A good solution.
- Find the UUID of the hung VM.
You can do this via the command line with ‘xe vm-list’ or via XenCenter. - Find the Domain ID of the hung VM.
Run ‘list_domains’ from the command line, and match the UUID with the ID numberid | uuid | state
0 | 2fe455fe-3185-4abc-bff6-a3e9a04680b0 | R
47 | 267227f3-a59e-dafe-b183-82210cf51ec4 | B
59 | 298817fb-8a3e-7501-11e0-045a8aa860ff | B
60 | 46e3d5aa-2f02-dfdc-b053-9a8ac56ec5d1 | B
61 | 16cf3204-eb17-5a12-e8d0-c72087bda690 | B
62 | 1f9053b5-c6ca-40bb-504e-3017c37e7281 | H
63 | ddaec491-097a-e271-362b-f2f985e26e4a | R
65 | 55f3b225-4f65-d1ea-aa19-add44c5acce7 | B
66 | 7adef6fd-9171-5426-b333-6fb1b57b8e60 | B H
67 | 6046dc13-f70b-8398-56fb-069c22440a7c | B
68 | f201cd94-a501-00c2-d21e-8c2f03ea167b | B H
In our case, UUID 1f9053b5-c6ca-40bb-504e-3017c37e7281 is hung, which is Domain ID 62.
- Run destroy_domain on the Domain ID.
# /opt/xensource/debug/destroy_domain -domid 62
- The VM will still show itself as running, so now, we need to reboot it.
# xe vm-reboot name-label='name of the VM' --force
- The VM is now rebooted, and you can bring it up as if you had just pulled the plug. That is, check for some disk corruption, etc.
Thanks go to the Citrix Forum folks for helping us come up with a solution.