GlusterFS 3.2.6 for XenServer 6.0
Posted by admin in 
I’ve been wanting to test GlusterFS running natively under XenServer for quite some time. In order to do so, I needed to compile GlusterFS specifically for XenServer 6.0. I wouldn’t do this on a production server.
Here’s how I did it, and the resulting RPM’s.
Compiling GlusterFS
Fuse is already included in XenServer 6.0, so all we really need to do is install the compiler. In order to get the compiler I folowed instructions found on the Citrix XenServer forums. Log into your XenServer and run the following:
# cd /etc/yum.repos.d # mv CentOS-Base.repo CentOS-Base.repo-old # mv CentOS-Base.repo.orig CentOS-Base.repo # yum install gcc make automake
Once the compiler is installed use wget to fetch the latest GlusterFS source. Do not untar and ungzip the archive. We’ll also install some support utilities needed for compiling. They are not needed for installing.
# wget http://download.gluster.com/pub/gluster/glusterfs/LATEST/glusterfs-3.2.6.tar.gz # yum install flex # yum install bison # yum install python-ctypes # yum install fusermount # yum install readline # yum install rpm-build # yum install libibverbs-devel
Once that is done, we’ll build the GlusterFS RPM’s.
# rpmbuild -ba glusterfs.spec # cd /usr/src/redhat/RPMS/i386/ # rpm -ivh glusterfs-core-3.2.6-1.i386.rpm glusterfs-fuse-3.2.6-1.i386.rpm
Installing glusterfs-georeplication requires a better version of rsync than XenServer ships with, you can just ‘yum install rsync’ if you need it.
The system is installed and almost ready to go, just load the FUSE module and run the gluster commands as per normal.
The RPM’s can be installed in a standard install of XenServer, if I remember correctly.
glusterfs-core-3.2.6-1.i386.rpm
glusterfs-fuse-3.2.6-1.i386.rpm
glusterfs-geo-replication-3.2.6-1.i386.rpm
glusterfs-rdma-3.2.6-1.i386.rpm
I have done no testing on these, besides verify that *core* and *fuse* install and mount a GlusterFS export.
XenServer, iSCSI, and GlusterFS(NFS)
I’ve been running XenServer on iSCSI for quite some time. Performance and reliability have not always been the best, and sometimes it seems as though XenServer gets confused with snapshots and it’s LVM management.
I decided I needed to either a) switch to NFS or b) change to a different iSCSI provider.
Before I decided anything, I performed some tests. What kinds of IOPS was I getting from NFS vs iSCSI? What about my transfer rate? All this information needed to be gathered from a VM. This is the result of my tests.
Before we start, I’d like to explain why I’ve chosen to use the GlusterFS NFS server over the kernel NFS server. There are three reasons: 1) the kernel NFS server was not stable in my situation, and XenServer lost the NFS share often enough to cause issues. 2) The kernel NFS server was slower in most cases than the Gluster NFS server. Unfortunately, I no longer have the numbers to prove it, so I’ll have to run those tests again another time. 3) I could get iSCSI failover using DRBD and Gluster failover using it’s replication (both over a separate NIC). In theory, I could have done that with Kernel NFS and DRBD as well, but I decided against it.
I was interested in both IOPS and speed of transfer. I needed good IOPS in order to host an SQL Server and a Zimbra server on the SR, while I needed good speeds to host a file server or two.
I did not do any tests using the GlusterFS replication or iSCSI/DRBD.
read more
Rebooting a hung HVM in XenServer
Citrix XenServer has a problem that seems to have become significantly more prominent in the latest 5.6 FP1 release.
HVM (Hardware Virtual Machine) is a true virtualized server. It uses the Intel VT-x and AMD-V extensions to provide virtualization.. XenServer provides a QEMU based emulation layer for devices.
PVM (Para-Virtualized Machine) is a virtualized server that requires modified kernel that is ‘Xen aware’ to run. This kernel has special ties into Xen’s ABI, and typically runs faster than an HVM.
Windows based VM’s are para-virtualized by default. Since Citrix/Xen cannot modify the Windows kernel, they use specific device drivers to provide the para-virtualization layer. Linux based VM’s can be PVM’s or HVM’s, based on their support for Xen. Since Citrix has access to the Linux kernel source, they can directly modify (or distributions can modify) the kernel to talk to the Xen ABI. What this basically ends up doing, is XenServer only supports para-virtualization with specific Linux distributions. If you want to run a distribution that is not supported by Citrix, it by default becomes an HVM.
The ‘hung HVM’ issue I’ve seen only affects Linux HVM’s. The Linux HVM essentially locks, becoming non-responsive to pings, local console, basically everything. Using the tools provided by XenServer (XenCenter or command line) have no effect. The VM cannot be shutdown or rebooted. If this was a real machine, you could pull the power cord and deal with the results later. At least you could get the machine up and running again.
My past efforts to reboot a hung HVM was to migrate or shutdown all the VM’s running on the XenServer, and rebooting the XenServer itself. Then bringing the VM’s back. This was a task that would usually have to wait until after business hours. The hit on server downtime and impact on users was usually too great to do the reboot during the day.
A discussion on the Citrix forums finally brought a solution to light. It’s entirely command line based, can be done during business hours, and doesn’t affect any other running VM. A good solution.
- Find the UUID of the hung VM.
You can do this via the command line with ‘xe vm-list’ or via XenCenter. - Find the Domain ID of the hung VM.
Run ‘list_domains’ from the command line, and match the UUID with the ID numberid | uuid | state
0 | 2fe455fe-3185-4abc-bff6-a3e9a04680b0 | R
47 | 267227f3-a59e-dafe-b183-82210cf51ec4 | B
59 | 298817fb-8a3e-7501-11e0-045a8aa860ff | B
60 | 46e3d5aa-2f02-dfdc-b053-9a8ac56ec5d1 | B
61 | 16cf3204-eb17-5a12-e8d0-c72087bda690 | B
62 | 1f9053b5-c6ca-40bb-504e-3017c37e7281 | H
63 | ddaec491-097a-e271-362b-f2f985e26e4a | R
65 | 55f3b225-4f65-d1ea-aa19-add44c5acce7 | B
66 | 7adef6fd-9171-5426-b333-6fb1b57b8e60 | B H
67 | 6046dc13-f70b-8398-56fb-069c22440a7c | B
68 | f201cd94-a501-00c2-d21e-8c2f03ea167b | B H
In our case, UUID 1f9053b5-c6ca-40bb-504e-3017c37e7281 is hung, which is Domain ID 62.
- Run destroy_domain on the Domain ID.
# /opt/xensource/debug/destroy_domain -domid 62
- The VM will still show itself as running, so now, we need to reboot it.
# xe vm-reboot name-label='name of the VM' --force
- The VM is now rebooted, and you can bring it up as if you had just pulled the plug. That is, check for some disk corruption, etc.
Thanks go to the Citrix Forum folks for helping us come up with a solution.