GlusterFS 3.2.6 for XenServer 6.0
Posted by admin in 
I’ve been wanting to test GlusterFS running natively under XenServer for quite some time. In order to do so, I needed to compile GlusterFS specifically for XenServer 6.0. I wouldn’t do this on a production server.
Here’s how I did it, and the resulting RPM’s.
Compiling GlusterFS
Fuse is already included in XenServer 6.0, so all we really need to do is install the compiler. In order to get the compiler I folowed instructions found on the Citrix XenServer forums. Log into your XenServer and run the following:
# cd /etc/yum.repos.d # mv CentOS-Base.repo CentOS-Base.repo-old # mv CentOS-Base.repo.orig CentOS-Base.repo # yum install gcc make automake
Once the compiler is installed use wget to fetch the latest GlusterFS source. Do not untar and ungzip the archive. We’ll also install some support utilities needed for compiling. They are not needed for installing.
# wget http://download.gluster.com/pub/gluster/glusterfs/LATEST/glusterfs-3.2.6.tar.gz # yum install flex # yum install bison # yum install python-ctypes # yum install fusermount # yum install readline # yum install rpm-build # yum install libibverbs-devel
Once that is done, we’ll build the GlusterFS RPM’s.
# rpmbuild -ba glusterfs.spec # cd /usr/src/redhat/RPMS/i386/ # rpm -ivh glusterfs-core-3.2.6-1.i386.rpm glusterfs-fuse-3.2.6-1.i386.rpm
Installing glusterfs-georeplication requires a better version of rsync than XenServer ships with, you can just ‘yum install rsync’ if you need it.
The system is installed and almost ready to go, just load the FUSE module and run the gluster commands as per normal.
The RPM’s can be installed in a standard install of XenServer, if I remember correctly.
glusterfs-core-3.2.6-1.i386.rpm
glusterfs-fuse-3.2.6-1.i386.rpm
glusterfs-geo-replication-3.2.6-1.i386.rpm
glusterfs-rdma-3.2.6-1.i386.rpm
I have done no testing on these, besides verify that *core* and *fuse* install and mount a GlusterFS export.
XenServer, iSCSI, and GlusterFS(NFS)
I’ve been running XenServer on iSCSI for quite some time. Performance and reliability have not always been the best, and sometimes it seems as though XenServer gets confused with snapshots and it’s LVM management.
I decided I needed to either a) switch to NFS or b) change to a different iSCSI provider.
Before I decided anything, I performed some tests. What kinds of IOPS was I getting from NFS vs iSCSI? What about my transfer rate? All this information needed to be gathered from a VM. This is the result of my tests.
Before we start, I’d like to explain why I’ve chosen to use the GlusterFS NFS server over the kernel NFS server. There are three reasons: 1) the kernel NFS server was not stable in my situation, and XenServer lost the NFS share often enough to cause issues. 2) The kernel NFS server was slower in most cases than the Gluster NFS server. Unfortunately, I no longer have the numbers to prove it, so I’ll have to run those tests again another time. 3) I could get iSCSI failover using DRBD and Gluster failover using it’s replication (both over a separate NIC). In theory, I could have done that with Kernel NFS and DRBD as well, but I decided against it.
I was interested in both IOPS and speed of transfer. I needed good IOPS in order to host an SQL Server and a Zimbra server on the SR, while I needed good speeds to host a file server or two.
I did not do any tests using the GlusterFS replication or iSCSI/DRBD.