First, a bit about GlusterFS:

GlusterFS is a scale-out NAS file system developed by Gluster. It aggregates various storage servers over Ethernet or Infiniband RDMA interconnect into one large parallel network file system. GlusterFS is based on a stackable user space design without compromising performance. It has found a variety of applications including cloud computing, biomedical sciences and archival storage. GlusterFS is free software, licensed under GNU GPL v3 license.

The Gluster company was recently purchased by RedHat.

Now a bit about DropBox:

Dropbox is a Web-based file hosting service operated by Dropbox, Inc. that uses cloud storage to enable users to store and share files and folders with others across the Internet using file synchronization.

The portion of DropBox I’ve tried to duplicate is the storage part.  File Sharing with other users is not a goal here.  I need access to my data no matter where I am, even if I don’t have my laptop with me, and I like my data backed up offsite.  In my case, the offsite data is behind a VPN, so all I need to get access to my latest data is something that can attach to the VPN.  My iPhone can do that.  Data syncing works only on the following OS’s (the OS needs good FUSE support:

Recommended: Redhat Enterprise GNU/Linux on x86-64 architecture and Ext3 file system
Tested and Supported: Fedora, Debian, CentOS, Ubuntu GNU/Linux on x86-64 architecture and Ext3 (or Ext4 from Linux-2.6.30 onwards) file system, Oracle Solaris 64bit + ZFS
Known to Work: Mac OS X 64bit + HFS+, FreeBSD 64bit + UFS

GlusterFS is fairly portable to any 64-bit POSIX compliant operating system and disk file system with extended attribute support. Native client port requires FUSE device support.

I run mine on Ubuntu 64-bit with EXT4 or XFS as the underlying filesystem.

So why don’t I use DropBox?  I really dislike having my data sitting on other people’s servers.  My data, my responsibility.

The setup I have is currently on three systems, my laptop (thinkpad), my desktop (quad), and a virtual server sitting in a colo (geo).  Thinkpad I have with me all the time.  It uses GlusterFS’s ‘replica’ mode to duplicate it’s data onto quad.  Quad then uses GlusterFS’s ‘geo-replication’ mode to get my data onto geo.

There are drawbacks to the method I’m using here, for example, if I’m at a client’s location and not on my internal network, thinkpad doesn’t sync with quad, and therefor, data doesn’t get to geo.  This is a limitation I can work with, and everything gets in sync eventually (see below).  Most of the time, I have some sort of VPN to my internal network, so syncs with quad work, albeit slowly.   I’m currently running GlusterFS 3.2.5, but the new GlusterFS 3.3 (now in beta) may solve some of my syncing issues with the new pro-active self heal.

First, we need to install GlusterFS onto all three systems: (thinkpad and quad are running Xubuntu 11.10, while geo is running Ubuntu Server 10.04)

# wget http://download.gluster.com/pub/gluster/glusterfs/LATEST/Ubuntu/11.10/glusterfs_3.2.5-1_amd64.deb
# apt-get install nfs-common
# dpkg --install glusterfs_3.2.5-1_amd64.deb

nfs-common is installed, since GlusterFS also provides an NFS Server component.  We won’t be using it, but it’s a dependency for the .deb.  We need to make sure the gluster daemon starts at system boot and is currently running(on quad and thinkpad, geo does not need it).

# update-rc.d glusterd defaults
# /etc/init.d/glusterd start 

Once GlusterFS is installed, we need to set up replication between thinkpad and quad.  In my case, I’m syncing all of /home.  It’s about 230 GB right now, but after the initial sync, things run very smoothly.  I started with an empty /home on quad.  Having existing data in quad‘s /home may cause problems.  The first thing we’ll do is tell the systems about each other:

thinkpad# gluster peer probe quad
Probe successful

That’s it, both quad and thinkpad know about each other.  Now we’ll create the sync volume.  But first, I unmounted /home on quad and thinkpad, and mounted them in a new position, in my case /gluster/export on thinkpad and since quad didn’t have a separate disk for /home, I create a directory called /raid/gluster/export on my RAID5 array.

thinkpad# gluster volume create sync1 replica 2 transport tcp thinkpad:/gluster/export quad:/raid/gluster/export
Creation of sync1 has been successful
Please start the volume to access data
thinkpad# gluster volume sync1 start 

We’ll just do a quick check to see how things look.

thinkpad# gluster volume info

Volume Name: sync1
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: thinkpad:/gluster/export
Brick2: quad:/raid/gluster/export

Now we’ll mount them at /home for both thinkpad and quad.  I’ll show the thinkpad fstab entry, and let you figure out the quad one.

/dev/sda6         /gluster/export   ext4         noatime,nodiratime 0 3
thinkpad:/sync1   /home             glusterfs    noatime,nodiratime 0 0

At this point, we need to force a sync to occur.  Depending on the amount of data you have, it could take awhile.

thinkpad# find /home -noleaf -print0 | xargs --null stat >/dev/null

Once that’s done, both systems will remain in sync, as long as they are on the same network.  I wouldn’t recommend using both systems at the same time.

One of the issues here is that thinkpad is a laptop.  I take it with me when I see clients, for vacations, to the coffee shop, etc.  That means it’s not always connected to a network, and if it is, not necessarily the same network quad is connected to.  I need to force a sync when thinkpad is back where it can see quad.  The way I deal with that is quite straight forward, I force a self heal whenever the laptop is powered back on.  I initially put the self heal in /etc/rc.local, but Ubuntu seems to run that fairly early in the boot process (S03).  So, I created /etc/init.d/local, and set it to run last in the start-up scripts.

#!/bin/bash

#
# this is done after everything else
#

mount /home

# if we are on AC power, we'll force a sync with glusterfs
if /usr/bin/on_ac_power; then
 nice find /home -noleaf -print0 | xargs --null stat >/dev/null &
fi

First I mount /home.  The glusterd daemon is started after the disks are mounted, so I force it to happen here.  Then, if we are plugged in to wall power, I start a self heal.  I should probably do a check for the correct network as well, but the process is easy enough to kill off if I don’t need it.

I do something similar with quad, but since quad rarely power cycles, I created a cron job that forces a sync at 1:00 AM.

Now I need to get my data off to geo.  Using GlusterFS’s geo-replication I have it sync the data from quad to geo.  I chose quad, since it’s essentially always connected to the Internet, with a static VPN to geo.

quad# ssh-keygen quad# cp .ssh/id_rsa /etc/glusterd/geo-replication/secret.pem quad# cp .ssh/id_rsa.pub /etc/glusterd/geo-replication/secret.pen.pub 

quad# ssh-copy-id root@geo password: quad# gluster volume geo-replication sync1 geo:/geo Starting geo-replication session between sync1 geo:/deo has been successful

The main difference between replication and geo-replication is that replication is synchronous.  Geo-replication is asynchronous, and the data on geo may be out of sync for short periods.  It works quite well over the Internet.  In fact, GlusterFS uses rsync as the underlying transport for geo-replication.  A status check will show you if everything is running.

quad# gluster volume geo-replication sync1 geo:/geo status

MASTER SLAVE STATUS 
--------------------------------------------------------------------------------
sync1 geo:/geo OK

And there you have it.  I have automatic syncing of data across systems, and access to all of my data from anywhere I need it.  It works well for me.

I am looking forward to GlusterFS 3.3, to see how it will help with disconnected/reconnected syncs.

I decided I needed to either a) switch to NFS or b) change to a different iSCSI provider.

Before I decided anything, I performed some tests.  What kinds of IOPS was I getting from NFS vs iSCSI? What about my transfer rate?  All this information needed to be gathered from a VM.  This is the result of my tests.

Before we start, I’d like to explain why I’ve chosen to use the GlusterFS NFS server over the kernel NFS server.  There are three reasons: 1) the kernel NFS server was not stable in my situation, and XenServer lost the NFS share often enough to cause issues.  2) The kernel NFS server was slower in most cases than the Gluster NFS server.  Unfortunately, I no longer have the numbers to prove it, so I’ll have to run those tests again another time.  3) I could get iSCSI failover using DRBD and Gluster failover using it’s replication (both over a separate NIC).  In theory, I could have done that with Kernel NFS and DRBD as well, but I decided against it.

I was interested in both IOPS and speed of transfer.  I needed good IOPS in order to host an SQL Server and a Zimbra server on the SR, while I needed good speeds to host a file server or two.

I did not do any tests using the GlusterFS replication or iSCSI/DRBD.

Storage Server Configuration

The system used at the storage server: AMD Athlon(tm) 7850 Dual-Core Processor at 2.8 GHz, with 4 GB RAM.  The disks were 7200 RPM Western Digital Black 2 TB SATA.  Linux software RAID was used to mirror 2 drives, and iSCSI was offered on one mirror, while Gluster had another.

The file server was running Ubuntu 10.04 LTS with GlusterFS 3.2 and Ubuntu’s standard iSCSI install.

The follow modifications were done in rc.local

echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor 

echo cfq > /sys/block/sda/queue/scheduler
echo cfq > /sys/block/sdb/queue/scheduler
echo cfq > /sys/block/sdc/queue/scheduler
echo cfq > /sys/block/sdd/queue/scheduler
echo cfq > /sys/block/sde/queue/scheduler
echo cfq > /sys/block/sdf/queue/scheduler

The following changes were to GlusterFS:

Options Reconfigured:
performance.read-ahead: on
performance.io-cache: on
performance.write-behind-window-size: 4MB
performance.io-thread-count: 64
performance.cache-size: 134217728
nfs.trusted-sync: on
nfs.port: 2049

Conclusions

Before you get to the data, here are my conclusions.  Gluster performs on par with iSCSI in my scenario.  It drops below iSCSI in some tests, and is above iSCSI in others.  The situation may be different if tested on a 10 GigE network instead of a 1 GigE network.

I’ll be placing Gluster with NFS into production, and hopefully get rid of some of my iSCSI issues.

IOPS

I did IOPS testing from both a Windows 7 VM, and an Ubuntu 10.04 VM.  For Windows, I used sqlio from Microsoft (http://www.microsoft.com/download/en/details.aspx?id=20163) and for Linux I used fio (from Ubuntu’s repository).  For both VM’s, I tested random IO using 8K chunks to simulate the SQL Servers, and sequential IO using 64K chunks to test transfer speeds.

Windows sqlio

Gluster NFS did surprisingly well, beating the iSCSI SR and local storage SR on writes, by a substantial margin.  Local storage writes (on hardware RAID1) were pretty slow in comparison.  The big surprise here was the iSCSI 64K sequential write IOPS.  This result was not duplicated under Linux.

Linux fio

This shows Gluster falling behind iSCSI in 8K reads and writes, but passing iSCSI on 64K sequential writes.

Speed

Speed tests were done with the same tools under Windows and Linux, with the addition of Citrix’s ‘Performace VM’ (http://support.citrix.com/article/CTX127065) doing tests as well.

Windows sqlio

Linux fio

XenServer Performance VM graphs

4K Sequential Reads

Both the iSCSI and Gluster servers showed initial transfers rates of 80+ MB/s, but they were not able to maintain that rate for the duration of the test.  They both maintained it for roughly 45-60 seconds, which would work well in a ‘bursty’ environment.

4K Random Reads

What can I say to this?  It took me by surprise.  The next step up for block size in the Performance VM was 32K, and Local Storage, iSCSI and Gluster performed as per the 1M tests below.

4K Sequential Write

4K Random Write

1M Sequential Read

1M Random Read

1M Sequential Write

Both iSCSI and Gluster are hitting the theoretical transfer rates of GigE.

1M Random Write

Muliple VM’s per drive share

I did some tests with multiple VM accessing the mounted drive at the same time.  The results are below.  (No graphs, there would be too many)

Raw Data

SQLIO Performance From a Windows 7 VM

The following sqlio commands were run on each partition

sqlio -kW -t8 -s120 -o8 -frandom -b8 -BH -LS -Fparam.txt
sqlio -kR -t8 -s120 -o8 -frandom -b8 -BH -LS -Fparam.txt
sqlio -kW -t8 -s120 -o8 -fsequential -b64 -BH -LS -Fparam.txt
sqlio -kR -t8 -s120 -o8 -fsequential -b64 -BH -LS -Fparam.txt

Gluster NFS

2 threads writing for 120 secs to file e:\testfile.dat  using 8KB random IOs 	enabling multiple I/Os per thread with 8 outstanding
	buffering set to use hardware disk cache (but not file cache)

using specified size: 5000 MB for file: e:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:  1419.13
MBs/sec:    11.08
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 10
Max_Latency(ms): 1794
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0  9 54 22  6  3  1  1  1  1  1  1  0  0  0  0  0  0  0  0  0  0  0  0  1

2 threads reading for 120 secs from file e:\testfile.dat using 8KB random IOs
	enabling multiple I/Os per thread with 8 outstanding
	buffering set to use hardware disk cache (but not file cache)
using specified size: 5000 MB for file: e:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:   128.24
MBs/sec:     1.00
latency metrics:
Min_Latency(ms): 11
Avg_Latency(ms): 124
Max_Latency(ms): 585
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 100

2 threads writing for 120 secs to file e:\testfile.dat using 64KB sequential IOs
	enabling multiple I/Os per thread with 8 outstanding
	buffering set to use hardware disk cache (but not file cache)
using specified size: 5000 MB for file: e:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:  1437.62
MBs/sec:    89.85
latency metrics:
Min_Latency(ms): 1
Avg_Latency(ms): 10
Max_Latency(ms): 1420
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0  0  0  1  1  2  2  3 51 32  1  1  1  1  1  1  1  0  0  0  0  0  0  0  1

2 threads reading for 120 secs from file e:\testfile.dat using 64KB sequential IOs
	enabling multiple I/Os per thread with 8 outstanding
	buffering set to use hardware disk cache (but not file cache)
using specified size: 5000 MB for file: e:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:   862.93
MBs/sec:    53.93
latency metrics:
Min_Latency(ms): 1
Avg_Latency(ms): 17
Max_Latency(ms): 138
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0  0  0  0  0  0  0  0 30 11  2  2  2  2  2  2  2  2  2  2  2  3  3  2 28

iSCSI

2 threads writing for 120 secs to file f:\testfile.dat using 8KB random IOs
	enabling multiple I/Os per thread with 8 outstanding
	buffering set to use hardware disk cache (but not file cache)
using specified size: 5000 MB for file: f:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:  1229.25
MBs/sec:     9.60
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 12
Max_Latency(ms): 611
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  8 80  1  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 11

2 threads reading for 120 secs from file f:\testfile.dat using 8KB random IOs
	enabling multiple I/Os per thread with 8 outstanding
	buffering set to use hardware disk cache (but not file cache)
using specified size: 5000 MB for file: f:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:   247.88
MBs/sec:     1.93
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 63
Max_Latency(ms): 14740
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  1  1  1  1  1  1  1 91

2 threads writing for 120 secs to file f:\testfile.dat using 64KB sequential IOs
	enabling multiple I/Os per thread with 8 outstanding
	buffering set to use hardware disk cache (but not file cache)
using specified size: 5000 MB for file: f:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:    49.44
MBs/sec:     3.09
latency metrics:
Min_Latency(ms): 2
Avg_Latency(ms): 322
Max_Latency(ms): 2369
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0  0  0  0  0  0  0  0  3  1  0  0  0  0  0  0  0  0  0  0  0  0  0  0 95

2 threads reading for 120 secs from file f:\testfile.dat using 64KB sequential IOs
	enabling multiple I/Os per thread with 8 outstanding
	buffering set to use hardware disk cache (but not file cache)
using specified size: 5000 MB for file: f:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:  1013.00
MBs/sec:    63.31
latency metrics:
Min_Latency(ms): 1
Avg_Latency(ms): 15
Max_Latency(ms): 678
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0  0  0  0 16 36 15  1  9  2  1  1  1  1  1  1  0  1  1  1  1  1  0  0 10

Local Storage

2 threads writing for 120 secs to file g:\testfile.dat using 8KB random IOs
	enabling multiple I/Os per thread with 8 outstanding
	buffering set to use hardware disk cache (but not file cache)
using specified size: 5000 MB for file: g:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:   146.64
MBs/sec:     1.14
latency metrics:
Min_Latency(ms): 1
Avg_Latency(ms): 108
Max_Latency(ms): 3601
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 99

2 threads reading for 120 secs from file g:\testfile.dat using 8KB random IOs
	enabling multiple I/Os per thread with 8 outstanding
	buffering set to use hardware disk cache (but not file cache)
using specified size: 5000 MB for file: g:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:   329.14
MBs/sec:     2.57
latency metrics:
Min_Latency(ms): 6
Avg_Latency(ms): 48
Max_Latency(ms): 1480
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0  0  0  0  0  0  0  0  0  0  0  1  1  1  2  2  3  3  3  3  3  3  3  3 71

2 threads writing for 120 secs to file g:\testfile.dat using 64KB sequential IOs
	enabling multiple I/Os per thread with 8 outstanding
	buffering set to use hardware disk cache (but not file cache)
using specified size: 5000 MB for file: g:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:   908.15
MBs/sec:    56.75
latency metrics:
Min_Latency(ms): 3
Avg_Latency(ms): 17
Max_Latency(ms): 3501
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0  0  0  0  0  0  2  1  3  3  0  8 50 18  7  0  1  0  1  0  1  1  1  0  4

2 threads reading for 120 secs from file g:\testfile.dat using 64KB sequential IOs
	enabling multiple I/Os per thread with 8 outstanding
	buffering set to use hardware disk cache (but not file cache)
using specified size: 5000 MB for file: g:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:  1238.51
MBs/sec:    77.40
latency metrics:
Min_Latency(ms): 1
Avg_Latency(ms): 12
Max_Latency(ms): 1006
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0  0  0  0  3  4  5  5  5  3  3  7 23 13  7  3  3  2  2  2  1  1  1  1  4

FIO Performance from an Ubuntu 10.04 64-bit VM

# Write IOPS test
fio --filename=/dev/xvde --direct=1 --rw=randwrite --bs=8k --size=5G --numjobs=2 --runtime=120 --group_reporting --name=file1 

# Read IOPS test
fio --filename=/dev/xvde --direct=1 --rw=randread --bs=8k --size=5G --numjobs=2 --runtime=120 --group_reporting --name=file1 

# Write Bandwidth test
fio --filename=/dev/xvde --direct=1 --rw=write --bs=64k --size=5G --numjobs=2 --runtime=120 --group_reporting --name=file1 

# Read Bandwidth test
fio --filename=/dev/xvde --direct=1 --rw=read --bs=64k --size=5G --numjobs=2 --runtime=120 --group_reporting --name=file1

Gluster NFS

file1: (g=0): rw=randwrite, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
Starting 2 processes

file1: (groupid=0, jobs=2): err= 0: pid=22231
  write: io=699488KB, bw=5829KB/s, iops=728, runt=120003msec
    clat (usec): min=636, max=3484K, avg=2739.59, stdev=14123.66
    bw (KB/s) : min=    2, max= 4278, per=51.36%, avg=2993.01, stdev=453.21
  cpu          : usr=0.15%, sys=0.72%, ctx=87467, majf=0, minf=265
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/87436, short=0/0
     lat (usec): 750=0.11%, 1000=24.36%
     lat (msec): 2=64.69%, 4=8.92%, 10=0.48%, 20=0.09%, 50=0.01%
     lat (msec): 100=0.64%, 250=0.68%, 500=0.01%, >=2000=0.01%

Run status group 0 (all jobs):
  WRITE: io=699488KB, aggrb=5828KB/s, minb=5968KB/s, maxb=5968KB/s, mint=120003msec, maxt=120003msec

Disk stats (read/write):
  xvdb: ios=21/87407, merge=0/0, ticks=10/237870, in_queue=237860, util=99.97%

file1: (g=0): rw=randread, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
Starting 2 processes

file1: (groupid=0, jobs=2): err= 0: pid=22311
  read : io=756128KB, bw=6301KB/s, iops=787, runt=120008msec
    clat (usec): min=82, max=2493K, avg=2534.10, stdev=13046.89
    bw (KB/s) : min=   59, max= 5800, per=51.70%, avg=3257.14, stdev=949.26
  cpu          : usr=0.23%, sys=0.69%, ctx=94527, majf=0, minf=211
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=94516/0, short=0/0
     lat (usec): 100=0.03%, 250=0.03%, 500=1.39%, 750=25.61%, 1000=43.37%
     lat (msec): 2=24.09%, 4=0.75%, 10=2.52%, 20=0.96%, 50=0.07%
     lat (msec): 100=0.52%, 250=0.63%, 500=0.03%, 750=0.01%, 2000=0.01%
     lat (msec): >=2000=0.01%

Run status group 0 (all jobs):
   READ: io=756128KB, aggrb=6300KB/s, minb=6451KB/s, maxb=6451KB/s, mint=120008msec, maxt=120008msec

Disk stats (read/write):
  xvdb: ios=94490/0, merge=0/0, ticks=237960/0, in_queue=237940, util=99.97%

file1: (g=0): rw=write, bs=64K-64K/64K-64K, ioengine=sync, iodepth=1
Starting 2 processes

file1: (groupid=0, jobs=2): err= 0: pid=22390
  write: io=3771MB, bw=32177KB/s, iops=502, runt=120006msec
    clat (msec): min=1, max=1299, avg= 3.97, stdev=10.74
    bw (KB/s) : min= 1682, max=23040, per=50.63%, avg=16289.98, stdev=1782.65
  cpu          : usr=0.09%, sys=0.80%, ctx=60343, majf=0, minf=66
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/60334, short=0/0

     lat (msec): 2=69.80%, 4=26.60%, 10=0.86%, 20=0.53%, 50=0.36%
     lat (msec): 100=0.66%, 250=1.20%, 2000=0.01%

Run status group 0 (all jobs):
  WRITE: io=3771MB, aggrb=32176KB/s, minb=32948KB/s, maxb=32948KB/s, mint=120006msec, maxt=120006msec

Disk stats (read/write):
  xvdb: ios=43/120622, merge=0/0, ticks=220/474620, in_queue=474830, util=99.97%

file1: (g=0): rw=read, bs=64K-64K/64K-64K, ioengine=sync, iodepth=1
Starting 2 processes

file1: (groupid=0, jobs=2): err= 0: pid=22473
  read : io=4742MB, bw=40466KB/s, iops=632, runt=120004msec
    clat (usec): min=146, max=334092, avg=3158.45, stdev=8708.99
    bw (KB/s) : min=  442, max=32768, per=50.30%, avg=20354.80, stdev=2713.53
  cpu          : usr=0.11%, sys=0.92%, ctx=75901, majf=0, minf=102
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=75876/0, short=0/0
     lat (usec): 250=1.45%, 500=0.03%, 750=2.32%, 1000=4.19%
     lat (msec): 2=87.02%, 4=1.05%, 10=1.27%, 20=0.88%, 50=0.28%
     lat (msec): 100=0.65%, 250=0.85%, 500=0.01%

Run status group 0 (all jobs):
   READ: io=4742MB, aggrb=40465KB/s, minb=41437KB/s, maxb=41437KB/s, mint=120004msec, maxt=120004msec

Disk stats (read/write):
  xvdb: ios=151707/0, merge=0/0, ticks=466670/0, in_queue=466660, util=99.97%

iSCSI

file1: (g=0): rw=randwrite, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
Starting 2 processes

file1: (groupid=0, jobs=2): err= 0: pid=23143
  write: io=1014MB, bw=8610KB/s, iops=1076, runt=120643msec
    clat (usec): min=289, max=1290K, avg=1852.73, stdev=14270.21
    bw (KB/s) : min=   53, max= 9728, per=54.28%, avg=4673.82, stdev=1717.51
  cpu          : usr=0.29%, sys=1.04%, ctx=129865, majf=0, minf=248
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/129846, short=0/0
     lat (usec): 500=45.69%, 750=32.09%, 1000=19.45%
     lat (msec): 2=0.46%, 4=0.29%, 10=0.89%, 20=0.21%, 50=0.16%
     lat (msec): 100=0.33%, 250=0.38%, 500=0.01%, 750=0.04%, 1000=0.02%
     lat (msec): 2000=0.01%

Run status group 0 (all jobs):
  WRITE: io=1014MB, aggrb=8610KB/s, minb=8816KB/s, maxb=8816KB/s, mint=120643msec, maxt=120643msec

Disk stats (read/write):
  xvdc: ios=25/129844, merge=0/0, ticks=50/237260, in_queue=238270, util=100.00%

file1: (g=0): rw=randread, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
Starting 2 processes

file1: (groupid=0, jobs=2): err= 0: pid=23223
  read : io=1053MB, bw=8987KB/s, iops=1123, runt=120014msec
    clat (usec): min=89, max=281301, avg=1775.15, stdev=7398.71
    bw (KB/s) : min=  115, max=10352, per=49.75%, avg=4470.80, stdev=2769.23
  cpu          : usr=0.32%, sys=0.97%, ctx=134833, majf=0, minf=252
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=134818/0, short=0/0
     lat (usec): 100=0.01%, 250=0.02%, 500=47.34%, 750=48.52%, 1000=0.41%
     lat (msec): 2=0.26%, 4=0.17%, 10=0.89%, 20=1.05%, 50=0.41%
     lat (msec): 100=0.43%, 250=0.51%, 500=0.01%

Run status group 0 (all jobs):
   READ: io=1053MB, aggrb=8986KB/s, minb=9202KB/s, maxb=9202KB/s, mint=120014msec, maxt=120014msec

Disk stats (read/write):
  xvdc: ios=134812/0, merge=0/0, ticks=237360/0, in_queue=237350, util=99.97%

file1: (g=0): rw=write, bs=64K-64K/64K-64K, ioengine=sync, iodepth=1
Starting 2 processes

file1: (groupid=0, jobs=2): err= 0: pid=23302
  write: io=2784MB, bw=23757KB/s, iops=371, runt=120004msec
    clat (usec): min=909, max=3022K, avg=5383.29, stdev=51258.84
    bw (KB/s) : min=   21, max=38195, per=78.56%, avg=18663.68, stdev=8802.51
  cpu          : usr=0.08%, sys=0.57%, ctx=44551, majf=0, minf=64
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/44545, short=0/0
     lat (usec): 1000=14.09%
     lat (msec): 2=83.96%, 4=0.13%, 10=0.19%, 20=0.26%, 50=0.07%
     lat (msec): 100=0.45%, 250=0.63%, 500=0.01%, 750=0.02%, 1000=0.01%
     lat (msec): 2000=0.15%, >=2000=0.03%

Run status group 0 (all jobs):
  WRITE: io=2784MB, aggrb=23756KB/s, minb=24326KB/s, maxb=24326KB/s, mint=120004msec, maxt=120004msec

Disk stats (read/write):
  xvdc: ios=36/89008, merge=0/0, ticks=200/473940, in_queue=474280, util=99.96%

file1: (g=0): rw=read, bs=64K-64K/64K-64K, ioengine=sync, iodepth=1
Starting 2 processes

file1: (groupid=0, jobs=2): err= 0: pid=23385
  read : io=5110MB, bw=43602KB/s, iops=681, runt=120009msec
    clat (usec): min=157, max=305786, avg=2930.78, stdev=8578.41
    bw (KB/s) : min= 7572, max=31360, per=50.33%, avg=21945.59, stdev=2723.04
  cpu          : usr=0.13%, sys=1.00%, ctx=81775, majf=0, minf=100
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=81760/0, short=0/0
     lat (usec): 250=0.22%, 500=0.71%, 750=6.77%, 1000=12.59%
     lat (msec): 2=74.95%, 4=0.34%, 10=1.26%, 20=1.10%, 50=0.67%
     lat (msec): 100=0.57%, 250=0.81%, 500=0.01%

Run status group 0 (all jobs):
   READ: io=5110MB, aggrb=43602KB/s, minb=44648KB/s, maxb=44648KB/s, mint=120009msec, maxt=120009msec

Disk stats (read/write):
  xvdc: ios=163448/0, merge=0/0, ticks=464580/0, in_queue=464560, util=99.97%

Local Storage

file1: (g=0): rw=randwrite, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
Starting 2 processes

file1: (groupid=0, jobs=2): err= 0: pid=23496
  write: io=64584KB, bw=550988B/s, iops=67, runt=120028msec
    clat (usec): min=282, max=617769, avg=29727.13, stdev=27055.25
    bw (KB/s) : min=  115, max=  660, per=50.12%, avg=269.65, stdev=36.97
  cpu          : usr=0.05%, sys=0.06%, ctx=8074, majf=0, minf=240
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/8073, short=0/0
     lat (usec): 500=0.83%
     lat (msec): 2=0.14%, 4=2.58%, 10=22.82%, 20=30.61%, 50=30.76%
     lat (msec): 100=2.03%, 250=10.19%, 500=0.02%, 750=0.02%

Run status group 0 (all jobs):
  WRITE: io=64584KB, aggrb=538KB/s, minb=550KB/s, maxb=550KB/s, mint=120028msec, maxt=120028msec

Disk stats (read/write):
  xvde: ios=25/8068, merge=0/0, ticks=10/239650, in_queue=239700, util=99.98%

file1: (g=0): rw=randread, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
Starting 2 processes

file1: (groupid=0, jobs=2): err= 0: pid=23576
  read : io=85768KB, bw=731832B/s, iops=89, runt=120009msec
    clat (usec): min=82, max=168917, avg=22386.39, stdev=22463.69
    bw (KB/s) : min=  236, max=  581, per=50.07%, avg=357.53, stdev=36.04
  cpu          : usr=0.01%, sys=0.12%, ctx=10722, majf=0, minf=205
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=10721/0, short=0/0
     lat (usec): 100=0.07%, 250=0.08%, 500=0.07%, 750=0.02%
     lat (msec): 2=0.02%, 4=3.10%, 10=32.09%, 20=46.12%, 50=9.06%
     lat (msec): 100=1.05%, 250=8.32%

Run status group 0 (all jobs):
   READ: io=85768KB, aggrb=714KB/s, minb=731KB/s, maxb=731KB/s, mint=120009msec, maxt=120009msec

Disk stats (read/write):
  xvde: ios=10714/0, merge=0/0, ticks=239470/0, in_queue=239560, util=99.97%

file1: (g=0): rw=write, bs=64K-64K/64K-64K, ioengine=sync, iodepth=1
Starting 2 processes

file1: (groupid=0, jobs=2): err= 0: pid=23655
  write: io=4630MB, bw=39507KB/s, iops=617, runt=120004msec
    clat (usec): min=617, max=813247, avg=3235.29, stdev=12106.59
    bw (KB/s) : min= 1581, max=31104, per=51.51%, avg=20349.41, stdev=3693.46
  cpu          : usr=0.18%, sys=0.88%, ctx=74094, majf=0, minf=65
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/74078, short=0/0
     lat (usec): 750=10.02%, 1000=38.75%
     lat (msec): 2=41.36%, 4=5.96%, 10=0.85%, 20=0.66%, 50=0.99%
     lat (msec): 100=0.59%, 250=0.78%, 500=0.01%, 750=0.04%, 1000=0.01%

Run status group 0 (all jobs):
  WRITE: io=4630MB, aggrb=39506KB/s, minb=40455KB/s, maxb=40455KB/s, mint=120004msec, maxt=120004msec

Disk stats (read/write):
  xvde: ios=50/148124, merge=0/0, ticks=270/471900, in_queue=472150, util=99.99%

file1: (g=0): rw=read, bs=64K-64K/64K-64K, ioengine=sync, iodepth=1
Starting 2 processes

file1: (groupid=0, jobs=2): err= 0: pid=23738
  read : io=6464MB, bw=55159KB/s, iops=861, runt=120007msec
    clat (usec): min=137, max=134005, avg=2315.94, stdev=7830.72
    bw (KB/s) : min=17107, max=43047, per=50.15%, avg=27661.51, stdev=3437.10
  cpu          : usr=0.10%, sys=1.27%, ctx=103448, majf=0, minf=99
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=103430/0, short=0/0
     lat (usec): 250=9.72%, 500=0.78%, 750=22.86%, 1000=50.09%
     lat (msec): 2=9.05%, 4=1.58%, 10=3.79%, 20=0.96%, 50=0.03%
     lat (msec): 100=0.26%, 250=0.89%

Run status group 0 (all jobs):
   READ: io=6464MB, aggrb=55159KB/s, minb=56483KB/s, maxb=56483KB/s, mint=120007msec, maxt=120007msec

Disk stats (read/write):
  xvde: ios=206818/0, merge=0/0, ticks=461050/0, in_queue=461030, util=99.97%

HVM (Hardware Virtual Machine) is a true virtualized server.  It uses the Intel VT-x and AMD-V extensions to provide virtualization..   XenServer provides a QEMU based emulation layer for devices.

PVM (Para-Virtualized Machine) is a virtualized server that requires modified kernel that is ‘Xen aware’ to run.  This kernel has special ties into Xen’s ABI, and typically runs faster than an HVM.

Windows based VM’s are para-virtualized by default.  Since Citrix/Xen cannot modify the Windows kernel, they use specific device drivers to provide the para-virtualization layer.  Linux based VM’s can be PVM’s or HVM’s, based on their support for Xen.  Since Citrix has access to the Linux kernel source, they can directly modify (or distributions can modify) the kernel to talk to the Xen ABI.  What this basically ends up doing, is XenServer only supports para-virtualization with specific Linux distributions.  If you want to run a distribution that is not supported by Citrix, it by default becomes an HVM.

The ‘hung HVM’ issue I’ve seen only affects Linux HVM’s.  The Linux HVM essentially locks, becoming non-responsive to pings, local console, basically everything.  Using the tools provided by XenServer (XenCenter or command line) have no effect.  The VM cannot be shutdown or rebooted.  If this was a real machine, you could pull the power cord and deal with the results later.  At least you could get the machine up and running again.

My past efforts to reboot a hung HVM was to migrate or shutdown all the VM’s running on the XenServer, and rebooting the XenServer itself.  Then bringing the VM’s back.  This was a task that would usually have to wait until after business hours.  The hit on server downtime and impact on users was usually too great to do the reboot during the day.

A discussion on the Citrix forums finally brought a solution to light.  It’s entirely command line based, can be done during business hours, and doesn’t affect any other running VM.  A good solution.

1. Find the UUID of the hung VM.
   You can do this via the command line with ‘xe vm-list’ or via XenCenter.
2. Find the Domain ID of the hung VM.
   Run ‘list_domains’ from the command line, and match the UUID with the ID number 

   id |                                 uuid |  state

    0 | 2fe455fe-3185-4abc-bff6-a3e9a04680b0 |     R

   47 | 267227f3-a59e-dafe-b183-82210cf51ec4 |    B

   59 | 298817fb-8a3e-7501-11e0-045a8aa860ff |    B

   60 | 46e3d5aa-2f02-dfdc-b053-9a8ac56ec5d1 |    B

   61 | 16cf3204-eb17-5a12-e8d0-c72087bda690 |    B

   62 | 1f9053b5-c6ca-40bb-504e-3017c37e7281 |      H

   63 | ddaec491-097a-e271-362b-f2f985e26e4a |     R

   65 | 55f3b225-4f65-d1ea-aa19-add44c5acce7 |    B

   66 | 7adef6fd-9171-5426-b333-6fb1b57b8e60 |    B H

   67 | 6046dc13-f70b-8398-56fb-069c22440a7c |    B

   68 | f201cd94-a501-00c2-d21e-8c2f03ea167b |    B H

   In our case, UUID 1f9053b5-c6ca-40bb-504e-3017c37e7281 is hung, which is Domain ID 62.

3. Run destroy_domain on the Domain ID.

   # /opt/xensource/debug/destroy_domain -domid 62

4. The VM will still show itself as running, so now, we need to reboot it.

   # xe vm-reboot name-label='name of the VM' --force

5. The VM is now rebooted, and you can bring it up as if you had just pulled the plug.  That is, check for some disk corruption, etc.

Thanks go to the Citrix Forum folks for helping us come up with a solution.

Linux maps it’s hard drives using UDEV, and doesn’t guarantee a drive that is mounted at sda today, will be mounted as sda tomorrow.  It may mount as sdb, or sdc.  So, when you get a failed drive, and /proc/mdstat tell you /dev/sdc1 failed, how do you know which physical drive it really is?  In RAID-5, if you pull the wrong physical drive, chances are you’ve lost all your data.  In RAID-6, you get an extra chance.

The answer all boils down to the drives serial number.  You need to find the mapping between the mounted drive and it’s serial number via ‘lshw’.

# lshw -class disk

*-disk:4
 description: ATA Disk
 product: WDC WD2001FASS-0
 vendor: Western Digital
 physical id: 0.0.0
 bus info: scsi@5:0.0.0
 logical name: /dev/sde
 version: 01.0
 serial: WD-WMAUR0279543
 size: 1863GiB (2TB)
 capabilities: partitioned partitioned:dos
 configuration: ansiversion=5 signature=0007bd23

The two lines from the above output that concern us are ‘logical name’ and ‘serial’  The serial number will match what is printed on the hard drives label, and the logical name will match the failed drive in /proc/mdstat.

Just to make things easier for me, rather than pull all the drives out of a box looking for the proper one, I keep a list of the drives, their positions and serial numbers.  I keep this list with the system, and reference it if I need to replace a drive.  I also make sure to update it with the new drives information.

Another way I do it, is to simply use a label maker to place the serial number into a more visible position on the drive.

All you have to be able to do is see the serial number clearly.  It’s way better than pulling them all until you find the right one.

• Download cygwin 1.7+ from the regular place
• Install the default system, plus the following: cygrunsrv, openssh, and rsync
• I also install joe (my editor of choice), and procps (for top)
• Start a cygwin shell and type:

# ssh-host-config -y
# cygrunsrv -S sshd

• I then attempt to ssh into my backupPC system from cygwin, just a nice test, and creates the .ssh directory for me.
• Copy BackupPC’s public key over

# scp root@backuppc:/var/lib/backuppc/BackupPCkey.pub ~/.ssh/authorized_keys

• Install the following scripts to Administrators home directory

pre-backuppc.sh

#!/bin/bash
# script to create shadow copies of Windows drives and export them to a
# drive letter for BackupPC backup
# the shadow copies get mount to c:\shadow\(drive letter).  The directory
# structure must exist

##############################################################################
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see
# http://www.gnu.org/licenses/
##############################################################################

# Launches passed input via 'at' to get around $USERNAME=SYSTEM
# problem under ssh login where the shell lacks permsisions to run
# commmands like vshadow or dosdev
function at_launch ()
{
    local h m s wait1 command
    if [ $3 != "" ] ; then
        command="${1} ${2} >> ${3}"
    else
        command="${1} ${2}"
    fi

    set -- $(date +"%H %M %S")
        h=$((10#$1))  #Note explicitly use base 10 so that 08 and 09 not interpreted as bad octal
    	m=$((10#$2 +1)) #Advance minutes by 1
    	s=$((10#$3))
    	wait1=$((60 - $s))
    [ $s -gt 55 ] && let "m += 1" "wait1 += 60" # Make sure >5 seconds left
    [ $m -ge 60 ] && let "m %= 60" "h += 1" #Overflow minutes
    let "h %= 24"
       	at $h:$m $(cygpath -w $(which bash.exe)) -c \"$command\"
#       	> /dev/null

    echo Running \'$command\' at $h:$m
    return $wait1
}

# create the command to shadow the drive, and wait 2 minutes plus the seconds
# before the at command runs before returning (make sure shadow copy is made)
function shadow_drive ()
{
    date
    echo Shadowing Drive $@
    local wait1
    drive=$@

    at_launch "/cygdrive/c/WINDOWS/vshadow.exe -p" $drive "/home/Administrator/vshadow-out"
    wait1=$?
    let "wait1 += 120"
    echo sleeping for $wait1
    sleep $wait1
    date
    echo done sleep
}

# get the guids from the vshow-out file and place them into the shadow-guids
# file for the post-backuppc scripts use (and hours for mapping to directory)
function get_guids ()
{
    echo Getting shadow copy GUIDS
    cat ~/vshadow-out | grep "* SNAPSHOT ID" | awk '{print $5}' >> ~/shadow-guids
}

function map_shadow ()
{
    echo Mapping GUID $1 to $2
    local wait1
    local guid="$1"
    local dir="$2"
    at_launch /cygdrive/c/WINDOWS/vshadow.exe -el=$guid,$dir "/home/Administrator/map.out"
    wait1=$?
    let "wait1 +=30"
    sleep $wait1
}

date

# get rid of the guids file if it exists
rm ~/shadow-guids
rm ~/vshadow-out
rm ~/map.out

sleep 10

# create the snapshots
shadow_drive c:
shadow_drive h:
shadow_drive j:

# get the guids into a single file
get_guids

loop=0

# create the shadow directory structure AFTER we make the shadow copies
# the post-backuppc.sh script deletes this tree after removing the mounts
mkdir /cygdrive/c/shadow
mkdir /cygdrive/c/shadow/c
mkdir /cygdrive/c/shadow/h
mkdir /cygdrive/c/shadow/j

# loop throuh the guids and map to mount point
# assumes guids in file are in order of shadows created
while read line ;
do
   if [ $loop == 0 ] ; then
       map_shadow $line "c:\\\\\shadow\\\\\c"
   fi

   if [ $loop == 1 ] ; then
       map_shadow $line "c:\\\\\shadow\\\\\h"
   fi

   if [ $loop == 2 ] ; then
       map_shadow $line "c:\\\\\shadow\\\\\j"
   fi

   let "loop += 1"
done < ~/shadow-guids

post_backuppc.sh

#!/bin/bash

# script to delete the shadow copies used by backuppc

while read line ;
do
  vshadow -ds=$line
done < ~/shadow-guids

# now clean up the directory structure
#rmdir /cygdrive/c/shadow

• In the BackupPC config for the system (I use the web page to edit the config) Add the following

DumpPreUserCmd: $sshPath -c blowfish -q -x -l Administrator <windows server name> /usr/bin/bash -l -c /home/Administrator/pre-backuppc.sh

DumpPostUserCmd:$sshPath -c blowfish -q -x -l Administrator <windows server name> /usr/bin/bash -l -c /home/Administrator/post-backuppc.sh

* sometime between the release of Cygwin 1.7 and the current version, ssh login became case sensitive.  I used to log in with ‘-l administrator’ and now I have to log in with ‘-l Administrator’.  That one set me back a bit.

Once you do a change to the backups, make sure you do a full backup right away.  It’s a good test, and backuppc doesn’t like it’s backup directory structure changed during incrementals.

And that’s it. I have BackupPC backup /cygdrive/c/shadow, and all is well. Now, if someone can tell me how to color/syntax highlight bash code in WordPress, I’d be happy.

Portions of GPLv3 code in this script were taken from http://sourceforge.net/apps/mediawiki/backuppc/index.php?title=User_Scripts_-_Client_-_Windows_VSS, which is a much more complicated script than I needed.