NearToTheSky's ramblings

Backing up windows with rsync and Volume Shadow copy Service.

Keeping backups is an important part of computer maintenance and one that I have been overlooking for too long. After an unfortunate disk crash that happened back around the beginning of the year, I've decided to get serious about it and decided to set up a dedicated machine for backups. After looking at various alternatives, I decided to buy some hardware and run FreeNAS on it.

Setting up FreeNAS went without a hitch and scheduling backups of my various FreeBSD computers to it went pretty smoothly, thanks to my familiarity with rsync.

Then came the more interesting issue of backing up windows. Now, one would imagine that backing up a windows computer should be easy, it turns out that it isn't. There were several backup tools I reviewed, but all of them failed in some fundamental way:

• Most tools assume that the backup medium is available locally and not over a network. This is not an issue in principle, since they'll support backing up to samba shares and those can be set up easily, but samba is not very fast and only enumerating files on samba, for large volumes is somewhat of a challenge.
• Some tools are image-based. This makes them decently fast over samba, especially with incremental backups, but it also makes it much harder to access the files afterwards. And ideally, I'd rather be able to read my files from the backup even without the corresponding restore tool.
• Some tools use a proprietary protocol and are not cross platform. Which makes them useless for backing up to FreeNAS.
• Some tools seem good, but they are expensive and in this day and age, not having a proper, free backup solution is ridiculous

So, I was left with trying to run rsync+ssh on windows.
Now, rsync is not exactly written for windows, but thankfully, there are ports of it that work decently. One solution is to simply install cygwin, but that tends to be messy and I generally don't need the whole thing.
I have used cwrsync in the past, a long time ago, which is basically a package of the cygwin, openssh and rsync dll from the cygwin package. It turns out that it's not free anymore because they've added a GUI, which I have no use for. Thankfully, it is possible to obtain something similar by downloading the cygwin installer, setting it to "download without installing" and then gather the necessary files from the downloaded archives. At the time of this writing, the list of needed files is:

cygasn1-8.dll
cygcom_err-2.dll
cygcrypt-0.dll
cygcrypto-1.0.0.dll
cyggcc_s-1.dll
cyggssapi-3.dll
cygheimbase-1.dll
cygheimntlm-0.dll
cyghx509-5.dll
cygiconv-2.dll
cygintl-8.dll
cygkrb5-26.dll
cygpopt-0.dll
cygroken-18.dll
cygsqlite3-0.dll
cygssp-0.dll
cygwin1.dll
cygwind-0.dll
cygz.dll
rsync.exe
ssh-keygen.exe
ssh.exe

The first step is to set up a home path for cygwin to use. To that effect, use the system control panel to create an environment variable called HOME and make it point to your user folder.

Optionally, you can then use ssh-keygen normally to generate a keypair allowing you to run the backups automatically without requiring a password.
For example:

Then copy the public key generated to the ssh server and test by establishing an ssh connection.

This was enough to have rsync working, but there remained another issue. Windows programs have the ability to open files in such a way that other programs are not allowed them to read them, which is inconvenient when it comes to backups. Thankfully, Windows features a Volume Shadow copy Service which allows to take snapshots of volumes at a given point in time, including locked files. This is used amongst other things for system recovery (in the form of permanent copy-on-write snapshots), but can also be used for backups using temporary copy-on-write snapshots. When such a snapshot is created, it is possible to mount it and read all files from it without restrictions. This can be done with the following two tools:

• vshadow, which is available, with its source as part of the Microsoft Windows SDK for Windows 7 and .NET Framework 4. If you don't like to install the whole SDK, it might be possible to find it as an individual download.
• dosdev, which can be found as part of the Microsoft Product Support ReportsNote that you do not need to install the whole package. You can extract dosdev from it instead

Also, I needed a tool to allow reporting failures to my mailbox. A quick search revelaed that Blat was a good choice.

With the different utilities available, I was left with the step of writing a script to hold it all together. It looks approximately like this, for a computer named xyz:

PATH=S:\backup_tools\;%PATH%
vshadow -script=vss-setvar.cmd -exec=do_rsync.cmd %1
if ERRORLEVEL 1 (blat -server mailserver -port 587 -f neartothesky@mailserver -to neartothesky@mailserver -subject "xyz backup failed" -body "Volume shadow copy creation failed")

set RSYNC_FAILED=FALSE
call vss-setvar.cmd
if exist Z: (blat -server mailserver -port 587 -f neartothesky@mailserver -to neartothesky@mailserver -subject "xyz Backup failed" -body "Z: already exists" & goto :end)
dosdev Z: %SHADOW_DEVICE_1%
rsync -rvvhHpognt --rsync-path="rsync --fake-super" --delete /cygdrive/z -e "\"/cygdrive/s/backup tools/ssh"" xyz@backupserver:~/ > rsync.log 2>&1
if ERRORLEVEL 1 (set RSYNC_FAILED=TRUE)
dosdev -r -d Z:
findstr /c:"rsync error" rsync.log
if not ERRORLEVEL 1 (set RSYNC_FAILED=TRUE)
if %RSYNC_FAILED%==TRUE (blat -server mailserver -port 587 -f neartothesky@mailserver -to neartothesky@mailserver -subject "xyz Backup failed" -body "Rsync failed")
:end

Finally, set backup.cmd to run as a scheduled task and Voila! working automated backup over rsync.

Note: This post was heavily inspired by the information found in another blog, treating the same topic from a slightly different perspective.