Wed, 22 Jun 2005
At customer request, we're going to start offering outbound SMTP service to
Postica customers. Doing so requires a much greater guarantee of availability
than is required when only accepting mail from other MTAs. MTAs are
able to use multiple MX records when attempting to deliver mail, and will queue
mail if none of the MX hosts are available. MUAs, on the other
hand, can generally only be configured with a single hostname to use as the
SMTP server for outbound mail, and tend to show the user an unpleasant error
message if there is a problem connecting to the SMTP server.
To provide high-availability, load-balanced SMTP service, I decided to use
round-robin DNS in combination with CARP,
the UCARP implementation specifically. CARP is
a protocol for supporting failover of an IP address, very similar to VRRP.
I installed the Debian ucarp
package on two servers. Each server is the preferred server for one
ucarp-managed IP address and the backup for the other; smtp.postica.net
points to both addresses. I also installed the iputils-arping package
which is used to send gratuitous arps when the IP address moves to a new server
thus causing the MAC address to change. Note that the arping
program in the iputils-arping package is different than the one in
the arping package.
I added two up options to /etc/network/interfaces on
each server to start one ucarp process for each IP address when the physical
interface to which the ucarp addresses are bound is brought up.
/etc/network/interfaces
auto eth0
iface eth0 inet static
address 192.168.1.101
netmask 255.255.255.0
gateway 192.168.1.1
up ucarp -i eth0 -s 192.168.1.101 -v 201 -p secretPassword -a 192.168.1.201 \
--upscript=/etc/ucarp/vip-201-up.sh --downscript=/etc/ucarp/vip-201-down.sh -P \
-z -k 10 --daemonize
up ucarp -i eth0 -s 192.168.1.101 -v 202 -p secretPassword -a 192.168.1.202 \
--upscript=/etc/ucarp/vip-202-up.sh --downscript=/etc/ucarp/vip-202-down.sh -P \
-z -k 0 --daemonize
down pkill ucarp
The interfaces file is essentially the same on the second server,
but the values of -k arguments, the advertisement skew which
determines priority, are swapped. If you were running ucarp on multiple
interfaces, you probably wouldn't want to kill all ucarp processes when
bringing an interface down; you might want to use start-stop-daemon with
--make-pidfile and --background instead of using ucarp's
--daemonize option.
The --upscript and --downscript arguments tell ucarp
what scripts to run when taking over or releasing an IP address, respectively.
Here's an example of each:
vip-201-up.sh
#! /bin/sh
exec 2> /dev/null
/sbin/ip addr add 192.168.1.201/24 dev "$1"
start-stop-daemon --start --pidfile /var/run/ucarp-arping.192.168.1.201 \
--make-pidfile --background --exec /usr/sbin/arping -- -q -U 192.168.1.201
vip-201-down.sh
#! /bin/sh
exec 2> /dev/null
/sbin/ip addr del 192.168.1.201/24 dev "$1"
start-stop-daemon --stop --pidfile /var/run/ucarp-arping.192.168.1.201 \
--exec /usr/sbin/arping
rm /var/run/ucarp-arping.192.168.1.201
In theory, it should only be necessary to send a single (or maybe a couple)
gratuitous arp. I had a problem when using vrrpd, though, in which the backup host
would briefly become the master, the arp table on the router would get updated
with the MAC address of the new master, then it would go back to being backup.
During this period, the other host would think it was the master the entire
time, and so would not send any arp updates making the IP address unreachable
until the router's arp table was updated. I don't know if this could occur
using CARP, but I prefer to play it safe and have the master continue to send
unsolicited arps by using start-stop-daemon to spawn a
long-running arping process.
In summary, round-robin DNS is used to balance the load across the two servers,
and in the event that one of the servers goes down, both IP addresses will be
handled by a single server.
tech » mail | Permanent Link
I recently had to replicate a subversion repository into an existing svn
repository. I wanted to import a vendor's svn repo into an existing repo,
recording the full history from the source into the destination. Development
was continuing in the source repo so I also needed a solution that would allow
repeated syncs from the source to the destination.
First, I looked at using svnsync to
do this. It handles the requirements for preservation of history and repeated
syncs, but it requires a brand new repository to write to, and only svnsync
should be allowed to write to the destination.
I also briefly looked at using svnsync to replicate to a new local repo, then
using svndumpfilter (or svndumpfilter3?) to generate a dump that I could load
into the existing destination repo, but doing this repeatedly didn't sound like
fun.
So I figured out how to replicate changes from one svn repository to another
using git-svn.
If you want to do something similar, here's how I did it.
# create a git repository to use for the migration
git init migration
cd migration
# git-svn doesn't manipulate revision properties to store the original author,
# so the commits in the destination repository will be attributed to the user
# committing to the repo. Setting svn.addAuthorFrom will add a From: line to
# the end of the log messages in the destination repo. The svn.useLogAuthor
# will read this value out of svn commits so git will report the correct
# author.
git config svn.addAuthorFrom true
git config svn.useLogAuthor true
# Initialize the source and destination repositories. Set a prefix for each since we're using two.
# Both source and destination repositories can be any valid svn URL.
# We're not doing development in git so we don't tell git to treat svn branches
# and tags like git ones. If we have a standard svn layout with trunk,
# branches, and tags, we just treat them like normal directories as far as git
# is concerned.
git svn init -R source --prefix source/ https://example.com/path/to/source
git svn init -R dest --prefix dest/ file:///path/to/dest
# Map svn usernames in the source repository to names and email addresses.
# These values will end up in the From: line in commits. You may want to edit
# the svn-authors before running `git svn fetch` if you want more descriptive
# names.
(for i in `svn log https://example.com/path/to/source | grep -P '^r\d' | cut -f2 -d'|' | sort -u`; do echo "$i = $i <$i@example.com>"; done) > .git/svn-authors
# Add the username of the user that will be performing the commits to the destination as well.
echo "cwarden = <cwarden@xerus.org>" >> .git/svn-authors
# Tell git to use the list of authors you just created.
git config svn.authorsfile .git/svn-authors
# Pull in both svn repositories to git. This might take a while.
git svn -R source fetch
git svn -R dest fetch
# Create local branches from the remote branches. `git cherry-pick` requires a
# local branch, and I like the symmetry of using one for the source as well.
git branch source remotes/source/git-svn
git branch dest remotes/dest/git-svn
The rest of the steps can be wrapped up in script to be run each time you want
to migrate changes from source to dest.
#!/bin/sh
# Update the source from svn.
git co source
git svn rebase || exit 1
# And update the destination repo in case anyone else is writing to it.
git co dest
git svn rebase || exit 1
# Take all of the changes that are in source, but not dest, and
# individually apply them to the local git copy of the repo.
git cherry -v dest source | grep ^\+ | awk '{print $2}' | xargs -I{} sh -c "git cherry-pick {}; git diff --quiet || exit 255" || exit 1
# Commit the changes that we just cherry-picked to svn.
git svn dcommit || exit 1
# Now the somewhat ugly part. Because the two histories remain separate, we
# tell git that we've applied commits from the source to the destination using
# the git grafts feature.
# This eliminates the previously applied commits from the list that
# git needs to look through the next time we run `git cherry`.
(git show-ref -s dest; git show-ref -s source) | paste -d' ' -s | xargs -d '\n' -I{} sh -c "test -f $(git rev-parse --show-toplevel)/.git/info/grafts && grep -q '{}' $(git rev-parse --show-toplevel)/.git/info/grafts || echo '{}' >> $(git rev-parse --show-toplevel)/.git/info/grafts"
As an alternative to cherry-picking commits, you could use git merge
source instead. This frees you from having to store the graft points,
but at the expense of having each merge appear as a single commit in the
destination svn repo, so you don't get the full history from the source.
Update: The script that's run for each synchronization now aborts if an error occurs, most likely a cherry-pick that results in a conflict. Thanks for fr0sty on #git for coming up with the git diff --quiet solution to detect failed cherry-picks because the exit status is ambiguous.
tech | Permanent Link
