Wed, 22 Jun 2005

High-Availability SMTP with UCARP on Debian

At customer request, we're going to start offering outbound SMTP service to Postica customers. Doing so requires a much greater guarantee of availability than is required when only accepting mail from other MTAs. MTAs are able to use multiple MX records when attempting to deliver mail, and will queue mail if none of the MX hosts are available. MUAs, on the other hand, can generally only be configured with a single hostname to use as the SMTP server for outbound mail, and tend to show the user an unpleasant error message if there is a problem connecting to the SMTP server.

To provide high-availability, load-balanced SMTP service, I decided to use round-robin DNS in combination with CARP, the UCARP implementation specifically. CARP is a protocol for supporting failover of an IP address, very similar to VRRP.

I installed the Debian ucarp package on two servers. Each server is the preferred server for one ucarp-managed IP address and the backup for the other; smtp.postica.net points to both addresses. I also installed the iputils-arping package which is used to send gratuitous arps when the IP address moves to a new server thus causing the MAC address to change. Note that the arping program in the iputils-arping package is different than the one in the arping package.

I added two up options to /etc/network/interfaces on each server to start one ucarp process for each IP address when the physical interface to which the ucarp addresses are bound is brought up.

/etc/network/interfaces
auto eth0
iface eth0 inet static
   address 192.168.1.101
   netmask 255.255.255.0
   gateway 192.168.1.1
   up ucarp -i eth0 -s 192.168.1.101 -v 201 -p secretPassword -a 192.168.1.201 \
     --upscript=/etc/ucarp/vip-201-up.sh --downscript=/etc/ucarp/vip-201-down.sh -P \
     -z -k 10 --daemonize
   up ucarp -i eth0 -s 192.168.1.101 -v 202 -p secretPassword -a 192.168.1.202 \
     --upscript=/etc/ucarp/vip-202-up.sh --downscript=/etc/ucarp/vip-202-down.sh -P \
     -z -k 0 --daemonize
   down pkill ucarp

The interfaces file is essentially the same on the second server, but the values of -k arguments, the advertisement skew which determines priority, are swapped. If you were running ucarp on multiple interfaces, you probably wouldn't want to kill all ucarp processes when bringing an interface down; you might want to use start-stop-daemon with --make-pidfile and --background instead of using ucarp's --daemonize option.

The --upscript and --downscript arguments tell ucarp what scripts to run when taking over or releasing an IP address, respectively. Here's an example of each:

vip-201-up.sh
#! /bin/sh
exec 2> /dev/null

/sbin/ip addr add 192.168.1.201/24 dev "$1"
start-stop-daemon --start --pidfile /var/run/ucarp-arping.192.168.1.201 \
  --make-pidfile --background --exec /usr/sbin/arping -- -q -U 192.168.1.201
vip-201-down.sh
#! /bin/sh
exec 2> /dev/null

/sbin/ip addr del 192.168.1.201/24 dev "$1"
start-stop-daemon --stop --pidfile /var/run/ucarp-arping.192.168.1.201 \
  --exec /usr/sbin/arping
rm /var/run/ucarp-arping.192.168.1.201

In theory, it should only be necessary to send a single (or maybe a couple) gratuitous arp. I had a problem when using vrrpd, though, in which the backup host would briefly become the master, the arp table on the router would get updated with the MAC address of the new master, then it would go back to being backup. During this period, the other host would think it was the master the entire time, and so would not send any arp updates making the IP address unreachable until the router's arp table was updated. I don't know if this could occur using CARP, but I prefer to play it safe and have the master continue to send unsolicited arps by using start-stop-daemon to spawn a long-running arping process.

In summary, round-robin DNS is used to balance the load across the two servers, and in the event that one of the servers goes down, both IP addresses will be handled by a single server.

tech » mail | Permanent Link

Replicating a Subversion Repository Using Git

I recently had to replicate a subversion repository into an existing svn repository. I wanted to import a vendor's svn repo into an existing repo, recording the full history from the source into the destination. Development was continuing in the source repo so I also needed a solution that would allow repeated syncs from the source to the destination.

First, I looked at using svnsync to do this. It handles the requirements for preservation of history and repeated syncs, but it requires a brand new repository to write to, and only svnsync should be allowed to write to the destination.

I also briefly looked at using svnsync to replicate to a new local repo, then using svndumpfilter (or svndumpfilter3?) to generate a dump that I could load into the existing destination repo, but doing this repeatedly didn't sound like fun.

So I figured out how to replicate changes from one svn repository to another using git-svn. If you want to do something similar, here's how I did it. 

# create a git repository to use for the migration
git init migration
cd migration

# git-svn doesn't manipulate revision properties to store the original author,
# so the commits in the destination repository will be attributed to the user
# committing to the repo.  Setting svn.addAuthorFrom will add a From: line to
# the end of the log messages in the destination repo.  The svn.useLogAuthor
# will read this value out of svn commits so git will report the correct
# author.
git config svn.addAuthorFrom true
git config svn.useLogAuthor true

# Initialize the source and destination repositories.  Set a prefix for each since we're using two.
# Both source and destination repositories can be any valid svn URL.
# We're not doing development in git so we don't tell git to treat svn branches
# and tags like git ones.  If we have a standard svn layout with trunk,
# branches, and tags, we just treat them like normal directories as far as git
# is concerned.
git svn init -R source --prefix source/ https://example.com/path/to/source
git svn init -R dest --prefix dest/ file:///path/to/dest

# Map svn usernames in the source repository to names and email addresses.
# These values will end up in the From: line in commits.  You may want to edit
# the svn-authors before running `git svn fetch` if you want more descriptive
# names.
(for i in `svn log https://example.com/path/to/source | grep -P '^r\d' | cut -f2 -d'|' | sort -u`; do echo "$i = $i <$i@example.com>"; done) > .git/svn-authors

# Add the username of the user that will be performing the commits to the destination as well.
echo "cwarden = <cwarden@xerus.org>" >> .git/svn-authors

# Tell git to use the list of authors you just created.
git config svn.authorsfile .git/svn-authors

# Pull in both svn repositories to git.  This might take a while.
git svn -R source fetch
git svn -R dest fetch

# Create local branches from the remote branches.  `git cherry-pick` requires a
# local branch, and I like the symmetry of using one for the source as well.
git branch source remotes/source/git-svn
git branch dest remotes/dest/git-svn
The rest of the steps can be wrapped up in script to be run each time you want to migrate changes from source to dest. 
#!/bin/sh
# Update the source from svn.
git co source
git svn rebase || exit 1

# And update the destination repo in case anyone else is writing to it.
git co dest
git svn rebase || exit 1

# Take all of the changes that are in source, but not dest, and
# individually apply them to the local git copy of the repo.
git cherry -v dest source | grep ^\+ | awk '{print $2}' | xargs -I{} sh -c "git cherry-pick {}; git diff --quiet || exit 255" || exit 1

# Commit the changes that we just cherry-picked to svn.
git svn dcommit || exit 1

# Now the somewhat ugly part.  Because the two histories remain separate, we
# tell git that we've applied commits from the source to the destination using
# the git grafts feature.
# This eliminates the previously applied commits from the list that
# git needs to look through the next time we run `git cherry`.
(git show-ref -s dest; git show-ref -s source) | paste -d' ' -s | xargs -d '\n' -I{} sh -c "test -f $(git rev-parse --show-toplevel)/.git/info/grafts && grep -q '{}' $(git rev-parse --show-toplevel)/.git/info/grafts || echo '{}' >> $(git rev-parse --show-toplevel)/.git/info/grafts"

As an alternative to cherry-picking commits, you could use git merge source instead. This frees you from having to store the graft points, but at the expense of having each merge appear as a single commit in the destination svn repo, so you don't get the full history from the source.

Update: The script that's run for each synchronization now aborts if an error occurs, most likely a cherry-pick that results in a conflict. Thanks for fr0sty on #git for coming up with the git diff --quiet solution to detect failed cherry-picks because the exit status is ambiguous.

tech | Permanent Link

Tue, 21 Jun 2005

I'm an Art Collector

On Saturday, Denisa and I bought our first pieces of art. I took her down to Scala Galleria to show her Carrie Graber's stuff that I liked. She wasn't quite as impressed as I was, but she thought some of pieces were pretty good.

We ended up buying two small, 9x12 pieces, one original and one print. The original is Room Full of Blues.



The print is The Letter.

culture | Permanent Link

Fri, 10 Jun 2005

Does Russell Get Frugality?

Apparently not.

Russell Beattie is a "work[er] at Yahoo! focusing on new mobile products," but seems *amazed* that somebody who works for a living might find that spending $500 on a mobile phone is extravagant. My goodness!

In his latest "Online notebook" post about Leslie Katz's post about Jimmy Buffet's lost Sony Ericsson mobile, Russell breathlessly recounts Leslie's "snarkiness and ignorance" as she "breathlessly recounts that a bus boy who found the phone may have crank called Bill Clinton, 'whose number was stored in the $500 phone's directory,' and warns there might be more phone calls coming since, 'Apparently, the fancy phone's memory card with the names, numbers and some addresses' is still missing." Emphasis mine. Double emphasis his.

Astounding. For some reason this rubbed me the wrong way - just the snarkiness and ignorance of it. Could someone at Yahoo please give this man a decent financial planner and tell him to get a clue? Thanks.

-Xn

tech | Permanent Link

Tue, 07 Jun 2005

Watered-Down California Wine

Intelligent Life, a new magazine from the publishers of The Economist, has a story about how much of the wine produced in California is "watered back", i.e. diluted with water. Apparently, this is done to keep alcohol content down or to make sure that fermentation occurs if there is too much sugar in the grapes. I'm no wine connoisseur, but I'd be surprised if very many wine drinkers were aware of this.

…French towns and villages are a great deal more picturesque than anything that northern California can offer. Calistoga and Yountville are not exactly Beaune and Châteauneuf-du-Pape. But for the winemaker the obvious difference is the weather: California has longer and warmer sunshine—which means that the fully ripened grapes may well produce a wine with a bit too much alcohol for comfort (too much, also, for the bottom line, since wine and liquor are taxed according to their alcohol level).

One obvious solution would be to pick the grapes earlier, except that that would be at the cost of the "full-bodied" flavour and "big" character of the typical Californian wine. Hence a preference for the other obvious solution: adding water during the fermentation process. At which point, everyone gets a little coy, even a trifle alarmed. Witness the concern of a spokeswoman for California's Wine Institute. "How did you hear about this?" she demanded. "This is a long, long story—it's kind of a stylistic thing."

business | Permanent Link

Apple Service Pretty Damn Good

In addition to not shuffling, my iPod Shuffle started having some problems being detected upon plugging it into a usb port. I tried multiple usb ports on multiple computers with which I had previously used the Shuffle, so I concluded that the problem must be with the iPod.

Friday afternoon, I opened a service request with Apple online. I was told they would send out a replacement with a postage-paid envelope in which I could send back the defective iPod. There would be no cost, but they took my credit card number in case I didn't send back the broken one.

My new Shuffle arrived this morning (Monday) around 10 am. That's a pretty amazing response time. I plugged the new one in, initialized it with gtkpod, and copied over some mp3's. The new one even works in shuffle mode. I put the old one back in the envelope and scheduled a pickup with DHL, who picked it up a couple hours later. I'm happy.

In other Apple news, new Macs will have Intel CPU's.

However, Schiller said the company does not plan to let people run Mac OS X on other computer makers' hardware. "We will not allow running Mac OS X on anything other than an Apple Mac," he said.

Nevermind, there's not much of a story here.

tech | Permanent Link

Sun, 05 Jun 2005

Credit Card Expiration Date Maxim

Steve Friedl has long maintained the "No Dashes Or Spaces" Hall of Shame which catalogs web sites that require visitors to enter their credit card number without spaces as dashes. As he points out, it's trivial to strip out the extraneous characters server-side so there's no reason to have this stupid restriction in web applications.

I would like to proffer an additional maxim of credit card entry. Credit card expiration dates should never be entered using the name of the month. While I don't have an extensive list of offenders like Steve, I would estimate that there are thousands of web sites that request your credit card expiration date with two pull-down menus, one for the month, January, February, etc., and one for the year. As far as I know, every credit card in existence has the expiration printed as either MM/YY or, less commonly, MM-YY.

When somebody is making a purchase online with a credit card, they simply need to copy the card number (hopefully, without stripping out the whitespace) and the expiration date into a form. The customer shouldn't need to mentally convert the format of the expiration date on the card to the format required on the web site. Additionally, there's no reason to use pull-downs to enter the expiration date. A small text box is the perfect format for entering a 5 digit string.

I suggest the following regex be used to validate expiration dates:
^((0?[1-9])|(1[0-2]))[/-]?(2[01])?\d\d$
This is probably even more liberal than necessary. It allows a one- or two-digit month and a two- or four-digit year, optionally separated by either a slash or a dash. Here's a PHP function to validate such a date string and normalize it to MM/YYYY: 

   define('MPE_CREDIT_CARD_EXPIRATION', '~^((0?[1-9])|(1[0-2]))[/-]?(2[01])?\d\d$~');

   /**
   *
   * Check whether a string is a valid expiration date.  Returns the
   * normalized string if it is valid; else returns false.
   *
   * @return   mixed
   * @access   public
   */
   function validateExpiration($expiration, $asOf = null, $expiredOK = false) {
      if (is_null($asOf)) {
         $yearMonth = date('Ym');
      } elseif (is_numeric($asOf)) {
         // unix timestamp
         $yearMonth = date('Ym', $asOf);
      } else {
         // assume mysql datetime (YYYY-MM-DD HH:MM:SS)
         $yearMonth = substr(str_replace('-', '', $asOf), 0, 6);
      }
      $expiration = preg_replace('/\s/', '', $expiration);
      if (! preg_match(MPE_CREDIT_CARD_EXPIRATION, $expiration)) {
         return false;
      }
      if (strlen($expiration) <= 4) {
         // [M]MYY => MM/YY
         $expiration = sprintf('%02d/%02d', substr($expiration, 0, -2), substr($expiration, -2));
      } elseif (! preg_match('~[/-]~', $expiration)) {
         // [M]MYYYY => MM/YYYY
         $expiration = sprintf('%02d/%02d', substr($expiration, 0, -4), substr($expiration, -4));
      }
      // [M]M-[YY]YY => [M]M/[YY]YY
      $expiration = str_replace('-', '/', $expiration);
      list($month, $year) = explode('/', $expiration);
      strlen($year) < 4 && $year += 2000;
      $month = sprintf('%02d', $month);
      if (! $expiredOK && $year . $month < $yearMonth) {
         return false;
      }
      return sprintf('%02d/%04d', $month, $year);
   }

It will also verify that the expiration date is in the future, as of the current date or an optional date passed as a second parameter. If the third parameter is true, dates in the past will not be rejected.

tech | Permanent Link

Great Beer and Great Art in Los Angeles

I went down to the Scala Galleria on Montana in Santa Monica yesterday to check out their art by Carrie Graber. They had about 10 pieces of hers including a couple small original paintings and a number of giclees on canvas. Her paintings are amazing.

I had never checked out that area of Santa Monica before, so I wandered down Montana towards the beach. It's mostly little boutiques, but there are also a number of coffee cafes and restaurants. I stopped at the bluffs and read for bit. On the way back, I stopped in a little pub called Father's Office. From the "Beer" sign outside, I thought it would be a dive, but it was actually a trendy little place.

It reminds me a bit of City Pub in Redwood City. They have 36 beers on tap, mostly California microbrews and a few international beers. I had a great IPA, Bear Republic Racer 5, and an even hoppier double IPA, Russian River Brewing's Pliny the Elder. Both were fantastic beers. Surprisingly, they had Greene King Abbot Ale. I've never seen Greene King in the states. I asked if they ever get the IPA, which is probably my favorite English beer, but the manager said that it doesn't travel well, which is kind of weird considering IPA's were originally made for export.

The only disappointment was the food. They have an absurd policy of "No substitutions, modification, alterations or deletions. Yes, really." When did beer snobs become food snobs? Since I couldn't get a burger without cheese, I ordered some chorizo which wasn't very good.

culture | Permanent Link

Fri, 03 Jun 2005

Novafoam Mattress Holding Up to the Heat

Memory foam is designed to soften based on your body temperature. I was a bit worried that my Novafoam mattress might start to feel like a waterbed as the weather got hotter. (The Sleep Number salesguy told me as much.) Summer has started here in the valley, and so far, I'm still sleeping comfortably.

misc | Permanent Link

Car Insurance

Jonathan, over at MyMoneyBlog, has a couple posts up about car insurance. His most recent is about collision and comprehensive. I dropped these on my 1999 Honda Civic a couple weeks ago. I'm saving $750 a year without it, and it wouldn't cause too much of financial burden if the car were involved in an accident. 21st Century fixed their web site so I was able to change my coverage without talking to Patrick. We dropped it on Denisa's car, which is only worth maybe $3k, a long time ago.

Jonathan's earlier post on liability coverage is interesting to me too. Since I started driving, I've always had the minimum liability coverage required by law. Since I didn't have many assets for someone to come after, this was fine. As I start to build up my net worth, I should definitely increase the amount of my bodily injury and property damage coverage.

I think Jonathan is a sucker to buy uninsured motorist coverage though. His collision insurance will cover his car in an accident, regardless of whether the other party has insurance, and his medical insurance should cover his medical bills. I don't think uninsured motorist coverage covers hit-and-runs either, so it's really a waste of money.

business | Permanent Link

Shorting Real Estate

It looks like we're one step closer to being able to short real estate. There's a company called HedgeStreet (Via The Big Picture) which allows trading futures on housing prices and mortgage rates. It's not quite the same as shorting a house as there's no underlying asset being bought and sold, nor even an option on an underlying asset; rather, it's basically a bet on which way prices (or rates) will go.

The web site is terribly slow so I haven't read all of the details. If it gets significant volume, it may become a valuable tool for predicting future prices, as markets seem to be pretty good at this type of thing.

business | Permanent Link

Idiots Lose Money "Investing" in Real Estate

Two of my favorite blogs are Housing Bubble and Housing Crash. Housing Bubble collects articles about the housing bubble in the United States. Housing Crash is mostly Patrick's chronicle of the housing bubble in the San Francisco area, but he also posts articles related to the housing bubble nationwide.

There seem to be an equal number of people who claim that there is no housing bubble as those who warn of it. The former group includes all of the real estate agents, mortgage lenders, and county tax assessors. The higher real estate prices go, the more money they make. The latter group is made of economists who point out that sales prices to rental rates ratio is out of whack, that housing prices and debt service costs have far outgrown household income, and that the residential real estate prices have been driven up by all of the easy money thrown at borrowers (well, easy until those 5/1 ARMs get past the 5 part). See The Coming Crash in the Housing Market : 10 Things You Can Do Now to Protect Your Most Valuable Investment for a thorough discussion.

Las Vegas seems to be leading the nation in the bursting of the bubble. Home builders have started lowering the prices of new homes. It's got to suck to know that your neighbors bought their houses for $100k less than you. Las Vegas has tons of open land to be developed so the surge in home prices was probably largely driven by speculators. Housing Crash points us to the story of some of those speculators.

"They call you and say 'you are so lucky .. this just came across.. it's going to be worth 100k before it closes,'"said Dyan Harmell. "We came with the hopes of buying two houses. We left the first day owning four. Within the next week, owning 6 -- all the way up to 19."
"We were told there were 80 people in the lottery for 15 homes. And lo and behold, every single person we knew got a home in that lottery,"said Pulte home buyer Cathy Wodka.

That's a pretty good marketing ploy. They must be targeting the same people who click on flashing banner ads that tell them they've won a new iPod.

Tips for Buying Real Estate

I know I wasn't the only idiot who lost a bunch of money a few years ago "investing" in equities. But at least I learned something. Don't buy something because it has gone up in value; it doesn't mean that it is going to keep going up in value indefinitely.

business | Permanent Link

Updated RSS Feeds

I've updated my screen-scraped rss feed of Northern Trust's Economic Research to include their International Comments.

Also, I don't think I mentioned it when I first set it up, but I also have a feed of the Prudent Bear news.

business | Permanent Link

The Housing Bubble in Pictures

From Calculated Risk

From Northern Trust

From The Miami Herald

business | Permanent Link

The state is that great fiction by which everyone tries to live at the expense of everyone else. - Frederic Bastiat