Fri, 15 Apr 2005
Amazon, Screen Scraping and More Yahoo Pipes
When playing with Yahoo Pipes the other day, I mentioned that it would be nice if it had a screen-scraping module to build feeds for pages that don't one. It turns out there are a handful of services that already do that. I used Feedity to create of feed of the Prudent Bear news since my script stopped working at some point.
This morning I decided to update my Amazon aStore since they now allow you to add more than just the nine items on the homepage. I wanted to add a bunch of stuff from my wishlist. So I decided to try to get a list of ASINs for the items in my wishlist using Yahoo Pipes. Amazon provides RSS feeds of wishlists, but they only contain the last 10 items, so I used Ponyfish to create a screen-scraped feed. (I triggered an application error trying to do so with Feedity.)
Then I used Yahoo Pipes rename module to change the title of each item to the link, and the regex module to change the title to ASIN by extracting it from the URL. This gives me a feed of ASIN's. Admittedly, this is quite a convoluted solution to the actual problem I was trying to solve; curl and sed would have worked fine in this instance. It gave me a chance to play around with Yahoo Pipes, though.
Copying a Crapload of Files is Hard
I was recently faced with what turned out to be an unexpectedly difficult task. I had to copy about 700GB of data from one filesystem to another. The data was comprised of the backups of about 40 servers as stored by BackupPC. BackupPC is a great piece of software for backing up a network of servers. It can backup machines using SMB, rsync, and tar over rsh/ssh. It compresses the backed up files and pools them so multiple copies of the same files are stored as hardlinks to the compressed files in the pool.
In order to increase the storage capacity of the backup server, I was tasked with adding a second RAID array. I set up an LVM volume on the new array, and then needed to copy all of the files from the old array to the new one.
First, I started up rsync and let it go. For a long time, the rsync process just kept growing and growing. Before it starts copying any files, it goes through all of the files to figure out what it's going to copy. A known limitation of rsync is that it uses a lot of memory when there are a lot of files involved. I didn't know how many files we had, but with 4GB of RAM in the machine, I was pretty confident that we would be ok. After more than 24 hours, before it had started actually copying any files, rsync died as it hit the 3GB per-process memory limit.
For my next attempt, I used cp -a. cp started out alright.
Throughput from source volume was pretty abysmal, but after about 5 days, the
entire pool of compressed files had been copied. As it was copying the
individual servers' backup directories (which contain the hardlinks to the
pool), cp also hit the 3GB limit and died.
I briefly thought about using sgp_dd to do a lower level copy of the data, but we really wanted to use XFS for the new filesystem; the original filesystem was EXT3. I didn't confirm whether it would work with LVM volumes either.
I ended up just starting with an empty BackupPC directory. We'll keep the old backups for a while until we have sufficient history. Then, we'll add the old array to the new volume for additional disk space.
An interesting project might be to add an optional argument to rsync to allow it to use a dbm file to maintain state rather than keeping everything in memory. I guess nobody anticipated terabyte-sized disk volumes being used with 32-bit processors.
Pricing That Doesn't Make Sense is Annoying
Seth Godin is trying to make a point about how people act differently when on camera.
Odd segue: Today, in anticipation of a dinner party, I stopped at a lobster seller in Chelsea Market in NYC. I asked for a six pound lobster. The pricing at the store is $9.95 a pound for small lobsters and $8.95 a pound for lobsters six pounds and up.
The lobster weighed (I'm not making this up), 5.97 pounds. For reference, that's just less than a pound by the weight of a penny. Feed the lobster a plankton and it would be six pounds.
He started to ring me up at $9.95 a pound. I pointed out the price breakdown and the guy shrugged and said, "It doesn't weight six pounds."
Two co-workers came over and with precisely the same uncomprehending grin, repeated his point. I added a penny to the scale but they weren't swayed.
So, the two questions are, "Do you think the owner wanted them to act this way?" and "Would they have acted differently if they were on camera?"
Maybe because it's obvious, but he doesn't mention the absurdity of the pricing scheme. The price for the lobster should, of course, be something like $9.95/lb for the first pound and $8.95/lb for each additional pound. Damn, it's annoying when you run into companies that have stupid pricing.
Mark Hurst describes a conversion with a Dell sales rep in which the sales rep explains that a computer without a monitor costs $400 more than the same computer with a monitor.
Taxes Kill Productivity
Cantillon's Paradise points to an AP story which says that Americans spend 6.6 billion hours on their taxes. It's rather ridiculous the lengths a citizen must go to in order to comply with the law. Luckily, our civil servants are working to resolve this issue.
Sensitive to the demands that tax laws put on weary taxpayers, the IRS has seven people working full time to reduce the anguish for filers. The IRS Office of Taxpayer Burden Reduction looks for requirements that can be streamlined, reduced or eliminated under the law.
Of course, there's only so much they can do considering the amount of tax law that are in charge of enforcing. Congress is responsible for the mess.
I've probably spent at least 60 hours on taxes so far this year, mostly reading IRS publications, including best sellers Business Expenses, Business Use of Your Home, and the much referenced tome, How to Depreciate Property.
My wife recently decided to change her major to accounting. I think I have the perfect internship for her.
business » taxes | Permanent Link
What To Do With The Big Moo?
The two books by Seth Godin that I've read, Purple Cow and The Big Red Fez, were both good. So, when Seth announced The Big Moo on his blog, I was excited about getting my hands on it. The Big Moo is a collaboration between Seth and 32 other business writers, with each writing contributing a chapter on how to make your business remarkable. As an interesting marketing idea, galleys (pre-release) versions of the book were offered in boxes of 50 for $2 each ($100 per box). Their goal was to get word of the book out early so big companies would buy truckloads of the book for their executives.
Back in 2000, the president of BNW bought everyone a copy of Tom Peters's The Professional Service Firm 50. The Big Moo is a similar type of book, but without the annoying typography (lots of different font sizes, bold, exclamation points, etc. I hate the style of writing in Peters's book. If you need to trick me into getting excited about your writing, you haven't written a very good book. Get me thinking, and I'll be excited.) It's not that it's a bad book, but it definitely under-delivers. Each of the authors contributed a single chapter, and most "chapters" are two or three pages long. It feels like you're reading a brain fart that the author couldn't bother to think through fully.
It's definitely not the type of book that forever changes your thinking: it wouldn't be included in a personal MBA list. So I'm not sure what I should do with 48 unread copies of the book (I did give one copy away). I can't give it to people saying, "You have to read this!" And I don't really want to give it someone and say, "Here's an OK book you might want to read if you don't have anything better to do."
If you want a copy, let me know. Some other readers enjoyed it more than I did. Meryl at Blogcritics.org gave it a decent review.
The book does what it sets out to do: motivate the reader to get out there to put ideas to work to develop a remarkable organization that gets everyone buzzing.
As does Jack at 800-CEO-READ. Though he seems to like the book mostly because the proceeds go to charity. That's not a very good reason to read a book. You'd be better off giving the money directly to charity and avoiding the printing costs.
Update (10/10/05): I managed to unload two copies. Thanks, Heather!
Update (10/18/05): John, who also took a copy off my hands wrote up a review on his blog.
business » marketing | Permanent Link
Sun, 03 Apr 2005
3ware 9500 Notes
I recently upgraded a Pogo Linux StorageWare server. The server came with a 3ware 8506-8 SATA controller and 7 250 GB drives which we have configured in a RAID 10 array with one spare (the OS is on a separate 80GB drive). The server has 16 drive bays in total, so we decided to buy a second controller and 7 more drives to be configured in another RAID 10 array with one spare.
I had a couple problems getting the new card working. First, I had to upgrade the kernel to get the 3w-9xxx module. The Debian package for 2.6.10 worked. The next problem was using the 3ware management utilities. 3ware provides two utilies, a command-line one called tw-cli, and a web-based interface called 3dm2. According to their web site, the "in Engineering Phase" software should be used with a 2.6 kernel. When I tried using the software though, both tw_cli and 3dm2 reported "Application too old for controller firmware". I contacted the always helpful Pogo support folks who advised me to use version 9.1.5.2 of the 3ware software, which is a released version. The 9.1.5.2 software was able to detect the 9500, but not the 8506, so I tried version 9.2 which was able to access both. So it seems that "in Engineering Phase" means outdated as it's older than the released version.
Having gotten the 3ware software working, I was able to create and delete
arrays from within Linux rather than having to use the BIOS utility. Then I
could benchmark various RAID configurations. I tested RAID 10, RAID 50, and
RAID 5 with 6 drives and with 7 drives. (RAID 10 is striping across
mirrored sets of disks and RAID 50 is striping across RAID 5 arrays) In
any configuration, the 9500 blows the 8506 out of the water. I ran my
benchmarks using bonnie++.
With RAID 10, I was able to get about 170 MB/s write speeds and 185 MB/s reads.
Setting a large read-ahead value of 16384 (blockdev --setra 16384
/dev/sdc1) significantly improves read performance so I will have to
tune it for the best performance once we put the new drives into production.
Striping across 2 3ware RAID 5 or 3 RAID 1 arrays
using LVM gave almost as
good performance as RAID 50 and RAID 10 3ware arrays, respectively.
With RAID 5, using 7 drives in the array rather than 6 gives better read
performance, but poorer write performance. The additional drive also
seems to adversely impact file creation and deletion speeds.
XFS seems to give the best overall performance. Ext3 gives good read performance, but mediocre write speeds. Reiserfs is very fast at creating and deleting files, but doesn't give as good throughput.
Housing Crash Links
I've created an rss feed of Patrick's Housing Crash
links. Many of the links don't make it to his blog.
Subscribe here
Scary Housing Numbers
(Via Housing
Bubble) LA
Times has a story about the increasing popularity of interest-only
mortgages. An astonishing 47.8% of homes sold last year in California were
purchased with interest-only, adjustable rate mortgages. In 2001, it was 1.4%, and I'd bet that almost all of those were purchased by
investors who wouldn't be holding the property very long.
Many of these people are buying with no money down so when interest rates rise and they can no longer afford their mortgage payments, there's no disincentive to simply walk away from the property. You don't need to have very good credit to buy a house these days; what's a little bankruptcy blemish on one's credit report.
"I have $40,000 in student loans from my master's degree," Herron said. "I have high credit card debt. I'm a typical American. And yet they wanted to give me more debt to buy a house."
She wonders sometimes if she'll end up in foreclosure, if the bank will take her beloved home away from her when she can no longer pay her bills. Maybe it was a mistake to give her this money, maybe it was a mistake for her to take it. But she wasn't about to protest.
"If you're like me, you're so incredulous that anyone would give you any money whatsoever, you just close your eyes and sign the papers," Herron said. "I would have signed anything."
Mon, 21 Mar 2005
Bow and Arrow Taxes
All of the archers out there will be happy to hear that the Arrow Tax has been repealed. Alas, the Bow Tax still applies.
Bows
The 11% tax on bows will apply to bows having a peak draw weight of 30 pounds or more. This replaces the current law which applies to bows having a peak draw weight of 10 pounds or more. Broadheads suitable for use with certain arrows will be taxed as a part and accessory at the 11% bow tax rate. The effective date for both provisions is for articles sold after November 21, 2004.
Repeal of Arrow Tax
The tax on arrows that was in effect for arrows sold after November 21, 2004, and before December 23, 2004, has been repealed. The rules used prior to November 22, 2004, for arrow components still apply for the 4th quarter of 2004. Any tax that was imposed by the manufacturer, producer, or importer on arrows after November 21, 2004, and before December 23, 2004, may have to be refunded to the purchaser. See Changes Effective for the Second Quarter of 2005 for details on the tax on arrow shafts effective after March 31, 2005.
How much are we paying the members of Congress to make these laws? (Hint: It's less than $155,101)
Update: Unfortunately, those arrows aren't really tax free. Beginning March 31, there will be a tax on the arrow shaft.
business » taxes | Permanent Link
How You Like Them Apples?
- I can't believe that once I've counted four apples, there are no more apples.
