Fri, 15 Apr 2005
Amazon, Screen Scraping and More Yahoo Pipes
When playing with Yahoo Pipes the other day, I mentioned that it would be nice if it had a screen-scraping module to build feeds for pages that don't one. It turns out there are a handful of services that already do that. I used Feedity to create of feed of the Prudent Bear news since my script stopped working at some point.
This morning I decided to update my Amazon aStore since they now allow you to add more than just the nine items on the homepage. I wanted to add a bunch of stuff from my wishlist. So I decided to try to get a list of ASINs for the items in my wishlist using Yahoo Pipes. Amazon provides RSS feeds of wishlists, but they only contain the last 10 items, so I used Ponyfish to create a screen-scraped feed. (I triggered an application error trying to do so with Feedity.)
Then I used Yahoo Pipes rename module to change the title of each item to the link, and the regex module to change the title to ASIN by extracting it from the URL. This gives me a feed of ASIN's. Admittedly, this is quite a convoluted solution to the actual problem I was trying to solve; curl and sed would have worked fine in this instance. It gave me a chance to play around with Yahoo Pipes, though.
Copying a Crapload of Files is Hard
I was recently faced with what turned out to be an unexpectedly difficult task. I had to copy about 700GB of data from one filesystem to another. The data was comprised of the backups of about 40 servers as stored by BackupPC. BackupPC is a great piece of software for backing up a network of servers. It can backup machines using SMB, rsync, and tar over rsh/ssh. It compresses the backed up files and pools them so multiple copies of the same files are stored as hardlinks to the compressed files in the pool.
In order to increase the storage capacity of the backup server, I was tasked with adding a second RAID array. I set up an LVM volume on the new array, and then needed to copy all of the files from the old array to the new one.
First, I started up rsync and let it go. For a long time, the rsync process just kept growing and growing. Before it starts copying any files, it goes through all of the files to figure out what it's going to copy. A known limitation of rsync is that it uses a lot of memory when there are a lot of files involved. I didn't know how many files we had, but with 4GB of RAM in the machine, I was pretty confident that we would be ok. After more than 24 hours, before it had started actually copying any files, rsync died as it hit the 3GB per-process memory limit.
For my next attempt, I used cp -a. cp started out alright.
Throughput from source volume was pretty abysmal, but after about 5 days, the
entire pool of compressed files had been copied. As it was copying the
individual servers' backup directories (which contain the hardlinks to the
pool), cp also hit the 3GB limit and died.
I briefly thought about using sgp_dd to do a lower level copy of the data, but we really wanted to use XFS for the new filesystem; the original filesystem was EXT3. I didn't confirm whether it would work with LVM volumes either.
I ended up just starting with an empty BackupPC directory. We'll keep the old backups for a while until we have sufficient history. Then, we'll add the old array to the new volume for additional disk space.
An interesting project might be to add an optional argument to rsync to allow it to use a dbm file to maintain state rather than keeping everything in memory. I guess nobody anticipated terabyte-sized disk volumes being used with 32-bit processors.
Pricing That Doesn't Make Sense is Annoying
Seth Godin is trying to make a point about how people act differently when on camera.
Odd segue: Today, in anticipation of a dinner party, I stopped at a lobster seller in Chelsea Market in NYC. I asked for a six pound lobster. The pricing at the store is $9.95 a pound for small lobsters and $8.95 a pound for lobsters six pounds and up.
The lobster weighed (I'm not making this up), 5.97 pounds. For reference, that's just less than a pound by the weight of a penny. Feed the lobster a plankton and it would be six pounds.
He started to ring me up at $9.95 a pound. I pointed out the price breakdown and the guy shrugged and said, "It doesn't weight six pounds."
Two co-workers came over and with precisely the same uncomprehending grin, repeated his point. I added a penny to the scale but they weren't swayed.
So, the two questions are, "Do you think the owner wanted them to act this way?" and "Would they have acted differently if they were on camera?"
Maybe because it's obvious, but he doesn't mention the absurdity of the pricing scheme. The price for the lobster should, of course, be something like $9.95/lb for the first pound and $8.95/lb for each additional pound. Damn, it's annoying when you run into companies that have stupid pricing.
Mark Hurst describes a conversion with a Dell sales rep in which the sales rep explains that a computer without a monitor costs $400 more than the same computer with a monitor.
Taxes Kill Productivity
Cantillon's Paradise points to an AP story which says that Americans spend 6.6 billion hours on their taxes. It's rather ridiculous the lengths a citizen must go to in order to comply with the law. Luckily, our civil servants are working to resolve this issue.
Sensitive to the demands that tax laws put on weary taxpayers, the IRS has seven people working full time to reduce the anguish for filers. The IRS Office of Taxpayer Burden Reduction looks for requirements that can be streamlined, reduced or eliminated under the law.
Of course, there's only so much they can do considering the amount of tax law that are in charge of enforcing. Congress is responsible for the mess.
I've probably spent at least 60 hours on taxes so far this year, mostly reading IRS publications, including best sellers Business Expenses, Business Use of Your Home, and the much referenced tome, How to Depreciate Property.
My wife recently decided to change her major to accounting. I think I have the perfect internship for her.
business » taxes | Permanent Link
What To Do With The Big Moo?
The two books by Seth Godin that I've read, Purple Cow and The Big Red Fez, were both good. So, when Seth announced The Big Moo on his blog, I was excited about getting my hands on it. The Big Moo is a collaboration between Seth and 32 other business writers, with each writing contributing a chapter on how to make your business remarkable. As an interesting marketing idea, galleys (pre-release) versions of the book were offered in boxes of 50 for $2 each ($100 per box). Their goal was to get word of the book out early so big companies would buy truckloads of the book for their executives.
Back in 2000, the president of BNW bought everyone a copy of Tom Peters's The Professional Service Firm 50. The Big Moo is a similar type of book, but without the annoying typography (lots of different font sizes, bold, exclamation points, etc. I hate the style of writing in Peters's book. If you need to trick me into getting excited about your writing, you haven't written a very good book. Get me thinking, and I'll be excited.) It's not that it's a bad book, but it definitely under-delivers. Each of the authors contributed a single chapter, and most "chapters" are two or three pages long. It feels like you're reading a brain fart that the author couldn't bother to think through fully.
It's definitely not the type of book that forever changes your thinking: it wouldn't be included in a personal MBA list. So I'm not sure what I should do with 48 unread copies of the book (I did give one copy away). I can't give it to people saying, "You have to read this!" And I don't really want to give it someone and say, "Here's an OK book you might want to read if you don't have anything better to do."
If you want a copy, let me know. Some other readers enjoyed it more than I did. Meryl at Blogcritics.org gave it a decent review.
The book does what it sets out to do: motivate the reader to get out there to put ideas to work to develop a remarkable organization that gets everyone buzzing.
As does Jack at 800-CEO-READ. Though he seems to like the book mostly because the proceeds go to charity. That's not a very good reason to read a book. You'd be better off giving the money directly to charity and avoiding the printing costs.
Update (10/10/05): I managed to unload two copies. Thanks, Heather!
Update (10/18/05): John, who also took a copy off my hands wrote up a review on his blog.
