Sat, 14 May 2005

JavaHMO Debian Package

JavaHMO is an application that allows you to use the Tivo Home Media Option to access music and photos stored on your Java-supported computer (Linux is a good choice) via your Tivo. I previously created a Debian init script for JavaHMO. Dave posted a comment on that page saying that he was having problems getting JavaHMO to start. It looks like the init script doesn't work with newer version of JavaHMO.

So I've started working on a more proper Debian package for JavaHMO. Rather than simply using alien to convert the RPM, I've tried to make it more of a proper Debian package. I've removed all the capital letters from directory and file names, so the config directory is /etc/javahmo rather than /etc/javaHMO, for example. It also creates the javahmo user and sets up the init script links.

I'm quite new to Debian packaging, and I haven't made it through the whole Policy Manual yet. Suggestions for improvements to the package are welcome.

Installation Instructions

You'll need to have a JVM installed. I recommend using make-jpkg with Sun's JRE.

Add the following line to /etc/apt/sources.list:
deb http://packages.pinkhamster.net/ unstable main
and run apt-get install javahmo. The JavaHMO configuration file, /etc/javahmo/configure.xml, is writable by group javahmo. Add any users who should be able to configure JavaHMO to this group. Run jhmo gui to start up the gui configuration tool and enable the plugins that you would like to use.

tech | Permanent Link

Tue, 26 Apr 2005

Email Security Silliness

It's amazing the number of times I've received email saying something like, "Here's the first eight digits of my credit card number. I'll send the second eight in another email (for security)."

This is not security. I wonder how people come up with these ideas. Do they think criminals accidentally intercept random emails sometimes? Or that they are able to intercept some messages, but there's no way they would be able to get two in a row?

If you're concerned about security, ask someone who is knowledgeable about the subject. In the case of email security, use PGP/GPG or S/MIME.

tech | Permanent Link

Tue, 19 Apr 2005

My iPod Shuffle Doesn't

I got my free iPod Shuffle yesterday. I spent way too many hours trying to get it working. On my laptop, with a 2.6.9 kernel, hotplug loaded the ub module instead of usb-storage. I upgraded makedev and created the /dev/ub* devices, but I couldn't access the /dev/uba device. Finally, I tried it on my desktop with a 2.6.10 kernel, and after plugging it in a second time, I was able to access the Shuffle's vfat filesystem as a scsi device.

I installed gtkpod, and was able to create the appropriate directory structure and copy over some mp3's to the Shuffle. I plugged in the headphones, turned it on, and it played. It sounds decent with the headphones Apple supplies, but better with my Sony ones. Apple really should have integrated the headphones into the lanyard it comes with. Or there should be a spring-loaded device to coil up the wires.

One thing my Shuffle doesn't do is shuffle. It works in normal, linear mode, but in shuffle mode, it won't play anything. Perhaps gnupod didn't create the directory structure correctly or something. I'm an album listener rather than a single listener so it's not a big deal, but I don't like knowing that my new gizmo isn't functioning properly. One other problem is that when it's plugged into my usb port, the light always blinks orange. It's supposed to be green when fully charged, but I left it plugged in all night, and it greeted me with flashing orange in the morning.

tech | Permanent Link

Fri, 15 Apr 2005

Amazon, Screen Scraping and More Yahoo Pipes

When playing with Yahoo Pipes the other day, I mentioned that it would be nice if it had a screen-scraping module to build feeds for pages that don't one. It turns out there are a handful of services that already do that. I used Feedity to create of feed of the Prudent Bear news since my script stopped working at some point.

This morning I decided to update my Amazon aStore since they now allow you to add more than just the nine items on the homepage. I wanted to add a bunch of stuff from my wishlist. So I decided to try to get a list of ASINs for the items in my wishlist using Yahoo Pipes. Amazon provides RSS feeds of wishlists, but they only contain the last 10 items, so I used Ponyfish to create a screen-scraped feed. (I triggered an application error trying to do so with Feedity.)

Then I used Yahoo Pipes rename module to change the title of each item to the link, and the regex module to change the title to ASIN by extracting it from the URL. This gives me a feed of ASIN's. Admittedly, this is quite a convoluted solution to the actual problem I was trying to solve; curl and sed would have worked fine in this instance. It gave me a chance to play around with Yahoo Pipes, though.

tech | Permanent Link

Copying a Crapload of Files is Hard

I was recently faced with what turned out to be an unexpectedly difficult task. I had to copy about 700GB of data from one filesystem to another. The data was comprised of the backups of about 40 servers as stored by BackupPC. BackupPC is a great piece of software for backing up a network of servers. It can backup machines using SMB, rsync, and tar over rsh/ssh. It compresses the backed up files and pools them so multiple copies of the same files are stored as hardlinks to the compressed files in the pool.

In order to increase the storage capacity of the backup server, I was tasked with adding a second RAID array. I set up an LVM volume on the new array, and then needed to copy all of the files from the old array to the new one.

First, I started up rsync and let it go. For a long time, the rsync process just kept growing and growing. Before it starts copying any files, it goes through all of the files to figure out what it's going to copy. A known limitation of rsync is that it uses a lot of memory when there are a lot of files involved. I didn't know how many files we had, but with 4GB of RAM in the machine, I was pretty confident that we would be ok. After more than 24 hours, before it had started actually copying any files, rsync died as it hit the 3GB per-process memory limit.

For my next attempt, I used cp -a. cp started out alright. Throughput from source volume was pretty abysmal, but after about 5 days, the entire pool of compressed files had been copied. As it was copying the individual servers' backup directories (which contain the hardlinks to the pool), cp also hit the 3GB limit and died.

I briefly thought about using sgp_dd to do a lower level copy of the data, but we really wanted to use XFS for the new filesystem; the original filesystem was EXT3. I didn't confirm whether it would work with LVM volumes either.

I ended up just starting with an empty BackupPC directory. We'll keep the old backups for a while until we have sufficient history. Then, we'll add the old array to the new volume for additional disk space.

An interesting project might be to add an optional argument to rsync to allow it to use a dbm file to maintain state rather than keeping everything in memory. I guess nobody anticipated terabyte-sized disk volumes being used with 32-bit processors.

tech | Permanent Link

Sun, 03 Apr 2005

3ware 9500 Notes

I recently upgraded a Pogo Linux StorageWare server. The server came with a 3ware 8506-8 SATA controller and 7 250 GB drives which we have configured in a RAID 10 array with one spare (the OS is on a separate 80GB drive). The server has 16 drive bays in total, so we decided to buy a second controller and 7 more drives to be configured in another RAID 10 array with one spare.

I had a couple problems getting the new card working. First, I had to upgrade the kernel to get the 3w-9xxx module. The Debian package for 2.6.10 worked. The next problem was using the 3ware management utilities. 3ware provides two utilies, a command-line one called tw-cli, and a web-based interface called 3dm2. According to their web site, the "in Engineering Phase" software should be used with a 2.6 kernel. When I tried using the software though, both tw_cli and 3dm2 reported "Application too old for controller firmware". I contacted the always helpful Pogo support folks who advised me to use version 9.1.5.2 of the 3ware software, which is a released version. The 9.1.5.2 software was able to detect the 9500, but not the 8506, so I tried version 9.2 which was able to access both. So it seems that "in Engineering Phase" means outdated as it's older than the released version.

Having gotten the 3ware software working, I was able to create and delete arrays from within Linux rather than having to use the BIOS utility. Then I could benchmark various RAID configurations. I tested RAID 10, RAID 50, and RAID 5 with 6 drives and with 7 drives. (RAID 10 is striping across mirrored sets of disks and RAID 50 is striping across RAID 5 arrays) In any configuration, the 9500 blows the 8506 out of the water. I ran my benchmarks using bonnie++. With RAID 10, I was able to get about 170 MB/s write speeds and 185 MB/s reads. Setting a large read-ahead value of 16384 (blockdev --setra 16384 /dev/sdc1) significantly improves read performance so I will have to tune it for the best performance once we put the new drives into production. Striping across 2 3ware RAID 5 or 3 RAID 1 arrays using LVM gave almost as good performance as RAID 50 and RAID 10 3ware arrays, respectively.
With RAID 5, using 7 drives in the array rather than 6 gives better read performance, but poorer write performance. The additional drive also seems to adversely impact file creation and deletion speeds.

XFS seems to give the best overall performance. Ext3 gives good read performance, but mediocre write speeds. Reiserfs is very fast at creating and deleting files, but doesn't give as good throughput.

tech | Permanent Link

Mon, 14 Mar 2005

Blosxom Static Rendering with Writeback Plugin

This site runs on a poor little Pentium MMX 200, so the performance of Blosxom, the software that drives the site, isn't very good. Luckily, Blosxom has a static rendering mode which allows Apache to serve flat html pages. In order to use the writeback plugin, though, POST requests to trackback an entry or post a comment must be handled directly by Blosxom.

So I set up Blosxom in a hybrid manner where new entries are rendered statically every 15 minutes, and the entire site is re-rendered every two hours to bring in comments and trackbacks to the flat pages. All POST requests are handled by Blosxom, as are requests for pages which haven't been pre-generated yet. I can also temporarily disable static rendering if I want to preview a new post without it showing up on the index pages.

My crontab

*/15 * * * * test -f /usr/local/www/xn.pinkhamster.net/.gen && /usr/lib/cgi-bin/blosxom -password=xxxxxxxxxxx -quiet=1

# regenerate all pages to pull in comments
15 */2 * * * test -f /usr/local/www/xn.pinkhamster.net/.gen && /usr/lib/cgi-bin/blosxom -password=xxxxxxxxxxx -all=1 -quiet=1

I can remove the .gen file and preview an entry by going directly to the entry's URL.

Apache Rewrite Rules

RewriteCond %{REQUEST_METHOD} !^POST [NC]
RewriteCond %{QUERY_STRING} ^$
RewriteCond %{DOCUMENT_ROOT}/static/$1 -f [OR]
RewriteCond %{DOCUMENT_ROOT}/static/$1 -d
RewriteRule ^/blog(/?.*)$ /static/$1

I test for an empty query string so that Blosxom will handle the request when I want to enable autotrack. Also, if I want to preview changes to an entry that has already been generated, I can just tack something on to the query string.

It looks like this is similar to Rael's setup, but he only serves the statically rendered index pages directly from Apache and has Blosxom handle all individual entry pages.

tech | Permanent Link

Fixing Web Sites with Javascript

Hat tip1 to John Battelle for pointing out Greasemonkey, a Firefox plugin that allows you to run Javascript scripts on arbitrary sites to improve their appearance or usability.

The one script I find most immediately useful is one that fixes some deficiencies of Allmusic. Allmusic is a fantastically useful site, but since it's redesign, it has the annoying trait that every biography is truncated and ends with a Read More link. The script replaces those links with Javascript links that pull in the full biography using XMLHttpRequest. It is remarkably fast considering how sluggish the site often is in general.

1. John must get a lot of hat tips considering he has the number two result on google for hat tip on a page for which the term only appears in links to it.

tech | Permanent Link

Thu, 03 Mar 2005

iPodder on Debian

I still don't quite understand what all the hoopla around podcasting is about, but since KCRW started offering feeds of their programs, I thought I'd check it out. I don't have an mp3 player, so I was thinking I might listen to some programs while working or listen to them through my Tivo.

I downloaded an RPM of iPodder, converted it to a Debian package using alien and installed it. It looks like it requires wxPython 2.5 so I installed the libwxgtk2.5.3-python package. Since I also have wxPython 2.4 installed, I had to update /opt/iPodder/iPodderGui.py to use the right version. Here's the diff:

--- iPodderGui.py.orig  2005-03-03 11:28:58.000000000 -0800
+++ iPodderGui.py       2005-03-03 11:29:06.000000000 -0800
@@ -1,3 +1,5 @@
+import wxversion
+wxversion.select('2.5')
 import wx
 import wx.xrc as xrc
 import  wx.lib.filebrowsebutton as filebrowse

I still don't see what's so great about this. I'd be happy if straw supported enclosures so I could just download the mp3s from there.

Update: Mostly as an exercise, I've created a Debian package for iPodder.

tech | Permanent Link

Tue, 01 Mar 2005

MySQL with SSL on Debian

Due to licensing issues, the Debian packages for MySQL are not built with OpenSSL support. In order to enable SSL, the packages must be recompiled.

server:/usr/src# apt-get build-dep mysql-server
server:/usr/src# apt-get install libssl-dev
server:/usr/src# apt-get source mysql-server
server:/usr/src# cd mysql-dfsg-4.0.23 (or whatever version you're building)
server:/usr/src/mysql-dfsg-4.0.23# vi debian/rules

Change --without-ssl to --with-ssl.

server:/usr/src/mysql-dfsg-4.0.23# vi debian/changelog

Add an entry to the top of the changelog.

mysql-dfsg (4.0.23-4-zerolag-1) unstable; urgency=low

  * Compile with OpenSSL support

 -- Christian Warden   Fri, 25 Feb 2005 12:32:05 -0800

Then build and install the packages.

server:/usr/src/mysql-dfsg-4.0.23# ./debian/rules binary
server:/usr/src/mysql-dfsg-4.0.23# dpkg -i ../*mysql*.deb

If you already an SSL certificate that you're using with mod_ssl, you can use that. Otherwise, generate a new key and certificate. See /usr/share/doc/mysql-server/SSL-MINI-HOWTO.txt.gz for information on how to generate a self-signed certificate. Add the key and certificate to /etc/mysql/my.cnf.

[mysqld]
...
ssl-key=/etc/mysql/server.key
ssl-cert=/etc/mysql/cert.key

Create a user in MySQL that requires SSL encryption.

mysql> GRANT ALL on db.* to user@10.0.0.10 IDENTIFIED BY 'password' REQUIRE SSL;

Now, you should be able to connect over SSL. Remember that the client must have been compiled with SSL support too. In order to use SSL, the client must use the --ssl-ca option, either on the command-line or in ~/.my.cnf (or [mysql] section of /etc/mysql/my.cnf).

client:~$ mysql --ssl-ca=/dev/null -h server -u user -p

As far as I can tell, there's no way to actually force the client to validate the servers certificate so it doesn't matter what value you set for ssl-ca. It doesn't even have to exist. This, of course, means that while the connection is encrypted, it's vulnerable to man-in-the-middle attacks.

See /usr/share/doc/mysql-server/SSL-MINI-HOWTO.txt.gz if you want to use client certificates to authenticate clients rather than, or in addition to, passwords.

Update: I originally forgot to note that you should update the changelog before building the packages.

tech | Permanent Link

Newer Posts | Older Posts

The state is that great fiction by which everyone tries to live at the expense of everyone else. - Frederic Bastiat