Mon, 28 Feb 2005
mutt-ng
It looks like mutt development is starting to pickup again. For those not familiar with it, mutt is the best email client out there. Development has forked and there is a new mutt-ng project. Kyle Rankin has written up a little summary. So far, it's mostly just integration of many of the third-party patches that have been available for a while. Since the Debian package includes many of these patches already, including one of the most important, header caching, that's not too exciting.
Two of the new features included in mutt-ng are a sidebar and nntp support.
The sidebar is similar to those in most gui mail readers which shows the number
of messages in the folders in your mailboxes. With
pager_index_lines set, mutt basically looks like a text-mode
version of the common three-pane interface in most gui clients. I'll probably
unsubscribe from the exim-users mailing list once the newsreader works since I
can just read the gmane group. Right now,
trying to read a usenet message causes a segfault unfortunately.
mutt-ng seems a bit slower too. Returning from the pager to the index takes an extra second or so.
Debian packages for sid are here:deb http://people.debian.org/~nobse/debian/ unstable/
tech » mail | Permanent Link
Show stdin in a Web Browser
This is brain-dead simple, but useful.
For example,
$ pandoc example.rst | stdin2wwwFri, 18 Feb 2005
Where Google Sucks
I was trying to think of Suze Orman's name while composing an email today. I could picture her face, and I know she's a popular speaker and author on personal finance. So I tried searching for different combinations of "female", "financial advice", "speaker", "author", "personal finance", etc. That didn't get me anywhere, so I tried searching on Google Images and threw "blond" in there. No luck.
Eventually, I gave up and went to Amazon to look for personal finance books. None of her books were in the Most Popular list, but a picture of one of her books happened to show up in the Listmania! section on the right.
I'm sure all of those social network companies maintain the kind of metadata on individuals that would be necessary for this type of search to work, but it needs to be web-accessible, and I'm not sure who would maintain such data on third-parties who aren't part of the network.
As an aside, Google also needs to update their advanced search so that date searches can be more granular than the current past 3/6/12 months options. Remarkably, Bloglines doesn't even support date ranges. Luckily, Straw, my aggregator, does. That allowed me to quickly find a post I had read this morning about yet another new social networking company.
Fri, 07 Jan 2005
Blacklisting Fax Machines
Fax machines hate me. I used to receive a few calls a week from fax machines on my land-line. Eventually, I Googled my number and found that it was listed as a fax machine for USC. I emailed the webmaster and got the number removed, but it didn't really end the persistent fax machines.
I moved a few months ago and was able to ditch the land-line. (My previous landlord had DirectTV dishes installed and cut the cable to the building making DSL the only affordable option for high-speed Internet access.)
Now, I've got a fax machine calling my cell phone and it won't give. It's
calling me every 60 seconds and leaving me voice mail. Damn you, 818-783-7049!
Why isn't there a button on my cell phone to block all calls from a number?
Tue, 21 Dec 2004
Blockbuster Could Use Better Developers
How many stars does Napoleon Dynamite get?
Hmm, the Enhanced Widescreen Letterbox DVD version must be much better.
Update (Fri, 24 Dec 2004): I received Napoleon Dynamite from
Blockbuster. The sleeve that DVD came said that it was disc "1 of null".
Thu, 18 Nov 2004
Google, Say It Isn't So Already
Some web hosting customers, especially those with dubious business plans who have hired SEO witch doctors, request that every web site be placed on a separate IP address. The theory is that Google is likely to discount the links between sites on the same IP address because such sites may be trying to artificially increase their PageRank. It would be silly for Google to do this since there could be hundreds or thousands of unrelated sites that share an IP address, simply because they are customers of the same web hosting company. Nonetheless, this seems to be a widely-held belief.
This is frustrating to those in the hosting business because IP(v4) addresses are a finite resource, and here in North America, requests for new IP addresses for IP-based web hosting must include a technical justification.
Today, Sean emailed Google to ask for clarification. I hope that Google will publicly state that using multiple IP addresses cannot be used to inflate PageRank.
Update (Thu, 09 Dec 2004): Google responds by saying, go ask someone else to speculate.
Date: Thu, 09 Dec 2004 14:33:35 -0800 From: help@google.com Subject: Re: [#17061378] Regarding the need for a static IP Address Hi Sean, Thank you for writing to us. Please note that we don't comment on webmaster techniques or the details of our search technology beyond what appears on our site. We've dedicated an entire section of our site to answering the most common questions from those who maintain and/or promote websites. You'll find all of our publicly available information posted at http://www.google.com/webmasters/index.html Besides this section of our site, we've created a newsgroup discussion forum for passionate Google users. At http://groups.google.com in the http://groups.google.com/groups?q=google.public.support.general group, many webmasters and Google users share their questions and expertise. We recommend performing an advanced search on this group if you feel your question is particularly challenging and you've been unable to find an answer on our site. To do so, go to http://www.google.com/advanced_group_search?hl=en and enter your search terms in one of the 'Find messages' fields at the top of the page. Type 'google.public.support.general' in the 'Newsgroup' search field, and click 'Google Search.' If you don't find an answer to your question, you can always post your question to the group to see if other newsgroup users have helpful advice. Please be aware that this content isn't posted by Google, and we cannot verify its accuracy. Regards, The Google Team
Apache Virtual Host Bandwidth Monitoring
Also, a good example of the READ-COMMITTED isolation level
mod_accounting is a simple
Apache module for recording bandwidth usage on a per-virtual host basis. It
can write to either a MySQL or PostgreSQL database. You just add a couple
directives to httpd.conf to tell mod_accounting which database to write to, the
query to run, and how often to write to the database.
Here's the configuration I use.
<IfModule mod_accounting.c>
AccountingQueryFmt " \
INSERT INTO \
vhost_accounting \
( \
bytes_in, \
bytes_out, \
virtual_host, \
host \
) \
values \
( \
%r, \
%s, \
LOWER('%h'), \
'host.example.com' \
);"
AccountingTimedUpdates 300
AccountingDatabase vhost_accounting
AccountingDatabaseDriver mysql
AccountingDBHost stats.example.com 3306
AccountingLoginInfo vhost_accounting XXXXXXXX
</IfModule>
And the corresponding table.
CREATE TABLE `vhost_accounting` ( `virtual_host` varchar(128) NOT NULL default '', `bytes_in` int(20) unsigned NOT NULL default '0', `bytes_out` int(20) unsigned NOT NULL default '0', `timestamp` timestamp(14) NOT NULL, `host` varchar(128) NOT NULL default '', `id` bigint(20) unsigned NOT NULL auto_increment, PRIMARY KEY (`id`) ) TYPE=InnoDB
If you have tables of hosts and virtual hosts, you'll probably want to
normalize the database and modify AccountingQueryFmt to use the foreign key to
the vhost. For example,AccountingQueryFmt "INSERT INTO vhost_accounting (bytes_in, bytes_out, virtual_host_id) SELECT %r, %s, id FROM vhost WHERE virtual_host_name = LOWER('%h');"
There are a couple issues that make deploying mod_accounting to multiple machines a bit tricky. Each apache process does its own logging, and thus, maintains its own connection to the database server. If you have hundreds of Apache processes and multiple servers, you'll quickly have thousands of open connections to the database. To reduce the number of concurrent connections, I made a one-line change to mod_accounting to close the connection to MySQL after writing to the database so a new connection is used each time data is written to the database.
The next problem is simply the large amount of data that is collected. It
quickly grows to the point where generating reports in real-time is too slow.
To solve this problem, I created a summary table, vhost_accounting_summary, to
consolidate the data produced by each Apache process. I also decided that
hourly statistics were sufficient for my needs.
The records in the vhost_accounting table are removed once the data has been
inserted into vhost_accounting_summary. One other table,
vhost_accounting_timestamp, is used to keep track of when the summary table was
last updated and to ensure that data from vhost_accounting does not
accidentally get written to vhost_accounting_summary multiple times. (More
details below.)
CREATE TABLE `vhost_accounting_summary` ( `virtual_host` varchar(128) NOT NULL default '', `bytes_in` int(20) unsigned NOT NULL default '0', `bytes_out` int(20) unsigned NOT NULL default '0', `timestamp` timestamp(14) NOT NULL, `host` varchar(128) NOT NULL default '', KEY `virtual_host` (`virtual_host`), KEY `idx_timestamp_host` (`timestamp`,`host`) ) TYPE=InnoDB CREATE TABLE `vhost_accounting_timestamp` ( `timestamp` timestamp(14) NOT NULL ) TYPE=InnoDB
And here's the SQL to update the vhost_accounting_summary table.
SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
BEGIN;
SELECT timestamp FROM vhost_accounting_timestamp FOR UPDATE;
SELECT @max_id:= max(id) FROM vhost_accounting;
INSERT INTO
vhost_accounting_summary
SELECT
virtual_host,
SUM(bytes_in),
SUM(bytes_out),
DATE_FORMAT(timestamp, '%Y-%m-%d %H:00:00') AS dayhour,
host
FROM
vhost_accounting
WHERE
id < @max_id
GROUP BY
virtual_host,
dayhour;
DELETE FROM vhost_accounting WHERE id < @max_id;
UPDATE vhost_accounting_timestamp SET timestamp = now();
COMMIT;
There are a couple important points to make about the SQL above. We want to
make sure that mod_accounting never has to wait to write to the
vhost_accounting because of a lock. In addition, we want to ensure that, if
multiple copies of the update script get run concurrently, it does not cause
the same data to be written to vhost_accounting_summary multiple times.
Mod_accounting does its work while processing a request. If mod_accounting is waiting for a lock, the Apache process will not finish processing the current request until the lock is removed or a lock timeout occurs. This can quickly cause all of the Apache children to be tied up and prevent new requests from being processed.
To prevent a lock from being set, we only act on records with a primary key one less than the maximum. This prevents a next-key lock from being set when we delete the records we have just processed.
To prevent multiple instances of the script from causing redundant data to be
stored, we do a locking read of the timestamp by using FOR
UPDATE. This lock only affects this summary script since
mod_accounting does not use the vhost_accounting_timestamp table.
Furthermore, we must use the READ-COMMITTED
isolation level to guarantee that once the script receives the lock, it
doesn't read any rows from vhost_accounting that have already been
deleted.
And finally, we're ready to generate some reports…
SELECT SUM(bytes_in) AS bytes_in, SUM(bytes_out) AS bytes_out, SUM(bytes_in + bytes_out) AS total, virtual_host, host FROM vhost_accounting_summary WHERE timestamp >= '2004-11-18 00:00:00' AND timestamp <= '2004-11-18 23:59:59' GROUP BY virtual_host ORDER BY total DESC
Tue, 16 Nov 2004
Exchange Replacements
I spent a few hours today researching Exchange replacements. These are products that are designed to replace Microsoft Exchange on the server, but still allow use of Outlook as a client, including the much-beloved calendaring features.
Here's what I came up with.
Date: Tue, 16 Nov 2004 16:24:55 -0800 From: "Christian G. Warden" <cwarden@zerolag.com> Subject: Exchange replacement analysis - phase 1 There are a handful of products that claim to be Exchange replacements. They all work in the same manner, using a custom MAPI connector, which is basically a plug-in for Outlook, to access the server. Each version of Outlook has different features so most of these products only work with certain versions of Outlook. Because I'm not very familiar with Outlook, it is difficult for me to tell if these products fully support the features of Outlook. We'll need to setup a test environment to fully evaluate any of these products. OpenGroupware[1] This was previously a closed source server that was open sourced a couple years ago. I evaluated it briefly a year or so ago, and it seemed stable and featureful, but had a bit of a clunky web interface. It looks like development is pretty active, though. I haven't evaluated the Outlook connector, ZideLook[2], which is a commercial product which costs about $50 per client. There is no demo of ZideLook available. ZideLook communicates with OpenGroupware using WebDAV. OpenGroupware just handles the groupware functionality and integrates with third-party IMAP servers. 1. http://www.opengroupware.org/en/index.html 2. http://esd.element5.com/product.html?cart=1&productid=517934&languageid=1&nolselection=1¤cies=EUR SUSE LINUX Openexchange Server[3] This is a commercial product. It is distributed a full linux distribution and cannot be installed on an existing Linux system. (Such an installation would not be supported at least.) Pricing is unclear. The product is supposed to be available for purchase online at novell.com, but isn't, perhaps because they are currently integrating the product with Novell's Groupwise. There is an online demo[4] and the Outlook connector is available for download[5]. Openexchange is made up of a number of open source components and comFire, the groupware component, which was licensed from a company called Netline. comFire has recently been open sourced by Netline as Open-Xchange[6], but the Outlook connector is not licensed for use with Open-Xchange. The Outlook connector communicates with the server using WebDAV. There is a good article about Openexchange[7]. 3. http://www.suse.com/us/business/products/openexchange/index.html 4. http://www.suse.com/us/business/products/openexchange/demo.html 5. http://www.suse.com/us/business/products/openexchange/download.html 6. http://mirror.open-xchange.org/ox/EN/product/ 7. http://www.linux-magazine.com/issue/48/Suse_Linux_Openexchange_41.pdf Bynari Insight Server[8] and Insight Connector[9] I believe Bynari was the first company with an "Exchange replacement on Linux" product. Their Outlook connector allows calendars and address books on an IMAP server. It claims to require the Insight Server, though Insight Server uses Cyrus as the IMAP server, so it may work with a normal Cyrus server. Insight Server is composed of a number of open source products such as Postfix, OpenLDAP, and Apache. Bynari seems to think most of the value is in the Connector since a 1000 user license for Insight Connector is $17,000, and a 1000 user license for a bundled Insight Server and Insight Connector is $18,000. (Insight Server without the Connector is also sold for $1,000.) A demo is available. 8. http://www.bynari.net/index.php?id=1169 9. http://www.bynari.net/index.php?id=7 BILL Workgroup Server[10]/Exchange4Linux[11] Documentation is kind of spotty on this one. I don't think it's worth evaluating except as a last resort. 10. http://www.billworkgroup.org/billworkgroup/home 11. http://www.exchange4linux.com/exchange4linux/Home None of the Above (IMAP/LDAP/SMTP/WebDAV or FTP) Depending on the customer's needs, perhaps Outlook in "Internet Mail Mode" will be sufficient. IMAP supports shared folders, but I don't know if it supports setting ACLs. Outlook also supports LDAP for address books, but I don't know if supports updating the directory. Outlook can send meeting requests and responses over email and publish free/busy time over FTP (and, I think, either WebDAV or HTTP PUT), but I don't know if this would meet the customer's needs. I recommend trying out Openexchange first as it seems to be the most open and widely deployed. Christian
Comments from anyone who has deployed one of these products for use with Outlook would be appreciated.
tech » mail | Permanent Link
Promises are Useful
Henrik Joretag tweeted that he doesn't like promises in javascript.
I’m just gonna say it:
I really don’t like promises in JS.
It’s a clever pattern, but I just don’t see the upside, sorry.
— Henrik Joreteg (@HenrikJoreteg) October 26, 2014I've been working on a large javascript application for the past two years, and I've found promises to be invaluable in expressing the dependencies between functions.
But Henrik's a smart guy, having authored Human Javascript and written lots of Ampersand.js, so I decided to look at a recent case where I found the use of promises helpful, and try to come up with the best alternative solution.
The Backstory
I recently discovered a bug in our app initialization. It's an offline-first mobile app. We have a function that looks like this:
App.init = function() {
return Session.initialize()
.then(App.initializeModelsAndControllers)
.then(App.manageAppCache)
.then(App.showHomeScreen)
.then(App.scheduleSync)
.then(_.noop, handleError);
};
When the app starts up, it needs to:
- Load some user info from the local datastore, if possible; otherwise, from the network
- Load metadata about the models used in the app and initialize controllers
- Set up polling for application cache updates and event handles to appcache events, to prompt the user to restart when a new version of the app is available, for example
- Show the initial application screen
- Schedule a sync of metadata and data
- and if an error occurs during any of these steps, handle it gracefully
The chaining of promise-returning functions in App.init nicely expresses the
dependencies between these steps.
We need valid session info for the user before we can initialize the models,
which may require fetching data over the network.
We can't show the initial home screen until the controllers have been
initialized to bind to DOM events and display information from the models.
If there's a new version of the app available, we want to prompt the user to
restart before they start using the app, and we don't want to schedule a sync
until the app is being used.
Unfortunately, during a recent upgrade some users ran into a bug in some code I
wrote to upgrade the local database, called from
App.initializeModelsAndControllers. The result was an uncaught javascript
error, Uncaught RangeError: Maximum call stack size exceeded, on
the initialization screen which is displayed before App.init is called.
I fixed the bug which broke upgrading some databases, but I also realized that the dependency between App.initializeModelsAndControllers and App.manageAppCache should be reversed. If there's an upgrade available, we want users to be able to restart into it before initializing the app further (and running more code with a larger surface area for possible bugs).
The diff to make this change with our promise-returning functions is nice and simple:
--- a/app.js
+++ b/app.js
@@ -1,7 +1,7 @@
App.init = function() {
return Session.initialize()
- .then(App.initializeModelsAndControllers)
.then(App.manageAppCache)
+ .then(App.initializeModelsAndControllers)
.then(App.showHomeScreen)
.then(App.scheduleSync)
.then(_.identity, handleError);
(The careful reader will notice that stuck users need a way of upgrading the
broken app stored in the app cache. The cache manifest gets fetched
automatically to check for and trigger an app update each time the page is
loaded, so even though we haven't set up an event handler to prompt the user to
restart after the update has been downloaded, if the user waits a few minutes
to ensure that the app update is complete, then kills and restarts the app, the
new version will be loaded.
The careful reader who happens to be a coworker or affected customer will note
that this didn't actually work because our cache manifest requires a valid
session and our app starts up without refreshing the OAuth session to speed up
start-up time. An upcoming release of GreatVines
app to the iOS app store will contain a new setting to trigger the session
refresh at app initialization. Affected customers have been given a
work-around that requires tethering the device to a computer via USB.
Developers using the Salesforce Mobile SDK are welcome to email me for more
info.)
Recapping the Question
App.init = function() {
return Session.initialize()
.then(App.initializeModelsAndControllers)
.then(App.manageAppCache)
.then(App.showHomeScreen)
.then(App.scheduleSync)
.then(_.noop, handleError);
};
Now, back to the issue at hand. What would this function look like using an alternative to promises. Ignoring the possible use of a preprocessor that changes the semantics of javascript, the two likely candidates are callback passing and event emitter models.
The typical callback hell approach looks something like this:
App.init = function(success, error) {
Session.initialize(function() {
App.manageAppCache(function() {
App.initializeModelsAndControllers(function() {
App.showHomeScreen(function() {
App.scheduleSync(success, error);
}, error);
}, error);
}, error);
}, error);
};
For simplicity, I've ignored the fact that the original version both had some error-handling and returns the error to the caller through a rejected promise, but I'm pretty sure this is not what Henrik had in mind. So what might an improved callback-passing approach look like?
App.init = function(success, error) {
var scheduleSync = App.scheduleSync.bind(App, success, error);
var showHomeScreen = App.showHomeScreen.bind(App, scheduleSync, error);
var initializeModelsAndControllers = App.initializeModelsAndControllers.bind(App, showHomeScreen, error);
var manageAppCache = App.manageAppCache.bind(App, initializeModelsAndControllers, error);
Session.initialize(manageAppCache, error);
};
By pre-binding the callbacks, we avoid the nested indentations, but we've reversed the order of the dependencies, so that's not a great improvement.
I've failed to come up with a good callback-passing alternative to promises, so let's look at what event emitter approach would look like.
App.init = function() {
BackboneEvents.mixin(App);
App.once('sessionInitialized', App.manageAppCache);
App.once('appCacheManaged', App.initializeModelsAndControllers);
App.once('modelsAndControllersInitialized', App.showHomeScreen);
App.once('homeScreenShown', App.scheduleSync);
var initialized = App.trigger.bind(App, 'initialized');
App.once('syncScheduled', initialied);
App.once('sessionInitializationError appCacheManagementError modelsAndControllersInitializationError homeScreenError syncSchedulingError', function(error) {
App.off('sessionInitialized', App.manageAppCache);
App.off('appCacheManaged', App.initializeModelsAndControllers);
App.off('modelsAndControllersInitialized', App.showHomeScreen);
App.off('homeScreenShown', App.scheduleSync);
App.off('syncScheduled', initialied);
App.trigger('initializationError', error);
});
Session.initialize();
};
This approach does allow us to list dependencies from top to bottom, and
doesn't take us into calllback hell, but it has some problems as well.
With the promise-returning and callback-passing approaches, each function was
loosly coupled to the caller, returning a promise or accepting success and
error callbacks. With this event emitter approach, we need to make sure both
the caller and called agree upon the names of events. Admittedly, most
functions are going to be more tightly coupled than these, regardless of the
approach, having to agree upon parameters to the called function and parameters
to the resolved promise or callback, but they don't need to share any special
information like the event name to determine when the called function is done.
This event emitter example also requires more work to clean up when an error
occurs.
Perhaps instead of using a single event bus, we could listen for events on multiple objects and standardize on event names. Events could be triggered on each function.
App.init = function() {
var initialized = App.init.trigger.bind(App.init, 'initialized');
var error = function(error) {
App.init.trigger('error', error);
Session.initialize.off('done', App.manageAppCache);
Session.initialize.off('error', error);
App.manageAppCache.off('done', App.initializeModelsAndControllers);
App.manageAppCache.off('error', error);
App.initializeModelsAndControllers.off('done', App.showHomeScreen);
App.initializeModelsAndControllers.off('error', error);
App.homeScreenShown.off('done', App.scheduleSync);
App.homeScreenShown.off('error', error);
App.scheduleSync.off('done', initialized);
App.scheduleSync.off('error', error);
};
// Assume every function mixes in BackboneEvents
Session.initialize.once('done', App.manageAppCache);
Session.initialize.once('error', error);
App.manageAppCache.once('done', App.initializeModelsAndControllers);
App.manageAppCache.once('error', error);
App.initializeModelsAndControllers.once('done', App.showHomeScreen);
App.initializeModelsAndControllers.once('error', error);
App.homeScreenShown.once('done', App.scheduleSync);
App.homeScreenShown.once('error', error);
App.scheduleSync.once('done', initialized);
App.scheduleSync.once('error', error);
Session.initialize();
};
This is very verbose, containing a lot of boilerplate. We could create a function to create these bindings, e.g.
var when = function(fn, next, error) {
BackboneEvents.mixin(fn);
fn.once('done', next);
fn.once('error', error);
};
when(Session.initialize, App.manageAppCache, error);
...
But it doesn't feel like we're on the right path. We'd still have to handle
unbinding all of the error handlers when an error occurs in one of the
functions.
It's also worth pointing out that using an event emitter isn't really
appropriate in a case like this where you want to get an async response to a specific
function call. If the called function can be called multiple times, you don't
know that the event that was triggered was in response to your call without
passing additional information back with the event, and then filtering.
There's Got To Be A Better Way
Unsatisfied with any of the alternatives to promises I came up with, I replied
to Henrik's tweet, and he said that he uses Async.js and Node's error-first
convention. So let's look at what using async.series would
look like.
App.init = function(callback) {
async.series([
Session.initialize,
App.manageAppCache,
App.initializeModelsAndControllers,
App.showHomeScreen,
App.scheduleSync,
], callback);
Finally, we've come to an alternative solution that feels equivalent to the
promise one. The async.series solution has at least one benefit
over the promise-chaining one: the Async library has a separate function,
async.waterfall, to use when the return value from one function
should be passed as a parameter to the next function in the series. In my
original solution, it's not clear without looking at the function definitions
whether the return value from each function is used by the next function.
Given the simple error handling in App.init, in which an error anywhere in the chain aborts the flow, and a single error handler function is responsible for handling the error from any of the functions, Async.js would be a decent replacement.
But for a more complicated flow, a promise-based approach would
probably come out on top. For example, here's what
App.initializeModelsAndControllers looks like:
App.initializeModelsAndControllers = function() {
return Base.init()
.then(Base.sufficientlyLoadedLocally)
.then(App.handleUpgrades, App.remoteInit)
.then(App.loadControllers);
};
This function introduces a fork in the chain of serialized functions:
- Initialize the models
- Check whether the app is ready to start up offline
- If so, kick off any upgrade processing necessary
- If not, start up the remote initialization process
- Initialize the controllers
It looks like this could be done with async.auto,
but it would probably start getting messy.
Fri, 01 Oct 2004
Airplay Exposes Videos Watched in Chrome's Incognito Mode
iOS may expose your guilty pleasures by showing the names of videos watched in
Chrome's Incognito mode.
After exiting incognito mode, the name of a video viewed is still shown.
Killing chrome removes the name of the video.
