Mon, 18 Jan 2016

Requirements in Micro-purchasing

I recently completed an 18F micro-purchase task. It turned out to be a very simple task, tweaking some CSS. I spent more time on bidding, setting up the development environment, and figuring out how to receive payment than on the actual work.

My pull request was accepted, and 18F paid me quickly, but after my changes were merged, Maya Benari pointed out that a simpler solution may have been acceptable, or perhaps even preferable.

It turns out that 18F have drafted U.S. Web Design Standards, and have already given a lot of thought to things like vertical spacing, the subject of my change.

As an independent contractor with no previous experience working with 18F, I didn't know there was a U.S. Web Design Standards document, much less its contents. Had this task been assigned to an internal staff member, they probably would have known that the page being updated should follow the standards, or that if the standards didn't adequately address the requirements for the task, that perhaps a future update to the standards should be planned.

Getting the right level of details in task requirements is often tricky. One doesn't want to spend days writing requirements for a task that can be completed in hours. But one also doesn't want what could have been an hour-long task drag on for days because the requirements were ambiguous.

When working within a long-lived team structure, there is some shared institutional knowledge that doesn't need to be made explicit in every task, like the fact that U.S. Web Design Standards exist. Usually when bringing in an outside contractor or new employee, one is starting a relationship that will last weeks, months, or years, and knowledge transfer is an expected part of the on-boarding. This investment makes sense given the size of the overall engagement.

Micro-purchasing of software development work is unique in this respect. Neither party is committing to a long-term relationship, and therefore unlikely to bear significant knowledge transfer costs. Had the acceptance criteria for my spacing fix task included compliance with the U.S. Web Design Standards, I would have had to bid more to cover the time to read and understand the document. And 18F wouldn't want to bear the cost of making me an expert on U.S. Web Design Standards if I were only going to complete a single task.

It's still early in 18F's micro-purchasing experiment, but if the experiment continues or if other organizations want to try it out, it's worth thinking about what types of tasks are best suited for micro-purchasing, and what level of detail in the requirements is optimal.

Some possible questions to ask about tasks you are considering outsourcing as micro-purchases:

As always when discussing project management, I suggest ordinal prioritization of tasks and confidence interval based estimates. If your existing tools don't support these, I recommend checking out LiquidPlanner (affiliate link).

I like the micro-purchasing model, and I hope that it succeeds and is adopted by commercial organizations.

I've gotten multiple emails from grad students doing research on Github. If you're an economics student interested in industrial organization or open source, micro-purchasing of open source software could be an interesting research project.

business | Comments | Permanent Link

Thu, 24 Sep 2015

Debugging Salesforce Apps with Flame Graphs

Salesforce applications can sometimes be difficult to debug. The debug log often has the information you need, but finding it isn't always easy.

For example, in a recent project, I needed to figure out why performing an action from a Visualforce page was causing more SOQL queries than expected, generating warning emails as we neared the governor limit. This was a large application that had been under development for about eight years; many different developers had been involved in various phases of its development.

As with many Salesforce applications that have been through many phases of development, it wasn't obvious how multiple triggers, workflow rules, and Process Builder flows were interrelated.

It was easy enough to figure out which SOQL queries were running multiple times by grepping the debug log, but it wasn't obvious why the method containing the query was being called repeatedly.

In researching ways to analyze call paths, I found Brendan Gregg's work on flame graphs, which visualizes call stacks with stacked bar charts. (You might also be familiar with Chrome's flame charts.)

Visualizing the call path through stacked bar charts turned out to be a good way to make sense of the Salesforce debug log.

So I created Apex Flame to visualize Salesforce debug logs using flame charts or flame graphs.

Here's an example based on the debug log generated from running one of the tests included in the Declarative Lookup Rollup Summaries tool.

Example Flame Graph Reset Zoom Search RollupSummariesSelector.RollupSummariesSelector() (1235996 ns, 0.10%) SObjectSelector.assertIsAccessible() (235478 ns, 0.02%) RollupSummariesSelector.getSObjectFieldList() (1036673 ns, 0.09%) StringBuilder.CSVBuilder.add(String) (4611602 ns, 0.38%) SObjectSelector.getFieldListBuilder() (6141498 ns, 0.51%) SObjectSelector.getFieldListString() (6768212 ns, 0.56%) select Name,Active__c,AggregateOperation__c,AggregateResultField__c,CalculationMode__c,ChildObject__c,ConcatenateDelimiter__c,FieldToAggregate__c,FieldToOrderBy__c,ParentObject__c,RelationshipCriteria__c,RelationshipCriteriaFields__c,RelationshipField__c,CalculateJobId__c,UniqueName__c,CalculationSharingMode__c from LookupRollupSummary__c where CalculationMode__c in :calculationModeNames and ChildObject__c in :childObjectNames and Active__c = true order by ParentObject__c, RelationshipField__c (17721838 ns, 1.47%) RollupSummariesSelector.selectActiveByChildObject(List<RollupSummaries.CalculationMode>, Set<String>) (25244446 ns, 2.09%) RollupService.describeRollups(Schema.SObjectType) (27229671 ns, 2.26%) RollupService.triggerHandler() (28183993 ns, 2.34%) RollupServiceTest3Trigger on Account trigger event AfterInsert for [0011a00000CUUaT] (216806872 ns, 17.98%) RollupServiceTest.. RollupService.RollupService() (4998569 ns, 0.41%) RollupServiceTest3Trigger on Account trigger event BeforeInsert for [new] (111753618 ns, 9.27%) RollupS.. Insert:Account (331363950 ns, 27.48%) Insert:Account SObjectDomain.MockDatabase.hasRecords() (318698 ns, 0.03%) RollupSummaries.Constructor.construct(List<SObject>) (252746 ns, 0.02%) ApexTriggersSelector.getSObjectFieldList() (494428 ns, 0.04%) StringBuilder.CSVBuilder.add(String) (2209819 ns, 0.18%) SObjectSelector.getFieldListBuilder() (2910150 ns, 0.24%) SObjectSelector.getFieldListString() (3082568 ns, 0.26%) select Name,UsageIsBulk,UsageBeforeUpdate,UsageBeforeInsert,UsageBeforeDelete,UsageAfterUpdate,UsageAfterUndelete,UsageAfterInsert,UsageAfterDelete,TableEnumOrId,SystemModstamp,Status,NamespacePrefix,LengthWithoutComments,LastModifiedDate,LastModifiedById,IsValid,Id,CreatedDate,CreatedById,BodyCrc,Body,ApiVersion from ApexTrigger where Name in :names order by Name (6633886 ns, 0.55%) ApexTriggersSelector.selectByName(Set<String>) (10219683 ns, 0.85%) LREngine.RollupSummaryField.isAggregateBasedRollup() (253205 ns, 0.02%) LREngine.RollupSummaryField.isQueryBasedRollup() (333290 ns, 0.03%) LREngine.Context.add(LREngine.RollupSummaryField) (683669 ns, 0.06%) LREngine.RollupSummaryField.isDateOrTime(Schema.DisplayType) (557102 ns, 0.05%) LREngine.RollupSummaryField.isNumber(Schema.DisplayType) (851243 ns, 0.07%) LREngine.RollupSummaryField.isText(Schema.DisplayType) (642769 ns, 0.05%) LREngine.RollupSummaryField.validate() (431676 ns, 0.04%) RollupSummaries.makeTriggerName(LookupRollupSummary__c) (18859039 ns, 1.56%) RollupSummaries.onValidate() (57860434 ns, 4.80%) Rol.. SObjectDomain.handleAfterInsert() (57952353 ns, 4.81%) SO.. SObjectDomain.triggerHandler(SObjectDomain.IConstructable, Boolean, Boolean, Boolean, Boolean, Boolean, List<SObject>, Map<Id,SObject>) (58249145 ns, 4.83%) SO.. SObjectDomain.triggerHandler(System.Type) (59311227 ns, 4.92%) SO.. RollupSummariesTrigger on LookupRollupSummary trigger event AfterInsert for [a051a000002ly2F] (115241071 ns, 9.56%) RollupS.. SObjectDomain.SObjectDomain() (10311621 ns, 0.86%) SObjectDomain.MockDatabase.hasRecords() (3485995 ns, 0.29%) RollupSummaries.RollupSummaries() (2235961 ns, 0.19%) RollupSummaries.Constructor.construct(List<SObject>) (2968808 ns, 0.25%) SObjectDomain.triggerHandler(SObjectDomain.IConstructable, Boolean, Boolean, Boolean, Boolean, Boolean, List<SObject>, Map<Id,SObject>) (3215443 ns, 0.27%) SObjectDomain.triggerHandler(System.Type) (6785400 ns, 0.56%) RollupSummariesTrigger on LookupRollupSummary trigger event BeforeInsert for [new] (48017934 ns, 3.98%) Ro.. Insert:LookupRollupSummary__c (166856163 ns, 13.84%) Insert:Looku.. RollupSummariesSelector.getSObjectFieldList() (452150 ns, 0.04%) StringBuilder.CSVBuilder.add(String) (1558019 ns, 0.13%) SObjectSelector.getFieldListBuilder() (2293678 ns, 0.19%) SObjectSelector.getFieldListString() (2649104 ns, 0.22%) select Name,Active__c,AggregateOperation__c,AggregateResultField__c,CalculationMode__c,ChildObject__c,ConcatenateDelimiter__c,FieldToAggregate__c,FieldToOrderBy__c,ParentObject__c,RelationshipCriteria__c,RelationshipCriteriaFields__c,RelationshipField__c,CalculateJobId__c,UniqueName__c,CalculationSharingMode__c from LookupRollupSummary__c where CalculationMode__c in :calculationModeNames and ChildObject__c in :childObjectNames and Active__c = true order by ParentObject__c, RelationshipField__c (7340162 ns, 0.61%) RollupSummariesSelector.selectActiveByChildObject(List<RollupSummaries.CalculationMode>, Set<String>) (10460284 ns, 0.87%) RollupService.describeRollups(Schema.SObjectType) (10976812 ns, 0.91%) RollupService.triggerHandler() (11852139 ns, 0.98%) RollupServiceTestTrigger on Opportunity trigger event AfterInsert for [0061a00000AjNjf] (377149414 ns, 31.28%) RollupServiceTestTrigger on Oppo.. RollupService.triggerHandler() (1149884 ns, 0.10%) RollupServiceTestTrigger on Opportunity trigger event BeforeInsert for [new] (188822108 ns, 15.66%) RollupServiceTe.. Insert:Opportunity (568896270 ns, 47.19%) Insert:Opportunity RollupActionCalculate.RollupActionCalculate() (1229813 ns, 0.10%) RollupActionCalculate.RollupToCalculate.toServiceRollupToCalculate() (914554 ns, 0.08%) LREngine.LREngine() (11705510 ns, 0.97%) LREngine.IsMultiCurrencyOrg() (1174095 ns, 0.10%) SELECT AccountId, Sum(Amount) lre0 FROM Opportunity WHERE AccountId in :masterIds GROUP BY AccountId (19113729 ns, 1.59%) LREngine.QueryExecutor.query(String, Set<Id>) (19272499 ns, 1.60%) LREngine.WithSharingQueryExecutor.query(String, Set<Id>) (19329423 ns, 1.60%) User Debug: New aggregarte value 100.0 for master 0011a00000CUUaTAAX (971765 ns, 0.08%) User Debug: SOQL is SELECT AccountId, Sum(Amount) lre0 FROM Opportunity WHERE AccountId in :masterIds GROUP BY AccountId (218734 ns, 0.02%) LREngine.rollUp(LREngine.Context, Set<Id>, Boolean) (23289994 ns, 1.93%) LREngine.rollUp(LREngine.Context, Set<Id>) (23341264 ns, 1.94%) LREngine.RollupSummaryField.isAggregateBasedRollup() (172964 ns, 0.01%) LREngine.RollupSummaryField.isQueryBasedRollup() (235926 ns, 0.02%) LREngine.Context.add(LREngine.RollupSummaryField) (520645 ns, 0.04%) LREngine.RollupSummaryField.isAggregateBasedRollup() (213647 ns, 0.02%) LREngine.RollupSummaryField.isDateOrTime(Schema.DisplayType) (433093 ns, 0.04%) LREngine.RollupSummaryField.isNumber(Schema.DisplayType) (291409 ns, 0.02%) LREngine.RollupSummaryField.isText(Schema.DisplayType) (527450 ns, 0.04%) LREngine.RollupSummaryField.validate() (296564 ns, 0.02%) RollupService.createLREngineContexts(List<LookupRollupSummary__c>) (7423909 ns, 0.62%) RollupSummariesSelector.getSObjectFieldList() (439088 ns, 0.04%) StringBuilder.CSVBuilder.add(String) (1518768 ns, 0.13%) SObjectSelector.getFieldListBuilder() (2216018 ns, 0.18%) SObjectSelector.getFieldListString() (2394601 ns, 0.20%) select Name,Active__c,AggregateOperation__c,AggregateResultField__c,CalculationMode__c,ChildObject__c,ConcatenateDelimiter__c,FieldToAggregate__c,FieldToOrderBy__c,ParentObject__c,RelationshipCriteria__c,RelationshipCriteriaFields__c,RelationshipField__c,CalculateJobId__c,UniqueName__c,CalculationSharingMode__c from LookupRollupSummary__c where UniqueName__c in :uniqueNames and Active__c = true order by ParentObject__c, RelationshipField__c (9208942 ns, 0.76%) RollupSummariesSelector.selectActiveByUniqueName(Set<String>) (12026548 ns, 1.00%) RollupSummariesSelector.getSObjectFieldList() (418751 ns, 0.03%) StringBuilder.CSVBuilder.add(String) (1555555 ns, 0.13%) SObjectSelector.getFieldListBuilder() (2244164 ns, 0.19%) SObjectSelector.getFieldListString() (2566370 ns, 0.21%) select Name,Active__c,AggregateOperation__c,AggregateResultField__c,CalculationMode__c,ChildObject__c,ConcatenateDelimiter__c,FieldToAggregate__c,FieldToOrderBy__c,ParentObject__c,RelationshipCriteria__c,RelationshipCriteriaFields__c,RelationshipField__c,CalculateJobId__c,UniqueName__c,CalculationSharingMode__c from LookupRollupSummary__c where CalculationMode__c in :calculationModeNames and ChildObject__c in :childObjectNames and Active__c = true order by ParentObject__c, RelationshipField__c (3684774 ns, 0.31%) RollupSummariesSelector.selectActiveByChildObject(List<RollupSummaries.CalculationMode>, Set<String>) (6684942 ns, 0.55%) RollupService.describeRollups(Schema.SObjectType) (6975134 ns, 0.58%) RollupService.triggerHandler() (7631690 ns, 0.63%) RollupServiceTest3Trigger on Account trigger event AfterUpdate for [0011a00000CUUaT] (21000032 ns, 1.74%) RollupService.triggerHandler() (689974 ns, 0.06%) RollupServiceTest3Trigger on Account trigger event BeforeUpdate for [0011a00000CUUaT] (7372667 ns, 0.61%) Update:SObject (28865245 ns, 2.39%) RollupService.rollup(List<RollupService.RollupToCalculate>) (83997802 ns, 6.97%) Rollu.. RollupActionCalculate.calculate(List<RollupActionCalculate.RollupToCalculate>) (85011182 ns, 7.05%) Rollu.. RollupActionCalculateTest.RollupActionCalculateTest() (3130766 ns, 0.26%) SELECT AnnualRevenue FROM Account WHERE Id = :tmpVar1 (5357048 ns, 0.44%) TestContext.TestContext() (1580402 ns, 0.13%) ApexTriggersSelector.ApexTriggersSelector() (1565665 ns, 0.13%) ApexTriggersSelector.getSObjectType() (246508 ns, 0.02%) SObjectSelector.assertIsAccessible() (746409 ns, 0.06%) ApexTriggersSelector.getSObjectFieldList() (807801 ns, 0.07%) StringBuilder.CSVBuilder.add(String) (9477382 ns, 0.79%) SObjectSelector.getFieldListBuilder() (10872873 ns, 0.90%) StringBuilder.CSVBuilder.getStringValue(String) (236619 ns, 0.02%) StringBuilder.CSVBuilder.getStringValue() (268153 ns, 0.02%) SObjectSelector.getFieldListString() (11221683 ns, 0.93%) SObjectSelector.getSObjectName() (197176 ns, 0.02%) select Name,UsageIsBulk,UsageBeforeUpdate,UsageBeforeInsert,UsageBeforeDelete,UsageAfterUpdate,UsageAfterUndelete,UsageAfterInsert,UsageAfterDelete,TableEnumOrId,SystemModstamp,Status,NamespacePrefix,LengthWithoutComments,LastModifiedDate,LastModifiedById,IsValid,Id,CreatedDate,CreatedById,BodyCrc,Body,ApiVersion from ApexTrigger where Name in :names order by Name (23913926 ns, 1.98%) ApexTriggersSelector.selectByName(Set<String>) (37030248 ns, 3.07%) SObjectSelector.SObjectSelector() (409239 ns, 0.03%) TestContext.isSupported() (41196903 ns, 3.42%) T.. RollupActionCalculateTest.testCalculateAction (1205664192 ns, 100.00%) RollupActionCalculateTest.testCalculateAction all (1205664192 ns, 100%) all

You can drill in by clicking on a statement, or click on Search to highlight statements that match a string.

I'd love to get some feedback from other Salesforce developers. Try out Apex Flame, and let me know what you think.

As for the debugging issue that initially inspired the creation of Apex Flame, it turned out to be the combination of a trigger that wasn't checking whether the old and new values of a field were different, causing it to run code unnecessarily, combined with a workflow rule that was causing the trigger to be called multiple times.

Another recent case in which it turned out to be helpful was in troubleshooting a process builder flow that was failing to update child records as expected. The flow assumed that if the parent record is new (ISNEW()) that it didn't have child records. It turns out this isn't necessarily true because a trigger on the parent object can create child records, and the flow runs after the after insert trigger. The order of execution is documented, but visualizing the execution order made it clear what was causing the unexpected behavior.

tech » salesforce | Comments | Permanent Link

Thu, 26 Feb 2015

Airplay Exposes Videos Watched in Chrome's Incognito Mode

iOS may expose your guilty pleasures by showing the names of videos watched in Chrome's Incognito mode.

After exiting incognito mode, the name of a video viewed is still shown.

Killing chrome removes the name of the video.

tech | Comments | Permanent Link

Tue, 11 Nov 2014

Net Neutrality

Thinking past hop 1

I had a great idea a few days ago, to start a website streaming cat videos called Petflix. I rented a server for $10/month that gave me 1Mbps of outbound bandwidth.

Growth has been exponential. I now have 20 customers that now pay me $1/month for streaming cat videos. Unfortunately, during peak cat video viewing hours, I've started getting complaints that the video streams are choppy.

Each stream uses 100kbps. So when I get more than 10 concurrent viewers, I max out my bandwidth, and packets start getting dropped.

I could buy another 1Mbps of bandwidth for $10/month, but that would make my website unprofitable. So I came up with a brilliant idea that would keep this operation going.

I have an old computer in my basement that I'm not using. I called up Comcast and offered to send them my computer, preloaded with all of my cat videos. They could install it in their data center, and I would configure my DNS servers to send all of my traffic originating from Comcast customers to the computer in their data center. Since 90% of my customers are on the Comcast network, I can grow my website without incurring any additional bandwidth costs.

Obama on net neutrality

The recent statement by Barack Obama on net neutrality suffers from the same error that most proponents make in arguing for net neutrality. It assumes away the problem of how the internet actually works.

Packets don't magically appear on an ISPs network, ready for the ISP to either send on to a customer, or throttle or block for some nefarious reason.

The Internet is a big interconnected network with lots of different agreements for connecting two networks together. As the owner of Petflix, I pay Tektonic for connecting my virtual server (which I also pay them for) with their network. Tektonic pays to connect their network with TierPoint (and for power and space in a data center). TierPoint pays to connect their network with Level 3. Level 3 and Comcast connect their networks, but the terms are contentious. And as a customer of an ISP, I pay for connecting my home network with Comcast.

Traditionally, large networks like Comcast and Level 3 would form settlement-free peering agreements. Both parties get about equal value out of connecting their networks, and it was deemed mutually beneficial to connect their networks without one party paying the other. Netflix started sending lots of traffic over the connection between Level 3 and Comcast, and Comcast no longer thinks they are getting equal value from peering, and wants to renegotiate so Level 3 pays them.

To be fair, Obama sort of recognizes that the problem isn't just between the ISP and their customers. One of his bullet points argues that net neutrality should apply between an ISP and other networks.

But if we apply the other rules of No Blocking, No Throttling, and No Paid Prioritization to the connections between ISPs and other networks, it's unclear what kind of arrangements would be legal. Would the government have to mandate the amount of bandwidth at these interconnects?

What We Talk About When We Talk About Bandwidth

Getting back to my parable, when a customer of Comcast pays for 20Mbps of bandwidth, and they can't even get 100kbps of throughput to Petflix, whose fault is it?

Is it Comcast's responsibility to ensure that there is enough bandwidth through Level 3, TierPoint, Tektonic, and Petflix?

Is it Petflix's responsibility?

Fortunately, we have a pretty good way of sorting it out: markets. If Petflix is so important to Comcast customers that Comcast will lose customers to competitors by not providing a satisfying cat video viewing experience, Comcast will likely work to ensure sufficient bandwidth is available, unless the cost to do so makes those customers unprofitable.

If Petflix customers start cancelling their subscriptions because they can't enjoy my videos, I will probably pay to ensure sufficient bandwidth is available, unless doing so makes the operation no longer profitable.

Net Competition

I'm sure readers will object that Comcast is nearly a monopoly in most markets, and therefore markets won't solve the problem. And I agree. So instead of championing net neutrality, I encourage readers to champion net competition.

Instead of adding federal regulation on top of local regulation which grants ISPs like Comcast monopolies, deregulate the market in ISPs. In Portland, there could be three high-speed ISPs competing for consumers' business soon.

Nobody knows what amount of bandwidth should exist between the millions of connections between networks, and who should pay how much to whom for these connections. Only the market can sort it out.

Obamacare for the Internet?

Senator Ted Cruz referred to net neutrality as Obamacare for the internet. Is that a valid analogy?

"Net Neutrality" is Obamacare for the Internet; the Internet should not operate at the speed of government.

— Senator Ted Cruz (@SenTedCruz) November 10, 2014

I would consider these the primary characteristics of Obamacare:

Obamacare doesn't seem like a great analogy for net neutrality. Both do add federal regulation on top of local/state regulation. And it's not inconceivable that if the FCC were to start regulating ISPs in the way that Obama suggests, that problems caused would lead to suggestions for more government "solutions" like government-run markets for network connectivity, but that's a bit of a stretch.

With a little research, Cruz probably could have come up with an act of Congress that would make a better analogy than the ACA.

tech | Comments | Permanent Link

Fri, 31 Oct 2014

Promises are Useful

Henrik Joretag tweeted that he doesn't like promises in javascript.

I’m just gonna say it: I really don’t like promises in JS. It’s a clever pattern, but I just don’t see the upside, sorry.

— Henrik Joreteg (@HenrikJoreteg) October 26, 2014

I've been working on a large javascript application for the past two years, and I've found promises to be invaluable in expressing the dependencies between functions.

But Henrik's a smart guy, having authored Human Javascript and written lots of Ampersand.js, so I decided to look at a recent case where I found the use of promises helpful, and try to come up with the best alternative solution.

The Backstory

I recently discovered a bug in our app initialization. It's an offline-first mobile app. We have a function that looks like this:

App.init = function() {
   return Session.initialize()
      .then(_.noop, handleError);

When the app starts up, it needs to:

  1. Load some user info from the local datastore, if possible; otherwise, from the network
  2. Load metadata about the models used in the app and initialize controllers
  3. Set up polling for application cache updates and event handles to appcache events, to prompt the user to restart when a new version of the app is available, for example
  4. Show the initial application screen
  5. Schedule a sync of metadata and data
  6. and if an error occurs during any of these steps, handle it gracefully

The chaining of promise-returning functions in App.init nicely expresses the dependencies between these steps.
We need valid session info for the user before we can initialize the models, which may require fetching data over the network.
We can't show the initial home screen until the controllers have been initialized to bind to DOM events and display information from the models.
If there's a new version of the app available, we want to prompt the user to restart before they start using the app, and we don't want to schedule a sync until the app is being used.

Unfortunately, during a recent upgrade some users ran into a bug in some code I wrote to upgrade the local database, called from App.initializeModelsAndControllers. The result was an uncaught javascript error, Uncaught RangeError: Maximum call stack size exceeded, on the initialization screen which is displayed before App.init is called.

I fixed the bug which broke upgrading some databases, but I also realized that the dependency between App.initializeModelsAndControllers and App.manageAppCache should be reversed. If there's an upgrade available, we want users to be able to restart into it before initializing the app further (and running more code with a larger surface area for possible bugs).

The diff to make this change with our promise-returning functions is nice and simple:

--- a/app.js
+++ b/app.js
@@ -1,7 +1,7 @@
 App.init = function() {
   return Session.initialize()
-     .then(App.initializeModelsAndControllers)
+     .then(App.initializeModelsAndControllers)
      .then(_.identity, handleError);

(The careful reader will notice that stuck users need a way of upgrading the broken app stored in the app cache. The cache manifest gets fetched automatically to check for and trigger an app update each time the page is loaded, so even though we haven't set up an event handler to prompt the user to restart after the update has been downloaded, if the user waits a few minutes to ensure that the app update is complete, then kills and restarts the app, the new version will be loaded.
The careful reader who happens to be a coworker or affected customer will note that this didn't actually work because our cache manifest requires a valid session and our app starts up without refreshing the OAuth session to speed up start-up time. An upcoming release of GreatVines app to the iOS app store will contain a new setting to trigger the session refresh at app initialization. Affected customers have been given a work-around that requires tethering the device to a computer via USB. Developers using the Salesforce Mobile SDK are welcome to email me for more info.)

Recapping the Question

App.init = function() {
   return Session.initialize()
      .then(_.noop, handleError);

Now, back to the issue at hand. What would this function look like using an alternative to promises. Ignoring the possible use of a preprocessor that changes the semantics of javascript, the two likely candidates are callback passing and event emitter models.

The typical callback hell approach looks something like this:

App.init = function(success, error) {
   Session.initialize(function() {
      App.manageAppCache(function() {
         App.initializeModelsAndControllers(function() {
            App.showHomeScreen(function() {
               App.scheduleSync(success, error);
            }, error);
         }, error);
      }, error);
   }, error);

For simplicity, I've ignored the fact that the original version both had some error-handling and returns the error to the caller through a rejected promise, but I'm pretty sure this is not what Henrik had in mind. So what might an improved callback-passing approach look like?

App.init = function(success, error) {
   var scheduleSync = App.scheduleSync.bind(App, success, error);
   var showHomeScreen = App.showHomeScreen.bind(App, scheduleSync, error);
   var initializeModelsAndControllers = App.initializeModelsAndControllers.bind(App, showHomeScreen, error);
   var manageAppCache = App.manageAppCache.bind(App, initializeModelsAndControllers, error);
   Session.initialize(manageAppCache, error);

By pre-binding the callbacks, we avoid the nested indentations, but we've reversed the order of the dependencies, so that's not a great improvement.

I've failed to come up with a good callback-passing alternative to promises, so let's look at what event emitter approach would look like.

App.init = function() {
   App.once('sessionInitialized', App.manageAppCache);
   App.once('appCacheManaged', App.initializeModelsAndControllers);
   App.once('modelsAndControllersInitialized', App.showHomeScreen);
   App.once('homeScreenShown', App.scheduleSync);
   var initialized = App.trigger.bind(App, 'initialized');
   App.once('syncScheduled', initialied);
   App.once('sessionInitializationError appCacheManagementError modelsAndControllersInitializationError homeScreenError syncSchedulingError', function(error) {'sessionInitialized', App.manageAppCache);'appCacheManaged', App.initializeModelsAndControllers);'modelsAndControllersInitialized', App.showHomeScreen);'homeScreenShown', App.scheduleSync);'syncScheduled', initialied);
      App.trigger('initializationError', error);

This approach does allow us to list dependencies from top to bottom, and doesn't take us into calllback hell, but it has some problems as well.
With the promise-returning and callback-passing approaches, each function was loosly coupled to the caller, returning a promise or accepting success and error callbacks. With this event emitter approach, we need to make sure both the caller and called agree upon the names of events. Admittedly, most functions are going to be more tightly coupled than these, regardless of the approach, having to agree upon parameters to the called function and parameters to the resolved promise or callback, but they don't need to share any special information like the event name to determine when the called function is done.

This event emitter example also requires more work to clean up when an error occurs.

Perhaps instead of using a single event bus, we could listen for events on multiple objects and standardize on event names. Events could be triggered on each function.

App.init = function() {
   var initialized = App.init.trigger.bind(App.init, 'initialized');
   var error = function(error) {
      App.init.trigger('error', error);'done', App.manageAppCache);'error', error);'done', App.initializeModelsAndControllers);'error', error);'done', App.showHomeScreen);'error', error);'done', App.scheduleSync);'error', error);'done', initialized);'error', error);
   // Assume every function mixes in BackboneEvents
   Session.initialize.once('done', App.manageAppCache);
   Session.initialize.once('error', error);

   App.manageAppCache.once('done', App.initializeModelsAndControllers);
   App.manageAppCache.once('error', error);

   App.initializeModelsAndControllers.once('done', App.showHomeScreen);
   App.initializeModelsAndControllers.once('error', error);

   App.homeScreenShown.once('done', App.scheduleSync);
   App.homeScreenShown.once('error', error);

   App.scheduleSync.once('done', initialized);
   App.scheduleSync.once('error', error);


This is very verbose, containing a lot of boilerplate. We could create a function to create these bindings, e.g.

var when = function(fn, next, error) {
   fn.once('done', next);
   fn.once('error', error);
when(Session.initialize, App.manageAppCache, error);

But it doesn't feel like we're on the right path. We'd still have to handle unbinding all of the error handlers when an error occurs in one of the functions.

It's also worth pointing out that using an event emitter isn't really appropriate in a case like this where you want to get an async response to a specific function call. If the called function can be called multiple times, you don't know that the event that was triggered was in response to your call without passing additional information back with the event, and then filtering.

There's Got To Be A Better Way

Unsatisfied with any of the alternatives to promises I came up with, I replied to Henrik's tweet, and he said that he uses Async.js and Node's error-first convention. So let's look at what using async.series would look like.

App.init = function(callback) {
   ], callback);

Finally, we've come to an alternative solution that feels equivalent to the promise one. The async.series solution has at least one benefit over the promise-chaining one: the Async library has a separate function, async.waterfall, to use when the return value from one function should be passed as a parameter to the next function in the series. In my original solution, it's not clear without looking at the function definitions whether the return value from each function is used by the next function.

Given the simple error handling in App.init, in which an error anywhere in the chain aborts the flow, and a single error handler function is responsible for handling the error from any of the functions, Async.js would be a decent replacement.

But for a more complicated flow, a promise-based approach would probably come out on top. For example, here's what App.initializeModelsAndControllers looks like:

App.initializeModelsAndControllers = function() {
   return Base.init()
      .then(App.handleUpgrades, App.remoteInit)

This function introduces a fork in the chain of serialized functions:

  1. Initialize the models
  2. Check whether the app is ready to start up offline
  3. Initialize the controllers

It looks like this could be done with, but it would probably start getting messy.

tech | Comments | Permanent Link

Tue, 30 Sep 2014

Changing the default browser in Debian

Note to self: the next time you want to change your default browser, here's how.

$ xdg-mime default chromium.desktop x-scheme-handler/http
$ xdg-mime default chromium.desktop x-scheme-handler/https
$ xdg-mime default chromium.desktop text/html
$ sudo update-alternatives --set x-www-browser /usr/bin/chromium

tech | Comments | Permanent Link

Wed, 17 Sep 2014

On Instacart Pricing Transparency

Stephen Klein, a biz dev guy from Instacart, emailed me to ask why I hadn't tried it yet. Here's my response:

Hi Stephen, I was excited to learn about Instacart offering deliveries from Costco. Unfortunately, your current pricing model makes deciding whether to use Instacart too expensive. It's not that your prices are too high. It's that I don't know if your prices are too high; that is, the cost of information is too high.

When deciding whether to use Instacart, my decision is between going to Costco or having Instacart go to Costco for me. In most cases, I would prefer to have Instacart go for me. (Sometimes, I might want to browse.) But to make the decision, I need to know how much it would cost me to use Instacart.

The current pricing model is not transparent enough to determine this cost. Your prices are not the same prices as those offered as Costco, so I don't know what the total cost of using Instacart is, and rather than spend the time trying to figure it out, I simply don't use Instacart.

My recommendation would be to change your pricing model to make explicit the cost of using Instacart. You could charge a fixed percentage above the Costco price, charge a higher delivery fee, or charge a per-item delivery fee, for example; whatever model would be both transparent and profitable.


business | Comments | Permanent Link

Mon, 08 Sep 2014

Building Multi-Container Apps with Panamax

CenturyLink released a new open source tool for building and managing applications composed of multiple docker containers called Panamax. It's similar to fig in providing a file format for describing the docker images which make up your application, and expressing the links between the containers.

The files used by Panamax to describe an application are called templates, and Panamax expands upon the model provided by fig by allowing applications to be built from existing templates, i.e. collections of existing images, and by providing a web interface for building templates. Templates can be fetched from and export to any github repo.

You create a new Panamax application by using an existing template or starting with a single docker image from Docker Hub.

Building an app using a template

Below I provide an example of how to build an application using a Panamax template. I created a template for the Cube event-logging and analysis server. (There's also a copy of the template in the panamax contest repo, but it was built using a MongoDB image which has since been deleted from Docker Hub.)

We'll create a simple Hello World node.js app, which logs each request to the cube server.

Install Panamax

First, install Panamax as described in the documentation. The current version of Panamax is distributed as a Vagrant VM, running CoreOS. Panamax itself is three docker apps that run in the VM: an API server, a web app that provides the main interface, and cAdvisor, used to monitor the docker containers. Panamax also includes a shell script, panamax, which is used to start up the VM.

Add a Panamax Template Source

Panamax is distributed with two github repos containing templates. Navigate to Manage | Sources to add my cwarden/panamax-templates repo as a source of templates.

Searching for Panamax Templates and Docker Images

Now, on the search screen, if you search for "cube", you'll find my template. (The first one is from the contest repo.)

Below the templates, you'll also find individual docker images. If you wanted to build up your application starting from a single image, you could start with one of these.

Click on More Details, and you'll see that my template is made up of three docker images: a MongoDB database, the cube collector, which accepts events and stores them in the database, and the cube evaluator, which reads data out of MongoDB and computes aggregate metrics from the events.

The details modal window also shows documentation I wrote up when creating the template.

Creating an Application

Click on Run Template. This will create a new application called "cube".

The Documentation link will show the same notes as on the More Details screen. The Port Forwarding section is important. Recall that Panamax and all of the docker containers it manages are running within a VM. If we want to access any of the services provided by these containers from outside the VM, we need to set up port forwarding.

In this case, we're going to add another docker container which sends data to the cube collector, but we'll want to access the cube evaluator from our host machine to make sure the logging is working correctly, so we need to set up port forwarding to VM for the evaluator:

$ VBoxManage controlvm panamax-vm natpf1 evaluator,tcp,,1081,,1081

(Instructions for sending data to the collector from the host machine are also included in the documentation.)

Prepare An App To Use With The Template

We'll use Valera Tretyak's simple hello world app for our application, making a small change to log each request to the cube collector.

Then we can create a docker image for this app and upload it to the Docker Hub.

$ docker build -t cwarden/hello-cube .
$ docker push cwarden/hello-cube

Extending The Template To Create a New Application

Now, let's actually use the template as a template by adding our new image as another container. From Manage | Manage Applications | cube, let's add the hello-cube service.

Next, we need to link the new container to the cube collector so it knows the hostname and port to use when sending events. We also need to expose the container's port to the VM. Click on the hourglass next to the hello-cube app.

And we need to expose the VM's port to the host machine.

$ VBoxManage controlvm panamax-vm natpf1 hello,tcp,,8080,,8080

Using The New App

When we access the app on localhost:8080, it will send an event to the cube collector. We can use the cube evaluator to monitor the number of events being generated.

Exporting An App As A Template

Now that we've finished building our app, we can export the app as a new template to a github repo using the Save as Template button.

tech | Comments | Permanent Link

Sat, 02 Nov 2013

Use Separate State Directory for Burp

I recently bought a RAID enclosure for my backup drives. One of the nice features is that it powers down the drives and fans when there is no activity for five minutes.

I use burp for backing up my laptops. Burp has a timed backup mode, in which a client can connect to the server, and start a backup if it's been too long since its last backup. This allows, for example, wife to suspend her laptop whenever she wants. When it is running, burp will check with the server every 20 minutes whether it needs to be backed up. If she suspends it in the middle of a backup, it will resume during the next check.

I've configured burp to only allow backups outside of my normal working hours so I don't need to hear the drives grinding away while I'm at my desk. (Pulling myself away from my desk at night is the next challenge.)

The only problem with this configuration is that when a client connects, the backup directory is accessed to figure out whether to start a backup, causing the drives in my RAID enclosure to spin up. The solution was to not have burp touch the backup directory until it's starting a backup.

When a client connects, burp creates a lockfile. By default, this lockfile is created in the backup directory, but there is an option to use a different directory, client_lockdir. Set this in the server's burp.conf.


The next change required is to have the timer script, which checks whether the server should start a backup when the client asks, use a separate directory to check the state of existing backups. First, enough data from the backup directory needs to be copied to a separate directory each time a backup is completed. To do this, I use the server_script_post option, which runs a script each time a backup completes.


My update_state_dir script copies the current symlinks and timestamp files from the backup directory to a separate state directory. I updated the timer script to check this state directory. Review all of the changes at github.

tech | Comments | Permanent Link

Thu, 05 Sep 2013

Estimating Tasks

As described in the previous post about our development process at GreatVines, we use LiquidPlanner for project management. Estimating how long tasks will take is important in most software development projects (and generally any project involving more than one person).

LiquidPlanner provides a great way of estimating tasks using an 80% confidence interval, but estimating tasks like this is a new concept for most developers. Below is a transcript with a developer that recently joined our project. We were able to get from "I have no idea how long it will take" to a decent estimate in about half an hour, but it should only take a minute or two for the next task.

2013-09-05 11:16:52: <Christian G. Warden> have you guys reviewed all of the
                     tasks for the september release?
2013-09-05 11:20:44: <Ben H> yes, but many of the tickets refer to parts of the
                     code I have no experience with so I have no idea how much
                     work they will be.
2013-09-05 11:23:11: <Christian G. Warden> you can put wide estimates on tasks
                     like that.  can you give me an example, and i'll walk you
                     through it?
2013-09-05 11:25:15: <Ben H> 10631525- Make Modal Form Fields scroll-able: I
                     have no idea where contacts are viewed in the application,
                     I haven't had any reason to look at their controller/model
2013-09-05 11:25:52: <Ben H> so any time estimate I would come up with would be
                     a sheer guess
2013-09-05 11:26:38: <Christian G. Warden> ok, no problem.  what's the
                     likelihood it will take you more than 2 weeks to complete
                     the task given what you currently know about the app?
2013-09-05 11:27:21: <Ben H> darn near 0% I would figure
2013-09-05 11:27:26: <Christian G. Warden> phew :) 
2013-09-05 11:27:31: <Christian G. Warden> how about 1 week?
2013-09-05 11:28:29: <Ben H> that's still very high, so 2%?
2013-09-05 11:28:38: <Christian G. Warden> how about 3 days?
2013-09-05 11:28:40: <Ben H> given that it looks like pure GUI work
2013-09-05 11:30:03: <Jim Thompson> yeah Ben the problem is that one customer
                     has put so many fields in the FieldSet that you can't see
                     the entire modal, the Save and Cancel buttons scroll below
                     the page
2013-09-05 11:30:33: <Ben H> I guess 3-4 days (if it is pure GUI work then 1-2
                     should be more then enough, unless there's something
                     unseemly complex)
2013-09-05 11:30:45: <Jim Thompson> I am going to move this task beneath the
                     Performance items Christian
2013-09-05 11:30:47: <Jim Thompson> (and Ben)
2013-09-05 11:32:20: <Christian G. Warden> would you say there's a 10%
                     probability that it will take more than 4 days?
2013-09-05 11:33:34: <Jim Thompson> at 12-32 hours it might have to drop off
                     the list, or at least to the bottom
2013-09-05 11:34:40: <Christian G. Warden> jim, we'll go through the tasks and
                     reprioritize after the estimates are updated.
2013-09-05 11:34:57: <Jim Thompson> ok, I also moved "New Task" and "New Event"
                     to October, which is when we promised it
2013-09-05 11:35:39: <Christian G. Warden> you're interrupting my estimation
                     lesson    :) 
2013-09-05 11:36:02: <Ben H> (yeah, he's been high balling for tutorial
2013-09-05 11:36:11: <Jim Thompson> ok im out
2013-09-05 11:37:50: <Christian G. Warden> no, i want to come up with realistic
                     estimates.  if they're wide right now because of
                     uncertainty as to what it will take to complete them,
                     that's fine.  once we have estimates, we can decide
                     whether to invest time to come up with tighter estimates.
2013-09-05 11:38:44: <Christian G. Warden> so, do you think 4 days as a high
                     estimate is accurate?
2013-09-05 11:39:29: <Ben H> for this example I put 12-24, it's really wide and
                     most likely most of that will be learning about how that
                     GUI element is working and making the tests
2013-09-05 11:39:57: <Ben H> the range is more to show that it could go
                     horribly wrong if there is something there I have no idea
2013-09-05 11:40:24: <Ben H> but 12 still seems really high
2013-09-05 11:40:30: <Christian G. Warden> what's the likelihood that you will
                     complete the task in 2 hours?
2013-09-05 11:41:14: <Ben H> the likihood I'll complete it in 6 hours I would
                     meet the 10% threshold
2013-09-05 11:41:26: <Ben H> *likelihood
2013-09-05 11:41:38: <Christian G. Warden> ok, so, let's make it 6-24
2013-09-05 11:45:29: <Christian G. Warden> and you can check yourself by
                     imagining we're going to a casino.  you can choose from
                     two wagers:
                     1) roulette - there are 10 numbers on the roulette wheel.
                     if 1 or 10 comes up, you lose. if 2 through 8 comes up,
                     you win $100.
                     2) the bookmaker - if you complete the task in less than 6
                     hours, you lose. if you complete the task in more than 24
                     hours, you lose.  if you complete the task between 6 and
                     24 hours, you win $100. which wager do you want to take?
2013-09-05 11:47:25: <Ben H> bookmaker, it also means I can spend some
                     speculative time trying different approaches to find one
                     that solves the problem better
2013-09-05 11:47:34: <Ben H> if things go well
2013-09-05 11:49:16: <Christian G. Warden> so it's not really an 80% confidence
                     interval.  it might be more like a 90% confidence
                     interval.  you should make the range narrower.
2013-09-05 11:49:52: <Ben H> how then do you show the risk if (I doubt in this
                     case) that 10% could be really bad
2013-09-05 11:52:19: <Christian G. Warden> it's ok if there's an outlier
                     occasionally.  we want to get an accurate estimate across
                     all of the tasks.
2013-09-05 11:56:09: <Christian G. Warden> you should narrow the range until
                     you're ambivalent between taking the two wagers.  and
                     widen the range when you'd rather play roulette.
2013-09-05 11:58:02: <Christian G. Warden> but we've already gone from "i have
                     no idea how long it will take" to "i'm 90% confident, it
                     will take between 6 and 24 hours".
2013-09-05 11:59:40: <Ben H> I'm just worried it's misplaced confidence
2013-09-05 12:00:03: <Ben H> either way, given the range, I am (90%) confident
                     that I can get it done within the range
2013-09-05 12:07:07: <Christian G. Warden> nobody dies if you're wrong.  you're
                     expected to be wrong 20% of the time (once you narrow it
                     to an 80% confidence interval).  if we decide we need to
                     know with more certainty when something will be done, we
                     might ask you to spend an hour to do a little research so
                     you can come up with a tighter estimate, after which you
                     would adjust the high and/or low estimate.  for now,
                     though, estimate the tasks based on what you currently
2013-09-05 12:07:50: <Ben H> ok

The roulette idea comes from Douglas Hubbard's How to Measure Anything.

business | Comments | Permanent Link

The state is that great fiction by which everyone tries to live at the expense of everyone else. - Frederic Bastiat