Some speculation and a vision of how to make finance safer.

What I know

• Something horrible happened
• Adequate testing did not occur
• Either there was no fire alarm for the event “hemorrhaging money” or the fire brigade was asleep

Regarding the last point, the New York Times reports — essentially — that there was no fire alarm, the fire brigade did notice, but there were no wheels on the firetruck.

What I don’t know

• I know nothing about the particular bug or the computing system
• I know nothing about Knight

If you assume that you know more facts about the situation than I do, you will almost surely be right.

What I imagine

bug size

I’ve seen the bug described as “a big bug”.  Bugs are hard to measure, but my guess is that it was a tiny bug.  I suspect it was the sort of bug that pretty much any programmer might create any day.  It just had big, big consequences.

solutions

There is talk of reeling in high frequency trading.  Certainly in terms of market-making stuffing that genie back into the bottle is more than Hercules will be able to handle.

But before instituting a fix, it is best to have isolated the bug.

causes

I hypothesize that the problem is that there is a pony express rider steering a rocketship, that it is easier to thread a camel through a needle than to promote a technical person to a position of real authority.

We humans are too stupid to be able to fully learn from the mistakes of others.  One of the interview questions to get a job overseeing computer trading should be: “What is your most embarrassing bug?”  If the answer is none or something minor, the candidate should be disqualified.

Someone who has had their knuckle hair singed would have:

• a seriously rigorous test suite for all code going into production
• a contingency plan if the code went berserk

A real solution

To my knowledge employment at large financial firms is all about politics.  It is the Peter Principle on steroids.  If they thought about it, I don’t think that the top financial people would want the airplanes that they ride around in to be built by a company that had the same promotion system.

My understanding of Boeing’s structure is that there is a parallel ladder: one side for management, one side for technical.  The managers take care of the political and bureaucratic elements, the technical people are given the respect and authority to take care of the technical aspects.  The managers are not bosses of their technical people, they are advocates and protectors.

Finance companies didn’t used to be technology companies.  Now they are, but their structures haven’t changed to match.  I’m doubtful that there’s a structure that can be successful for large companies that doesn’t have a separate technical track.  If politics and technical are combined, politics is always going to win out.