What happened to Knight Capital: $460 million lost in 45 minutes
Knight Capital lost more than $460 million in about 45 minutes on August 1, 2012, and nobody broke in.[1] One of its 8 trading servers was still running code that had sat unused for years. This case file covers how a routine software release went wrong, why the warnings were missed, what it cost, and the control that would have stopped it.
What happened
In 2012 Knight Capital was one of the biggest market makers on the U.S. stock market, handling a large share of retail orders routed from online brokers. Its order-routing system, called SMARS, ran on 8 servers.[3] In late July 2012 Knight rolled out new code for a new program for retail orders at the New York Stock Exchange, which was due to start on August 1.[1][3]
A technician copied the new code to the servers but missed one of the 8. Nobody else checked the work, because Knight did not require a second technician to review a deployment.[2][3] That eighth server still held an old function, known internally as Power Peg, that had not been used since 2003 but had never been removed.[3]
Before the market opened on August 1, Knight's system sent 97 automated emails to staff flagging an error. They were not set up as alerts, and nobody acted on them.[1][2] When trading began at 9:30, the 7 updated servers behaved normally. The eighth began firing orders without stopping. To fill just 212 small customer orders, it sent more than 4 million orders into the market in about 45 minutes.[1][2]
By the time the system was shut down, Knight had made more than 4 million trades in 154 stocks, covering more than 397 million shares. It was left holding about $3.5 billion in stocks it had bought and about $3.15 billion in stocks it had sold short, none of which it wanted.[3]
How it worked
No outsider was involved. The damage came from 3 old decisions meeting 1 new mistake.
First, the dead code stayed. Power Peg was retired in 2003 but left in place and still able to run.[3] Second, in 2005 Knight moved a piece of it, the counter that told Power Peg when an order was fully filled, and never retested the old function afterward. Without that counter, Power Peg had no way to know when to stop.[3] Third, the new 2012 code reused a setting, or flag, that used to switch Power Peg on. On 7 servers the flag now meant "use the new retail program." On the eighth, it woke up the old code.[3]
The new mistake was the missed server. One person copied the release by hand, there was no written procedure for doing it, and no one confirmed that all 8 servers matched.[2][3] Knight also had no automated control to halt a system that was behaving wildly, so staff had to diagnose the problem live while the market moved.[2]
What it cost
The trading loss was more than $460 million.[1] An analyst estimated the hit at about 30 percent of the firm's shareholder equity, and Knight's stock fell sharply in the days that followed.[4] Within a week, Knight took a $400 million rescue from a group of investors that included Getco, Blackstone and TD Ameritrade, in exchange for a majority stake.[4][5] In December 2012 Getco agreed to buy the whole company in a deal valued at about $1.4 billion.[5]
On October 16, 2013, the SEC charged Knight with violating the Market Access Rule, which requires brokers to have controls that stop erroneous orders before they reach the market. It was the first enforcement case under that rule. Knight agreed to pay $12 million and hire an independent consultant to review its controls.[1][2] The SEC's market abuse chief said firms must ask what would happen if each component in their systems malfunctions.[1]
The missing control
The missing control: verified, controlled code deployment. That means a written release procedure, a second person checking that every server received the new version, and an automatic way to stop the system when it misbehaves.
Any one of those would have changed the outcome. A second check would likely have caught the server that was missed. A simple comparison of what was running on all 8 machines would have shown one did not match. Removing dead code, or retesting it when the parts it relied on moved, would have left nothing old to wake up. And a kill switch tied to trading limits would have stopped the flood of orders in seconds rather than 45 minutes.[2][3]
What to do in your business
- Write down how updates go live. For your website, point-of-sale, booking or payment tools, keep a one-page checklist of steps, including how to confirm the update actually landed everywhere.
- Have a second person confirm. After any change to a system that touches money or customers, ask someone who did not make the change to check it works as expected.
- Delete what you no longer use. Old plugins, retired payment integrations and unused user accounts should be removed, not just switched off.
- Treat automated warnings as alarms. Make sure error emails from your systems go to a named person who reads them, not to a shared inbox nobody watches.
- Set limits that stop things automatically. Use daily caps on payouts, refunds, ad spend and transfers so a runaway process hits a ceiling before it hits your bank balance.
Check your business for this control
The free Heist Control Checklist walks through the controls behind every case on this site in about ten minutes. For ready-made policies, the Policy Pack has five editable templates, and the Insider Threat Kit covers risks from inside your own team.
- Guide: How a small business keeps software and devices patched, and why default passwords must go
- What caused the Equifax breach? An unpatched website and an expired certificate
- How the Capital One breach happened: one misconfigured cloud firewall
- How WannaCry hit the NHS: the fix existed 2 months before the attack
- How the Heartland breach happened: 130 million cards and an informant
- How the HSE cyber attack happened: one spreadsheet and 8 weeks of ignored alerts
- How a Raspberry Pi let hackers into NASA's Jet Propulsion Laboratory
- Every patching and monitoring control
- All case files
Facts are drawn from court records, government reports, company statements and reputable reporting, listed below. People are named only where they were convicted, pleaded guilty or spoke publicly in an official role.
- Mondo Visione (SEC press release 2013-222 reprint): SEC charges Knight Capital with violations of market access rule
- WilmerHale: Knight Capital settles Rule 15c3-5 violations with SEC, agrees to pay $12 million
- Prof. Jayanth R. Varma's blog: SEC order explains Knight Capital systems failure
- NBC News: Knight Capital gets $400 million lifeline
- CNN Money: Getco to buy Knight Capital