I consult, write, and speak on running better technology businesses (tech firms and IT captives) and the things that make it possible: good governance behaviors (activist investing in IT), what matters most (results, not effort), how we organize (restructure from the technologically abstract to the business concrete), how we execute and manage (replacing industrial with professional), how we plan (debunking the myth of control), and how we pay the bills (capital-intensive financing and budgeting in an agile world). I am increasingly interested in robustness over optimization.

I work for ThoughtWorks, the global leader in software delivery and consulting.

Wednesday, March 31, 2021

The Ever Given is Not a Black Swan Event

By transporting a container more cheaply than any other vessel afloat, [Emma Maersk] and her six sister ships were expected to stimulate even faster growth in international trade, lowering the cost of moving goods through the supply chains that had reshaped the global economy…

An extremely large container ship - the Ever Given - has been in the news this past week for running aground and blocking traffic through the Suez Canal. This has caused considerable disruption to global supply chains. Some have incorrectly referred to the incident and the aftermath as a “black swan” event.

The disruption is not simply the result of a ship running aground and blocking traffic in a busy, narrow passageway. The disruption is a function of the scale of the container ships themselves. In the mid-2000s, with global trade growing by leaps and bounds every year, shipping companies opted for larger and larger vessels. The largest of these ships each carry the equivalent of over 10,000 truckloads of goods. With trade growing, the value proposition of ever-larger ships was self evident in 2007: “On a per-container basis, a larger vessel cost less to build and operate than a smaller one, allowing the owner to undercut competitors’ cargo rates and still earn a healthy profit.”

Such scale led to optimization among tightly-coupled participants in the global supply chain. “Operating on regular schedules - such that an identical vessel departs Shanghai every Wednesday, stops in Singapore nine days later and arrives in Antwerp five weeks hence, with tight connections to barges and freight trains - intermodal container transport gave manufacturers and retailers the confidence to plan tightly organized long-distance supply chains.”

But as Nassim Taleb pointed out well over a decade ago in his book The Black Swan, “Globalization [of financial markets] creates interlocking fragility, while reducing volatility and giving the appearance of stability. In other words it creates devastating Black Swans.”

And so it has happened with container shipping.

Though supremely efficient at sea, Emma and the even larger ships that followed in her wake became a nightmare. By making freight transportation slower and less reliable than it had been decades earlier, they helped to stifle the globalization of manufacturing…

Local optimization - in this case, minimizing the cost of transporting a container of goods over the sea - distorted the economies of scale of global shipping. Shippers, facing excessive capacity after the financial crisis of 2008, either went out of business, merged, or did deals to fill their ships to capacity. Optimization of the oceangoing containership fleet came at the cost of the efficiency of the rest of the supply chain into which it was integrated.

Discharging and reloading the vessel took longer as well, and not only because there were more boxes to put off and on. The new ships were much wider than their predecessors, so each of the giant shoreside cranes needed to reach a greater distance before picking up an inbound container and bringing it to the wharf, adding seconds to the average time required to move each box. Thousands more boxes multiplied by more handling time per box could add hours, or even days, to the average port call. Delays were legion.
Freight railroads staggered under the heavy flow of boxes into and out of the ports. Where once an entire shipload of imports might be on its way to inland destination within a day, now it could take two or three. Queues of diesel-belching trucks lined up at terminal gates, drivers unable to collect their loads because the ship lines had too few chassis on which to haul the arriving containers. And often enough, the partners in one of the four alliances that came to dominate ocean shipping didn’t use the same terminal in a particular port, requiring expensive truck trips just to transfer boxes from an inbound ship at one terminal to an outbound ship at another.

But the disruption to supply chains created by the beaching of the Ever Given is not a black swan event. First, the Ever Given was involved in an incident in 2019 in which it was caught in high winds while operating at slow speed and collided with a pleasure ferry in the Elbe River in Hamburg, Germany. The same conditions have been cited as contributing factors to the vessel’s recent beaching in the Suez, and it is fair to say those conditions are not exactly the stuff of “one hundred year events”. Second, the supply chain bedlam triggered by the Ever Given’s beaching exposed the asymmetric downside risk endemic to the systemic fragility - that is, the woefully inadequate robustness - in tightly-coupled supply chains. However, this asymmetric risk - as evidenced by the previous two paragraphs - was hidden in plain sight.

An incident is not a “black swan event” simply because some people could not fathom the possibility of it occurring. The triggering event had happened before. The systemic fragility was as plain as day. In fact, the article I’ve quoted throughout this post describing that fragility - titled, appropriately enough, The Megaships That Broke Global Trade - was not published in the wake of the Ever Given’s beaching: it appeared last October in the Wall Street Journal. This is not a black swan event. This was a fragile system operating on borrowed time.

I kept a copy of the October WSJ article intending to use it as a metaphor for large software development processes designed to optimize labor unit costs, specifically developer labor unit costs. The pattern is the same, the risks are the same, the consequences are the same. But with hindsight the article reads much better as a chronicle of one-way risk: a tightly-coupled system with highly concentrated activity in the value stream that, in turn, is locally optimized to a single variable is a powder keg that can be detonated by the most mundane of sparks.

As written, the October WSJ article about the Emma Maersk is a story of misplaced local optimization creating systemic inefficiency and undesirable outcomes. Ironically, the author laments the factors that do not “flatter the bottom line” resulting from local optimization of the oceangoing container fleet: the need for “keeping more inventory, shipping via multiple routings and producing in multiple factories rather than in giant sole-source plants”. While these measures certainly increase costs, they also make manufacturing and distribution far more robust (that is, less fragile). The Ever Given incident points out the benefits of doing so. Cheap insurance policies against one-way downside risk such as multiple facilities and alternative shipping routes are a far preferable price to pay than to have factors outside of your control - a beached container vessel in the Suez - make a mockery of fragile optimization.

While the Emma Maersk article is a compelling story of systemic distortions resulting from local optimization, it is really the story of a critical yet fragile system waiting for a simple, pedestrian event to realize asymmetric downside risk.