Zombie Businesses

The Lehman bankruptcy is best known as the event that triggered a financial crisis. For many firms, it also sowed the seeds of an operating crisis.

Revenues plummeted at the end of 2008. Companies retrenched by laying people off. Managers coped with smaller staffs by asking employees to perform multiple jobs and to work longer hours. With remaining employees grateful to have kept their jobs, and with the economy leveling off rather than staying in freefall, corporate profitability rebounded as early as 2009.

Smart business leaders knew this wouldn't last, because you can't run a business for very long by running people into the ground. True, jobs weren't a-plenty and a depressed housing market meant employees weren't going to chase dream jobs. Plus, economic indicators gave no reason to believe things were going to improve any time soon. Still, the employer's risk of losing people who held the business together increased every day that the "new normal" set in. Smart leaders got in front of this.

From early 2009, healthy companies boosted their capital spending.¹ They used their capital in three ways.

The first was defensive. Managers classified as much work activity as "capital improvement" as they could. Doing so meant labor costs could be capitalized for up to 5 years. This let businesses retain people without eroding profitability. This prevented companies from losing employees with systemic knowledge of the business and intimate knowledge of customers.

The second was offensive, investing in core business operations. Companies invested in technology² to lock in those post-layoff productivity gains, and to improve customer self-service offerings since they had fewer employees to service customers. These investments made operations a source of competitiveness by lowering costs, increasing efficiency, and making businesses more responsive to customers. This made these firms better able to compete for market share - essential in a slow-growth world.

The third use was financial restructuring and building reserves. This meant issuing debt, and retiring equity. Debt was cheap, as interest rates hit record lows during the crisis. Debt was also in demand, as market liquidity was making a "flight to quality" and many corporations sported high credit ratings and had large cash balances that could comfortably cover interest payments. Debt-for-equity restructuring lowered the total cost of capital. It also benefited boards and CEOs by concentrating ownership of the company in fewer people's hands.

Smart business leaders responded to the financial crisis not only by protecting operations, but by improving and reforming them, taking advantage of cheap capital during the crisis to pay for it.

But many small businesses, high-risk businesses, and poorly capitalized businesses had neither capital cushions nor creative accounting to protect their operations. All they could do was cut costs and hope for the best. And cut they did. The people who got salary bumps in the boom years from 2006 through 2008 became "high-salary outliers" in 2009. It didn't matter that those were the people at the core of the company's capability and drive. When facing financial ruin, the CFO calls the shots, and the pricey people are the first to go regardless the impact to operations.

These cuts may have staved off bankruptcy, but set the stage for an operating crisis by depleting firms of core operating knowledge and contextual business understanding. Cuts made at the beginning of the crisis left few people - and often no single person - fluent in the details of business operations. Those who remain can mechanically perform different tasks, but don't understand why they do the things they do. The business has continued to run, but it runs on momentum. It doesn't initiate change. It erodes a little bit here and there, as employees exit and clients find the offerings stale and go elsewhere. Costs increase as salaries and stay bonuses are showered on those with the most experience in the mechanics of the business. Pricing power decreases as employees lose the ability to articulate the value of what they provide to their customers. As more time passes, the more acute this crisis becomes: margins get squeezed while the business itself becomes operationally sclerotic.

Just as there are zombie loans (banks keep non-performing loans on their books because they don't want to take the writedown), there are zombie businesses. They transact business and generate revenue. They have years of history and long-standing client relationships. Such a business may look like it can be successful with some investment, but lacking that core operating knowledge, it's a zombie: animated but only semi-sentient.

These firms will only have attracted risk capital late in the post-Lehman investment cycle. Because they haven't been making investments in efficiency or customer service, their first use of fresh capital will be to hire new operational support people in an attempt to get caught up. That's costly and inefficient, and it just adds capacity of people who know how to perform the same mechanical tasks. It won't change the fact that austerity depleted the firm of fundamental operating knowledge. New managers brought in with this investment will struggle to unwind the piled on (and often undocumented) complications of the business, while new people in execution roles will get no further than replay of the mechanical processes for running the business.

Resuscitating these businesses - bringing much needed innovation and structural reform to a firm that has been starved of it for a long time - is a time-consuming and costly proposition. First, nobody has time to spare: they're constantly on fire trying to control operations and contain costs of the fragile machinery of the business. Second, they don't know what to do: because nobody knows the business context very well, there isn't anybody who can competently partner on initiatives to reform the business. Third, middle managers lack the will: the trauma of the cuts and years of thin investment will have rendered decision makers reactive (keep operational flare ups under control) instead of aggressive (reinvent the business and supporting systems). Fourth, those same middle managers can only conceptualize the business as what it has always been; they'll lack the imagination to see what it could be.

Surviving a prolonged downturn is not necessarily the mark of a strong business. As Nassim Taleb pointed out in The Black Swan, a lab rat that survives multiple rounds of experimental treatment (say, exposure to high dosages of radiation) isn't "stronger" than rats that do not. The survivor will be in pretty bad shape for the experience. The heroic story of the tough and resourceful survivor isn't necessarily applicable to the business that survives tough times. The apocryphal story of the zombie is a better fit.

¹ Many businesses reported a significant uptick in capital spending from 2009-2011 compared to 2006-2008.

² Strong corporate IT spending is one reason why the tech sector was counter-cyclical from 2009-2011.

The Return of Financial Engineering - and What it Means for Tech

Soon after the financial crisis began in 2008, companies shifted attention from finance to operations. Finance had fallen out of favour: risk capital dried up, asset prices collapsed and capital sought safe havens like US Treasurys. The crisis also ushered in a period of tepid growth undermined by persistent economic uncertainty. This limited the financial options companies could use to juice returns, so they focused on investing in operations to create efficiencies or fight for market share. In the years following the crisis, the "operations yield" - the return from investing in operations - outstripped the "financial yield" - the return from turning a business into a financial plaything.

Since the start of the crisis, central banks have pumped a lot of money into financial markets, principally by buying up debt issued by governments and financial assets held by banks. This was supposed to spur business activity, particularly lending and investing, by driving down the cost of both debt (lower lending rates) and equity (motivating capital to seek higher returns by pursuing riskier investments).

Whether lending and investing have increased as a result of these policies is debatable. One thing they have done is encourage companies to change their capital structure: low interest rates have made debt cheaper to issue than equity, so companies have been busy selling bonds and buying back their own stock. There are financial benefits to doing this, namely lowering capital costs. It benefits the equity financiers by concentrating ownership of the company in fewer hands. But beyond reducing the hurdle rate for investments by a few basis points - and not very much at that given low interest rates - this doesn't provide any operational benefit.

What's not debatable is that it's brought about a return of financial engineering. CLOs and hybrids are back. Messers Einhorn and Buffet are engineering what amount to ATM withdrawals through preferred share offerings from Apple and Heinz, respectively. The OfficeMax / Office Depot merger is addition through subtraction: projected synergies exceed the combined market cap of the two firms.

When financing yields are higher than operating yields, business operations take a back seat to financial philandering. Consider Dell Computer. Dell's business lines are either in decline (services) or violently competitive (PCs and servers). Does it matter whether Dell is funded with public or private money? Will being a private firm make Michael Dell better able to do anything he cannot do today? It is hard to see how it does. But it does let him play tax arbitrage.

This suggests a bearish environment for investment in operations. The shift to debt in favour of equity, the rise in share buybacks, dividend payouts and M&A suggest that companies have run out of ideas for how to invest in themselves. Cash flow that would otherwise be channeled into the business is betrothed to financiers instead. Debt yields are kept low and roll-overs made easier by stable and consistent cash flows, and equity easier to raise when cash flows from operations are both strong and consistent. This compels executives to set a "steady as she goes" agenda - not an aggressive investment agenda - for business operations.

A reduction in business investment is not particularly good news for tech firms selling hardware, software or services to companies. But it's not all bad news. Rewarding financiers by a starving business for investment makes a company sclerotic. That clears the way for innovators to disrupt and grow quickly. But until those innovators rise up, finance is positioned to stymie - not facilitate - business innovation.

Investing in Software Through Experts, Analysis or Discovery

Whether investing in equities or in software, there are three distinct approaches to how we make an investment decision: based on our intuition or experience, based on our analysis of an opportunity, or based on our ability to rapidly discover and respond to market feedback. Let's look at each in turn.

When we invest based on experience, our decisions are rooted in the expertise we've gained from things we've done before. When based on intuition, our decisions are based on strongly held convictions. An equity investor familiar with retail might recognize demographic changes in specific geographic areas (e.g., migration of families with young children to Florida and Texas) and intuitively invest in firms exposed to those changes (such as commercial construction businesses operating in those areas). In the same way, somebody who has first hand experience can invest in developing a technology solution: the inspiration for Square, Inc. came from a small firm that couldn't close a sale because it couldn't accept credit cards. Although expert investing will likely involve a little bit of analysis, for the most part the investor relies on gut. Because we invest primarily on the courage of our convictions, capital controls on intuitive investments tend not to be very strict. In absolute terms, the capital commitment is generally small, although in relative terms the committed capital may be quite large especially if it is a private placement. As a result, the capital tends to be impatient: trust in an expert lasts only so long; investors will get cold feet quickly if there aren't quick results.

We can invest based on research and analysis. Value investors in equities study company fundamentals looking for firms with share prices that undervalue the assets or the durability of cash flows. In the same way, we can look for value gaps in business operations or market opportunities and identify ways that technology can deliver value to fill those voids. The foundation of the analysis are things such as value-stream mapping, or competitive analyses of solutions developed by sector and non-sector peers. From this, we can produce a financial model and, ultimately, a business case. We need expertise to develop a solution, but by and large we make our investment decision based on the strength of our analysis. In absolute terms, the amount of committed capital can be quite large. But, having rationalized our way to making an investment, the capital controls tend to be strict, and the capital tends to be patient.

Finally, we can invest based on our ability to discover and adjust based on market or user feedback. Traders move in and out of positions, adjusting with changes in the market and hedging based on how the market might change. Over a long period of time, the trader hopes to end up with large total returns even if any given position is held for only a short period of time. We can do something similar with software, using approaches like Continuous Delivery and Lean Startup. In this approach, we aren't just continuously releasing, but rapidly and aggressively acquiring and interpreting feedback on what we've released. We can also use things like A/B testing to hedge investments in specific features. When we invest this way, we do so based not so much on our expertise or analysis, but based on our willingness and ability to explore for opportunities. Capital controls are strict because we have to explain what features we're spending money on and how we're protecting against downside risk of making a mistake. The capital backing a voyage of discovery will be impatient, wanting frequent assurances of positive feedback and results. But at any given time, the amount of committed capital is small, because investors continually evaluate whether to make further investment in the pursuit.

Each of these approaches makes it easy to answer "why" we are making a particular investment. Why should we part with cash for this particular feature? Go ask the expert, or see how it fits in the business case, or go get user feedback on it. "Why" should drive every decision and action made in pursuit of an investment. Without the "why" there is no context for the "what". In its absence, the "what" will suffer.

No approach is universally superior to another. The approach we take has to play to our strengths and capabilities. Either we have people with expertise or we don't. Either we have people with analytic minds and access to data or we don't. Either we have the ability to rapidly deliver and interpret feedback or we don't. The approach we take must also be suitable to the nature of the investment we're making. A voyage of discovery is well suited for developing a product for a new market, but not for an upgrade to a core transactional system. The business case for investing in a customer self-service solution is going to be much more compelling than a business case for developing a product for an emerging market segment.

Just because we take one of these approaches is no guarantee of success. Not all investments are going to pay off: our experts may turn out to have esoteric tastes that aren't appealing to a mass audience. Our thoroughly researched market analysis might very well miss the market entirely. We might deliver lots of features but not find anybody compelled to use them.

Worse still, each of these approaches can be little more than a veneer of competency to unprofessional investing. A hired-in expert may be a charlatan. Many a manager has commissioned a model that inflates benefits to flatter an investment - only for those benefits to never be realized. Just because we can get continuous feedback from users does not mean that we can correctly interpret what that feedback really means.

Most of the time, of course, we take a hybrid approach to how we invest. We supplement expertise with a business case, or we charter an investment with a business case but use Continuous Delivery to get feedback. However we go about it, we need to get the essential elements of the approach right if we're to have any chance of success. Otherwise, we're just unprofessional investors: investing without experience, thoughtful analysis or an ability to respond quickly is reckless use of capital.

Entirely too much software investing fits this bill.

Sector Seven Is Clear

Many years ago, there was a television ad that showed an intruder being chased through a building by two security guards. The guards chase him from room to room, and ultimately down a long hallway. At the mid-point of the hallway, there's a line painted on the floor and wall. On one side of the line is a large number 7, on the other is the number 8. The intruder runs down the hallway, over the line. The two security guards come to a sudden stop right at the line. They watch as the intruder continues to run down the hallway into some other part of the building. After a pause, one of the security guards grabs his walkie-talkie and announces: "sector seven is clear". The intruder is still in the building, but the security guards no longer consider him to be their responsibility.

Every now and again I reference this ad in a meeting or presentation, in the context of Industrial IT. I've been reminded of it again recently.

Industrial IT encourages specialization: It is easier for HR to hire, staff, train and procurement to contract for people in specialist roles. And specialization is comfortable for a lot of people. Managers - particularly managers who have a poor grip on the work being done - like specialization because it is easier to assign and track tasks done by specialists. People in execution roles take comfort in specialization. It's easy to become proficient in a narrow field, such as a specific database or programming language. Given the slow rate of change of any given technology, you don't have to work too hard to remain acceptably proficient at what you do. You only face a threat of obsolescence. A commercial technology with sufficient market share and investment mitigates that risk to the individual.

Specialization means the most critical information - systemic understanding of how a solution functions and behaves from end-to-end - will be concentrated in a few people's heads. This knowledge asymmetry means those few people will be overwhelmed with demands on their time, creating a bottleneck while others on the team are idle. There will be a lot of hand-offs, which increases the risk of rework through misunderstanding. Because no single specialist can see a solution through to completion, nobody will have ownership for the problem. At least, not beyond making sure Sector 7 is clear.

I've written about it many times before, but Industrial IT prioritizes scale over results, specialists over professionals, predictability over innovation, and technology over value. Industrial IT is large but fragile: it struggles to get anything done, there aren't enough heroes to go around, its delivery operations are opaque, and it produces high-maintenance assets.

Even when there is executive commitment to change, it takes a long time and concentrated effort to change the industrial mind-set at a grass roots level.

We have to reframe problems above the task level. Everything we do should be framed as a meaningful business result or outcome, complete with acceptance criteria against which we can verify success in the business context. For example, the problem isn't to fix the payload of a specific webservice, the problem is to allow multiple systems to integrate with each other so that sales transactions can be exchanged. Agile Stories are particularly helpful for defining actions this way, whether new feature or defect. Stories make it possible for each person to explain why something is important, why something is valuable, why they are working on it. Back to our example, I'm fixing this webservice because until I do, there won't be order flow from a principal commercial partner. Stories are also helpful as they let us measure the things we do in terms of results, and not effort.

But there's more to this than process. Each person must feel personal ownership for the success of their actions. The job isn't to code to a specification, or to test against a test case. The job is to create a solution that enables a business outcome. Each person must ask questions about the completeness of the solution, and be motivated to verify them in the most complete manner possible.

Which makes the point that this is, fundamentally, a people challenge. We're asking people to change how they understand the problems they're solving and what "done" means. We're asking them to change their behaviours in how they investigate, test and verify what they do. More broadly, we're asking them to build a contextual understanding for the work they do, and more importantly why they are doing it. And we are asking them to take responsibility for the outcome: I will know this is complete when ...

Do not under-estimate the challenge this represents. The transition from industrial to professional is amorphous. There are false positives: people who sign up for this too quickly don't understand what's going to be required of them. It isn't long before the never ending chorus of "I don't" starts: I don't know how to do something, I don't have that information, I don't know how to find that out. And we can't take anything for granted. We must constantly challenge people's contextual understanding: can they explain, in a business context, why they are working on something, who it benefits, why it is important.

Not everybody will make this transition. For some, because this isn't their calling: not all assembly line workers can become craftsmen. Others will self-select out, preferring the comforts afforded by a specialist's cocoon.

All of these things - changes in process, practice, behaviours and people - require a tremendous amount of intestinal fortitude. The would-be change agent must be prepared for a frustratingly slow rate of change, and to invest copious amounts of time into people to help them develop new context and new muscle memories. On top of it, leaders are in short supply and mentors are even scarcer in Industrial IT shops. Legacy assets and systems (and their legacy of patchwork integration, bandaged maintenance and situational knowledge) will slow the rate at which you can make new hires productive.

The benefits of changing from industrial to professional are obvious. While the destination is attractive, the journey is not - and be under no illusions that it is. But who we work with, how we work, and what we get done in a professional IT business make it worth doing.

Engaging Auxiliary Forces for Strategic Software Solutions (Part II)

I wrote previously that "if one is to compete, one has no choice but to rely on auxiliaries or mercenaries" when you have to respond quickly to a competitive threat. Before looking further at engaging auxiliaries, it's worth considering the "if one is to compete" statement. As unattractive as it may sound, a firm can opt out of competition. Sometimes, the most compelling business option is to run the business for cash. How have late moving legacy firms fared in industries that have undergone strategic shift? E.g., will RIM and Nokia destroy more value than they will create by being late followers of Apple and Android?

Opting out is strategically unattractive. (In fact, it's downright Machiavellian.) But it should never be ruled out. And there isn't much to be said for "putting up the good fight" if you're ill prepared to bring it. It's simply a very public form of corporate seppuku that vapourizes equity and destroys careers.

However, if a firm chooses to compete, and has neither the capability nor the luxury of time to create a capability, it has no choice but to rent that capability by entering into an agreement with an auxiliary or mercenary force. As I pointed out in Part I, this relationship favours the seller.

How can the buyer mitigate the risks? By knowing when and how he or she wants to exit the relationship.

Buyers of auxiliary forces are tempted by what appears to be the best of both worlds: contracting allows the buyer to get an asset developed with minimal effort on behalf of the buyer. But the economics work against the buyer as time passes. The longer a supplier relationship lasts, the greater the dependency of the buyer on the seller, the stronger the seller's negotiating position over time. And development doesn't end with a single act of delivery: there is considerable activity required at the margins, things ranging from data migration to usage analytics. These costs are the buyer's to underwrite. Suppliers are thus able to expand their range of services and derive more cash from the relationship. This increases the costs to the buyer, which erodes the viability of the sourcing strategy.

Anticipating the terms and conditions of the exit are subsequently of prime importance to the buyer.

If the buyer has no alternative but to engage auxiliaries - if, for example, the buyer is purely a marketing company taking a flyer on a technology investment - it faces a long-term relationship with a supplier. The buyer's best bet is to engage auxiliaries as long term equity holders with minority rights in the relationship. This aligns the seller with the buyer and reduces (but most likely will not eliminate) cash bled from the buyer to the seller. By contrast, if the buyer intends to derive significant benefit from the intangible (technology) assets and, by extension, leverage its capability in technology, the buyer must engage auxiliaries for a short period on a fixed income basis, all the while preparing to transition away from the seller to "one's own" forces.

Supplier relationships are economically sticky. Switching from one supplier to another is generally a poor exit strategy. Equity relationships are difficult to unwind amiably (that is, without attorneys). Fixed income relationships come at the cost of the buyer, who will be bled to death pumping cash into multiple suppliers who will not underwrite the cost of a transition.

Thus the onus is on the buyer to make quick but considerate decisions when engaging an auxiliary. In the case of an equity relationship, the buyer must convince the seller to accept a minority equity position, and determine the viability of the investment quickly (reward or wind it up) so that minority position doesn't languish. In the case of a fixed-income relationship, the buyer needs to be able to exit the supplier relationship for a team of "one's own" forces, relegating the supplier to a minimal role at a transitional moment.

But there are often circumstances that muddle a buyer's judgment. With a gullible or desperate supplier, a buyer can prolong a supplier relationship in the hope that an investment will prove viable. By playing labour arbitrage, a buyer can defer the difficult task of building one's own forces. But whether equity or fixed income, the buyer has to remember that the economics of an auxiliary relationship are in decline for the buyer the minute the ink is dry on a contract. When engaging auxiliaries, the buyer must make quick investment decisions and take quick action. The longer a supplier relationship lasts, the more the power in the relationship shifts to the seller.

No matter the nature of the relationship - equity or fixed income - the buyer must not enter into an "arms length" relationship with the seller. The buyer must be engaged with the seller, constantly monitoring and auditing the deliverables over the life of the relationship. A buyer must be capable of competently auditing the work being done by the supplier before entering into a supplier relationship. If he can't, he is not a qualified buyer and must be regarded as a principal source of risk to the capital being invested. Investors in strategic software are wise to challenge the capability of the buyer as well as the seller.

Entering into a supplier relationship buys time for the buyer to build his or her own forces. These forces must be of equivalent or superior capability to those of the supplier. It might not be of equal size - an indigenous team may be smaller in size than a rented team - but it must be of equal capability. An own force inferior to auxiliaries will be manipulated by the auxiliary to the disadvantage of the buyer. With a team of equal capability, the buyer can quickly eclipse the influence of the supplier in the fulfillment chain. A buyer can preserve credibility with a slower velocity from an indigenous team, but not lower quality.

Auxiliaries can be useful to buyers to compensate for a short-term vulnerability, but only if the buyer has an exit strategy. Sellers, not buyers, benefit from long-term supplier relationships for strategic solutions. Buyers must make quick decisions about investment viability, and take competent actions in building the forces necessary to sustain them.

Engaging Auxiliary Forces for Strategic Software Solutions (Part I)

At one point or another, most firms will engage "auxiliary forces" - contract with firms for development teams - to develop software for them. If Machiavelli were sourcing software projects, he wouldn't approve.

"These arms [auxiliaries] may be useful and good in themselves, but for him who calls them in they are always disadvantageous; for losing, one is undone, and winning, one is their captive."

Niccolo Machiavelli, The Prince, Chapter XIII

Machiavelli counsels the prince against the use of auxiliary forces in battle. An auxiliary is not the same thing as a mercenary. A mercenary is a hired gun, loyal only to him or her self. A prince engages an auxiliary when he arranges for another principality to supply forces. The members of an auxiliary force will be loyal to their leader. The risk to the warring prince is that the kingdom with which he has entered into partnership will change terms during or after the battle.

Thus Machiavelli favours the use of one's own forces for important work. It is easier for the prince to align his force's interests with his own interests, and subsequently count on their loyalty and service because everybody stands to gain the same things for the same risks. In the heat of battle, mercenaries are likely to flee, while an auxiliary is likely to seek negotiations that minimize losses. After the battle, the auxiliary is likely to seek negotiations with the prince who invited them into the campaign. (More about this, below.)

Of course, business isn't warfare. But there are some business lessons we can learn just the same.

In software development, auxiliary forces have their place, particularly for utility services sourced for utility solutions. Consider an ERP implementation, complete with code customization. There is a large, diversified sell-side labor market, many alternative sources of that labor, pricing is more formulaic, risks and success criteria are known to some degree of specificity, and the work is not deemed strategically critical (not when everybody has an ERP). This commoditizes the sell-side offering, which favours the buyer.

The sell side has power in utility work only in times of acute labour shortage, which gives the sell side pricing power. But tight supply doesn't make a commodity into a strategic resource; it simply makes it a more expensive commodity. Like any commodity, shortages tend not to last very long: buyers will be priced out of the market (curtailing demand), while suppliers will find ways to create new labour capacity such as training new people (increasing supply). And sellers of services deal in a perishable commodity: time. Only in periods of very high demand will sellers press forward a negotiating advantage borne of tight supply. A billing day lost forever is strong incentive to a seller to do a deal. Whether a buyer engages for a utility service by contracting mercenaries (hired guns under the direction of the buyer) or by hiring an auxiliary (a partnership for delivery of the solution as a whole), the buyer has the upper hand when buying utility services. Or, perhaps, it is more accurate to say that the advantage is the buyer's to lose.

On consideration, Machiavelli wouldn't mind auxiliary forces when they're deployed for utility purposes. It is foolish to distract your best people with non-strategic pursuits.

But he would not approve the use of auxiliaries to achieve strategic solutions. The buyer of a strategic, unique, competitively differentiated solution must put more scrutiny on suppliers, navigate custom pricing, underwrite general and ambiguous risks, and has less certainty of the outcome and the business impact. The buyer will also be weighing intangible advantages in potential sellers, such as business domain or advanced technology knowledge. This makes for a small, opaque and poorly diversified market for any given strategic investment. This favors the seller.

By engaging auxiliary forces, the buyer can get a head start on a strategic solution. But then, it doesn't matter how you start, but how you finish.

A territory conquered by an auxiliary leaves the auxiliary with a strong negotiating position: as the saying goes, possession is nine-tenths of the law. The auxiliary force, in possession of the conquered land, can set terms for turning it over to the prince. This can also be true for strategic software investments. Strategic software doesn't end with a single act of delivery. It will need further investment and development. Their knowledge of the created software - the business domain, the code, the deployment scripts, and so forth - give the auxiliary force a familiarity of "the terrain" that the buyer simply doesn't have. This gives the auxiliary firm an outsized position of power to renegotiate terms with the buyer (e.g., usurious terms for ongoing development and maintenance), or to negotiate with firms who are competitors to the buyer to create a similar solution.

But what about intellectual property and contract laws? Don't they protect the buyer? When the focus shifts to contracts, it means both parties have brought in the lawyers. Lawyers are simply another type of armed force, one that can be both highly destructive and very costly. Lawyering-up simply escalates an arms race between the buyer and seller (or in Machiavelli's parlance, the prince and the auxiliary).

It's no surprise that Machiavelli concluded that auxiliary forces were not worth the risk in strategic undertakings. Anything of strategic importance to the prince must ultimately be done by the prince and his own forces.

This is an easy rule to write, but not an easy rule to live. Building a force (or in software terms, a team) capable of creating a strategic software solution requires world-class knowledge, experience, and facilities. It requires trained personnel and the ability to continue training. It requires the ability to recognize greatness and potential, and the ability to hire both. It requires innovation in tools, techniques and results.

It was Peter the Great's aspiration for Russia to have an ocean going navy. But he didn't start by emptying his treasury on local contractors to build his boats and hire people off the streets to serve as his officers. He studied English ship building and hired German Naval officers to train his officers and enlisted ranks. He invested in developing a capability. He played the long game to allow Russia to become a capable naval power, and not simply a nation with a floating armed force. When time permits, when one has the luxury of playing the long game of disruption, one can own, rather than rent, capability.

Time does not always permit the long game, especially in businesses whose models are being stampeded by software. If the competition has moved first, if one has to respond swiftly, one has no choice but to rely on auxiliaries or mercenaries if one is to continue to compete.

How best, then, for the buyer to engage an auxiliary or mercenary force, given Machiavelli's counsel? We'll look at the terms for this kind of an engagement in Part II.

Patterns for Buying and Selling Software Services

How we contract for software services has a big impact on how successful a software team will be.

There are three common ways of buying development and related services: we can contract for bodies, experts or a solution.

Contracting for Bodies

The simplest way to contract for services is a body shop or staff-augmentation contract. We're developing software in Java, we need Java developers, so we ring up a staff-aug firm and we rent Java developers. Fortune 100 companies rent people by the tens of thousands on staff aug contracts. Staff aug represents a substantial amount of revenue to firms ranging from the largest IT consultancies to local boutique shops. Body shopping is a big business: the margins are thin, but the volumes more than make up for it.

A volume buyer will have Vendor Management Office that consolidates buying requests from across the company, validates the time for they'll be needed (either project based or BAU based), makes the request consistent with position specs they put out to tender (Senior Developer, Lead QA, that sort of thing), and sources a body from the cheapest provider. This gives the scale buyer the appearance of quality control over the bodies they rent. It also eliminates all negotiation: sellers are given a position to fill at a price point; negotiation is simply "take it or leave it" to the seller.

In staff-aug, the seller isn't underwriting delivery risk (that is, of a project), they're only underwriting competency risk of the person they staff. Competency is established relatively quickly, within the first month or so: the staffed body is found to be either acceptable or, at the minimum, not unacceptable. For the seller, although the margins aren't very good, body shop contracts are low touch, it's consistent cash flow, and the easy money is hard for many to resist. Best of all to the seller, delivery risk is completely underwritten by the buyer: the buyer is hiring people on a contract basis to execute the tasks they've determined need to be done. If the project tanks, the buyer has to orchestrate and cover the costs of the rescue, whereas the seller stands to gain through an extension or expansion, and faces no downside risk of a clawback. Terms are often unfavourable to the seller: net 60 isn't uncommon. But cash flows are temporal: once or twice a month, just like payroll. You can set your watch by it. Body shop contracts are like ATMs to sellers.

Contracting for Experts

Another way to contract for services are expert or consulting contracts. A team wants an expert to solve a particular problem, something quite difficult or esoteric: we need a build guy, so we contract for a build guy. The contract may or may not be results based: in general terms, the buyer knows the outcome they want (a reliable, consistent build), but doesn't know exactly what that should look like (a build triggered with every commit, a build pipeline to stage automated test execution) or how to achieve it (developers need to code to keep the build working, not change the build so their code compiles). Execution tends to be highly collaborative: the buyer wants the expert rubbing elbows with the people in the team so that his knowledge gets transferred.

Risk is shared between buyer and seller. If the buyer sees things improve or if the expert gives insight that the team were never going to come up with on their own, they believe they got value for money and they pay. If the buyer doesn't see things improve or if the expert tells them things that they already knew, they don't believe they got value for money and they don't pay. The seller runs the risk that their expertise is sufficient for the situation, that there won't be personality conflicts that undermine their effectiveness, and that the buyer is sophisticated enough to consume the seller's expertise. Thus the risk is shared, and it's up to both parties to get to an acceptable outcome.

Expert consulting is appealing to independent contractors or boutique firms that are essentially collections of branded independent contractors. It is also appealing to large sellers if they believe leasing out an expert will lead to a large body shop or solution contract later.

Buyers pay dearly for experts, which means sellers get fat margins off the work. But sellers charge a premium for a lot of reasons. It is mercenary work to large firms because it doesn't build the seller's business (renting experts generally doesn't scale) and comes at high risk that the expert will quit to join or contract directly with the buying firm. Not everybody is cut out for expert consulting: in addition to needing marketable expertise, a good consultant requires a high EQ and a high degree of situational awareness. Contracts tend to be short duration, but take a long time to define (specify deliverables) and negotiate (intense rate negotiation). They are also high-touch, since the price attracts scrutiny from a lot of people on the buy side. On small contracts (measured in days or weeks), contracts are fixed price and cash flows are end-of-contract. When they're low six figures (such as a large team for a short time), it's money down with the remainder at completion. They're temporal if the engagement lasts for a long time.

Contracting for Solutions

The third form of contract is a project or solution contract. We're implementing SAP, so we need both the core modules and customization done to those modules. We know accounting, but we don't know how to implement SAP, or migrate our data into it, or code in ABAP. So we contract with an implementation and development firm. The ideal firm knows our line of business: e.g., if we're a retailer, we want a solution firm that knows how to configure SAP for retail, knows some really neat shortcuts and tricks, and some obscure 3rd party vendors with some interesting tools targeted at retail. That vertical expertise isn't strictly necessary, but at the least the buyer expects the seller will come to grips with the business domain pretty quickly.

In solution contracts, the sell side underwrites the risk of delivery. Of course, that's true only as far as the contract is concerned: in the event of a project failure, even if the buyer can claw back money from the supplier, the buying business is denied the working software it wanted at the time it wanted it. And the seller will have some people involved in development - people who know legacy systems, and various SMEs. But within the context of the services contract itself, the buyer turns over responsibility for development entirely to the supply firm: project management, user experience, requirements analysis, development, testing and sometimes even deployment and post-implementation services. If the project craters, it is the seller who must orchestrate and cover the costs of the rescue. Whatever marginal amounts the seller can squeeze out of the buyer during the rescue won't be enough to cover the entirety of the downside. Thus the seller underwrites the risk.

To the sell side, the margins are generally good but depend on the seniority of who gets staffed. The temptation is there to juice margins by staffing the delivery team with less experienced people (skill leverage is no different from financial leverage this way). But by the same token, sellers can be talked into "buying the business" - agreeing to enter a solution contract at a lower amount - if they believe that an initial solution sale will lead to subsequent (and significant) revenue. Cash flows can be either lump sums that start with prepayment, or time & material. Very large T&M projects will have a hold-back on each invoice and a rate discount at specified spend thresholds. Sellers can be rewarded for early delivery.

The Right Contract for the Right Situation

Each of these types of contracts have their place. A buyer who is managing and staffing a project team can successfully rent people, especially if the buyer has the means of thoroughly screening and assessing candidates and the willingness to reject people who are not up to scratch. A buyer who recognizes an acute need can contract an expert for a diagnosis and a solution path. Expert engagements are successful when the buyer understands what they're buying, is willing to accept that their self-diagnosis may be wrong and that the expert may steer them into a completely different direction than expected, and has the intestinal fortitude to follow-through. Solution contracts work when the supplier firm has the expertise in end-to-end delivery, a full set of capabilities, and depth of experienced personnel to come to their own rescue if the project ends up in trouble.

Commercial relationships hit the skids when buyer or seller - or both - fail to recognize the appropriate commercial relationship and the nature and responsibility for the risk. A seller who only employs developers - no UX, BAs, PMs, QA, etc. - cannot responsibly enter into a solution contract because they are not contributing enough to the entire solution. A buyer who wants to cherry pick people from multiple suppliers to form a single project team cannot hold any one supplier firm responsible for delivery. A seller supplying experts or bodies cannot compensate for inadequacies of the buyer, whether that's subject matter expertise or technical knowledge.

Sellers have to have enough self awareness to know the types of contracts they can responsibly enter into. Solution firms struggle with bodyshop contracts because the firm's unique value (in what they know or how they work) is neutered in a "rent a body" commodity sale. Solution firms also have low tolerance for long-running expert engagements, as they tie up expertise needed in high-risk solution contracts. Body shop firms tend to have very narrow experts who struggle in consulting engagements. They also lack management expertise or any "greater than the sum of our parts" value to be able to provide a business solution.

Buyers need to understand what's appropriate for them to buy. A buyer with little experience managing software development can't rent bodies because they can't manage them effectively. A buyer with a large, complex captive IT organization that contracts with a solution firm will suffer endless headaches caused by turf battles and disputes over "how to do things" among the buyer's and seller's staff.

Commercial relationships work best when both buyer and seller clearly understand the extent of each party's obligations, the risks that each are assuming, and how they're each mitigating those risks. In the end, a seller may not make a sale, and a buyer has to look for another supplier, because what's good for one is not good for the other. But it's a far, far better thing, over short term and long, that buyer and seller do good business than bad business together.

The Quality Call Option

On time. On scope. On budget. The holy grail of industrial, cost-center IT. Plan your work and work your plan. Be predictable.

As a set of goals for software development, this troika is incomplete because it neglects quality. That is a costly omission. Poor quality will ultimately steamroll the other three project variables: we'll be late, we'll remove features, and we'll spend a lot more money than we want to when we are forced to chase quality. We assume quality at our own risk.

Why do so many otherwise smart people take quality for granted? Mainly because they can't see how easily quality is compromised in development. It's easy to assume that requirements and specifications are sufficiently thorough, the developer just needs to work to the spec. It's easy to assume that we're hiring competent developers who write high quality code, and most buyers don't know what to look for or how to look for technical quality. It's easy to assume our Quality Assurance phase will guarantee solution quality by inspecting it in, instead of agonizing over how to build it in through every action we take in development.

And we tend not to realize that quality is in the eye of the beholder. When we have long feedback cycles - when it could be days, weeks or even months before one person's work is validated by another - the person responsible for creation is de facto the person responsible for validation. When "on time / on budget / on scope" is the order of the day, nobody is going to cast a critical eye over their own work.

It takes a while before we find out that one person's "done" is another person's "work-in-progress". The moment when a quality problem makes itself known is the moment when our implicit assumptions about quality start to become expensive.

There are a few simple things managers can do to give quality the same prominence as time, cost and scope. The first is to reduce the number of hand-offs in the team, and to strengthen those hand-offs.

Every hand-off injects incremental risk of misunderstanding. The greater the number of hand-offs, the greater the risk that we will have integration problems or understand the problem differently or solve the wrong problem. Development teams have a lot of specialists, both role (business analyst, developer, user experience designer, quality assurance analyst) and technical (mobile client developer, server-side Java developer, web client developer, etc.) Our ideal team is staffed by generalists who can each create a whole business solution. If we don't have people with generalist skills today, we must invest in developing those capabilities. There's no way around this. And there's no reason to accept technical specialization: we don't need to have mobile client developers as well as services developers and database developers. We can invest in people to become mobile solution developers who can write client and server code. Generalization versus specialization is our choice to make.

It takes time to professionalize a development team that has too many industrial characteristics. Until we've got a team of generalists, we have to make do with our specialists. As long as we do, we have to improve the fidelity of our hand-offs.

Rather than performing technical tasks in isolation, we can have specialists collaborate on delivering functional requirements. Have developers - even if they are fluent in different technologies - actively work together on completing an entire functional solution. That's easy to write but difficult to do. We must expect that we'll need to facilitate that collaborative behavior, particularly in the early going: collaborating on delivery of a functional solution is not natural behavior for people accustomed to going flat out to complete code by themselves.

We can increase collaboration across roles, too. We can have business analysts and developers perform a requirement walk-through before coding starts. We can have a business analyst or QA analyst desk check software before a developer promotes it as "development complete". We can insist on frequent build and deployment, and iterative QA. All of these shorten our feedback cycles, and gives us an independent validation of quality nearer to the time of creation.

This type of collaboration might seem matter-of-fact, but it doesn't come naturally to people in software development. A manager has to champion it, coach people in it, witness it first hand to assess how effectively people are performing it, and follow through to make sure that it takes place on its own.

Stronger hand-offs will slow down coding. The act of creating code will take longer when we insist that BAs do walkthroughs with developers, that developers collaborate on business solutions, and that QA desk-checks before code is promoted. But the velocity at which a team can code is not the same as the velocity at which a team can deliver software at an acceptable level of quality. It's easy to cut code. It's not easy to cut code that is functionally and technically acceptable. That means the former is a lousy proxy for the latter, and our management decisions are wildly misinformed if we focus on "dev complete".

Knowing the velocity at which a team can deliver a quality product gives us far more useful managerial information. For one thing, we have a much better idea for how long it's likely going to take to deliver our desired scope to an acceptable level of quality. This makes us less vulnerable to being blindsided by an overrun caused by lurking quality problems. We also have a better idea for the impact we can expect by adding or removing people. A team focused just on delivering code might deliver more code in less time if it has more people. But quality problems tend to grow exponentially. When quality is not a consideration in project planning, we tend to forget the dichotomy that a larger team can need more time to complete software of acceptable quality than a smaller team.

Another thing we can do is make our quality statistics highly visible. But visibility of the data isn't enough; we have to visibly act on that data. It's easy enough to produce information radiators that profile the state of our technical and functional quality. But an information radiator that portrays a decaying technical quality picture will have a negative effect on the team: it will communicate that management doesn't really care. Something we're prepared to broadcast is something that we have to be prepared to act on. When we expose quality problems we have to insist on remediation, regardless if that comes at the cost of new requirements development. That puts us face-to-face with the "quality call option": developing more bad stuff is less valuable than developing less good stuff. That might seem obvious, but it doesn't fit into an "on time / on scope / on budget" world. It takes courage to exercise the quality call option.

Finally, we have to look not just at project status data, but at flow data. Projects throw off copious amounts of flow data - trends over time - that most managers ignore. These are a rich source of information because they indicate where there are collaboration - and therefore quality - problems. For example, we can easily track a build-up in the number of defects that developers have tagged as ready for retest, but are not retested in a timely fashion by QA. We can look for a pattern of defects raised that are dispositioned as "not a defect". We can monitor defects that are reopened after retest. And we can look at the overall ratio of defects to test activity as an indicator of fundamental asset quality. The patterns of change in this data over time will indicate performance problems that undermine quality.

As people who understand how to deliver software, we have to insist on quality being on an equal footing with cost, scope and time. Getting that footing is one thing, keeping it is another. We need to measure both asset quality and team performance in terms of results of acceptable quality. We need to scrutinize our project data for threats to quality. We need to foster individual behaviors that will reduce self-inflicted quality problems. And we have the courage to broadcast our state of quality and act decisively - to exercise the quality call option - to defend it.

Strategic Software Investing: Chase the Ends, Not the Means

When a company engages in custom software development, it is making an investment into itself. Company executives decide to convert capital into software because they believe that the resulting software will make the business better.

How we make that investment decision is crucial to the success of the investment.

Our first investment decision is based solely on an idea. We spend some money to research an idea that looks promising to see if there is a business case for making further commitment. That business case comes down to the expected impact from the features we want at an expected cost by an expected time. We also want to understand the investment strategy: How will it be developed? By whom? Where will they be working? How well do they know these technologies? How well do they understand our business domain? What is the release schedule? How will we assess quality of the asset? We also want a risk analysis of both our business case and investment strategy, so that we understand the many different forms of risk there are to the capital the firm is committing.

This gives us enough information to make our second (and usually larger) investment decision: either we have an opportunity that meets our investment criteria that can be fulfilled at reasonable risk, or we don't.

Even if we secure financing and approval, sign the contracts, and get on with development, that's not the end of our investment decision-making. The investment rationale needs to be revisited continuously over the course of the investment. Once we begin executing on an investment, our objective is to maintain its viability. If reality turns out to be significantly different from the strategy (people quit, scope turns out to be larger than anticipated, business needs change), we either go back to our investment committee to ask for an adjustment to the investment (more money, rescope functionality, recast the expected business impact), or we have to recommend that they scuttle the investment.

We do this with any financial investment we make. Before we invest in a company, we do our research into the industry, business, and management team. We scrutinize quarterly reports for results and indicators. We withdraw our investment if we are not confident that management can turn it around. This is classic investing behaviour, and it is applicable to strategic software investments.

IT projects tend to lack investment rigour. Managers beholden to an annual budget cycle will concoct wildly optimistic cost forecasts based on flimsy details in the hope of getting a project funded in the annual budget lottery. Instead of using project inception as an investment qualifying activity, it's used strictly as a way to refine scope, time and cost estimates as a precursor to development. And once development is under way, the management mantra is to be "on time / on budget / on scope"; management will do whatever they can to bend reality to match the project plan to give the appearance of control.

What we end up with is budget money chasing any means of development that pledges it can fit under the spending cap, rather than corporate capital chasing a business outcome. Development isn't invited into the process of shaping the investment parameters, they're expected to live with them. That leads to compromise in team construction (availability over capability), vendor partners (low bid), and standards of performance (showing progress is more important than verifying completeness).

When money is chasing the means and not the ends, quality is the first casualty, followed quickly by scope. Costs will rise to meet budget, at which point a project sponsor is forced either to dial up more money or dial down expectations.

Software is an investment, not a cost. Good investment outcomes require good investment rigour.

Trust, but Verify

“If a builder builds a house for a man and does not make its construction firm, and the house which he has built collapses and causes the death of the owner of the house, that builder shall be put to death.”

Clearly, the Babylonians understood that the builder will always know more about the risks than the client, and can hide fragilities and improve his profitability by cutting corners—in, say, the foundation. The builder can also fool the inspector (or the regulator). The person hiding risk has a large informational advantage over the one looking for it.

-- Nassim Taleb and George Martin, How to Prevent Other Financial Crises, SAIS Review, Winter-Spring 2012

The interests of buyer and builder of any asset are fundamentally divergent. The builder, who has to keep workers engaged and cash flowing into the business, has a shorter horizon than the buyer, who enjoys the fruits of the asset for a long period of time. The buyer knows only whether the asset appears to do what it is expected to do after it is delivered, whereas the builder has more knowledge of the care that has gone into the asset: the quality of raw materials and the thoroughness of the workmanship. By necessity, there is a fundamental trust relationship between buyer and seller. The Babylonians understood the criticality of this trust relationship to society and thus set a very high expectation that it would not be violated.

In software development, information is highly asymmetric between buyer and seller. The buyer knows a particular business such as selling toys or flying people from place to place, while the seller knows how to develop and deliver software. A buyer knows whether software is functionally complete (can I transact business using this software?) but most buyers are not experts at software or software development. Few know the questions to ask relative to non-functional requirements, or how to validate that the answers to those questions are satisfactory. How does a buyer know that a solution is being developed so that it is highly secure from attack? or will perform reliably and satisfactorily? or will scale? Buyers assume the asset will have these characteristics. Yet buyers tend to find out whether they do only after the fact, and it is the buyer who is stuck with the outcome: delays due to unforeseen technical work, or higher than expected hardware costs. Most buyers of custom software are underwriting risks which they do not fully appreciate.

There is not an equivalent in software development to the ancient Roman practice of bridge engineers spending nights sleeping under their construction. The closest thing we have are commercial clawbacks and the threat of long and painful slogs to rescue a distressed investment. Clawbacks can be painful for a selling firm, but unless both parties contractually agreed a definition of a possible "rescue" state and set as a term that the seller is responsible for all costs incurred during rescue, the full force of the clawback is easily blunted through negotiation. Rescue operations are painful for the individuals involved in them, but are not not often led or staffed by the people who created the mess in the first place. Thus the builders responsible for the slipshod quality rarely feel the full force of the consequences.

Therefore, Mr. Kay sensibly suggests that active investors should have concentrated portfolios. This encourages long-term thinking, and detailed research. Such a concentrated investor will be an active steward of the company, making life difficult for the management if it errs. In short, such managers may behave like "owners" of the stocks in the portfolio.

-- John Authers, writing in the FT

If we can't get builders more closely aligned with buyers, perhaps we can get buyers more closely aligned with builders. Investing in custom software is active by nature. There are no passive IT investments, no index funds or ETFs through which we can invest in software. Investing in custom software is a business of picking winners. The things that Mr. Authers and Mr. Kay point out about financial investing are applicable to IT investing. We benefit when buyers or buyer's agents (e.g., IT portfolio managers) behave as "active stewards" of the investments they make.

To be effective stewards, a buyer must do their research on the things they are investing in. Part of that research is knowing the domain of software and software development: about technologies the team is using, about the level of confidence behind progress reports (can we validate results or can we merely validate effort has been expended?), about the fullness of the solution (what specific NFRs matter to us and how specifically are we validating that they are satisfied), and above all about the people responsible for delivery (do we have people with sufficient capability on the team). Where the buyer may not have the knowledge, they can assemble a board to oversee their investments, consisting of vested stakeholders and independent directors with expertise in areas where the buyer's knowledge and experience are lacking.

This puts the buyer closer to the builder while the asset is being produced. Not in the weeds of technical decisions, but in a position to set and reaffirm expectations with the team, and to validate that results are consistent with those expectations. The buyer trusts, but verifies.

As Mr. Authers points out, such a buyer will "[make] life difficult for the management if it errs." Better for an errant management to be corrected while there is value in an investment to the buyer, not after that value has been lost by the builder.

Resiliency, Not Predictability

A couple of months back, I wrote that it is important to shorten the results cycle - the time in which teams accomplish business-meaningful things - before shortening the reporting cycle - the time in which we report progress. Doing the opposite generates more heat than light: it increases the magnification with which we scrutinize effort while doing nothing to improve the frequency or fidelity of results.

But both the results cycle and reporting cycle are important to resiliency in software delivery.

A lot of things in business are managed for predictability. Predictable cash flows lower our financing costs. Predictable operations free the executive team to act as transformational as opposed to transactional leaders. Predictability builds confidence that our managers know what they're doing.

The emphasis on predictability hasn't paid off too well for IT. If the Standish (and similar) IT project success rate numbers are anything to go by, IT is at best predictable at underperforming.

When we set out to produce software, we are exposed to significant internal risks: that our designs are functionally accurate, that our tasks definitions are fully defined, that our estimates are informationally complete, that we have the necessary skills and expertise within the team to develop the software, and so forth. We are also subject to external risks. These include micro forces such as access to knowledge of upstream and downstream systems, and macro forces such as technology changes that obsolete our investments (e.g., long-range desktop platform investments make little sense when the user population shifts to mobile), and labor market pressures that compel people to quit.

We can't prevent these risks from becoming real. Estimates are informed guesses and will always be information deficient. Two similarly skilled people will solve technical problems in vastly different ways owing to differences in their experience. We have to negotiate availability of experts outside our team. People change jobs all the time.

Any one of these can impair our ability to deliver. More than one of these can cause our project to crater. Unfortunately, we're left to self-insure against these risks, to limit their impact and make the project whole should they occur. We can't self-insure through predictability: because these risks are unpredictable, we cannot be prepared for each and every eventuality. The pursuit of predictability is a denial of these risks. We need to be resilient to risks, not predictable in the face of them.

This brings us back to the subject of result and reporting cycles: the shorter they are, the more resilient we are to internal and external risks.

Anchoring execution in results makes us significantly less vulnerable to overstating progress. Not completely immune, of course: even with a short results cycle we will discover new scope, which may mean we have more work to do than we previously thought. But scope is an outcome, and outcomes are transparent and negotiable. By comparison, effort-based execution is not: "we're not done coding despite being at 125% of projected effort" might be a factual statement, but it is opaque and not negotiable.

In addition, a short results cycle makes a short reporting cycle more information rich. That, in turn, makes for more effective management.

But to be resilient, we need to look beyond delivery execution and management. A steady diet of reliable data about the software we're developing and how the delivery of that software is progressing allow a steering committee to continuously and fluidly perform its governance obligations. Those obligations are to set expectations, invest the team with the authority to act, and validate results.

When project and asset data are based in results and not effort, it is much easier for a steering committee to fulfill its duty of validation. And it helps us out with our other two obligations as well. We can scrutinize which parts of the business case are taking greater investment than we originally thought, and whether they are still prudent to pursue as we are making them. We also see if we are taking technical shortcuts in the pursuit of results and can assess the long term ramifications of those shortcuts nearer the time they are made. We are therefore forewarned as to whether an investment is in jeopardy long before financial costs and technical debt rise, and change or amplify our expectations of the team as we need to. This, in turn, gives us information to act on the last governance responsibility, which we do by changing team structure, composition and even leadership, and working with the investment committee to maintain viability of the investment itself.

A short results cycle, reporting cycle and governance cycle make any single investment more resilient. They also enable a short investment cycle, which makes our entire portfolio more robust. From an execution perspective, we can more easily move in and out of projects. Supported by good investment flow (new and existing opportunities, continuously reassessed for timeliness and impact), hedging (to alleviate risks of exposure to a handful of "positions" or investments), and continuous governance assessing - and correcting - investment performance, we can make constant adjustments across our entire portfolio. This makes us resilient not only at an individual investment level, but at a portfolio level, to micro and macro risks alike.

IT has historically ignored, abstracted or discounted risks to delivery. Resiliency makes us immune to this starry-eyed optimism which is at the core of IT's chronic underperformance. That makes IT a better business partner.

Rules versus Results

Assessments are based, not on whether the decisions made are any good, but on whether they were made in accordance with what is deemed to be an appropriate process. We assume, not only that good procedure will give rise to good outcome, but also that the ability to articulate the procedure is the key to good outcomes.

-- John Kay, writing in the Financial Times

A common error in both the management and governance of IT is an over-reliance on rules, process and "best practices" that portend to give us a means for modeling and controlling delivery. We see this in different ways.

• Project managers construct elaborately detailed project plans that show business needs decomposed into tasks which are to be performed by specialists in a highly orchestrated effort. And detailed they are, often down to the specific task that will be performed by the specific "resource" (never "person", it's always "resource") on a specific day. This forms a script for delivery. The tech investor without a tech background cannot effectively challenge a model like this, and might even take great comfort in the level of detail and specificity. They have this broken down into precise detail; they must really know what they're doing.
• Companies invest in "process improvement" initiatives, with a focus on establishing a common methodology. The belief is that if we have a methodology - if we write unit tests and have our requirements written up as Stories and if we have a continuous build - we'll get better results. Methodology becomes an implicit guarantor of success. If we become Agile, we'll get more done with smaller teams.
• "Best practices" are held out as the basis of IT governance. Libraries of explicit "best practices" prescriptively define how we should organize and separate responsibilities, manage data centers, contract with vendors and suppliers, and construct solutions. If there is a widely accepted codification of "best practices", and we can show that we're in compliance with those practices, then there's nothing else to be done: you can't get better then "best". We'll get people certified in ITIL - then we know we'll be compliant with best practices.

* * *

To see how stultifying such behaviour can be, imagine the application of this emphasis on process over outcome in fields other than politics or business. Suppose we were to insist that Wayne Rooney explain his movements on the field; ask Mozart to justify his choice of key, or Van Gogh to explain his selection of colours. We would end up with very articulate footballers, composers and painters, and very bad football, music and art.

-- John Kay

It is tempting to try to derive universal cause-and-effect truths from the performance of specific teams. Because this team writes unit tests, they have higher code quality. Or, Because we ask users to review security policies and procedures at the time their network credentials expire, we have fewer security issues. Does unit testing lead to higher quality? Does our insistence on policy result in tighter security? It may be that we have highly skilled developers who are collaborative by nature, writing relatively simple code that is not subject to much change. It may be that our network has never been the target of a cyber attack. A "rule" that dictates that we write unit tests or that we flog people with security policy will be of variable impact depending on the context.

There are no such things as best practices. There are such things as practices that teams have found useful within their own contexts.

@mikeroberts

Suppose a team has low automated unit test coverage but high quality, while another team has high automated unit test coverage but low quality. A rule for high unit test coverage is no longer indicative of outcome. The good idea of writing unit tests is compromised by examples that contradict lofty expectations for going to the trouble for writing them.

It isn't just the rules that are compromised: rule makers and rule enforcers are as well. When a team compliant with rules posts a disappointing result, or a team ignorant of rules outperforms expectation, rule makers and rule enforcers are left to make ceteris peribus arguments of an unprovable counterfactual: had we not been following the rule that we always write unit tests, quality would have been much lower. Maybe it would. Maybe it would not.

Rules are not a solid foundation for management or governance, particularly in technology. For one thing, they lag innovation, rather than lead it: prescriptive IT security guidelines circa 2007 were ignorant of the risks posed by social media. Since technology is a business of invention and innovation, rules will always be out of date.

For another, rules create the appearance of control while actually subverting it. Rules appear to be explicit done-or-not-done statements. But rules shift the burden of proof of compliance to the regulator (who must show the regulated isn't following the rule), and away from the regulated (who gets to choose the most favourable interpretation of the rules which apply). Tax law which explicitly defines how income or sales transactions are to be taxed encourage the individual to seek out the most favourable tax treatment: e.g., people within easy driving distance of a lower cost sales tax jurisdiction will go there to make large purchases. The individual's tax obligation is minimized, but at a cost to society which is starved for public revenues. In IT terms, the regulated (that is, the individual responsible for getting something done) holds the power to determine whether a task is completed or a governance box can be ticked. I was told to complete this technical task; I have completed it to a point where nobody can tell me I am not done with it. The individual effort is minimized, but at a cost to the greater good of the software under development which is starved for attention. It is up to the regulator (e.g., the business as the consumer of a deliverable, or a steering committee member as a regulator) to prove that it is not. 

When the totality of the completed technical tasks does not produce the functionality we wanted, or the checboxes are ticked but governance fails to recognize an existential problem, our results fall short of our expectations despite the fact that we are in full compliance with the rules.



* * *

Instead, we claim to believe that there is an objective method by which all right thinking people would, with sufficient diligence and intelligence, arrive at a good answer to any complex problem. But there is no such method.

-- John Kay

Just because rules-based mechanisms are futile does not mean that all our decisions - project plans, delivery process, governance - are best made ad hoc.

Clearly, not every IT situation is unique, and we can learn and apply across contexts. But we do ourselves a disservice when we hold out those lessons to be dogma. Better that we recognize that rigorous execution and good discipline are good lifestyle decisions, which lead to good hygiene. Good hygiene prevents bad outcomes more than it enables good ones. Not smoking is one way to prevent lung cancer. Not smoking is not, however, the sole determinant of good respiratory health.

Principles give us an important intermediary between prescriptive rules and flying by the seat of our pants. Principles tend to be less verbose than rules, which makes them more accessible to the people subject to them. They reinforce behaviours which reinforce good lifestyle decisions. And although there is more leeway in interpretation of principles versus rules, it is the regulator, not the regulated, who has the power to interpret results. Compliance with a principle is a question of intent, which is determined by the regulator. If our tax law is a principle that "everybody must pay their fair share", it is the regulator who decides what constitutes "fair share" and does so in a broader, societal context. Similarly, a business person determines whether or not software satisfies acceptance criteria, while a steering committee member assesses competency of people in delivery roles. Management and governance are not wantonly misled by a developer deciding that they have completed a task, or an auditor confirming that VMO staffing criteria are met.

"We always write unit tests" is a rule. "We have appropriate test coverage" is a principle. "Appropriate" is in the eye of the beholder. Coming to some conclusion of what is "appropriate" requires context of the problem at hand. Do we run appreciable risks by not having unit test coverage? Are we gilding the lily by piling on unit test after unit test?

We are better served if we manage and govern by result, not rule. Compliance with a rule is at best an indicator; it is not a determinant. Ultimately, we want people producing outstanding results on the field, not to be dogmatic in how they go about it. Rules, processes, and "best practices" - whether in the form of an explicit task order that tells people what they are to do day in and day out or a collection of habits that people follow - do not compensate for a fundamental lack of capability.

Adjudication of principles requires a higher degree of capability and situational awareness by everybody in a team. But then we would expect the team with the best players to triumph over a team with adequate players in an adequate system.

Shorten the Results Cycle, not the Reporting Cycle

A big software development project collapses just weeks before it is expected to be released to QA. According to the project leadership team, they're dogged by integration problems and as a result, the software is nowhere close to being done. It's going to take far more time and far more money than expected, but until these integration problems get sorted out, it isn't clear how much more time and how much more money.

The executive responsible for it is going to get directly involved. He has a background in development and he knows many of the people and suppliers on the team, but he doesn't really know what they're doing or how they're going about doing it.

The team situation is complicated. There are the consultants from a product company, consultants from two other outsourcing firms, several independent contractors, plus a few of our own employees. QA is outsourced to a specialist firm, and used as a shared service. All these people are spread out across the globe, and even where people are in the same city they may work out on different floors, or in different buildings, or simply work from home. Teams are organized by technology (e.g., services developers) or activity (analysts, developers, QA). Project status data is fragmented: we have a project plan, organized by use cases we want to assemble and release for staged QA. We have the developer tasks that we're tracking. We have the QA scripts that need to be written and executed. We have test data that we need to source for both developers and QA. And we have a defect list. Lots of people, lots of places, lots of work, lots of tracking, but not a lot to show for it.

The executive's first action will be to ask each sub-team to provide more status reports more frequently, and report on the finest details of what people are doing and how long it will be before they're done. New task tracking, daily (perhaps twice daily) status updates, weekly status updates to stakeholders to show progress.

* * *

The common reaction to every failure in financial markets has been to demand more disclosure and greater transparency. And, viewed in the abstract, who could dispute the merits of disclosure and transparency? You can never have too much information.

But you can.

So wrote John Kay in the Financial Times recently. His words are applicable to IT as well.

Gathering more data more frequently about an existing system merely serves to tell us what we already know. A lot of people are coding, but nothing is getting done because there are many dependencies in the code and people are working on inter-dependent parts at different times. A lot of use cases are waiting to be tested but lack data to test them. A lot of functional tests have been executed but they're neither passed nor failed because the people executing the tests have questions about them. The defect backlog is steadily growing in all directions, reflecting problems with the software, with the environments, with the data, with the requirements, or just mysterious outcomes nobody has the time to fully research. When a project collapses, it isn't because of a project data problem: all of these things are in plain sight.

If getting more data more frequently isn't going to do any good, why do the arriving rescuers always ask for it? Because they hope that the breakthrough lies with adjusting and fine tuning of the existing team and organization. Changing an existing system is a lot less work - not to mention a lot more politically palatable and a lot less risky - than overhauling a system.

* * *

There are costs to providing information, which is why these obligations have proved unpopular with companies. There are also costs entailed in processing information – even if the only processing you undertake is to recognise that you want to discard it.

More reporting more often adds burden to our project managers, who must now spend more time in meetings and cobbling together more reports. Instead of having the people close to the situation look for ways to make things better, the people close to the situation are generating reports in the hope that people removed from the situation will make it better. It yields no constructive insight into the problems at hand. It simply reinforces the obvious and leads to management edicts that we need to "reduce the number of defects" and "get more tests to pass."

This reduces line leaders into messengers between the team (status) and the executive (demands). As decision making authority becomes concentrated in fewer hands, project leadership relies less on feedback than brute force.

* * *

[M]ost of the rustles in the undergrowth were the wind rather than the meat.

Excessive data can lead to misguided action, false optimizations and unintended outcomes. Suppose the executive bangs the drum about too many defects being unresolved for too long a period of time. The easiest thing for people to do isn't to try to fix the defects, but to deflect responsibility for them, which they can do by reassigning those in their queue to somebody else. Some years ago, I was asked by a client to assess a distressed project that among other things had over 1,000 critical and high priority defects. It came as no surprise to learn that every last one of them was assigned to a person outside the core project team. The public hand wringing about defects resulted in behaviours that deferred, rather than accelerated, things getting fixed.

* * *

The underlying profitability of most financial activities can be judged only over a complete business cycle – or longer. The damage done by presenting spurious profit figures, derived by marking assets to delusionary market values or computed from hypothetical model-based valuations, has been literally incalculable.

Traditional IT project plans are models. Unfortunately, we put tremendous faith in our models. Models are limited, and frequently flawed. Financial institutions placed faith in Value at Risk models, which intentionally excluded low probability but high impact events, to their (and to the world's) detriment. Our IT models are similarly limited. Most IT project plans don't actively include the impact analysis of reasonably probable things like the loss of key people, of mistakes in requirements, or business priority changes.

In traditional IT, work is separated into different phases of activity: everything gets analyzed, then everything gets coded, then everything gets tested, then it gets deployed. And only then do we find out if everything worked or not. It takes us a long, long time - and no small amount of effort - to get any kind of results across the finish line. That, in turn, increases the likelihood of disaster. Because effort makes a poor proxy for results, interim progress reports are the equivalent of marking to model. Asking project managers for more status data more frequently burns the candle from the wrong end: reducing the project status data cycle does nothing if we don't shorten the results cycle.

* * *

It is time companies and their investors got together to identify information [...] relevant to their joint needs.

We put tremendous faith in our project plans. Conventional thinking is that if every resource on this project performs their tasks, the project will be delivered on time. The going-in assumption of a rescue is that we have a deficiency in execution, not in organization. But if we are faced with the need to rescue a troubled project, we must see things through the lens of results and not effort. Nobody sets out to buy effort from business analysts, programmers and QA analysts. They do set out to buy software that requires participation by those people. This is ultimately how any investment is measured. This and this alone - not effort, not tasks - must be the coin of the realm.

As projects fail for systemic more than execution reasons, project rescues call for systemic change. In the case of rescuing a traditionally managed IT project, this means reorganizing away from skill silos into teams concentrating on delivering business-specific needs, working on small business requirements as opposed to technology tasks, triggering a build with each code commit and immediately deploying it to an environment where it is tested. If we do these things, we don't need loads of data or hyper-frequent updates to tell us whether we're getting stuff done or not. Day in and day out, either we get new software with new features that we can run, or we don't.

It is irrational to organize a software business such that it optimizes effort but produces results infrequently. Sadly, this happens in IT all the time. If we believe we can't deliver meaningful software in a short time frame - measured in days - we've incorrectly defined the problem space. Anything longer than that is a leap of faith. All the status reports in the world won't make it anything but.

When the Unstoppable Force of Growth Meets the Immovable Object of Control

You create a new software application. It grows, rapidly. And it keeps growing. You add tools. You add people. You add roles and structure. You split the codebase into different technical components. You divide teams. You add environments. You make rules for merging and deploying.

One day, you look round and realize you have 20 times the staff but deliver only a fraction of what you used to when you had only a handful of people. Demand is still growing, but you can't keep up.

On top of it, everybody is quarreling.

There are the old-timers, the people who were there at the beginning, who know the code backwards & forwards, who were there for the early triumphs. They're still the people who get called in when a deployment is in trouble, when there's a mystery problem, when it's already well past the 11th hour. Which, of course, means they're called in for every deployment. They want to carry on with the "trust me" ways. "Trust me" when I tell you that I'll get it done, just leave me to it. "Trust me" that if I have to, I'll go to heroic lengths to make my deadline. "Trust me" that we'll pull it off - we always have. This is how we've always done it. This is what works.

Then there are the people hired in the past year to run QA and create a PMO. They want control. "I want estimates, and task orders, and a project plan. I want specifications and approvals. I want to maximize the time programmers spend programming and testers spend testing. I want the cheapest resources I can get. I want test scripts. I want documentation. I want process." This is how we did it at my last firm. This is what works.

Then it happens again. Another botched deployment. Several days of downtime. Angry customers. Day after day of crisis calls and coordinated recovery efforts. Too much drama makes management nervous. We can't go on like this. This doesn't work.

But even management is divided. Some of the managers have been around since the early days: We operate in a fast & furious environment, this comes with the territory. Some of the managers are new: You can't run a business of this size by the seat of your pants. The rhetoric heat up and escalates. Neither side convinces the other. Disagreements become arguments become accusations. "Cowboys". "Control freaks". Impasse. Stalemate. Nothing changes.

The bickering continues, until the moment when the decision is made for everybody. Another deployment failure. This one in spectacular style: very public, and very severe, and very embarrassing, and very bad.

Time for new leadership. Call a search firm. Hire some hotshot from a firm like the one we want to be, one that grew and scaled. Give him a mandate. Have him report directly to the President.

The hired-in management hopes this new leader will bring deliverance from the chaos. The old-timers hope that "how he did it at his last firm" is "how we've always done it here".

Whatever the case, it is now entirely out of their hands.

* * *

I've seen this same pattern in a lot of growth business, both captive tech and tech start-ups: its market is still growing, but operations have become sclerotic and performance erratic.

By the time it gets to this point - lots of people, little getting done, low confidence, open bickering - the overriding mission has already started to change from innovating in the business through technology to The Website Must Not Go Down.

This mission change is driven from the top. Leadership feels the pain of operational failure more acutely as they come to see the business as an established competitor rather than a plucky start-up. Whether a start-up with investor money, or a captive IT project that has prominence within a large corporate, leaders are held to the expectation that operations will be predictable. This is how they are measured, so this is how they will manage.

Caution rules the day. We deploy less often and with more ceremony. We err on the side of not making a mistake. The fear of a service interruption causes organizational seizure. The price of innovation is subconsciously deemed too high.

We didn't used to be like this.

* * *

Let's dissect the situation.

On the plus side, you still have a core of people who are fluent in the code because they were among the principal authors. You've lost some key people over the years, but you still have people with in-depth knowledge of the whats and the whys of the code. And, the hero culture means they are personally invested in success: this is their code, this is personal. You also have hired in new people - a lot of new people - some of whom can become fluent in the product, the customers, and the business over time.

Most everybody will have a job on the line, and many senior people are still "close to code". There won't be much time for luxury positions, people in jobs off the line focused on things like process and group standards.

Strange as it may seem, another plus is that you operate in a fast-paced business environment. This is counter-intuitive: the environment seems to be the source of the problem. But it is your greatest asset. The critical success criteria are not costs but growth. Riding that growth will depend on innovation and time-to-market more than predictability and control.

Then there are the minuses.

You are beholden to the knowledge of the people who form your core group, and that knowledge exists exclusively in their heads. All those people you added to scale your capacity have given you a bloated team; if costs aren't a concern now they will be soon. Worse, you're not getting value from those new hires. Many are net negative contributors. With a bit of time and attention, some can become net positive, but the rest - maybe 30%, maybe 80% - are just plain bad hires. This happens when people are hired for capacity as opposed to capability.

You've added new roles, new structure, and new formality, in an attempt to gain control. That's given you a more complex business. It also creates mission confusion: as much as people are trying their level best they're adding as much confusion and delay as clarity and certainty because the structure is at odds with results.

Your core team of "heroes" have had a lot of freedom and independence for a long time. They will generally resist any change you try to make that they perceive will curtail that freedom.

People may be close to code, but if you've split the code into technical silos, your people will be pretty far removed from how and why your customers use the software.

Many of these minuses are by-products of the responses to growth: hire more people, add structure, divide and conquer the problem. But the fundamental hero culture that is resistant to any change is a hold-over from the early, free-wheeling days.

If the business is still growing, the heroes should have a case for remaining aggressive and getting stuff done. But priorities change when business leaders think they've got something - market share, outside investors - to lose. And the credibility of the hero culture erodes with every production snafu, every minute of unscheduled downtime, every unforced error.

* * *

The cowboy approach puts stability at risk. Control will stifle growth. So what can you do? It seems you are forced to choose between "responsibly stable" and "recklessly aggressive".

You are not. You must unwind and rebuild.

Fundamentally, things in this organization still get done in an ad-hoc way. Layers of control, scale, and structure have been grafted onto this. They are a mismatch. We know this because all those people and processes are shunted aside when stuff absolutely needs to get done, when a new release absolutely needs to get deployed.

Unwind

Here are things we can do to unwind these layers.

Furlough the net negative people. Having net negative contributors isn't good for anybody: not for the business, not for the net positive contributors, not for the people who are net negative. Frustration and disappointment are lousy characteristic of operations and careers.

Institute a policy of "do no harm". Introduce greater rigor in how you work - in how you analyze, code, build and test. Publicly expose quality statistics. Every new line of code, every new requirement, every new test, should be performed to a higher level of simplicity, clarity, ease of maintenance. Agile is pretty good for this.

Practices aren't enough. You need to instill value systems and social norms that reinforce them. Agile is pretty good for this, too. If you haven't done so in a while, re-read the Agile manifesto. It offers a core set of values. And a policy of "do no harm" is a good starting point for a value system.

These things gives us a foundation of excellence. They reduce the noise and distractions, and makes quality central to, not a potential byproduct of, what we do.

Rebuild

The unwinding work strips out the misfit layers and lets us improve our fundamentals. But that isn't enough. We also have some rebuilding to do.

Organize people by how your customers work with you, not how the technology works together. It doesn't make sense to organize a business orthogonally to the way it makes money, by product, or by customer-activity model.

Hire in new people who are not only smart and talented but specifically have the completion and collaboration genes in their DNA. Then, pair. Then rotate pairs. Then pair across discipline - BAs with Developers, Developers with QAs, UX with QAs. Do not underestimate how difficult it will be for people to pair. People will quit as a result of this. But this is a highly effective way to unwind knowledge silos. Unit tests are helpful, Stories with acceptance criteria are helpful, but nothing reduces proprietary knowledge as effectively as people actively working together on a specific solution.

Advance the state of your practice as far down the path of Continuous Delivery as you can. Make many small deliveries (Agile Stories are fantastic for enabling this). Commit code frequently. Build continuously. Automate everything. Deploy many times a day.

Results

This will leave you more resilient to downside risks (less susceptibility to catastrophic events) and able to capitalize on upside opportunities (quickly satisfy needs). Stability and innovation are no longer trade-offs.

You can start to rebuild while you are still unwinding. Just know why you are doing what you are doing. Reorganizing without behaviour change is just going to add layers and confusion. Similarly, introducing better practices will make your organization less bad, but won't make it world class.

Do not under-estimate the work and commitment this requires This is a major overhaul. This requires changing people and changing mind-sets, a lot of communication and reassurance, a suspension of disbelief that this will work, and tenacity during the trough of despair when it looks like things are getting worse, not better.

Most of all, remember that the most critical acts of leadership are the repeated commitments not to a vision, but to the people responsible for executing it. They'll make mistakes. They'll make plenty of mistakes. Make it safe to make mistakes. Do not hold people to an expectation of perfection, but that they act, iterate, and learn.