We knew what didn’t work in software development a generation ago. They’re still common practices today.

In the not too distant past, software development was notorious for taking too long, costing too much, and having a high rate of cancellation. Companies spent months doing up-front analysis and design before a line of code was written. The entire solution was coded before being deemed ready to test. Users only got a glimpse of the software shortly before the go-live date. No surprise that there were a lot of surprises, all of them unpleasant: a shadow “component integration testing” phase, an extraordinarily lengthy testing phase, and extensive changes to overcome user rejection. By making the activities of software development continuous rather than discrete, collaborative rather than linear, and automated rather than repeated meant more transparency, less time wasted, and less time taken to create useful solutions through code.

It was pretty obvious back then what didn’t work, but it wasn’t so obvious what would. Gradually people throughout the industry figured out what worked less bad and, eventually, what worked better. Just like it says in the first line of the Agile manifesto. But I do sometimes wonder how much of it really stuck when I see software delivery that looks a lot like it did in the bad old days. Such as when:

• "Big up front design" is replaced with … "big up front design": pre-development phases that rival in duration and cost those of yesteryear.
• It takes years before we get useful software. The promise during the pitch presentation was that there would be frequent releases of valuable software… but flip down a few slides and you’ll see that only happens once the foundation is built. It’s duplicitous to talk about “frequent releases” when it takes a long time and 8 figures of spend to get the foundation - sorry, the platform - built.
• We have just as little transparency as we did in the waterfall days. Back then, we knew how much we’d spent but not what we actually had to show for it: requirements were “complete” but inadequate for coding; software was coded but it didn’t integrate; code was complete but was laden with defects; QA was complete but the software wasn’t usable. The project tasks might have been done, but the work was not. Today, when work is defined as tasks assigned to multiple developers we have the same problem, because the tasks might very well be complete but "done" - satisfying a requirement - takes more than the sum of the effort to complete each task. Just as back then, we have work granularity mismatches, shadow phases, deferred testing, and rework cycles. Once again, we know how much we’re spending, but have no visibility into the efficacy of that spend.
• The team is continuously learning … what they were reasonably expected to have known before they started down this path in the first place. Specifically, that what the team is incrementally "learning" is only new to people in the development team. Significant code refactoring that results from the team "discovering" something we already knew is a non-value-generative rework cycle.
• We have labor-intensive late stage testing despite there being hundreds and hundreds of automated tests, because tests are constructed poorly (e.g., functional tests being passed off as unit tests), bloated, flaky, inert, disabled, and / or ignored. Rather than fixing the problems with the tests and the code, we push QA to the right and staff more people.
• Deadlines are important but not taken seriously. During the waterfall days, projects slipped deadlines all the time, and management kept shoveling more money at it. Product thinking has turned software development into a perpetual investment, and development teams into a utility service. Utility service priorities are self-referential to the service they provide, not the people who consume the service. The standard of timeliness is therefore "you’ll get it when you get it."
• Responsibility isn’t shared but orphaned. Vague role definitions, a dearth of domain knowledge, unclear decision rights, a preference for sourcing capacity (people hours) and specialist labor mean process doesn’t empower people as much as process gives people plausible deniability when something bad happens.

Companies and teams will traffic in process as a way to signal something, or several somethings: because of our process we deliver value, our quality is very high, our team is cost effective, we get product to market quickly. If in practice they do the things we learned specifically not to do a generation ago, they are none of the things they claim to be.

The era of ultra low interest rates is over. Tech has painful adjustments to make.

Interest rates have been climbing for two years now. The Wall Street Journal ran an article yesterday with the headline the days of ultra low interest rates are over. Tech will have to adjust. It’s going to be painful.

When capital is expensive, we measure investments against the hurdle rate: the rate of return an investment must satisfy to exceed to be a demonstrably good use of capital. When capital is ridiculously cheap, we no longer measure investment success against the hurdle rate. In practice, cheap capital makes financial returns no more and no less valuable than other forms of gain.

There are ramifications to this. As fiduciary measures lapse, so does investment performance. We go in pursuit of non-financial goals like "customer engagement rate". We get negligent in expenditure: payrolls bloated with tech employees, vendors stuffing contracts with junior staff. We get lax in our standard of excellence as employees are aggressively promoted without requisite experience. We get sloppy in execution: delivery as a function of time is simply not a thing, because the business is going to get whatever software we get done when we get it done.

Capital may not be 22% Jimmy Carter era expensive, but it ain’t cheap right now. Tech has to earn its keep. That means a return to once familiar practices, as well as change that orchestrates purge of tech largesse. Business cases with financial returns first, non-financial returns second. Contraction of labor spend: restructuring to offload the overpromoted, and consolidation of roles or lower compensation for specialization. Transparency of what we will deliver when for what cost, and what the mitigation is should we not. An end to vanity tech investments, because the income statement, much less the balance sheet, can no longer support them.

Some areas of the tech economy will be immune to this for as long as they are thematically relevant. AI and GenAI are TINA (there is no alternative) investments: a lot of firms have no choice but to spend on exploratory investments in AI, because Wall Street rewards imagination and will reward the remotest indication of successful conversion of that imagination that much more. Yet despite revolutionary implications, AI enthusiasm is tempered compared to frothy valuations for tech pursuits of previous generations, a function of investor preference for, as James Mackintosh put it, profits over moonshots.. Similarly, businesses where there is a tech arms race on because innovation offers competitive advantage, such as in-car software, it will be business as usual. But these arms races will end, so it will be tech business as usual until it isn’t. (In fact, in North America, this specific arms race may not materialize for a long, long time as EV demand has plateaued, but that’s another blog for another day.)

Tech has had the luxury of not being economically anchored for a long time now. If interest rates settle around 400 bps as the WSJ speculated yesterday, those days are over. The adjustment to a new reality will be long and painful because there’s a generation of people in tech who have not been exposed to economic constraints.

This is the Agile Manager blog, as it has been since I started it in 2006. Good news, this change doesn’t mean a return to the failed policies of waterfall. Agile had figured out how to cope with these economic conditions. Tech may not remember how to use those Agile tools, but it has them in the toolkit. Somewhere.

That said, I also blog about economics and tech. If the Fed funds rate lands in the 400 bps range, tech is in for still more difficult adjustments. More specifically, the longer tech clings to hopes for a return to ultralow interest rates, the longer the adjustment will last, and the more painful it will be.

The ultralow rate party is over. It’s long past time for tech to sober up.

Patterns of Poor Governance

As I mentioned last month, many years ago I was toying around with a governance maturity model. Hold your groans, please. Turns out there are such things. I’m sure they’re valuable. I’m equally sure we don’t need another. But as I wrote last month there seemed to be something in my scribbles. Over time, I’ve come to recognize it not as maturity, but more as different patterns of bad governance.

The worst case is wanton neglect, where people function without any governance whatsoever. The organizational priority is on results (the what) rather than the means (the how). This condition can exist for a number of reasons: because management assumes competency and integrity of employees and contractors; because results are exceedingly good and management does not wish to question them; because management does not know the first thing to look for. Bad things aren’t guaranteed to happen in the absence of governance, but very bad things can indeed (Spygate at McLaren F1; rogue traders at Société Générale and UBS). Worse still, the absence of governance opens the door to moral hazard, where individuals gain from risk borne by others. We see this in IT when a manager receives quid pro quo - anything from a conference pass to a promise of future employment - from a vendor for having signed or influenced the signing of a contract.

Wanton neglect may not be entirely a function of a lack of will, of course: turning a blind eye equals complicity in bad actions when the prevailing culture is “don’t get caught.”

Distinct from wanton neglect is misplaced faith in models, be they plans or rules or guidelines. While the presence of things like plans and guidelines may communicate expectations, they offer no guarantee that reality is consistent with those guidelines. By way of example, IT managers across all industries have a terrible habit of reporting performance consistent with plans: the “everything is green for months until suddenly it’s a very deep shade of red” phenomenon. Governance in the form of guidelines is often treated as “recommendations” rather than “expectations” (e.g., “we didn’t do it that way because it seemed like too much work”). A colleague of mine, on reading the previous post in this series, offered up that there is a well established definition of data governance (DAMA). Yes there is. The point is that governance is both a noun and a verb; governance “as defined” and “as practiced” are not guaranteed to be the same thing. Pointing to a model and pointing to the implementation of that model in situ are entirely different things. The key defining characteristic here is that governance goes little beyond having a model communicating expectations for how things get done.

Still another pattern of bad governance is governance theater, where there are governance models and people engaged in oversight, but those people do not know how to effectively interrogate what is actually taking place. In governance theater, some governing body convenes and either has the wool pulled over their eyes or simply lacks the will to thoroughly investigate. In regulated industries, we see this when regulators lack the will to investigate despite strong evidence that something is amiss (Madoff). In corporate governance, this happens when a board relies almost exclusively on data supplied by management (Hollinger International). In technology, we see this when a “steering committee” fails to obtain data of its own or lacks the experience to ask pertinent questions of management. Governance theater opens the door to regulatory capture, where the regulated (those subject to governance) dictate the terms and conditions of regulation to the regulators. When governance is co-opted, governance is at best a false positive that controls are exercised effectively.

I’m sure there are more patterns of bad governance, and even these patterns can be further decomposed, but these cover the most common cases of bad governance I’ve seen.

Back to the question of governance “maturity”: while there is an implied maturity to these - no controls, aspirational controls, pretend controls - the point is NOT to suggest that there is a progression: i.e., aspirational controls are not a precursor to pretend controls. The point is to identify the characteristics of governance as practiced to get some indication of the path to good governance. Where there is governance theater, the gap is a reform of existing institutions and practices. Misplaced faith requires creation of institutions and practices, entirely new muscle memories for the organization. Each represents a different class of problem.

The actions required to get into a state of good governance are not, however, an indication of the degree of resistance to change. Headstrong management may put up a lot of resistance to reform of existing institutions, while inexperienced management may welcome creation of governance institutions as filling a leadership void. Just because the governance gap is wide does not inherently mean the resistance to change will be as well.

If you’re serious about governance and you’re aware it’s lacking as practiced today, it is useful to know where you’re starting from and what needs to be done. If you do go down that path, always remember that it’s a lot easier for everybody in an organization - from the most senior executive management to the most junior member of the rank and file - to reject governance reform than to come face to face with how bad things might actually be.

Governance Without Benefit

I’ve been writing about IT governance for many years now. At the time I started writing about governance, the subject did not attract much attention in IT, particularly in software development. This was a bit surprising given the poor track record of software delivery: year after year the Standish CHAOS reports drew attention to the fact that the majority of IT software development investments wildly exceeded spend estimates, fell short of functional expectations, were plagued with poor quality, and as a result quite a lot of them were canceled outright. Drawing attention to such poor results gave a boost to the Agile community who were pursuing better engineering and better management practices. Each is clearly important to improving software delivery outcomes, but neither addresses contextual or existential factors to investments in software. To wit: somebody has to hold management accountable for keeping delivery and operations performing within investment parameters and, if it is not, either fix the performance with or without that management or negotiate a change in parameters with investors. Governance, not engineering or management, is what addresses this class of problem.

If IT governance was a fringe activity twenty years ago, it is everywhere today: we have API governance and data governance and AI governance and on and on. Thing is, there is no agreement as to what governance is. Depending on who you ask, governance is “the practice” of defining policies, or it “helps ensure” things are built as expected, or it “promotes” availability, quality and security of things built, or it is the actual management of availability, quality and security. None of these definitions are correct, though. Governance is not just policy definition. Terms like “promote” and “helps ensure” are weasel words that imply “governance” is not a function held accountable for outcomes. And governance intrinsically cannot be management because governance is a set of actions with concomitant accountability that are specifically independent of management.

That governance is still largely a sideline activity in IT is no surprise. For years, ITIL was the go-to standard for IT governance. ITIL defines consistent, repeatable processes rooted in “best practices”. The net effect is that ITIL defines governance as “compliance”. As long as IT staff follow ITIL consistent processes, IT can’t be blamed for any outcome that resulted from its activity: they were, after all, following established “best practices.” As there is not a natural path from self-referential CYA function to essential organizational competency, it is unrealistic to expect that IT governance would have found one by now.

I’ve long preferred applying the definition of corporate governance to IT governance. Corporate governance boils down to three activities: set expectations, hire managers to pursue those expectations, and verify results. When expectations aren’t met, management is called to task by the board and obliged to fix things. If expectations aren’t met for a long period of time, the managers hired to deliver them have to go or the expectations have to go. And if expectations aren’t met after that, the board goes. Before it gets to anything so drastic, governance has that third obligation, to “verify results.” Good governance sources data independently of management by looking directly at artifacts and constructing analyses on that data. In this way, good governance has early warning as to whether expectations are in jeopardy or not, and can assess management’s performance independently of management’s self-reporting. Governance is not “defining policies” or “helping to ensure” outcomes; governance is actively involved in scrutinizing and steering and has the authority to act on what it has learned.

Governance is concerned with two questions: are we getting value for money, and are we receiving things in accordance with expectations. Multiple APIs that do the same thing, duplicative data sources that don’t reconcile, IT investments that steamroll their business cases, all make a mockery of IT governance. We’ve got more IT “governance” than we’ve ever had, yet all too often it just doesn’t do what it’s supposed to do.

I’m picking up the topic of IT governance again because it does not appear to me that the state of IT governance is materially better than it was two decades ago, and this deserves attention. Soon after I started down this path, I thought it would be helpful to have a governance “maturity model.” No, the world does not need another maturity model, let alone one for an activity that is largely invisible and only conspicuous when it fails or simply isn’t present. It doesn’t help that good governance does not guarantee a better outcome, nor that poor governance does not guarantee a bad outcome. Governance is a little too abstract, difficult to describe in simple and concrete terms, and subsequently difficult for people to wrap their heads around. That, in turn, renders any “maturity model” an academic exercise at best.

Still, there is room for something that characterizes all this governance on an IT estate and frames it as an agent for good or bad. That is, in the as practiced state, is governance of this activity (say, API or appdev) materially reducing or increasing exposure to a bad outcome. That’s a start.

* * *

Dear readers,

I took extended leave from work last year, and decided to also take a break from writing the blog. I’m back.

Also, I do want to apologize that I’ve been unable all of these years to get this site to support https. It’s supposed to be a simple toggle in the Google admin panel to enable https, but for whatever reason it has never worked, which I suspect has to do with the migration of the blog from Blogger into Google. Despite admittedly tepid efforts on my part, I've not found a human who can sort this out at Google. I appreciate your tolerance.

What Does God Need With a Starship?

Andy Kessler wrote in the WSJ this month about the value of being a contrarian. Contrarians have a reputation for being cynics or curmudgeons because they’re out of step with mainstream thinking. And it’s true that being contrarian solely for the purpose of resisting or denying change is generally not helpful. But contrarian thinking can bring a lot of constructive insight.

For quite a few years now, I’ve written about the value of activist investing, which at its best challenges institutional thinking - and, when necessary, institutional reporting - for the benefit of those invested in the business outcome. Activist investors are contrarian thinkers. An effective activist investor sources their own ground truths, creates their own hypotheses from the data, and advocates for those alternative hypotheses. This is true for public company investors and captive IT investors alike. The activist investor in a public company visits company installations, talks to customers, analyzes the footnotes of SEC filings, and develops hypotheses that management may not see or may be choosing not to report. The activist investor in a captive IT investment does the same things: interviews members of the team, reviews code, and analyzes status reports to develop alternative interpretations about the actual progress of and threats to a program or product. The formula is the same: scrutinize the data you’re provided, get some of your own, recontextualize it, and draw your own conclusions. This is critical thinking technique as we were all taught in high school.

But every silver lining has a cloud. The activist isn’t always right. David Einhorn raised questions derived from ground truths and got it right about Allied Capital, St. Joe Company, and Lehman Brothers, but he got it wrong about Keurig. The value of Mr. Einhorn’s contrarian thinking wasn’t in its accuracy as much as it was in offering a fact-based challenge to management’s narrative. The activist articulates a narrative that reframes a situation and draws attention to a risk or deficiency that others don’t see or that management may be obfuscating. The thought exercise is helpful for all investors to re-evaluate what they believe their risk exposure in that specific position to really be.

And it’s important to note that, like anything else, investor activism can be a charade. The public company investor may simply be generating doubt about a company to bolster a short position that can be quickly liquidated. Similarly, the captive IT steering committee member who is also a vendor rep may simply be fostering fear, uncertainty and doubt to drive more services revenue from an existing customer.

Perhaps most important of all, the activist investor isn’t very popular. Contrarian thinking takes us out of our comfort zone, makes us consider difficult possibilities, forces us to have data to support the thing that we desperately want to be true, and reminds us that we’re not as smart as we want to believe that we are. But more banally, challenging the board and management meeting-in and meeting-out wears on people. Contrarian thinkers are irritating. 'tis best that you enjoy dining alone.

Being a contrarian is not the easiest path to take. John Kay once wrote that regulators, if they are not to be co-opted by the regulated, require "...both an abrasive personality and considerable intellectual curiosity to do the job." Contrarian thinking at its best.

Because sometimes, when everybody is too mesmerized or beat down or overwhelmed, or simply can't be bothered, for the sake of everybody concerned, somebody has to ask: “what does God need with a starship?”

What Can You Do With Less?

In the heady days of the dot-com boom, it became common for investors to challenge would-be entrepreneurs with the question, “what can you do with more?” To the aspirant (and typically struggling) startup, the question was profound, if for no other reason than they were buried in the realities of trying to keep their own narrow universe from imploding on itself.

The question is still asked today, just in different ways. For example, the Wall Street Journal recently profiled the relationship between Masayoshi Son of Softbank and Adam Newmann of WeWork, specifically how Mr. Son prodded Mr. Newmann to pursue ever more ambitious goals in 2018. That ambition culminated in a goal for WeWork to grow revenue from $2b to over $350b in 5 years (making it larger than Apple), a mooted valuation of $10t (making it equal to about 1/3rd the total market valuation of all US equities), and a pitch for $70b in financing. It was certainly more; although, what actually followed was certainly less.

While the question remains the same, the question behind the question is not. A quarter of a century ago, it wasn’t clear what tech companies would succeed, and what success would look like, if in fact any would succeed at all. Spending more in the pursuit of success was a way to have less dependency on serendipitous technology and market phenomenon in the pursuit of what everybody knew was the future. Owning more of the value chain, spending more on marketing and awareness campaigns, signing up more partners, and so forth were ways to project influence over the things out of one company’s direct control. From a finance perspective, this was small beer: the price to try everything and fail wasn’t a whole lot more than the price to try a few things and fail.

Today, if every business must be a digital business, the question of success or even survival is not what is being put to the test. Instead, it is a test of one’s of ambition: if the future of [insert your industry name here] is digital, the question isn’t whether [insert your digital strategy here] has the potential to succeed or not, but whether it will become the dominant digital path in its industry or just an also ran that becomes a footnote in history. You must think bigger than a digital strategy: what are you going to do to impose your vision of the future on the commercial and non-commercial ecosystems relevant to your future? “What could you do with more?”

In rapidly growing markets, there is some wisdom in this. Industry lifecycles are characterized by relatively short periods of rapid growth pursued by hundreds of equity financed competitors that are followed by long periods of slow growth dominated by a handful of oligopolistic market participants sucking cash flow from operations to service debt, finance buybacks and pay dividends. An overwhelming majority of the small competitors that exist during the rapid growth phase won’t survive and the small ones that do won’t matter much. As Larry Ellison pointed out years ago, “The No. 1 software company in every segment makes all the money. We never buy anything where it doesn‟t put us in the No. 1 position or get us in such a strong No. 2 position that we think we can get to No. 1 very quickly.” When a clear market opportunity emerges, the stakes are very high indeed.

Of course, not all tech is about potential for world domination. Sometimes it is about utility. Utilities - think electricity, water, and the like - are taxes on a business. As we saw a couple of months ago, one of the overriding questions that dogs utility tech investments is, “do we have to do it now?” Another is, “what could you do with less?”

This latter question can be responded to as an appeal to value more than to cost. Even within the most mundane of utility tech opportunities are innovations, sometimes small, but innovations nonetheless that are legitimate sources of value. While many (if not most) utility tech investments will never have a comprehensive value proposition, often they can be unpacked so that the utility investment can lead with value realization. Decoupling legacy technologies through APIs and abstraction layers allows for creativity not just in how utility tech is delivered, but how fundamental business problems can be solved. When successful, a utility tech investment can be reframed from a single all-in commitment to a series of investment tranches that deliver both near-term value and long-term utility. When we do this type of analysis, we very often find there is quite a lot that can be done with less.

This draws a great deal of ire, of course. Enterprise IT doesn’t much like requesting a little more funding for the same initiative year after year after year, much less the specter of a long-lived hybrid tech landscape resulting from only partial modernization. Tech vendors prefer large commitments from their customers before they will offer discounts or commit top people. And, this appears to legitimize the lack of confidence that corporate capital allocators have in an IT function’s competency.

But when the relationship between enterprise IT and the rest of the business is characterized by low trust - still all too common to this day - it behooves IT to meet the trust deficit head on. Doing so demonstrates good stewardship of capital and provides transparency into why and how IT spends that capital. It also makes utility tech spend far less self-referential (I still see “we’re moving to the cloud because the cloud is better” as a business case justification) and far more aligned with business goals. And leading with value while asking for tranches of “less” is not an acknowledgement that IT isn’t trustworthy as much as it insists on a high-trust partnership with business and IT on achieving the outcomes. IT doesn’t get a blank check to do tech things, but then neither does the business get anything it might ever want. Both are equal partners in shared outcomes, and partnerships cannot function without trust.

“What can we do with less?” is a good question to ask. Because sometimes, less really is more.

Armchair Regulator

Bernie Madoff, one of the greatest financial fraudsters of all time, died earlier this month. That his was possibly the largest Ponzi scheme ever created a sense of history unfolding in front of our eyes as it was exposed in 2008. That his fraud was exposed at the height of the greatest economic crisis since 1929 gave the financial crisis a name and a face that the legions of bankers, politicians, Treasury and Fed officials could not. That this was a fraud hidden in plain sight for years, probably decades, that had duped so many cemented the popular impression that nobody had their hands on the wheel of the financial economy; it was a runaway train that spelled doom for its willing and unwilling passengers alike.

My interest in the Madoff fraud has always had little to do with the fraud itself or the context of the times, and more to do with the failure of regulators to detect and investigate Madoff’s activities. The regulatory lapses would be comical if their effects weren’t so tragic.

Through the eyes of regulatory practice, the Madoff fraud is a story of what regulators did not do. The SEC received a half-dozen detailed allegations about Madoff’s business over a 16 year span, allegations that the SEC either dismissed without investigation, or investigated in a half-hearted fashion. In light of what was ultimately learned, the latter are egregious lapses in regulatory conduct. SEC officials accepted implausible statements from Madoff as fact. They failed to perform the most rudimentary of independent research. To wit: a single call to the DTCC would have revealed Madoff hadn’t executed a trade in years.

From a regulatory perspective, the primary conclusions have been (a) shame on the SEC and (b) that regulators are not looking for fraud while it is happening, only describing it in detail once it has been exposed.

Yet there are deeper, personal, questions to reflect on.

The most popular - and the most useless - of these questions is whether or not you would have been a willing investor. The “would you have invested” question is useless in no small part because it implies that the victims are complicit in the perpetuation of the fraud by their own blind faith borne entirely of greed. And it is true that for decades, Madoff’s returns were outrageous and inexplicable by all standards, yet he persisted year after year after year, attracting new investors in his ever-growing Ponzi scheme. But if invited, coerced, hounded, and even shamed by people in your trust network, would you have put money in? Of course you would have. Very few individual investors have the sophistication and skepticism, let alone the cuts and bruises and calluses, of an institutional investor. Add the marketing that Madoff employed - personal relationships, exclusivity, secrecy, and the twisted implication that because the SEC had investigated his firm no less than six times and found nothing it was quite clearly legitimate - and the individual with capital to invest didn’t have much of a chance. As the 19th century sage PT Barnum pointed out, the coefficient of suckers to birth rates has always been nearly 1.0. Very few have special immunity.

The better questions have to do with how you would have behaved as a regulator. The popular posture toward the SEC is one of disappointment, that they failed the very investors they are paid to protect, that the institution itself is corrupted by regulatory capture. Yet had you been a SEC investigator, would you have recognized the impossibility of Madoff’s returns when presented with little more than other people’s theories? Would you have been diligent with inquiries after the boss told you to drop it? Would you even have been just a little bit more hygienically thorough?

Answer those questions in the context of business-as-usual at any regulatory agency. How understaffed is the SEC? How many plausible allegations of fraud is the SEC presented with each month? And the allegations they receive, are they truly the product of independent researchers seeking to right wrongs, or are they motivated by some nefarious intent such as professional jealousy? How does the SEC balance the quantity versus the quality of investigations they undertake? How routine do these investigations become to the investigators themselves? What makes any of us think we would not ourselves be burdened by these - and many, many more - factors in the disposition of our jobs as investigators? This line of questioning is not intended to suggest the SEC was not derelict in its duties, but to point out how difficult the job is.

If the regulatory failures surrounding Madoff are examples of bad governance, what does good look like? I have written elsewhere that we can take cues from the positive behaviors and actions of the activist investor playbook. But if the regulatory lapses in the Madoff case tell us anything, it is that a playbook is one thing, a persona is another. Dr. John Kay summed this up best when he described the person best suited to do this kind of job:

You require both an abrasive personality and considerable intellectual curiosity to do the job.

It is easy to criticize the regulators in hindsight from the comfort of one’s armchair. It is another to possess the personality traits, the discipline to know when to bring them to bear, and the energy to sustain them, in every situation every day in which we find ourselves.

If there was ever a good time to be sure you have good governance, this is it

Since 2006, I've written multiple blog posts, a few articles and self-published an e-book on governing investments in strategic software. Software development is unique from most every other office-based occupation because it is the conversion of capital into intangible assets by way of human effort, at a level of effort that remains labor intensive to this day. Although other white-collar occupations such as back-office accounting can be labor intensive, their costs flow through to the income statement as SG&A rather than depreciation. Blue-collar occupations such as manufacturing labor can be capitalized (the labor that goes into building a car is a cost that is factored into finished goods inventory, which is a balance sheet phenomenon), but decades of capital investment in robotics have reduced the labor intensity of most manufacturing functions. True, many firms (particularly nascent tech firms) expense their software development labor costs; yet development is still a capital allocation function if they're spending investor capital or retained earnings rather than cash flow from operations to finance the development. Restated, software development is discretionary spend of balance sheet cash that could be directed to other investments or returned to investors.

Being an act of capital allocation, it has always struck me odd that most corporate captive IT organizations and tech firms self-govern the transformation of capital into software. In practice, software development governance is clubby, even more clubby than most corporate boards where the CEO has hand-picked the directors. For purposes of ceremony, IT governance is heavy-handed in reporting, but light touch in execution, loaded as it typically is with vendor reps with sales revenue on the line, delivery managers with bonuses on the line and business sponsors with promotions on the line. This isn't an environment where hard questions are tolerated, let alone asked. Some years ago, I was working with a regulated healthcare company replatforming its customer-facing solutions at all locations worldwide. It's target state one month after launch was to have 5 locations running the new software in parallel with legacy, processing 0.15% of corporate transactions, with another 5 locations ready for go-live the following month. In the event, the software had to be shut down after two locations were live for a few days because the new platform was dispensing client instruction incompatible with and contradictory to the healthcare products being provided. Sixty days after go-live, no locations were operating on the new platform, there was no resolution for this potentially life-endangering flaw, and therefore no path to production for the initial 2, or 5, or 10 locations, let alone any of the thousands of locations beyond that. The PMOs self-reported program status to the governance board? Green.

"We're all ok here" has been the modus operandi for IT governance for the past two decades. A lot of this is a function of the fact that despite the dot-com bust and the 2008 financial crisis, the first two decades of the 2000s have been characterized by overall global economic growth. Mid-way through 2020, we are now in a period of economic uncertainty. COVID-19 has initiated socioeconomic change that will have lasting effect on consumer behaviors, business practices and public policy. Uncertainty remains as the nature and trajectory that containment and combatitive policies to defeat the virus remain unknown. A lengthy period of containment and combat won't simply harden new business patterns such as work-from-home policies and consumer preference for take-out dining: it will unleash entirely new patterns as socioeconomic pressures build concomitant with human frustration at restrictions and seasonal change incompatible with policy relaxation.

The uncertainty factor matters because it creates new pressures for captive IT and tech firms alike to show diligence with every dollar spent. No matter the exposure to socioeconomic disruption your company now faces and balance sheet strength your company had entering into this crisis, the CEO was not and is not too keen on capital spend right now. While there are good cases to be made for investments in strategic software, particularly those that are reasonable preparations for how a society and economy function in the future, the tolerance for both wasteful allocation and misallocation is low.

Which brings us to the question of software delivery governance. If governing software delivery was hard during stable economic times, it is that much much harder during unstable economic times. Weak governance is highly vulnerable to threats old and new that cannot be swept under the rug worn threadbare by sudden economic convulsions. There are three threats worth highlighting: overspend, vendor fraud, and employee misalignment.

The first threat is little to no tolerance for overspending. I've written previously that CFOs earn their stripes on Wall Street by creating the appearance of control. Control arises from operational predictability: predictable operations create stable cash flows that finance healthy dividends and buoy the credit rating. In good times, perhaps the chief accountant was able reclassify a lot of maintenance work as capital improvement to bury that $2m IT project overrun a few years ago, and a late-year sales flourish buried that $4m overrun last year. In a sustained downturn, there's no accounting trickeration or sales windfall to come to the rescue. Not to mention, the CEO really wants to tell Wall Street that despite the economic upheaval, the company results were so good and the indicators so overwhelmingly positive, we're increasing the dividend. No CIO in their right mind will jeopardize the CEO's victory lap.

The second threat is an increase in fraud. Vendors have always exported overhead labor from their income statement to that of their customers. At a tactical level, vendor overhead employees are written into client contracts in loosely-defined roles that permit them to bill a few hours every billing cycle. Vendor direct labor rates are slightly inflated as a means of deflecting attention from the fact that vendor direct labor hours are significantly inflated. Surcharges such as "vendor administration services" - which amount to vendors charging their customers to make certain that timesheets are entered, invoices are generated and payments are collected - are added as service fees. And, of course, vendors cheerfully enter into one-way contracts to deliver CEO vanity projects of "re-imagination" and "market disruption" with no clawback clauses for results or outcomes. There is also the more pedestrian version where vendors enter into one-way contracts that create disincentive for improvement and (perversely) create incentive for perpetual failure: e.g., a vendor that charges per bug fix incident has no incentive to remediate the underlying cause, and actually has incentive to perpetuate the underlying cause.

These are all phenomenon that happen in good times, the product of lax contract administration, asymmetric knowledge of software delivery that favors the vendor over the buyer, and the starry-eyed "true north" ambitions of buyers. The boss wants to do this, we're not a software firm, procurement was told to get the deal done, and the contract gives little room for pushback, so everybody goes along for the ride. In unstable times, the pressures for vendors to inflate contracts to meet revenue targets, stave off layoffs, and sign obligation-light contracts to create flexibly tappable cash flow intensify that much more. Vendors have many more problems to try to export to their customers.

The third threat to look out for is the mismatch between individual and organizational goals. I wrote about this in 2013 in a two-part series on conflict. The phenomenon materializes as contrived complexity: obfuscated domain knowledge, jealously protected relationships, and unnecessarily complicated code are all examples of acts of individual self-preservation of employment that undermine effective governance of a strategic software investment. The board member who everyone knows is beholden to the knowledge of the governed will be played. If there is nobody with the domain knowledge, relationships, and algorithmic know-how to call bullshit on the executors, then the bullshitters will rule the day every day.

Today, governance is increasingly being put to the test, specifically at a time when governance standards have become institutionally lax and have been so for a long period of time. Governance will tighten, as described by John Maynard Keynes in his book The Great Crash of 1929:

In depression all this is reversed. Money is watched with a narrow, suspicious eye. The man who handles it is assumed to be dishonest until he proves himself otherwise. Audits are penetrating and meticulous. Commercial morality is enormously improved.

Keynes' statement that "all of this is reversed" is all well and good, but the fact is that regulators always lag the regulated, and governance is a regulatory phenomenon. By the time regulators catch up with the regulated, the damage is done to regulators' reputations for control, scrutiny and alignment. Rest assured that your successor will bask in the glory of being a more diligent investor, a more scrupulent auditor, and a more competent leader than you were.

If you prefer not to allow your career gravestone to be a platform for your successor to dance upon, what does it take to know that your governance practices are up to the task?

It's worth pointing out that you never will. Governance is something that is only evident by its absence. In addition to being intangible, there is no infallibility to governance: in practice, even the most prudent governance structures fail to detect problems, while the most lax governance structures suffer no damage from problems that do not metastasize. Governance is not a precise science. Governance is only obvious when it succeeds as a counterfactual (hindsight makes clear a crisis was intentionally avoided, such as investing in Bernie Madoff's fund in the summer of 2008) or when it fails (a fund piles in all of its money into Madeoff's fund in the summer of 2008).

That said, what should IT be doubling-down on to increase the effectiveness of its governance practices?

1. Strengthen the cost audit function. Ask people everywhere from the accounting fraternity to the program management office to procurement to increase the scrutiny of every dollar that goes out the door for a strategic software investment, whether for badged employee payroll, vendor services, cloud services, or licensed product. The objective is to identify non-value-generative chargebacks. Either the person responsible for the charge can answer "what value did we get for this" or they cannot. The point isn't to spot administrative errors, but inexplicable costs and patterns of charges that point to budget leakage or legitimately high costs that point to accelerated budget drainage, and to do so while threat to budget can be contained.
2. Strengthen the solution audit function. Engage a captive technical audit function or an external vendor with no connection whatsoever to a program of work to perform audits and peer reviews of tests, code, and team practices. Is the code written to a degree of simplicity that it is comprehensively testable, and easily maintainable by other developers? Can the piece-parts of the solution be tested in isolation and without an environment, or do they require expansive infrastructure and consistent data fixtures? Are there transitive dependencies that create false testing positives or negatives? Are integrations validated through contract tests before arriving in an environment? Is the team working in very narrow silos of domain knowledge or technical expertise? Are some people acting as bottlenecks? Have the number of hand-offs increased to complete requirements or fix defects? Have the number of defects increased, or has the time to resolution of them increased? Keeping a close eye on both the what and the how provides an early warning to delays in delivery or higher cost of maintenance than planned.
3. Increase the threshold of acceptance of management status reports. In medicine, patient testing is frequently done to confirm or deny a threshold of "no evidence of disease." This involves taking a sampling of cells and subjecting them to tests to ascertain a percentage present per the statistically significant sample, and a trend of counts of samples made over time. A higher standard of acceptance is "no evidence of disease", but that's an impractical threshold as 100% testing of cells would be fatal to the patient. In strategic software investing, the lower threshold of "no evidence of failure" is a proxy of convenience to the board: if the program manager says the status is green and the summary indicators support that, the program manager has given no reason not to believe the status is green, but not reason to believe that it well and truly is. Program governance must raise the threshold on program management to provide "evidence of no failure". That means program management conclusively demonstrates that effort is not passing for results, tasks are not masquerading as requirements, and work is not being deferred that conflates "developer complete" with "user acceptance".
4. Be an activist investor. As a board member, get your own data, form your own contra hypotheses, and engage in direct, constructive interrogation at board meetings. As a board member, you can talk to users, shadow teams, analyze delivery data, look at code and run analyses over that code. Nothing else will provide more effective leadership in these times from a governance role than investor activism that respectfully challenges those on the line creating a strategic software asset.
5. Have at least one independent director on the governing board. Break up traditional governance structures by having independent directors with no connection to solution sponsorship, vendor representatives, or delivery managers. With no political, financial or emotional investment in any one or any thing, independent directors are better positioned to provide critical insight of the state and trajectory of the investment, as well as clinical recommendations for potential change.

Governance of strategic software investments does well to apply the standard of US <-> Soviet arms reductions treaties in the 1980s of "trust, but verify." Creating a no-trust governance environment is counterproductive. Low-trust environments are extremely dysfunctional and chaotic, and at best offer an anti-pattern for governance. That clubby team of vendor reps and ambitious managers doesn't perform well when at each other's throats like the crew of the ISS Enterprise in Star Trek: TOS. But a board driven by an activist chair and flanked by an independent director with a penchant for healthy skepticism creates an environment where trust is earned and re-earned, rather than tacitly given or outright kicked to the curb. That sets the tone for the board, delivery leaders and delivery partners, a tone especially valuable during a time when control is at a premium and trust is in rather short supply.

I'll be Gone, You'll be Gone

The Financial Times recently ran a long article describing the breakdown of governance over the Crossrail development, the first new underground railway in London in over a century. Good governance is usually only appreciable by its absence, but the Crossrail case allows a before / after contrast between the presence of good and, after the sacking of a key board member, the presence of bad. Early on, "TFL's representative on the Crossrail board ... had kept a close eye on costs, and asked all 'the difficult questions' according to one individual close to the board". Yet after his departure, "It [the board] endorsed everything rather than challenging [management] and asking questions.". As a case study in governance, the FT article is an analysis very much worth reading.

The article also describes a nefarious management phenomenon common to complex, long-term investments: "And the management often doesn't care because they know they won't be there when the project isn't delivered on time and to budget."

I recently read John Kay's book Other People's Money: The Real Business of Finance. In Chapter 4, Dr. Kay describes the trading mentality that replaced that of stewardship across much of the banking sector from the 1980s onward. He introduces the subject using a quote from a senior industry figure: "''We are investment bankers. We don't care what happens in five years.'" The people who make the promises and ink the deal are not the people responsible for fulfilling the promises and seeing the deal through.

The FT article describes this phenomenon within Crossrail: "...like all long-term projects, it suffered from management changeovers at contractors and suppliers. 'You always have a lot of baton changes on large-scale projects like this... So although they will have been suppressing bad news, none of the people who were round the table at the beginning are still there - it's the nature of careers that no one stays for 10 years.'"

Or, as Dr. Kay put it, "Traders need not wait to see when or whether the profits materialise. I'll be gone, you'll be gone."

A little over a decade ago, a collapsing mortgage market in the United States metastasized into a global financial crisis when the value of complex structured products (think collateralized debt obligations backed by sub-prime mortgages) on bank balance sheets suddenly collapsed. Chuck Prince, CEO of Citigroup during the run-up to the global financial crisis of 2008, will forever be enshrined in the annals of finance history for his regrettable statement that 'as long as the music is playing, you've got to get up and dance.' "Soon after," writes Dr. Kay, "Prince was forced from office and Citigroup was bailed out by the US taxpayer."

Earlier this year, Crossrail development, which had appeared to be moving successfully along, fell off a financial cliff, requiring two capital injections in rapid succession, just so they could keep paying the bills. Per the FT: "'It feels as if they clung on to the schedule they had until it became clear that it absolutely couldn't be delivered,' he says. 'But when they dropped it, it seems that they had nothing left to work to. That would explain what looks like a sudden collapse.'" The chairman of Crossrail has been forced to resign (its chief executive having exited just before the bad news broke), and Crossrail has been bailed out by the UK taxpayer.

In the post-collapse analysis of financial crises, Dr. Kay points out that the principal actors are not particularly adept at critical self-assessment. "Rogue traders normally protest that the activity would eventually have been profitable if the bank had not closed its position, just as the gambler dragged home from the casino tells his wife and the world that he would have come out on top only if he had been allowed to stay longer." Additionally, the principal actors in the early stages - who are the primary beneficiaries of "cash borrowed from destiny with some random payback time" - have a tendency to deflect any doubts about the merits of the rewards they reaped in the time before the crash. "With the cognitive dissonance of the tailgater, he [Mozilo] would explain that the considerably larger amount he had received for his services as chief executive of Countrywide was justified by the profits that his company had reported from the sale of mortgages before the borrowers failed to pay them back." While deflection on the grounds that the collapse was episodic rather than systemic may be delusional, it is a convenient pound-the-fist-on-the-table argument to justify individual financial rewards of questionable merit.

In the banking sector, the problem was borne of a combination of the agency problem (the objectives of managers are materially divergent from those of the board) twined with moral hazard (no downside exposure to idiotic risks). It remains to be seen whether or not this proves to be the case with Crossrail. But with a management extracting high compensation derived from other people's money, a large number of contractors each working in a silo, an established pattern of suppressing bad news, and a forced buyer of last resort, Crossrail certainly appears to possess many of the same characteristics.

Per the FT, "People are always adventerous on costs and time because they don't want to tell the truth." The first and last line of defense against the agency problem and moral hazard is good governance. Poor governance plagued the banking sector in the years up to the 2008 financial crisis: few directors of financial institutions had any "sophisticated understanding of banking and risk", consisting as they did of "leading clients, ex-politicos and community leaders". Crossrail appears to have become plagued by the same, with one of it's principal stakeholders having the attitude of "'give him a call when it's time to cut the ribbon.'"

There is no easy fix to poor governance because governance tends to be self-referential among its members and lacking in tools for self-correction and continuous improvement. As Dr. Kay writes, "Their attitude was not the comprehensive immorality of the overt fraudster but the willful blindness of those who do not ask questions when it would be embarrassing, or at least inconvenient, to know the answer. Upton Sinclair's remark is again relevant: 'it is difficult to get a man to understand something, when his salary depends on his not understanding it.'"

Governance is only as effective as each governor, which is why I advocate for everybody in a governing role to study and practice the positive behaviors of activist investors. Given the variable - and generally poor - standard of governance in place over most investments, there needs to be a standard of excellence in governance practice. Activist investing is the closest thing we have to such a standard.

In Tech, Portfolio Management is a Metaphor, Not a Way of Life

A few years ago, I wrote about the chronic misuse of the word "governance" in technology. The word "portfolio" is suffering the same fate.

The reason for introducing the word "portfolio" into tech management is that some portion of tech spend is really an investment in the business, something that differentiates it and gives it a competitive edge - for example, mobile or web client-facing apps, or internal software that codifies workflows to capture efficiencies. This is fundamentally different from utility tech, the basic technology a company needs to function, such as ERP or email. To get a better return from our strategic opportunities, we should think in terms of investing-like behaviors and outcomes, as opposed to traditional project-oriented ones (such as time and budget). The word "portfolio" enters the lexicon because when we have multiple technology investments, we logically have a collection of them. Since most firms have more opportunities for investment than capital to invest, there is a degree of selectivity of what gets included in the investment portfolio.

While I was writing chapter 8 of Activist Investing in Strategic Software, it occurred to me that the use of word "portfolio" in technology has increased in recent years. Unfortunately, the activity described by aspirant "technology investment portfolio managers" are a very small fraction of those characteristic of financial portfolio management. The principal problem is that the word "portfolio" suggests a degree of decision-making flexibility that the captive technology investor doesn't really enjoy. Consider:

1. Although we can create diversity of our investment outcomes, the strategic software investor is limited to a single investment type - e.g., an operationally-based delivery. A financial investor has many more choices of vehicles and outcomes than a captive investor does. There are no fixed income products available to the captive technology investor, it's all equity. Plus, although we can run multiple experiments to qualify an investment and accelerate our delivery frequency to get things in production faster, all of our investment positions are inherently long. The only short position we can take in something is not to invest.
2. While there are countless investment opportunities, it's rare that a company can pick and choose every investment it makes. Some investments are forced on it by regulation; others by competition; still others by reliability of dilapidated legacy assets; and sometimes because the boss says this or that is what we're going to do and that's all there is to it. A captive technology investment portfolio isn't as discretionary as a financial one.
3. Our investment goals are denominated in different and sometimes only quasi-economic measures of value. All financial investments denominate their performance in the same measure, even those that have explicit non-financial goals. As I've written before, it's futile to concoct synthetic measures like "business value".
4. Most often, we can't measure the impact of any given investment in isolation of all other changes a business makes to itself and those that happen in its commercial ecosystem: we change the economics, processes, technology, and policy of our business all the time, while our commercial partners are also making the same changes to theirs. Isolating an outcome to a single decision or action (like a specific tech investment) is very difficult. In addition, because counterfactuals are unprovable, we can't measure whether an alternative investment would have yielded a better or worse outcome. In contrast, we can measure our results of financial portfolios against how well "Hindsight Capital Partners" performed over the same time period.
5. Venture capital is a high risk business that has more misses than hits. The success rate doesn't improve when the VC is corporate rather than financial. A company doesn't have unlimited capital to experiment at scale and can't afford to have a low success rate on investment decisions.

As much as we may want to be investors, the portfolio metaphor is very limited in captive investing situations. It lacks diversity, is inherently imprecise, and both performance and competency as investors are as much a matter of opinion as fact. It's potentially dangerous, too, because managed poorly it can damage its operational solvency (that is, the capability to get things done through technology) by making a mess of its capital management.

The premise of the book I'm writing - that activist investing behaviors yields better performing tech investments - is also substantially a metaphor. There are real aspects to it, specifically that shareholder activist behaviors - being investigative, inquisitive, interrogative and invasive - are highly suitable to captive technology investing. But "shareholder activism" only goes only so far: we can't buy out other "investors" and we don't gain control of the board through proxy fights. We have fewer levers to throw to change outcomes, and virtually all of them are operational (process, scope, personnel) in nature: few are the captive investments that can show better performance on the basis of financial engineering alone.

While metaphors are limited, they do help us to interpret our world differently. When we think in investment terms, we see financial expectations and possibilities much more clearly than when we think strictly in operating terms. When we think in shareholder activism terms, we understand the importance of good governance structures and mechanics, and the need for diligence by those investing in the business through technology. Interpreting delivery of strategic software through these lenses adds dimensions that make the operations that create them more value-generative to their host businesses.

But we do ourselves no favors by getting carried away with it. As helpful as the portfolio metaphor is, it's just a metaphor - not a way of life.

Counterfactuals

Earlier this year, my house should have burned to the ground.  A CR2032 battery exploded and caught fire in a confined place dense with flammable objects.  But my house didn't burn down: at the moment the battery exploded, I was sitting a few feet away from it. I heard a loud bang, investigated, and stamped out the fire within a few seconds.

I wasn't planning to be there at the time. A series of minor reschedules and reprioritizations had me sitting in my home office at the moment the battery exploded. I happened to be in the right place at the right time to prevent something from happening.  It was business as usual for the rest of the day.

There is no law or reason governing why I was there at the moment the battery exploded, or for the battery to have exploded at precisely that moment.  What if my schedule isn't rearranged and I'm not home when the battery explodes? What if it happens hours earlier or later when everybody is asleep in the house? I can't help but think that there are more quantum realities where something bad happened than there are where something bad did not happen.

But that isn't necessarily true.  The battery could have exploded at a time when nobody was around to put out the fire, but the ensuing fire may have simply died out causing no damage.  And maybe the chemical processes that triggered the explosion was itself an anomaly of circumstances set in motion by my presence in the room at that particular moment.  I have to be content with the only certainty I have, which is that a battery exploded and nothing bad happened.

Counterfactuals are messy because they're not provable.  Strategy and investing, execution and operations are loaded with counterfactuals.  They're each similarly messy because we don't really know what would have happened had we chosen another course of action.  Conventional wisdom says that Kodak should have bet on the digital technology they invented, and not dithered around with its legacy analogue business. But what if Kodak were to have invested heavily in digital and their distribution partners made good on their threats to shift the lucrative film processing business to Fuji?  Would Kodak's bet on the future have imploded for being starved of cash flow?  Hindsight isn't 20/20, it's narrative fallacy.

Just as it's difficult to have retrospective strategic certainty, it's next to impossible to forecast the outcomes of alternative paths, decisions or circumstances.  It sounds simple enough: invest in the highest-return opportunities, and consistently assess the performance of and the risks to delivery.  But we're forced to interpret and project from incomplete and imperfect information. Our plans and forecasts are just as likely to be narrative fallacy as fact-based assessments.  Microsoft paid dearly for Skype, but we can't know what might have been had it been acquired by Google or Facebook - and the possibilities may very well have justified the premium Microsoft paid.

The landscape of strategic software investing and governance is riddled with speculation.  As much as we would like to think we've made the best decision available to us, or dodged a bullet, we rarely know with certainty.  No matter how formulaic we're told that portfolio management or delivery operations can be, they will never be as optimal or predictable as promised, because business is imprecise and untidy.

Activist Investing in Strategic Software Chapter 1 Excerpt - Why Governance Matters

I've published drafts of the introduction and first 4 chapters of my book, Activist Investing in Strategic Software. I still have some citations to finalize, several visuals to integrate and a lot of editing to do. But the foundation is there. A sample of the book (currently, just the introduction) is available on from the site.

Here's an excerpt from Chapter 1, Why Governance Matters.

* * *

That there are abundant examples of bad governance but few examples of good ones comes as no surprise. What passes for "good" in governance is not all that remarkable. Boards, being the representatives of investors, are expected to be independent, diligent, and uncorruptable. Independent members of corporate boards are assumed to be people of capability and integrity. People in governance roles are expected to discharge their duties competently; meeting expectation is not exceptional. Hence we have few examples of good governance but many examples of bad.

This doesn't help us understand why governance is important.

Poor governance, such as in the cases of Olympus, Hollinger, Madoff, or the United Nation's Oil for Food program, can lead to devastating outcomes that are plain as day after the fact. Yet as we established in the prior section, there are objective characteristics of good governance: set expectations, invest authority, and validate results. And there is research to suggest that the presence of these have a significant impact on bottom-line results.

In a study of 1,500 companies by Harvard professor Paul Gompers, well governed organizations outperformed poorly governed ones by 8.5% annually. “Well governed companies face the same kind of market and competitor risks as everybody else, but the chance of an implosion … by ineffective management is way less.”

-- Gavin Anderson, Chairman, Governance Metrics International

Good governance reduces risk of bad things happening, and there is reason to believe it is a contributing factor to superior performance

Just as a corporation is an investment of capital, so, too, is strategic software. What is true for a company at a macro level should be true at a micro level. The characteristics of good corporate governance should be present in IT governance: an independent board that sets expectations, chooses and changes leadership as necessary, and validates results reported by that leadership.

Activist Investing in Strategic Software

A few years ago, I felt I had enough experience - and had put enough thought into the subject - to write a book on governance in software development. I had observed that most tech firms and captive IT organizations are largely left to self-govern, and both are pretty light touch about it. I had also observed that governance is widely misunderstood and the term is used in technology in a lot of different ways, almost universally incorrectly. With more ambitious investments being made in software and the success rate of large projects not getting all that much better, the need for better governance was there. Plus, it seemed to me that if IT didn't get its act together soon enough, the CFOs of the world were going to get IT's act together for it, so it also made sense to write it from a bit more of a financial rather than an operations perspective.

The timing was good, too, because the prior decade had given us some very well documented examples of egregiously bad corporate governance, ranging from isolated cases of accounting fraud (Worldcom and Parmalat) to the near-collapse of the banking industry that resulted from so many firms taking too much risk onto their balance sheets, their boards having absolutely no idea they had done so. There was some nascent research suggesting that good governance has a measurably positive impact on business outcomes, and also that "activist investors" - people who force their way onto the board of a company to agitate for change - were by and large net positive to a business. All together, we had great examples of how not to govern as well as behaviors we could use as a standard to define what governance really means - practiced or otherwise - in software development.

Within a couple of years, I had written enough material to compose a draft. Then I took a hiatus from it.

In the years since I wrote the first draft, a tension has developed between advocates of "innovation" and champions of "scale" in software development. In one corner are the enterprise development people who say control over operations yields predictable results. In the other are the lean startup people who say that discipline in execution twined with feedback loops is control enough, so just let people go on voyages of discovery and you'll have far better business results. The control camp claims that innovation is possible in their way because it embraces Agile (evidently, all you need are "innovation sprints"). The lean camp claims they can scale to the size of a large business (which may be true, but they're light on practical details). Both say they can revolutionize business itself.

But both camps take an execution (that is, operations) perspective. Execution is important, but if we're going to make organizational impact, we have to do it from a financial, not an operational, perspective. Operations are cost. Investments are value. To change the business of software, we have to speak the language and act the part of the latter. It doesn't matter how revolutionary or how beneficial a different way of delivering software can be to a company: nothing that comes out of either camp is going to cause the authors of Fundamental Accounting Principles to overhaul their text.

I am once again writing the book. I will publish it iteratively. It is still early days - no cover art, not much of a landing page.

But today, the first draft of the first chapters of my book, Activist Investing in Strategic Software, is available on Leanpub.

Why Commercial Contracts Matter In Agile Software Development

"We value customer collaboration over contract negotiation." -- The Agile Manifesto

Contracts for software development have historically included language that specifically defines the software being developed. This protects the buyer from paying for an asset that does not serve its business needs, the seller from requirements drift or expansion, and allows both parties to agree to duration and cost.  Traditional development contracts also stipulate the process by which the software is to be developed and tested, and how changes will be accommodated.  Since the way something is produced has direct bearing on the quality of what gets produced, specifying the process protects the buyer from slipshod work practices, and gives the seller a formal framework to control the development lifecycle.  The parties are trading a long-dated asset for a series of short-dated cash flows; being specific about the work being produced and the means of production is a means of protecting each party's economic interests throughout.

But with Agile becoming more and more widespread, contracts that stipulate requirements, team composition and change control processes lock both buyer and seller into a commercial arrangement that is an encumbrance (requiring constant amendment to accommodate changes) and may actually interfere with delivery.  Agile development requires more flexible contracting.  So it comes little surprise that a core tenet of the Agile manifesto is that people involved in developing software should constructively collaborate with one another to deliver a valuable business asset, not conform to a strict protocol that defines allowable behaviours.

The rigidity of traditional contracts has led firms in the Agile development business to experiment with looser contractual language.  In principle, this makes sense.  Agile teams deliver more frequently, so the lag between the buyer's cash and the seller's delivery isn't as great as it is with traditional software development.  Frequent showcases and deliveries improve trust and confidence between buyer in seller in ways that can't be codified in the language of a contract.  The benefit of looser language is that as teams learn more about the actual business needs and technical complexity of what they are developing, they have more freedom to act (and react) as the situation warrants.  There should also be limited downside. Even if the contract is vague, a buyer won't pay if she doesn't like what the seller has produced, and a seller will suspend work if he has an unwilling or incompetent buyer.  The people involved have maximum leeway to get stuff done and they'll let each other know in the most direct means possible when they're not happy.

In place of precisely defined language, it isn't uncommon to see development contracts that capture none of the intent whatsoever, and define only a supply of an unspecified number of people for an indefinite period of time.  If the understanding ex-contract is that the seller is there to develop software for a particular purpose in a particular manner, the contract is, in theory, at best a formality and at worst takes too much time.  If the development work is treated as R&D rather than a capital investment, doing this doesn't flaunt accounting practices (which require strict definition of capital work).  And if one party is disappointed with another during development, they'll make that abundantly clear by suspending performance until they are happy.  This is the triumph of the desire to get stuff done over the formality of contract law.

"That is, while there is value in the items on the right, we value the items on the left more."

But contracts do matter.

A contract expresses the value each party places in the other and the respect they have for one another.  The preamble language defines who the parties are or think they are as an acknowledgement of the strengths that each bring to the relationship: they are disruptors ("re-imagining the classroom for the 21st century"), or market leaders ("the nation's largest provider of financial services to retirees"), or specialists ("the leading provider of software development services to municipal governments in the tri-state area").  This underscores goals of the buyer (the disruptor is buying an innovative solution; the market leader wants cost efficiencies) and the capability of the seller (technical, process or subject matter expertise) and why the two parties want to work together.  In the contract, if the buyer is just a business no different from any other, and the seller is just a provider of people, the relationship will eventually come to reflect this, too.

A contract communicates the outcome the parties are working toward.  If the buyer wants an asset, they are committing to developing an asset in conjunction with a partner, and both buyer and seller are drivers in achieving that goal.  If the buyer wants only to rent capacity from the seller, the seller will be a passenger in the buyer's goals.

A contract defines a bond between two organizations, a bond that is meant to be durable in good times and in bad.  The more closely twined buyer and seller, the more likely they are to resolve their differences and difficulties. The more disposable a relationship, the more likely one party will dispense with the other when greener pastures beckon.

We don't want contracts that are ignorant of the need for flexibility.  But convenience erodes commitment: flexibility achieved through ambiguity undermines a sense of partnership. It is better to achieve flexibility through provisions that define the parties, define the mission, and define a bond.  This creates a commitment to the principles of a relationship.

Contracts exist between companies; relationships exist among people.  A relationship will always trump what's written in a contract.  But people come and go, and relationships are constantly tested.  A contract easily exited undermines the commitment of a relationship.  Good contracts are not an encumbrance to delivery: they strengthen the commercial ecosystem through which delivery happens.

The Persistent Imbalance Between Supply and Demand for Software Development Labor

The growth in demand for software has consistently outpaced the growth in the supply of software developers. This has been the case for well over half a century. It's worth looking at why.

Each major expansion in software development - automation (60s), productivity (80s), internet (90s), mobile (00s) - has been additive to the total stock of software in the world. The old stuff doesn't go away: software is still an enabler of labor productivity (office & communications), and a weapon for market share (online customer interaction). Yet we continue to find new applications for software: it is increasingly a product differentiator (embedded systems) or a product category of its own (social networking). While some segments have retrenched (companies license rather than write their payroll systems), the proliferation of new forms of software has more than compensated for consolidation in others. And the more it proliferates, the greater demand for integration, security, and other ancillary software.

Each new wave represents a structural increase in the demand for labor: the old stuff has to be maintained or replaced, while new applications not only bring new demand, they bring new tools and languages which require new skills and capabilities.

From a labor market perspective, the software economy has been expanding for decades. As a result, it marches to the beat of its own drum. Software development is generally counter-cyclical to the broader economy: it does well when the economy is down because software locks in productivity gains desired after layoffs & cutbacks. Software also makes its own opportunities, because it is inherently a business of invention and innovation. There are peaks and valleys: a structural change in demand for labor can sew the seeds of a bubble and its inevitable collapse. But bubbles in tech are bubbles of tech's own making: the video game / home computer bubble (1983), and the Y2K / dot-com bubble (2000) each resulted from irrational expectations and dubious business models created by people within the tech sector itself. The Y2K / dot-com bubble bursting coincided with increased accessibility to a global labor supply of software developers (offshoring was all the rage in the early 00s). Although the US experienced an acute contraction for software development labor, the global labor pool grew, and the regional contraction in the US proved to be short lived. Today, although the software labor market remains inefficient (context still doesn't travel), there are no easy cost savings to be gained (no large labor pools of skilled labor remain untapped). Global supply has been substantially eclipsed by global demand.

We're currently in the midst of another structural increase in the demand for software development labor, this time being driven by analytics and smart devices (the alleged "internet of things", from cars to coffee pots), with the odd halo application (e.g., wearable tech) thrown in for good measure. Every indication is that for the foreseeable future, demand for software developers will continue to increase at a rate faster than the supply of software developers available to develop it.

What does this mean to the business of software?

1. Ambition will outpace capability. Any business plan that comes down to "hire a bunch of smart engineers" - be it developing a new product or rescuing a distressed IT organization - is doomed. There is too much money chasing too few people. A company's labor timeline has to expand: it will take longer to hire experienced engineers, and firms will increasingly need to invest in incubating new developers. Labor scarcity poses a vulnerability to employers: a company known to have capable engineers is wearing a target for recruiters. When jobs outnumber candidates, jobs become commodities to employees. To differentiate from other employers, a firm must be highly attractive to the specific strata of the labor market that it wishes to employ. It does this by developing unique culture and values, and professional and societal aspirations that make it destination employer for those people. Without these things, it can only compete for labor on comp and career. It's difficult for a firm to maintain competitive advantage for labor solely on the price it is willing to pay for it.

2. Employers will pursue labor industrialization over tradecraft. Software development is labor intensive: the productivity enhancers that do exist such as automated testing and automated build are still poorly implemented, when used at all. Plus, the diversity of programming languages and the complexity of environments encourages labor specialization and task management. Still, people investing in software assets will not take "can't find competent people" for an answer. As the old saying goes, if you can't raise the bridge, lower the water. On a person-by-person basis, it is faster, easier, and cheaper to hire, train, and staff industrial workers to work on a software development "factory floor" where they perform coding tasks in an assembly-line like way than it is to recruit, develop and mentor polyskilled software developers. New labor formation will be largely industrial in nature, not tradecraft.

3. The risk of spectacular software failure will increase. The horrific explosion of the oil tank train that devastated Lac-Mégantic in 2012 was in no small part the result of demand exceeding supply. North American oil production has risen dramatically in the past half-decade. All that oil coming out of shale fields will find its way to refineries. Since there aren't pipelines to carry it, it's going by rail. The rail industry was in decline in North America for many years. A sudden uptick in demand can't be quickly satisfied by skilled labor. The net result is that railroads are hauling increasing volumes of a volatile commodity, but their capability to handle it isn't maturing at the same rate. In software, the demand/supply imbalance increases the risk of significant operating or project failure - that is, massive delivery overruns or post-delivery operating problems - as skills fail to mature in step with demand.

4. As the skills brought to bear on any given software investment deteriorate, software asset quality - particularly technical quality - will deteriorate. Industrial labor produces volume, not quality. The glut of software assets being produced will be toxic by technical quality standards. As it happens, it will go largely unnoticed because neither the concept of technical debt nor its commercial ramifications are well understood by the (average) business buyer of software, and because IT governance remains weak in practice. However, poor asset quality will become visible in maintenance and operating costs, and the occasional write-off. Any firm forced to make too many write-offs due to poor technical quality will cause it to see software as disposable rather than durable. That would create deflationary price pressure for labor and increase the demand for industrialization.

As long as the applications for software continue to expand, insufficient numbers of software engineers come into the work force, and software development remains labor intensive, there will be a fundamental supply / demand imbalance. But demand tends to be impatient. Economic and perhaps even political pressure will intensify to industrialize software development. This implies expansion of the secondary labor market, which is less skilled, educated, compensated and mobile than the primary labor market. That would be a lost opportunity: rather than fostering a global wave of knowledge workers, software development will simply bring the next wave of wage workers. We'll look at the reasons for that in the next post.

The Corrosive Effects of Complexity

"Much complexity has been deliberately created, to encourage consumers to pay more than they need, or expected." John Kay, The Wrong Sort of Competition in Energy

Modern software assets are complex in both their technical composition and their means of creation.  They are built with multiple programming languages, are expected to conform to OO standards and SOA principles, make use of automated tests and a progressive build pipeline, require a diverse set of skills (UX, developers, QA analysts, etc.) to produce, are used on a multitude of clients (different browsers or native client apps on PC, tablet and smartphone form factors), and are deployed using automated configuration management languages to a combination of physical and virtual environments (cloud and captive data centers).  Software is more complex today than it was less than a generation ago.

Complexity compromises both buyers and sellers of technology services.

Buyers suffer an information gap.  Few managers and fewer buyers have first-hand experience in one, let alone all, of the technologies and techniques a team is - or should - be using.  This creates information asymmetry between buyer and seller, manager and executor.   The more diverse the technologies, the more pronounced the asymmetry.

Sellers suffer a skill gap. Because the demand for software is outstripping the supply of people who can produce it, experienced people are in short supply.  There are more people writing their first Android app than their second, more people making their first cloud-based deployment than their second.  There are more blog posts on Continuous Delivery than there are people who have practiced it.  There are more are people filling the role of experience design than have designed software that people actually use.  And while long-standing concepts like OO, SOA and CI might be better understood than they were just a few years ago, a survey of software assets in any company will quickly reveal that they remain weakly implemented.  In a lot of teams, the people are learning what to do as much if not more than doing what they already know.

"Such information asymmetry is inevitable. Phone companies have large departments working on pricing strategies. You have only a minute or two to review your bill."

Information asymmetry favours the seller.  Sellers can hide behind complexity more easily than the buyer can wade through it: the seller can weave a narrative of technical and technology factors which the buyer will not understand.  The buyer must first disentangle what they've been told before they can begin to interpret what it actually means. This takes time and persistence that most software buyers are unwilling to make. Even if the buyer suspects a problem with the seller, buyer hasn't any objective means of assessing the seller's competency to perform.  Where complex offering are concerned, the buyer is at an inherent disadvantage to the seller.

"When you shop, you mostly rely on the reputation of the supplier to give you confidence that you are not being ripped off."

Technology buyers compensate by relying on proxies for competency, such as brand recognition and professional references of a selling firm. But these are controlled narratives: brands are aspirational visions created through advertising (although "Go ahead, be a Tiger" didn't end well...), while references are compromised by selection bias.  A buyer may also defer judgment to a third party, hiring or contracting an expert to provide expertise on their behalf. In each case, the buyer is entering into a one-way trust relationship with somebody else to fulfill their duty of competency.

An buyer inexpert in technology can compensate better by staying focused on outcomes rather than means.

Match cash flows to the seller with functionality of the asset they deliver. You're buying an asset.  Don't pay for promises, frameworks and infrastructure for months on end, pay for what you can see, use and verify.

Look under the hood.  There are plenty of tools available to assess the technical quality of just about any codebase that will provide easy-to-interpret analyses.  A high degree of copy-and-paste is bad.  So is a high complexity score.  So are methods with a lot of lines in them.

Spend time with the people developing your software.  Even if you don't understand the terminology, what people say and how they say it will give you a good indication as to whether you've got a competency problem or not.  Competency is not subject-matter-expertise: technology is a business of open-ended learning, not closed-ended knowledge.  But the learning environment has to be progressive, not blind guesswork.

Accept only black-and-white answers to questions.  Most things in software really are black and white.  Software works or it does not.  The build is triggered with every commit or it is not.  Non-definitive answers suggest obfuscation. "I don't know" is a perfectly valid answer, and preferable to a vague or confusing response.

An inexpert buyer is quickly overwhelmed by the seller's complexity. A buyer who stays focused on business outcomes won't dispel the seller's complexity, but will clarify things in favor of the buyer.

Governing IT Investments

In the previous blog, we looked at common misconceptions of IT governance. We also looked at how corporate governance works to better understand what governance is and is not. In this blog, we'll look at how we can implement more comprehensive governance in tech projects.

For corporate tech investments, the top most governing body is the firm's investment committee. In small companies, the investment committee and the board of directors are one and the same. In large companies, the investment committee acts on behalf of the board of directors to review and approve the allocation of capital to specific projects, usually on the basis of a business case. They also regularly review the investment portfolio and make adjustments to it as the firm's needs change.

The investment committee is composed of senior executives of the firm. Although executives are managers hired by investors to run the business, in this capacity they are making a specific allocation of capital that is generally of too low a level for board consideration. This is not a confusion of responsibilities. The board will have previously approved capital expenditure targets for the year as well as the strategy that makes the investment relevant, and any investment made by the investment committee has to stand up to board scrutiny (e.g., the yield should exceed the firm's cost of capital, or it should substantially remove some business operating risk). The investment decision is left to a capital committee composed of the firm's executives - who always have a fiduciary responsibility to shareholders - for sake of expediency.

The individual shareholders of a company have multiple investments and have limited amounts of time, so they rely on a board of directors to act on their behalf. In the same way, the investment committee members are the shareholders of an IT investment. They invest the firm's capital in a large and diverse portfolio above and beyond just IT investments. They will not have time to hover over each investment they make. So, just as investors form a board to govern a corporation, the investment committee forms a board to govern an individual investment.

In technology projects, we usually associate a "steering committee" with the body that has governance responsibilities for a project. As mentioned in the prior blog, steering committees are too often staffed by senior delivery representatives. This is a mistake. People who govern delivery do so on behalf of investors, not delivery. They must be able to function independently of delivery.

We'll call our governing body a "project board" so as not to confuse it with a traditional "steering committee". A project board that represents investors is composed of:

• a representative of the corporate investment committee (e.g., somebody from the office of the CFO)
• a representative from the business organization that will be the principal consumer of the investment (e.g., somebody from the COO's organization)
• a senior representative of the IT organization (e.g., somebody from the office of the Chief Information Officer or Chief Digital Officer)
• at least one independent director with experience delivering and implementing similar technology investments.

The program manager responsible for delivery and implementation of the investment is the executive, and interacts with the steering committee in the same way that the CEO interacts with the board of directors.

Again, notably absent from this board are the delivery representatives we normally associate with a steering committee: technical architects, vendors, infrastructure, and so forth. They may be invited to attend, but because they represent the sell side of the investment and not the buy side, they have no authority within the board itself. Investing them with board authority invites regulatory capture, which undermines independent governance.

The project board has an obligation to make sure that an investment remains viable. It does this primarily by scrutinizing project performance data, the assets under development and the people responsible for delivery. In addition, the board is given some leeway by the investment committee to change the definition of the investment itself.

Let's first look at how the board scrutinizes performance. The board meets regularly and frequently, concentrating on two fundamental questions: will the investment provide value for money? and is it being produced in accordance with all of our expectations? The program executive provides data about the performance of the project and the state of the assets being acquired and developed. The board uses this data, and information about the project its members acquire themselves, to answer these two governance questions. It also reconciles the state of the investment with the justification that was made for it - that is, the underlying business case - to assess whether it is still viable or not. The project board does this every time it meets.

The project board is also granted limited authority to make changes to the definition of the investment itself. It does not need to seek investment committee approval for small changes in the asset or minor increases in the cash required to acquire it if they do not alter the economics of the investment. This enables the project board to negotiate with the delivery executive to exclude development of a relatively minor portion of the business case if the costs are too high, or approve hiring specialists to help with specific technical challenges. The threshold of the project board's authority is that the sum of changes it approves must not invalidate the business case that justified the investment.

Scrutinizing performance and tweaking the parameters of the investment are how the board fulfills the three governance obligations presented in the previous blog. It fulfills its duty of verification by challenging the data the executive provides it and asking for additional data when necessary. It also has the obligation and the means to seek its own data, by e.g., spending time with the delivery team or commissioning an independent team to audit the state of the assets. It fulfills its duty of setting expectations by changing the parameters of the investment within boundaries set by the investment committee (e.g., allowing changes in scope that don't obliterate the investment case). It fulfills its duty of hiring and empowering people by securing specialists or experts should the investment run into trouble, and changing delivery leadership if necessary.

If the board concludes that an investment is on a trajectory where it cannot satisfy its business case, the board goes to the investment committee with a recommended course of action. For example, it may recommend increasing the size of the investment, substantially redefining the investment, or suspending investment outright. The board must then wait for the investment committee decision. The presence of a member of the investment committee on the project board reduces the surprise factor when this needs to happen.

This model of governance is applicable no matter how the investment is being delivered. Teams that practice Agile project management, continuous integration and static code analyses lend themselves particularly well to this because of the frequency and precision of the data they provide about the project and the assets being developed. But any team spending corporate capital should be held to a high standard of transparency. Delivery teams that are more opaque require more intense scrutiny by their board. And, while this clearly fits well with traditional corporate capital investment, it applies to Minimum Viable Product investing as well. MVP investments are a feedback-fueled voyage of discovery to determine whether there is a market for an idea and how to best satisfy it. Without independent governance, the investor is at risk of wantonly vaporizing cash on a quixotic pursuit to find anything that somebody might find appealing.

This is the structure and composition of good governance of an IT investment. Good structure means we have the means to perform good governance. But structure alone does not guarantee good governance. We need to have people who are familiar with making large IT investments, how those investments will be consumed by the business, what the characteristics of good IT assets are, and above all know how to fulfill their duty of curiosity as members of a project board. Good structure will make governance less ineffective, but it's only truly effective with the right people in governance roles.