We knew what didn’t work in software development a generation ago. They’re still common practices today.

In the not too distant past, software development was notorious for taking too long, costing too much, and having a high rate of cancellation. Companies spent months doing up-front analysis and design before a line of code was written. The entire solution was coded before being deemed ready to test. Users only got a glimpse of the software shortly before the go-live date. No surprise that there were a lot of surprises, all of them unpleasant: a shadow “component integration testing” phase, an extraordinarily lengthy testing phase, and extensive changes to overcome user rejection. By making the activities of software development continuous rather than discrete, collaborative rather than linear, and automated rather than repeated meant more transparency, less time wasted, and less time taken to create useful solutions through code.

It was pretty obvious back then what didn’t work, but it wasn’t so obvious what would. Gradually people throughout the industry figured out what worked less bad and, eventually, what worked better. Just like it says in the first line of the Agile manifesto. But I do sometimes wonder how much of it really stuck when I see software delivery that looks a lot like it did in the bad old days. Such as when:

• "Big up front design" is replaced with … "big up front design": pre-development phases that rival in duration and cost those of yesteryear.
• It takes years before we get useful software. The promise during the pitch presentation was that there would be frequent releases of valuable software… but flip down a few slides and you’ll see that only happens once the foundation is built. It’s duplicitous to talk about “frequent releases” when it takes a long time and 8 figures of spend to get the foundation - sorry, the platform - built.
• We have just as little transparency as we did in the waterfall days. Back then, we knew how much we’d spent but not what we actually had to show for it: requirements were “complete” but inadequate for coding; software was coded but it didn’t integrate; code was complete but was laden with defects; QA was complete but the software wasn’t usable. The project tasks might have been done, but the work was not. Today, when work is defined as tasks assigned to multiple developers we have the same problem, because the tasks might very well be complete but "done" - satisfying a requirement - takes more than the sum of the effort to complete each task. Just as back then, we have work granularity mismatches, shadow phases, deferred testing, and rework cycles. Once again, we know how much we’re spending, but have no visibility into the efficacy of that spend.
• The team is continuously learning … what they were reasonably expected to have known before they started down this path in the first place. Specifically, that what the team is incrementally "learning" is only new to people in the development team. Significant code refactoring that results from the team "discovering" something we already knew is a non-value-generative rework cycle.
• We have labor-intensive late stage testing despite there being hundreds and hundreds of automated tests, because tests are constructed poorly (e.g., functional tests being passed off as unit tests), bloated, flaky, inert, disabled, and / or ignored. Rather than fixing the problems with the tests and the code, we push QA to the right and staff more people.
• Deadlines are important but not taken seriously. During the waterfall days, projects slipped deadlines all the time, and management kept shoveling more money at it. Product thinking has turned software development into a perpetual investment, and development teams into a utility service. Utility service priorities are self-referential to the service they provide, not the people who consume the service. The standard of timeliness is therefore "you’ll get it when you get it."
• Responsibility isn’t shared but orphaned. Vague role definitions, a dearth of domain knowledge, unclear decision rights, a preference for sourcing capacity (people hours) and specialist labor mean process doesn’t empower people as much as process gives people plausible deniability when something bad happens.

Companies and teams will traffic in process as a way to signal something, or several somethings: because of our process we deliver value, our quality is very high, our team is cost effective, we get product to market quickly. If in practice they do the things we learned specifically not to do a generation ago, they are none of the things they claim to be.

The era of ultra low interest rates is over. Tech has painful adjustments to make.

Interest rates have been climbing for two years now. The Wall Street Journal ran an article yesterday with the headline the days of ultra low interest rates are over. Tech will have to adjust. It’s going to be painful.

When capital is expensive, we measure investments against the hurdle rate: the rate of return an investment must satisfy to exceed to be a demonstrably good use of capital. When capital is ridiculously cheap, we no longer measure investment success against the hurdle rate. In practice, cheap capital makes financial returns no more and no less valuable than other forms of gain.

There are ramifications to this. As fiduciary measures lapse, so does investment performance. We go in pursuit of non-financial goals like "customer engagement rate". We get negligent in expenditure: payrolls bloated with tech employees, vendors stuffing contracts with junior staff. We get lax in our standard of excellence as employees are aggressively promoted without requisite experience. We get sloppy in execution: delivery as a function of time is simply not a thing, because the business is going to get whatever software we get done when we get it done.

Capital may not be 22% Jimmy Carter era expensive, but it ain’t cheap right now. Tech has to earn its keep. That means a return to once familiar practices, as well as change that orchestrates purge of tech largesse. Business cases with financial returns first, non-financial returns second. Contraction of labor spend: restructuring to offload the overpromoted, and consolidation of roles or lower compensation for specialization. Transparency of what we will deliver when for what cost, and what the mitigation is should we not. An end to vanity tech investments, because the income statement, much less the balance sheet, can no longer support them.

Some areas of the tech economy will be immune to this for as long as they are thematically relevant. AI and GenAI are TINA (there is no alternative) investments: a lot of firms have no choice but to spend on exploratory investments in AI, because Wall Street rewards imagination and will reward the remotest indication of successful conversion of that imagination that much more. Yet despite revolutionary implications, AI enthusiasm is tempered compared to frothy valuations for tech pursuits of previous generations, a function of investor preference for, as James Mackintosh put it, profits over moonshots.. Similarly, businesses where there is a tech arms race on because innovation offers competitive advantage, such as in-car software, it will be business as usual. But these arms races will end, so it will be tech business as usual until it isn’t. (In fact, in North America, this specific arms race may not materialize for a long, long time as EV demand has plateaued, but that’s another blog for another day.)

Tech has had the luxury of not being economically anchored for a long time now. If interest rates settle around 400 bps as the WSJ speculated yesterday, those days are over. The adjustment to a new reality will be long and painful because there’s a generation of people in tech who have not been exposed to economic constraints.

This is the Agile Manager blog, as it has been since I started it in 2006. Good news, this change doesn’t mean a return to the failed policies of waterfall. Agile had figured out how to cope with these economic conditions. Tech may not remember how to use those Agile tools, but it has them in the toolkit. Somewhere.

That said, I also blog about economics and tech. If the Fed funds rate lands in the 400 bps range, tech is in for still more difficult adjustments. More specifically, the longer tech clings to hopes for a return to ultralow interest rates, the longer the adjustment will last, and the more painful it will be.

The ultralow rate party is over. It’s long past time for tech to sober up.

How Agile Management Self Destructs

I’ve been writing about Agile Management for over 15 years. Along the way, I’ve written (as have many others) how to get Agile practices into a business, how to scale them, how to overcome obstacles to them, and so forth. I’ve also written about how Agile gets co-opted, and a few months ago I wrote about how Agile erodes through workforce attrition and lowered expectations. I’ve never written about how Agile management can self-destruct.

The first thing to go are results-based requirements. Stories are at the very core of Agile management because they are units of result, not effort. When we manage in units of result, we align investment with delivery: we can mark the underlying software asset to market, we can make objective risk assessment and quantify not only mitigation but the value of mitigation. Agile management traffics in objective facts, not subjective opinion.

The discipline to capture requirements as Stories fades for all kinds of reasons. OKRs become soft measures. “So that” justification become tautologies. Labor specialization means that no developer pair, let alone a single person, can complete a Story. Team boundaries become so narrow they’re solving technical rather than business problems. And, you know what, it takes discipline to write requirements in this way.

Whatever the reason, when requirements no longer have fidelity to an outcome, management is back to measuring effort rather than results. And effort is a lousy proxy for results.

The next thing to go is engineering excellence. Agile management implicitly assumes excellence in engineering: in encapsulation, abstraction, simplicity, build, automated testing, and so forth. Once managers stop showing an active interest in engineering discipline, the symbiotic relationship between development and management is severed.

The erosion of engineering discipline is a function - directly or indirectly - of a lapse of management discipline. Whereas a highly-disciplined team decides where code should reside, an undisciplined team negotiates who has to do what work - or more accurately, which team doesn’t have to do what work. This is how architectures get compromised, code ends up in the wrong place, and abstraction layers create more complexity than simplicity.

The loss of engineering excellence is traumatic to management effectiveness. How something is built is a good indicator of the outcomes we can expect. Is the software brittle in production? Expensive to maintain? Does it take forever to get features released? Management has to reinforce expectations, verify that things are being built in the way they’re expecting them to be built, and make changes if they are not. When excellence in engineering is gone, management is no longer able to direct delivery; it is instead at the mercy of engineering.

The third thing to go is active, collaborative management. I’ve previously described what Agile management is and is not, so I’ll not repeat it here. The short version is, Agile management is a very active practice of advancing the understanding of the problem (and solution), protecting the team’s execution, and adjusting the team as a social system for maximum effectiveness. Now, management can check-out and just be scorekeepers even when there is engineering excellence and results-based requirements. But suffice to say, when saddled with crap requirements and becoming a vassal to engineering, management is reduced to the role of passenger. There is no adaptability, no responding to change, beyond adding more tasks to the list as the team surfaces them. Management is reduced to administration.

Agile requires discipline. It also requires tenacity. If management is going to lead, it has to set the expectations for how requirements are defined and how software is created and accept nothing less.

Give us autonomy - but first you've gotta tell us what to do

The ultimate state for any team is self-determination: they lead their own discovery of work, self-prioritize that work, self-organize their roles and self-direct the delivery.

Self-determination requires meta awareness. The team knows the problem space - the motivations of different actors (buyers, users, influencers), the guiding policies (regulatory and commercial preference), the tech in place, and so forth. Conversely, they are not whipsawed by external forces. They do not operate at the whim of customers because they evolve their products across the body of need and opportunity. They do not operate at the mercy of regulation because they know the applicable regulation and how it applies to them. They know their integrated technology’s features and foibles and know what to code with and code around. They know where the bodies are buried in their own code. And the members of the team might not be friends or even friendly to one another, but they know that no member of the team will let them down.

In the early 1990s, a Chief Technology officer I knew once replied to a member of his customer community during the Q&A at their annual tech meetup thusly: “there is nothing you can suggest that we’ve not already thought of.” Arrogance incarnate. But he and his entire team knew the product they were trying to build and they product they were not trying to build. They were comfortable with the tech trends they were latching onto and those they were not. They had not only customer intimacy but intimacy with prospective customers. They sold what they made; they did not make what they sold. Best of all, they’re still in business today, over 30 years later. They achieved a state of sustainable economic autonomy.

Freedom is most often associated with financial independence. There is a certain amount of truth to this. Financial independence means you can reach as far as “esteem” in Maslow’s hierarchy without doing much more than lavish spending. Unfortunately, money only buys autonomy for as long as the money lasts. History is littered with case studies where it did not. Sustained economic performance - through business cycles and tech cycles - yields the cash flow that makes self-determination possible. That requires evolution and adaptability, and those are functions of meta awareness.

Which brings us to the software development team that insists on autonomy. The team wants the freedom to tell their paymasters how a problem or opportunity space is defined, what to prioritize, how to staff, how much funding they need, and when to expect solution delivery will begin. That’s a great way to work. And, for decades now, management consultants have advised devolving authority to the people closest to the need or opportunity. But that proximity is only as valuable as the team’s comprehension of the problem space, familiarity with the domain, experience with similar engineering challenges, and the ability to think abstractly and concretely. When a team lacks in these things, devolving authority will simply yield a long and expensive path of discovery while the team acquires this knowledge. The less a priori knowledge the team has, the less structured and more haphazard the learning journey; so when the problem space is complex, this becomes a very long and very expensive discovery path indeed - and one that sometimes never actually succeeds.

Autonomy increasingly became the norm in tech as a result of the shortage of capable tech people, driven by the combination of cheap capital and COVID fueling tech investments. Under those conditions, a long learning journey was the price of admission; meta awareness was no longer a requirement for autonomy. With capital a lot more expensive today, tech spending has cooled and returns on tech investments are under much tighter scrutiny, and the longer the learning journey the less viable the tech investment. This is having the effect of exposing friction between how tech expects to operate and how business buyers expect for it to operate. Business buyers financing tech investments want tighter business cases that define returns and provide controls for capital spend. Tech employees want the space to figure out the domain, increasing expense spend and lengthening the time (and therefore cost) to deliver. With capital now having the upper hand, it is not uncommon for tech to be dichotomously demanding both autonomy and to be told exactly what to do.

While autonomy is the ultimate state of evolution for any team, the prerequisites to achieve it are extraordinarily high. It doesn’t require omnipotence, but it does require sufficient fluency with the tech and the domain to know the question to ask, to make appropriate assumptions, to anticipate the likely risks, and to know the sensible defaults to make in design. Devolved authority is a fantastic way to work, but autonomy must be earned, never granted.

Competency Lost

The captive corporate IT department was a relatively early adopter of Agile management practices, largely out of desperation. Years of expensive overshoots, canceled projects, and poor quality solutions gave IT not just a bad reputation, but a confrontational relationship with its host business. The bet on Agile was successful and, within a few years, the IT organization had transformed itself into a strong, reliable partner: transparency into spend, visibility into delivery, high-quality software, value for money.

Somewhere along the way, the “products not projects” mantra took root and, seeing this as a logical evolution, the captive IT function decided to transform itself again. The applications on the tech estate were redefined as products, assigned delivery teams responsible for them with Product Owners in the pivotal position of defining requirements and setting priorities. Product Owners were recruited from the ranks of the existing Business Analysts and Project Managers. Less senior BAs became Product Managers, while those Project Managers who did not become part of the Product organization were either staffed outside of IT or coached out of the accompany. The Program Management Office was disbanded in favor of a Product Portfolio Management Office with a Chief Product Officer (reporting to the CIO) recruited from the business. Iterations were abandoned in favor of Kanban and continuous deployment. Delivery management was devolved, with teams given the freedom to choose their own product and requirements management practices and tools. With capital cheap and cashflows strong, there was little pressure for cost containment across the business, although there was a large appetite for experimentation and exploration.

As job titles with "Product" became increasingly popular, people with work experience in the role became attractive hires - and deep pocketed companies were willing to pay up for that experience. The first wave of Product Owners and Managers were lured away within a couple of years. Their replacements weren't quite as capable: what they possessed in knowledge of the mechanical process of product management they lacked in fundamentals of Agile requirements definition. These new recruits also had an aversion to getting deeply intimate with the domain, preferring to work on "product strategy" rather than the details of product requirements. In practice, product teams were "long lived" in structure only, not in institutional memory and capability that matter most.

It wasn't just the product team that suffered from depletion.

During the project management years of iterative delivery, something was delivered every two weeks by every team. In the product era, the assertion that "we deploy any time and all the time" masked the fact that little of substance ever got deployed. The logs indicated software was getting pushed, but more features remained toggled off than on. Products evolved, but only slowly.

Engineering discipline also waned. In the project management era, technical and functional quality were reported alongside burn-up charts. In the product regime, these all but disappeared. The assumption was, they had solved their quality problems with Agile development practices, quality was an internal concern of the team, and primarily the responsibility of developers.

The hard-learned software delivery management practices simply evaporated. Backlog management, burn-up charts, financial (software investment) analysis and Agile governance practices had all been abandoned. Again, with money not being a limiting factor, research and learning were prioritized over financial returns.

There were other changes taking place. The host business had settled into a comfortable, slow-growth phase: provided it threw off enough cash flow to mollify investors, the executive team was under no real pressure. IT had decoupled itself from justifying every dollar of spend based on returns to being a provider of development capacity for an annual rate of spend. The definition of IT success had become self-referential: the number and frequency of product deployments and features developed, with occasional verbatim anecdotes that highlighted positive customer experiences. IT's self-directed OKRs were indicators of activity - increased engagement, less customer friction - but not rooted in business outcomes or business results.

The day came when an ambitious new President / COO won board approval to rationalize the family of legacy of products into a single platform to fuel growth and squeeze out inefficiency. The board signed up provided they stayed within a capital budget, could be in market in less than 18 months, and could fully retire legacy products within 24 months, with bonuses indexed to every month they were early.

About a year in, it became clear delivery was well short of where it needed to be. Assurances that everything was on track were not backed up by facts. Lightweight analysis led to analysis work being borne by developers; lax engineering standards resulted in a codebase that required frequent, near-complete refactoring to respond to change; inconsistency in requirements management meant there was no way to measure progress, or change in scope, or total spend versus results; self-defined measures of success meant teams narrowed the definition of "complete", prioritizing the M at the expense of the V to meet a delivery date.

* * *

The sharp rise of interest rates has made capital scarce again. Capital intensive activities like IT are under increased scrutiny. There is less appetite for IT engaging in research and discovery and a much greater emphasis on spend efficiency, delivery consistency, operating transparency and economic outcomes.

The tech organization that was once purpose built for these operating conditions may or may not be prepared to respond to these challenges again. The Agile practices geared for discovery and experimentation are not necessarily the Agile practices geared for consistency and financial management. Pursuing proficiency of new practices may also have come at the cost of proficiency of those previously mastered. Engineering excellence evaporates when it is deemed the exclusive purview of developers. Quality lapses when it is taken for granted. Delivery management skills disappear when tech's feet aren't held to the fire of cost, time and, above all, value. Domain knowledge disappears when it walks out the door; rebuilding it is next to impossible when requirements analysis skills are deprioritized or outright devalued.

The financial crisis of 2008 exposed a lot of companies as structurally misaligned for the new economic reality. As companies restructured in the wake of recession, so did their IT departments. Costly capital has tech in recession today. The longer this condition prevails, the more tech captives and tech companies will need to restructure to align to this new reality.

As most tech organizations have been down this path in recent memory, restructure should be less of a challenge this time. In 2008, the tech playbook for the new reality was emerging and incomplete. The tech organization not only had to master unfamiliar fundamentals like continuous build, unit testing, cloud infrastructure and requirements expressed as Stories, but improvise to fill in the gaps the fundamentals of the time didn't cover, things like vendor management and large program management. Fifteen years on, tech finds itself in similar circumstances. Mastering the playbook this time round is regaining competency lost.

Management is Getting Things Done Through People

Last year, I wrote that one of the core capabilities of an Agile manager is to "create and adapt the mechanical processes and social systems through which the team gets things done." I went on to describe a little bit of what allows the Agile manager to succeed at this; it merits a bit more commentary.

The mechanical processes a team performs matter because orchestrating the right activities in the right sequence at the right times are essential to exploring a problem space and evolving solutions. The Agile toolkit is well established and doesn't merit detailing here, but it is worth pointing out there are a lot of different techniques, activities, and ceremonies (and a multitude of variations therein) that the Agile manager can reach for; the Agile manager must be sufficiently fluent in the mechanics to know which are appropriate, and which are not, for the circumstances. Yes, process matters.

But the mechanical processes themselves aren't enough: the manager has to create the right social system in which people can participate effectively. That's much harder than executing to a script prescribed by a mechanical process because the manager has to understand the skills and aptitudes, strengths and weaknesses, competencies and limitations of the people within the team.

Sometimes people are exactly as advertised: a subject matter expert, a department director, a knowledge worker. And sometimes they're not. This person isn't an expert in supply chain, familiar with the abstract patterns and business processes of supply chain management, with first hand experience in a variety of implementations; they actually only have experience in how this one company manages its supply chain, and only in how it operates, not how it was designed in the first place. That department director is director in title only, because they've delegated all responsibility to subordinates and require decisions they are required to make to be framed as single-alternative choices. And that knowledge worker is really only fluent in which buttons to press at which part of a process and how to fix common exceptions, but has no idea why they do what they do.

To create functional social systems, the Agile manager must be able to meet the people in their team where those people are at. That means some degree of fluency in the subject matter for which an expert has been staffed, some degree of familiarity with the types of decisions the department director makes, some recognition of the wisdom a seasoned knowledge worker possesses. In last year's post, I suggested this is a function of both EQ and professional experience. EQ is key to awareness, but not technical understanding. Professional experience can be the source of techncial understanding, but is limited because no manager has experience with every domain and every context they're asked to manage.

What the manager does not know through experience the manager must try to learn through theory by conducting independent research and investigation into the respective domains to understand the context of the various participants. The important thing for the manager isn't to become a domain expert, but to be sufficiently fluent in the terminology to use and the questions to ask to assess, engage and manage people of various backgrounds and various capability levels.

Finding the right level of fidelity with which to engage members of the team is a critical component of social system formation. To wit: asking people without first hand knowledge of “what good looks like” in supply chain management to design a next generation process will design a "faster horses" solution that is, at best, less bad than what the company has today. Suppose the manager is able to recognize this deficiency; that recognition is fantastic, but it doesn't give the manager license to tell the team to down tools until the SME who isn't quite a SME is replaced. The fact is, no team is perfect, and the manager has to work with the people they’re staffed with. If a genuine SME can’t be sourced full time, it is incumbent on the manager to (a) create a social system within which the not-quite-a-SME is able to contribute to the best that their knowledge allows them to so that the team can make meaningful progress; and (b) in recognition of the not-quite-a-SME's limitiations, to see that the team validates proposed solutions and models against established industry models (that the manager may very well have to self-source), potentially supplemented with slivers of time from experts from analyst organizations or specialists (that may have to be sourced from the manager's network).

How the manager deals with circumstances like this is the difference between a person mindlessly executing a mechanical process and a person steering a team toward an outcome. The prior is an executor; the latter is a manager.

Sometimes, of course, it just isn't there to be done. About a decade ago, I was part of a technology team partnering with a financial services data business to replatform its operations and core systems. Before the first day was out, it was patently obvious the SMEs and knowledge workers could regurgitate the keys they pressed in the monthly process they followed, but had no understanding of why they did the things they did, nor could they articulate the value of the data they provided to the services their customers consumed that data for. Unsurprisingly, the management - none of whom had first hand knowledge of the business itself, having been installed following the acquisition of the company by private equity - refused to accept our day one conclusion. So on day two we performed workshops that laid bare the knowledge deficit. We abandoned the inception because the people they had brought simply lacked the wisdom that comes with knowledge, and no amount of sourced content and slivers of expert time could compensate: we concluded we weren’t even going to get a faster horses solution out of that group. Can't get blood from a stone, so there's no point continuing to squeeze.

Trust the process? Sure. But any process is only as effective as the people participating in it, and participation is a function of the underlying social system of the team. Creating an effective social system is at the heart of the definition of what management is: getting things done through people.

Taking people at face value based on their title is an abrogation of management's responsibility. There are no free rides in management, be it project, program, product, department, division, or executive. You have to know your people; to know your people you have to meet them where they’re at; to meet them where they’re at you have to understand their context; to understand their context you have to have some familiarity with their domain. The manager who fails to do these things is not a manager, they’re an individual contributor with a highfalutin' title.

Shadows

One of the benefits of being an agile organization is the elimination of IT shadows: the functions and activities that crop up in response to the inadequacy of the plans, competency and capacity of captive IT.

IT shadows appear in a lot of different forms. There are shadow IT teams of developers or data engineers that spring up in areas like operations or marketing because the captive IT function is slow, if not outright incapable, of responding to internal customer demand. There are also shadow activities of large software delivery programs. The phases that get added long after delivery starts and well before code enters production because integrating the code produced by dependent teams working independently is far more problematic than anticipated. The extended testing phases - or more accurately, testing phases that extend far longer than anticipated - because of poor functional and technical quality that goes undiscovered during development. The scope taken out of the 1.0 release resulting in additional (and originally unplanned) releases to deliver the initially promised scope - releases that only offer the promise to deliver in the future what was promised in the past, at the cost of innovation in the present.

None of these functions and activities are planned and accounted for before the fact; they manifest themselves as expense bloat on the income statement as no-alternative, business-as-usual management decisions.

The historical response of captive IT to these problems was to pursue greater control: double down on big up-front design to better anticipate what might go wrong so as to prevent problems from emerging in the first place, supplemented with oppressive QA regimes to contain the problems if they did. Unfortunately, all the planning in the world can’t compensate for poor inter-team collaboration, just as all the testing in the world can’t inspect quality into the product.

Agile practices addressed these failures through teams able to solve for end-to-end user needs. The results, as measured and reported by Standish, Forrester, and others, were as consistent as they were compelling: Agile development resulted in far fewer delays, cost overruns, quality problems and investment cancellations than their waterfall counterparts. With enough success stories and experienced practitioners to go round, it’s no surprise that so many captive IT functions embraced Agile.

But scale posed a challenge. The Agile practices that worked so well in small to midsize programs needed to support very large programs and large enterprise functions. How scale is addressed makes a critical distinction between the truly agile and those that are just trying to be Agile.

Many in the agile community solved for scale by applying the implicit agile value system, incorporating things like autonomous organizations (devolved authority), platforms (extending the product concept into internally-facing product capabilities) and weak ownership of code (removing barriers of code ownership). Unfortunately, all too many went down the path of fusing Agile with Waterfall, assimilating constructive Agile practices like unit testing and continuous build while simultaneously corrupting other practices like Stories (which become technical tasks under another name) and Iterations (which become increments of delivery, not iterations of evolving capability), ultimately subordinating everything under an oppressive regime pursuing adherence to a plan. Yes, oppressive: there are all too many self-proclaimed "Agile product organizations" where the communication flows point in one direction - left to right. These structures don’t just ignore feedback loops, they are designed to repress feedback.

If you’ve ever worked in or even just advocated for the agile organization, this compromise is unconscionable, as agile is fundamentally the pursuit of excellence - in engineering, analysis, quality, and management. Once Agile is hybridized into waterfall, the expectation for Agile isn’t excellence in engineering and management and the like; it is instead a means of increasing the allegenice of both manager and engineer to the plan. Iteration plans are commitments; unit tests are guarantees of quality.

Thus compromised, the outcomes are largely the same as they ever were: shadow activities and functions sprout up to compensate for IT’s shortcomings. The captive IT might be Agile, but it isn’t agile, as evidenced by the length of the shadows they cast throughout the organization.

The Manager-Leader versus the Manager-Administrator

We saw last month that just because somebody is the manager of an Agile team does not make that person an Agile manager. The Agile manager does specific things: advances the understanding of the domain, has a symbiotic relationship with the do-ers, creates and adjusts the processes and social systems, and protects the team’s execution. By virtue of doing these things, the Agile manager is a leader, whereas the traditional manager is just an administrator.

I have been fortunate to watch the rise of Agile competencies over the last two decades. I have been less fortunate to watch Agile management competencies erode as Agile has spread in popularity. Although there are undoubtedly numerous reasons why, one immediately stands out: while the value system that underlies the behaviors described in the previous post is highly compatible with the creative company mindset, it is highly incompatible with the operating company mindset.

The creative company - e.g., a studio that yields original work, anything from entertainments to custom software assets - benefits from how much it exposes itself to environmental uncertainty and how effectively it internalizes relevant learnings from it to create a successful, if unique and one-off, solution. In contrast, the operating company - a firm that mass produces products or provides mass volumes of labor to deliver solutions - benefits from environmental certainty that allow it to apply patterns that create consistency enabling repeatable solutions at scale. The greater the consistency, the greater the automation, the lower the labor proficiency level required, the lower the cost of execution, the higher the margin and cash flow. Whereas the creative studio model thrives on chaos, the operating company thrives on consistency.

This is a tectonic fault line in the application of Agile management practices. When Agile is brought to bear in an operating company context with an overriding mission to provide consistency of outcomes, the value system that fosters the management behaviors that get things done through people in the face of a volatile set of circumstances is simply ignored. The words remain - adaptive planning, continuous feedback, and the like - but the values that give rise to them in the first place simply dissipate.

The presence of Agile terminology twined with the absence of the Agile value system gives license to people in management roles to do pretty much anything under the aegis of “being agile”. Take “adaptive planning”. In practice, “adaptive” is used to mean anything on a spectrum from no management and no plan (“we’ll figure it out as we go, on somebody else’s dime”) to dictatorial management-by-plan (“the team is free to meet the commitments they make during the planning exercise.”) Planning itself is an exercise in plausible deniability for managers: if the do-ers create the plan, management is the act of holding people accountable for the plan they came up with, not for the continuous adjustment of the plan or refinement of the business outcomes in the face of what is learned through execution. And reporting against a plan is somewhat perversely passed off as “governance”, itself an overloaded term with no actual meaning beyond “fancy word for management reporting.”

The net result is that managers of Agile teams have found ways to make themselves passengers in Agile teams because they only do administrative, communication, and reporting tasks. Former IBM CEO Lou Gerstner referred to these kinds of managers as “presiders”. Mr. Gerstner deemed presiders to be useless. Do-ers in Agile teams deem presides to be useless as well.

I wrote a decade ago that Agile gets corrupted when it goes corporate. That phenomenon is not unique to Agile, of course. By way of example, look at cloud computing: who knew that “migrating to the cloud” was as simple as moving a data center from owned on-prem to leased in somebody else’s facility? Yes, people still pass this off as “cloud migration”. Enterprise scale, enterprise politics and enterprise vendors have a tendency to dilute concepts - cloud, Agile, you name it - to a point of rendering them inert. ‘Twas ever thus.

Yet while Agile concepts are bound to be co-opted, this does not have to be the case for Agile execution. Managers can choose to be drivers of outcomes rather than plans, work with the team rather than outside of it, create social systems rather than schedule meetings, and protect the team’s execution from external forces rather than allowing them to steamroll the team. These, among other things too numerous to list here, define excellence in management.

Pursuing excellence is a choice that always rests with practitioners. There is a reason why the administrative burden of Agile has always been defined as “lightweight”, and isn’t to ease the workload of managers: it is to give managers the bandwidth to take a leadership role in a peer relationship with engineers, designers, analysts, and all the other do-ers in a team. That door is always open to managers in an Agile team, but the decision to walk through it or not rests with the individual manager.

Choose wisely.

What, Exactly, Is Agile Management?

Engineers have long had a poor relationship with managers. There are the stories long told of engineers unconstrained by management executing high stakes skunkworks projects, or more modestly using guerilla tactics to sneak unauthorized features into products that delight users. There are popular caricatures such as the pointy-haired manager in Dilbert, or Ford Motor’s management as portrayed in the movie Ford v Ferrari. Historically, management has never been loved, and in fact is often loathed, by engineers.

This all seems different in Agile. Management appears to be well integrated in Agile teams, not an overhead, nuisance, or encumbrance as it is in traditional project management. If we accept the fact that Agile is a value system and not a set of mechanical processes, it stands to reason that there must be something different about the norms and behaviors of Agile managers vis-a-vis traditional managers. What is it that makes Agile management different from traditional management?

A good place to start is by looking at the fundamentals of Agile team dynamics. First and foremost, as Paul Hammant is fond of saying, there are no passengers in an Agile team.

An Agile story is an expression of end-to-end business need. Although completing a story may require contribution from people in different roles - QA analyst and developer and experience designer, for example - each person is still responsible for the entire outcome of the story. Each person is responsible for the outcome because team performance is measured on collective output (specifically, stories in production) as opposed to the sum of individual output (tasks completed by individuals). Every member creates the most comprehensive understanding they can of each story, which they express both through artifact (story narrative or code for example) and collaboration with other team members (story walkthroughs and desk checks). A person driving goals orthogonal to the team goal, or not driving at all, does not last for very long in an Agile team.

A less charitable corollary to Paul’s statement is that nobody can hide in an agile team. What every person does and how they do it is fully exposed to every other member of the team. This is why safety is a key characteristic of an Agile team: by making it safe for any person to admit they don’t know how to do something, it is easier for the team to collectively adapt and make the most of the strengths of the people it has. It does mean, however, that mismatches - people who overstate their skills and competencies - will be exposed very quickly.

Combined, this means that no individual can phone it in. One person who does a substandard job affects the performance of the entire team. Consider a team using story narratives to surface complexities in business requirements. A business analyst who consistently does a lackadaisical job will effectively export the analyst responsibilities to a developer. Developer productivity - and therefore story card throughput - will suffer as developers compensate for substandard analysis work.

The obvious conclusions are (a) every individual drives to the collective output of the team; (b) every individual must strive to perform a high state of excellence; and (c) a team cannot maintain a level of performance any higher than the level of excellence achieved by its weakest performing member.

For people working in roles directly related to creation and evolution of a software asset - experience designers, developers, QA analysts - this is easy to understand. We all understand the consequences to an Agile team trying to compensate for inadequate design artifacts, vague requirements, poor quality code, and ineffective tests: rework cycles, additional handoffs, and poor quality, among other things.

But what do these three things - drive, excellence, and minimum collective effectiveness - mean for people in management roles?

I was fortunate to have worked with very strong Project Managers in my early Agile projects, people who understood the Agile value system and knew how to apply it as managers in a delivery team context. They all had characteristics that are applicable to all management roles. Among them:

1. Managers advance the understanding of the problem / opportunity / solution domain. The best Agile managers are outstanding at scope definition and scope control. They do this by intelligently questioning different people in the team: business partners to ascertain what is truly important to the near-term business goals and is not, developers to ascertain what is tech goldplating and what is not, product managers on how can a story be split and part of it deprioritized, and so forth. It’s worth noting that codified accounting standards treat managers as overheads, for the simple reason that management is largely administrative work. By demonstrably working the details, some portion of an Agile manager's time is spent genuinely advancing the understanding of the problem/solution domain as part of a team collective. For this reason, a portion of an Agile project manager’s time can be capitalized, unlike traditional project managers who are purely expense. Traditional PMs create project plans, staffing plans and reports, schedule meetings and facilitate ceremonies. All useful, but not directly contributory to the technology asset itself. The difference is that the Agile project manager is hands-on with the problem and not simply performing tasks of administrative convenience - contracts, meetings, project plans - on the periphery of those hands on with the problem.

   It’s worth pointing out that this doesn’t happen by accident. Previous experience in a do-er role such as business analyst or developer certainly helps the manager to know the types of questions that need to be asked. But the key characteristic is both the ability and willingness to immerse and comprehend the details of the stories, the architecture, the code, the design, the tests. Good managers know how to work the problem and solution space. This applies to all levels of management, because “ground truths” always triumph over abstractions, right up to CEO and activist investor. Conversely, the manager who lacks the will or skill to internalize a domain is completely beholden to (and subsequently held hostage or simply played by) those in the team who do. This manager is the textbook definition of “overhead.”

2. A symbiotic relationship with the do-ers in the team. PMs have to provide a variety of status reports to various sponsors and buyers. In the early days of Agile development, there were all kinds of new and novel measures: velocity for measuring throughput of stories, load factor for assessing the actual capacity available to the team, story points (or equivalent) for measuring relative story sizes, and many more. It was easy - and very satisfying - to nerd out on the management metrics, creating forecasts of scope expansion and capacity and so forth. But one thing that distinguishes the Agile manager from the traditional manager is that the team drives the metrics. The metrics are only very rarely used to drive the team. The Agile manager uses the metrics to detect a change in the situation such as domain complexity that was not previously understood, or team member skill that was over- or under-stated. Project management metrics are used by managers to ask “what has changed”, not “why are we off plan”. The objective of the Agile metrics has never been to keep the team on a plan and held to a plan. The objective of the Agile metrics is to increase the understanding of how execution is unfolding and make the necessary adjustments in one of the four variables - time, scope, capacity or quality - to respond. That’s a big difference between traditional project management and Agile project management.

   Metrics being all the rage among the management class these days, this bears a bit of analysis. Of course there are instances where metrics are used to drive the team, such as when three of the project management variables - time, scope and quality - require compensation from the fourth - capacity. By way of example, when the number of sev 2 and 3 defects escaping to production rises a bit - not unsustainable or threatening, just something that stands out - while the team focus is still on new story development, it isn’t uncommon to ask everybody in the team to try to resolve one defect every day before finishing up. True, in this case a rise in defects exposes something about how execution is unfolding, and the team needs to adapt (put in a little additional time to address quality). But in practice, this is an instance where the metrics are used to drive the team (e.g., to maintain story velocity while paying down defects), if for no other reason than the optics of few defects to go along with story throughput puts stakeholder minds at ease.

   The same nuance applies to measuring story throughput. A release plan that slots specific stories for specific future iterations sounds like a fixed delivery plan, but when scope and time are the priorities this is a reasonable exercise. It helps to answer the “will the team make it“ question, and as long as capacity and quality variables can be relaxed it is a way of depicting scenarios and anticipating responses. However, when stories are slotted to future iterations and all 4 variables are held constant by the manager, we no longer have Agile project management, we have traditional project management that has co-opted Agile terms in the pursuit of control. Obviously, this is not Agile.

   The four variables combined with appropriate interpretation and application of the metrics enable the manager to have a symbiotic relationship with executors. This manager is an Agile manager. However, when the metrics themselves are used as drivers, management has a one-way relationship with do-ers. This manager is a tyrant.

3. Create and adapt the mechanical processes and social systems through which the team gets things done. Roy Sigham, the founder of ThoughtWorks, used to refer to the company as a “social experiment.” All teams are social experiments, random strangers brought together in combinations of varying skills, capabilities and personalities. There will be tension and conflict. There will be misrepresentations and misunderstandings. There will be volatility.

   Traditional project managers reach for plans against which they can hold people accountable. But this is not how managers fulfill their primary duty of “getting things done through people.” A high EQ allows a manager to recognize how people interact with one another. Professional experience allows the manager to recognize the skills people have and aspire to have and will never have. These and other factors let the manager create the circumstances for the right type of interactions and format - workshop, fishbowl, small group exercise, etc. - for the right subject in the right sequence at the right time to enable the team to get things done. Among other things, this takes listening and observation skills, the ability to design the mechanics and visualize how various activities will go, plus the ability to prioritize, facilitate, coach and intervene. This is management. This is how managers “get things done through people.” People who manage a plan are curators of documentation, not managers of a team of people.

4. Protect the execution of the team. Creating a functional team is one thing. Protecting the team’s execution is entirely another. There are constant challenges to the integrity of the workings of a team. People ghosted to work on multiple teams (a “10% allocation” of a person’s time to a team is utterly meaningless). Stakeholders with differing and highly volatile priorities. Fear - or just tepid commitment - to an Agile way of working. Corporate politics. Low trust corporate cultures. The Agile manager manages upwards and outwards, keeping these threats and many others at bay, preventing them from invading the team.

   There are times when external forces must invade the team, such as changing stakeholder priorities. The Agile manager creates a constructive framework and mechanical process for the team to ingest and respond to things that cannot be deflected without creating chaos with how the do-ers are otherwise do-ing. The manager who does not protect the execution of the team from external drama is a puppet to any and all external stakeholder.

I wrote above that each person in an Agile team drives to the team outcome; that each person must strive to achieve excellence; and that the team will move only as effectively as their weakest performing member. We’ll look at patterns and antipatterns of management behavior in the next post.

Echoes

In recent days, I’ve been reminded of some core company values, once fresh and different, later taken for granted. It was good to be reminded of them.

* * *

I was on a call the other day when a member of the client’s team said, “we don’t prioritize because something is easy, we prioritize something because it is hard.” They went on to acknowledge that some of my colleagues had counseled them to do this some months before I started working with them. I was overjoyed to hear the client say this.

When I first joined ThoughtWorks in 2005, one of our core messages was to solve an “outer quadrant” problem first - something in the problem space that is difficult and complex. Something small and discreet of course, as opposed to trying to boil the ocean, but very much in the category of “hard” than “easy”. This was contradictory to the prevailing wisdom at the time (that persists to this day), which was to lead with quick wins, pilots or proofs of concept.

By and large, none of these latter approaches address the core of the problem that needs to be solved. “Quick wins” attack the margins. They might very well be useful, but quick wins are just as likely to contribute to systemic complexity by layering on new points of integration, adding logic and system redundancies, and creating confusion over veracity of data or transaction success. I once came into contact with a company whose primary engagement pattern was to create duplicative data warehouses by copying data from existing data sources and writing reports off it. They’d done this 6 or 7 times at one company alone. It isn’t difficult to imagine the “single source of truth” problem that arose from timing discrepancies and reliability problems among all of those data warehouses. All those quick wins did was make them more sclerotic.

Pilots and proofs of concept tend to be evaluated in environments heavily favored to the success of the pilot, not in environments representative of the actual problem space that needs to be solved. The result of quick wins and pilots and proofs of concept is a false positive with regard to the affordability and even solve-ability of a particular problem space. You might feel good for having done something, but the something doesn’t make material progress toward addressing the underlying problem.

It seems to me that the “quick win” and “pilot” philosophy became increasingly prominent in American business in the 1980s. In the 1970s, American corporations had earned a reputation for poor quality. They also had become gripped by analysis paralysis fueled by, among other things, a fear of making a mistake. Management consultants like Tom Peters struck a nerve on this conflation of “poor quality” and “doing nothing about it”. In the book “In Search of Excellence” published in 1982, Peters and Robert Waterman listed what they called the “8 characteristics of excellent companies”. Top of the list was “a bias for action:” “a preference for doing something - anything - rather than sending a question through cycles and cycles of analyses and committee reports.” Peters, Waterman and others advocated “a bias for action” to cut the Gordian knot of inaction.

At the same time, Japanese management techniques were in the ascendency. Japan’s economy was booming, in no small part because products made in Japan had earned a reputation for both quality and affordability. American managers wanted to know what their Japanese counterparts did differently. Among the things that came to the surface in what made Japanese management different was to solve small problems and learn before solving a big problem. Whereas an American company would attempt to solve Big Problems with Big Solutions, the Japanese management philosophy attacked Big Problems by solving Small Problems with Small Solutions. In the American company, more often than not the Big Solution yielded a Big Mistake: massive cost overruns, excessive time delays, and all-too-frequently outright failure. In the Japanese company, The Big Problem was resolved over time as a sum of the evolution of Small Solutions.

Iterative solution development resonated with the “bias for action” theme because it provided a means of overcoming the collective learned helplessness among employees of American companies. A systemic problem in an American corporation was larger than any one person or small group of people (or their budgets) within an enterprise could solve, but small problems could be solved through simple, affordable solutions. The emerging personal computer technology of the time enabled this tremendously. The result was a brief management revolution of the 1980s, where pent up frustration among employees of being trapped in oppressive systems that yielded poor quality met up with the evangelical management themes of excellence that stressed “action” and “devolved decision making”, which in turn were operationalized by the liberating personal technologies of the day.

The themes of a “bias for action” and “iterative solutions” and many other things stressed by Peters and others under the aegis of “excellence” live on in many forms today, not least of which are Lean and Agile software development as well as product management practices. All well and good, but somewhere along the way the messages get corrupted. All too often, a “bias for action” prioritizes “quick wins” which are interpreted as “go after the low-hanging fruit” and is applied as “do the easy stuff first.” The easy stuff is almost always at the margins and not the core, and are not representative of the need. Similarly, even the most well-intentioned proof-of-concept misses the mark if it intentionally excludes the challenges central to the problem space.

To hear somebody say “we’re prioritizing the hard stuff first” is rewarding indeed. It’s an indicator that they’re serious about solving the problem at hand.

* * *

When I joined ThoughtWorks, one of the things that many of the experienced software developers stressed was mentorship. They would seek out less experienced colleagues who were difficult to staff on a client project and mentor them in everything ranging from technical skills, to the value system for how we worked, to ways in which they could be effective consultants.

Mentoring can be a slow process, the results difficult to measure, it doesn’t scale, and it doesn’t garner a whole lot of attention. Mentoring is not something that comes naturally to a lot of people, and by and large there are more people interested in doing things that are noisy and that scale - go on sales calls, speak at conferences - than in doing the humdrum work of mentoring a person.

Mentoring does more than just help people acquire skills and consulting acumen. It gives new people first hand experience in practicing the values that a company of people holds. This builds muscle memories that become their default way of working. That, in turn, builds the culture of a company, better than any email or podcast or missive ever can.

We had virtual leaving drinks for a long-time colleague particularly adept at doing this kind of mentoring, as he is joining another firm. I will miss him.

Autonomy Now

Distributed software development has been practiced for decades. Companies with global footprints were experimenting with this at least as far back as the 1970s. Skilled labor, global communication networks and collaborative tools made "offshore development" possible at scale from the mid-1990s onward. Improved skills, faster networks and more sophisticated collaboration tools have made distributed development practical for very complex software initiatives.

There can be significant differences in the way a team collaborates internally, and the way it collaborates with other teams across a program. Consider a distributed Agile program consisting of multiple teams based in different countries around the world. Under normal circumstances, individual teams of 8 to 12 people work day-in and day-out with colleagues in the same physical location. Intra-team events take advantage of the team's close proximity: the team room, collaborative practices like pair programming and desk checks, team ceremonies like stand-ups, and low-fidelity information radiators such as card walls are all high-bandwidth collaboration techniques. In-person communication is robust, spontaneous and fluid, so it makes sense to take full advantage of it. Conversely, inter-team events such as a Scrum-of-Scrums involve only key team members such as the project manager and lead developer, and are scheduled to take advantage (or at least minimize the inconvenience) of time zone overlap. In practice, any single team in a large program - even an Agile team - can function internally in a tightly coupled manner even though it is loosely coupled to other teams in the same program of work.

The COVID-19 pandemic has a lot of the global work force working in physical isolation from one another; this pushes distributed work models to their extreme. Yes, of course, it has long been possible for teams of individuals to work completely remotely from one another: e.g., tenured experts in the relevant technology who are also fluent in the business context and familiar with one another. But most teams don't consist of technology experts who know the domain and one another. In the commoditized IT industry, people are are staffed as "resources" who are qualified based on their experience with relevant technologies. Domain expertise is a bonus, and interpersonal skills (much less familiarity with team-mates) never enter the equation. A good line manager and competent tech lead know how to compensate for this through spontaneous, high-bandwidth interaction: if somebody's work is going adrift, pull them aside, ask the business analyst or product owner to join you, whiteboard and code together for a bit, and get it fixed. A good line manager and tech lead compensate for a lot of the messiness intrinsic to a team of commodity-sourced people. The physical isolation much of the world is experiencing makes this compensation more difficult.

There are lots of companies and individuals self-publishing practical advice for remote working. Many are worth reading. Although the recommendations look hygienic, good remote collaboration hygiene reduces individual frustration and maximizes the potential communication bandwidth. An "everyone is 100% remote" from one another model has scale limitations, and poor hygiene will quickly erode whatever scale there is to be had.

My colleague Martin Fowler posted a two-part series on how to deal with the new normal. The posts have a lot of practical advice. But the concluding paragraphs of his second post address something more important: it is imperative to change management models.

Being independent while working remotely is not working remotely in an independent manner. The more tightly coupled the team, the more handoffs among team members; the more handoffs, the more people will have to engage in intra-team communication; the lower the fidelity of that communication, the higher the propensity for mistakes. More mistakes means lower velocity, lower quality, and false positive status reports. In practice, the lower the fidelity of intra-team collaboration of a tightly coupled team, the lower the fidelity of inter-team collaboration regardless they are tightly or loosely coupled.

This is where a distributed program of truly Agile teams has a resiliency that Agile-in-name-only teams, command-and-control SAFe teams, and waterfall cannot intrinsically possess by their very nature. A requirement written as a Story that fulfills the INVEST principle is an autonomous unit of production. A development pair that can deliver a Story with minimal consultation with others in the team and minimal dependencies on anybody else in the team is an autonomous delivery team. A Quality Assurance Analyst working from clear acceptance criteria for a Story can provide feedback to the development pair responsible for the development of the Story. Stories that adhere to the INVEST principle can be prioritized by a product owner and executed in a Kanban-like manner by the next available development pair.

A tightly coupled team operating in a command-and-control style of management doesn't scale down to a more atomic level of the individual or pair. The program manager creates a schedule of work, down to the individual tasks that will fulfill that work and the specialist roles that will fulfill those tasks. Project managers coordinate task execution among individual specialists in their respective teams. One project manager is told by three people working on tasks for a requirement that their respective tasks are complete, yet the whole of their work is less than the sum of the parts. Now the manager must chase after them to crack their skulls together to get them to realize they are not done, and needs to loop in the tech lead to figure out where the alignment problems(s) are. This is difficult enough to do when people are in distributed teams in a handful of office buildings; it's that much more difficult when they are working in isolation of one another. Product quality, delivery velocity, and costs all suffer.

Command-and-control management creates the illusion of risk-managed delivery at large scale with low overheads. Forget about scaling up with efficiency; to be robust, a management paradigm needs to be able efficiently to scale down to deliver meaningful business results at the atomic level of the individual or pair. Micromanagement does not efficiently scale down because of the inherently high overheads. Self-directed autonomous teams do efficiently scale down because of the inherently low overheads.

In 2013, I spilled a few photons on the management revolution that never happened: for a variety of reasons in the 1980s, we believed we were on the cusp of a devolution of authority; instead, we got much denser concentration of authority. In 2018, I spilled a lot of photons on autonomous teams at enterprise scale being an undiscovered country worth the risk of exploring.

The COVID-19 pandemic is creating intense managerial challenges right now. It is important to note that there are likely to be long-term structural effects on businesses as well. Perhaps companies will encourage employees to work from home more regularly so the company can permanently reduce office square footage and therefore lease expense. Perhaps a new generation of secure mobile technologies will make it seem idiotic that large swaths of workers are office rather than home based. Perhaps companies will revise their operating models and position specs, requiring greater individual role autonomy to maintain high degrees of productivity in regular and irregular operating conditions. Perhaps metrics for contract labor - metrics that are not attendance based - will emerge to satisfy expectations of value delivery.

Perhaps, with the potential for long-term effects looming, it is time to go explore that undiscovered country of autonomy.

Muddling Through

British trade in the West was never an instrument of empires. It had no headquarters like the compounds at Montreal and no capitol like the rock-rooted fortress at Quebec. It had no seasonal rhythm like the canoe caravans coming down the rivers on the spring race of water to the tall ships above the quaysides loading their pyramids of peltry. It had no trace of the pagentry of black-robed priests visioning by their campfires in the forest a new empire for the Church, and no imperial plan of feudal seigniories and savage nations submissive to the rule of the Rock. The British trade was sporadic, individual and motivated by simple greed. When forty dollars' worth of trade goods would secure a thousand dollars' worth of peltry, there was incentive enough to bring pack trains to Ohio.

Walter Havinghurst, Land of Promise

In 1959, Dr. Charles Lindblom posited that companies that had disciplined processes for experimentation and discovery had an advantage over those that operated, in the words of Dr. John Kay, against "a single comprehensive evaluation of all options in light of defined objectives". The argument Doctors Lindblom and Kay make is common sense. For those of us without the gift of omnipotence, the ability to reassess objectives and tactics with the benefit of first-hand market knowledge is a handy capability to have.

While this is intuitively appealing to entrepreneurially-minded managers, it is decidedly unappealing to risk-averse capital. Complex corporate strategy derived from analysis and modeling creates the appearance of authoritative expertise. Authoritative experts are in the business of selling predictability. Debt capital likes predictability. As long as there is cheap capital to be raised from risk-averse investors there will be complex business strategies. This doesn't make for good strategy or good investment. This just makes it something that happens.

There is an obvious contemporary comparison between Amazon (financed with equity capital) and WeWork (financed largely with debt capital) and no doubt there will someday be a good case study that sheds light on the differences between the two firms. Suffice to say for now that in the latter's case, somebody tells a good story (Adam Neumann at WeWork), somebody has money burning a hole in their pocket (Masayoshi Son at Softbank). The story makes for an eye-popping valuation, but the story doesn't make it a rational valuation.

Human history is rife with examples of big, ambitious, up-front design that fails to live up to the hype and reward the capital behind it. I've been doing some research into the development of North America from the time of the Renaissance. The opening of the North American continent created a massive trade opportunity for European countries. Although the Spaniards were the first to arrive during the Renaissance, the French were first to pursue opportunities in the North American interior. The interior was rich with wildlife. The indigenous peoples were adept at harvesting the pelts of that wildlife. European manufactured products could be profitably traded for those pelts: $40 of trade goods to $1,000 of prepared pelts, per the opening quote.

As lucrative as the trade opportunities were, the French dreams for the development of the interior were even more ambitious.

Having seen the immense forests of the Ohio and the broad prairies of the Mississippi, La Salle formulated his life's ambition. He would establish French civilization in the rich country between the two rivers. He would begin by systematizing and enlarging the fur trade, using cargo vessels on the lakes, erecting fur depots on the rivers, establishing a chain of warehouses and magazines. It was a bold undertaking that looked to the protection of French interests all the way from the mouth of the Mississippi to the Great Lakes and the establishing of stations of prestige and power at a dozen strategic points.

[...]

It was a dimly comprehended country, but La Salle saw it more clearly than any man of his age. He made the restless [Governor] Frontenac envision it like a panorama from the Rock above the St. Lawrence. And it made the ambitious governor grasp his program of a commercial empire with French goods going systematically to the strategic posts and the fur caravans drawing in over the great web of rivers to the broad sea lanes of the Lakes. Together they drew up a design of occupation, fortification and settlement.

La Salle's big vision needed big execution. It did not suffer from a lack of big execution. He oversaw the construction of the 60-foot long Griffon, the first ship ever on the great lakes beyond Ontario; a vessel of that size could transport larger quantities of trade goods than the largest fleet of canoes. He established trade relationships with interior peoples. He established forts throughout the present day states of Ohio, Indiana and Illinois, reaching as far west as the Mississippi.

What the big vision did suffer from was unforeseen circumstances, to which La Salle was simply unable to respond. Advance traders squandered their French-made trade goods, trading for themselves rather than for their employer. One of La Salle's trusted lieutenants - Louis Hennepin - was taken prisoner by the Sioux. La Salle's men abandoned his key fortification of Fort Crevecoeur. A key trading partner - a village of the Illinois - was defeated by the Iroquiois. A storm over Lake Michigan claimed the Griffon, laden with pelts from the North American interior for Europe. Ultimately, La Salle's detractors were successful in discrediting him in Montreal and Paris, and his creditors closed in on him. While he would ultimately gain support from the French crown for one final expedition, it would fail for the same reasons his previous expedition did: ambition impervious to ground truths.

Though the ensuring years saw scattered new trading posts and mission stations - at Chicago and St. Joseph, at Detroit, Cahokia and Vincennes - there was no bold design of a French civilization encompassing the whole interior country.

The legacy of the French vision lingers on in the names of rivers, cities and counties that bear the names of the influence of the French explorers and of the explorers themselves: Eau Claire, Fond du Lac and St. Louis; Hennepin, Joliet and Marquette; and of course, La Salle himself. But these are just echoes of the past. As Dr. Havinghurst points out, the North American market for European trade goods fell into the hands of those who were not executing in pursuit of a grand design, but content to muddle through.

No surprise, then, that the 20th century instantiations of these 18th century ambitions - Singer Industries and TRW - if they are remembered at all, it is as legacy place names and not 21st century concerns.

The Fallacy of Composition

That six month experiment with Agile practices yielded off-the-charts results in time-to-market with quality, as well as sponsor and user satisfaction. It cost a lot, but everybody is smitten, and word has gone round the company. Everyone wants to know: what was different? Well, everything: automated build pipelines, stories, stand-ups, showcases, spikes, desk checks, you name it. Ok, great. Let's make sure everybody in the organization is doing those things everywhere, all the time.

That "Agile pilot" was a deliberate investment in the formation of behaviors designed to yield local (that is, team-level) optimization. Sustained, intense concentration on collaboration among consumers, designers, builders, testers and managers resulted in more experiments, more builds, more deployments, and more frequent feedback. This resulted in not only a better software asset, but an outcome that everybody is emotionally invested in.

The immersive nature in which these new muscle memories were developed came at a cost. You took the flack for using unapproved tools. You paired up your staff, effectively doubling the size of the team and more than doubling the run rate. You had to deal with the daily drama of your employees coping with the muscle confusion of learning new ways of working as well as having to deal with strong consultant personalities. A long-time employee quit - and with him, years of tribal knowledge walked out the door - because he was demoralized by the experience. No one seems to remember that your business partner all but demanded that you to pull the plug on the entire exercise just a few weeks in. With success comes selective amnesia.

Still, it was a success, if a success on a small scale. The experience of success in the small makes us prone to gross misunderstanding of what success looks like in the large. It is easy to fall victim to the fallacy of composition: the inference of the properties of the whole from the properties of its parts. There are two obvious ways this manifests itself: in not being aware of characteristics of success at scale, and in not being aware of the characteristics of what it takes to scale it up.

A small product team with no external dependencies is a closed ecosystem, like a terrarium. A terrarium ecosystem is not particularly diverse, and any deficiencies are easily corrected through minor adjustments a manager can directly make through direct intervention. An enterprise-scale ecosystem has a geometrically larger number of dependencies and interactions; these are not very easily compensated for by management-level employees through direct intervention.

Success at scale requires understanding an organization as a whole, through different lenses. "How do we spread these practices throughout the organization" looks at mechanical activities that yield local (team-level) optimization in a fixed span of time. They make no provision for what happens at an organizational level over a long span of time: different parts of the organization will move at different speeds, held back by legacy constraints that are not so easily undone; inter-team and inter-organizational dependencies are not simply solved through wistful promises of "alignment"; people with scarce skill sets cannot be co-located with every delivery team that needs them; how hiring is done and who is hired will determine whether change is sustained and evolves; procurement practices that buy for unit cost optimization rather than outcomes reward vendors for the wrong behaviors; operating company norms that prioritize efficiency at the expense of learning will render feedback loops inert.

In failing to see the characteristics of the organization that are not present in the parts we are looking at, we are blind to the organizational phenomenon that will impair and ultimately reverse the benefits of change.

It is also important to recognize that the characteristics of change at large scale are not the same as the characteristics of change at small scale.

When answering "what is different", the rigor, discipline and commitment necessary to raise the bar tend not to be part of the answer. This separation of "ends" and "means" invites dilution of skill transfer and subsequently dilution of the skills themselves: coaching replaces immersion, decks and self-service wikis replace coaching. In addition, the implicit reduction of risk - having been "proven out" in a pilot, the practices must no longer require a heavy investment - recasts Agile as an operational problem rather than one of learning and growth. It's just a different process from what we follow today, right?

"What is feasible, or beneficial, in the small may be infeasible, or harmful, in the aggregate."¹ Heavy investment in immersive activities at scale - in effect, making an investment to "push" change - is prohibitively expensive at enterprise scale. The sustained period of immersive skill development that was highly beneficial in a single team is infeasible at scale. Diluting change makes it outright harmful, as it can yield false positives of everything from collaboration to quality. That whole "we're going Agile" thing will become a thin veneer over the chronic problems that plague your organization, and once that happens something that could be of genuine benefit is corrupted to a point of uselessness.

The failure to recognize that an enterprise is a geometric rather than a linear degree of difference from a single team dooms a lot of well intentioned change programs. We also see the effect in maturity models and enterprise frameworks that are precise at atomic levels of change but vague or misleading at organizational levels of change. We'll look at the latter in a future post.

¹ Kay, John. Other People's Money: The Real Business of Finance.

Innovation Versus Control

Firms in industries ranging from financial services to retail pharmacy to fast food aspire to be "platform companies." In the minds of their chief executives, the emergence of Amazon and the evident superiority of platform economics make this necessary for their continued survival. It is also a good story to tell Wall Street as it allows a firm to create the aura of being the technology leader in their space while trafficking in the success of companies like Amazon.

"Platform" is conceptually conveyed as a technological phenomenon. But it stands to reason that the defining characteristic of the platform organization is neither the technology assets that they produce (e.g., friction-free consumable primitives), nor how they produce them (e.g., lean and agile process). The benefits of a platform are only yielded if the creativity and imagination of the rank and file can be unleashed through those assets, to experiment, learn, and implement quickly. This means devolving decision rights far down into the organization, a.k.a. autonomous teams.

I've been brushing up on organizational behavior theory, and during my research I came across this paragraph. There is a lot of wisdom condensed into these two sentences:

In general, the longer the time period required for the consequences of strategic decisions to be realized and evaluated, the less flexible are resources for commitment to alternative objectives. Furthermore, (1) the longer the time period in which strategic decisions operate as constraints on the decisions made by technical-level personnel and (2) the lower the complexity of the tasks required to carry out operational plans, the more likely that operational planning will take place at a higher level.

-- Gerald I. Susman, Autonomy at Work

The first sentence neatly captures why things like Agile and Continuous Delivery and Lean Startup are so appealing. We reach critical mass of feedback on a strategic imperative - and therefore judgment on the wisdom of that imperative - more quickly with lots of frequent deliveries of small but business-valuable things than we do with infrequent, large deliveries of comprehensive business solutions. The sooner we reach the inflection point where a body of feedback confirms or contradicts a strategic decision, the more quickly we can move on to the next phase of our strategy, or change course. This separates the sclerotic laggards from the adaptive innovators. In addition, the presence of continuous market intelligence serves to separate the agile and adaptive from the strategic flailers.

This is intuitively obvious, but seeing it in black and white serves as a means test for the fulfillment of business strategy: is a firm asserting, confirming, or just guessing at what the market will buy?

The second part of the paragraph helps us to better understand the organizational dynamics within a small and innovative company versus those within a large integrated program team or an enterprise.

The first part is simple enough: the longer it takes to realize a strategic imperative... Longer is bad, check; already established in the first sentence. The second part is where it becomes interesting: and, the simpler the tasks required to deliver that strategic imperative... This statement is an indictment of the labor carrying out those tasks and the management defining them.

All together, the second sentence tells us that a long-lived initiative expected to be fulfilled through simple tasks relegates executives to the role of supervisor.

This is a damning statement in a number of ways.

The moniker "executive" is highly relative, potentially to a point of meaninglessness. The greater the degree to which technical execution is decomposed into simple tasks, the higher up the responsibility for operational planning. The higher up the responsibility for operational planning, the less meaningful the title of the person doing that planning. C-levels engaged in day-to-day prioritization and resource allocation are not executives. They are mid-level managers who have benefited from title inflation. It also means that the scope of executive decision-making - strategy - is concentrated in just a few hands. This renders quite a few people executives in title only, and deprives a company of its next generation of leadership by stifling their formation.

Anyone touting the potential for innovation from a delivery team engaged in task execution is living in a world of make-believe. Innovation stems from the combination of autonomy and complexity: give a team the freedom to solve a complex problem any way they see fit, and they are likely to come up with something novel. A system based on completion of simple tasks deprives a team of any complexity to sink their teeth into. Additionally, a system of rudimentary task completion is inherently a control system, which has zero tolerance for independent thought or action that is off-plan. Innovation is scarce where control is the priority.

Enterprise-y Agile processes function as systems of control, not innovation. Any system that adjusts the work to suit the labor instead of adjusting the labor to suit the work will require a high degree of centralized control. Enterprise Agile processes tolerate, and even advocate, decomposing work into tasks and assigning them to specialist labor. This values the control of labor over the creativity of labor. Per the previous point, technical-level employees are systemically disenfranchised. A system based on control through tasks offers no leeway for devolved decision rights; the only right an individual has is to complete the tasks they've been told to complete. This makes enterprise Agile processes more prone to suppressing than unleashing innovation.

The dynamics of small teams in small companies are not directly transferable to small teams in large enterprises. Small teams in small companies have high degrees of overlapping responsibility, little tolerance for specialization, light processes, and engage in high-bandwidth, omni-directional communication. Large organizations codify things such as roles and responsibilities, career development, processes, and work (e.g., technology) guidelines, and engage in low-bandwidth, hierarchical communications. In small companies, trust is largely based on the expectation that everybody will do whatever it takes to achieve a common outcome; in large companies, trust is largely based on the expectation that specialized people respond to precise requests with precise responses. Team dynamics are functions of HR structures, organizational values and systems, communication patterns, and ingrained behavior patterns, all of which are highly resistant and even subersive to change when they have decades to develop within a company. The executive in a legacy enterprise who says they want to transform the company into a "start-up" betrays their naïveté of the magnitude - and unlikeliness - of that task.

The paragraph at the beginning of this post captures what many in the tech biz have experienced for decades. Since the 1990s, enterprise IT has been a story of scale. As it scaled, it became more prominent on the income statement, and was forced to place a premium on control. Occasionally, it basks in the reflected glory of innovative consumer technology firms, or gets elevated by a CEO as a source of untapped potential. Unfortunately, enterprise IT has never been able to reconcile an expectation for innovation with the fact an over-emphasis on control gives everybody in management a demotion, suppresses innovation, and stifles attempts at organizational renewal, all while holding a company back from fulfilling its strategic potential because it takes such a long time to get anything done.

The most interesting thing about that paragraph? It was first published in 1976. Industrial, not tech firms, were the prominent companies of the time. The lessons remain the same.

The Patrol Method and Objections to Self-Directed Agile Teams

In the previous post, we saw there are quite a few similarities between the Patrol method and self-directed Agile teams. It stands to reason that the resistance, doubt and objections faced by each from sponsors, leaders and members alike will be very similar. If that's the case, one can learn from the other.

These excerpts from the 1950s edition of the Scoutmaster's Handbook will sound familiar if you've ever tried to implement a self-directed Agile team:

"Some don't grasp the possibilities of the patrol method, and subsequently don't see the importance of it." If you can't appreciate the adaptability and agility of an organization characterized by strong capability and strong leadership throughout the ranks - one where people are continuously completing, learning, and adjusting to an extent that they perform best with team-level autonomy rather than top-down hierarchy - you won't see the value in a method designed to bring that about.

"Some lack faith in boys' ability to carry out responsibilities." You have to be willing to trust in people's capacity to both learn and make good decisions. Of course, trust is earned and not given, and while it takes time to develop it takes only seconds to erode. But an organization is a lot more efficient when it gets by on trust, because trust requires far less supervision and administration. Again, you have to be committed to what the right organization can deliver, and not make the organization and it's people hostage to the deliverables themselves.

"Some give up if it doesn't function perfectly right off the bat." Self-directed teams are a departure from traditional command-and-control hierarchy. The initial experience - and the initial results - can be very poor if self-direction is adopted swiftly and suddenly: self-direction requires very different behaviors and these can take a long time to develop. Until they do, teams will experience mission confusion as they come to grips with new expectations, interruption caused by external dependencies and boundaries they now have the responsibility to negotiate and manage, and seizure as people struggle to come to terms with responsibility for an entire solution rather than discreet, small tasks. How the team struggles with Tuckman's stages of group development will be mirrored in its results: at best a roller coaster where results are up one iteration then down the next, at worst a flatline where they struggle to get anything across the line. If we're not cognizant of (or better still, actually measuring) the development of new organizational "muscle memory", then the appearance of chaos within the team twined with few deliverables to show will cause the faint of heart to bail before the team gets through forming, norming and storming, to actually performing.

"Others don't like to part with authority. They found a chance to play and show off their specialties and don't realize they're stifling leadership development." Some people responsible for delivery of large programs or simple projects may be more comfortable concentrating authority rather than distributing it. This is an indication that they're more worried about the success of what they're responsible for than they are building up the people who can successfully deliver. While they may be good executors - a safe pair of hands you can trust to get something across the finish line - they're not good organization builders who can sustain what they create. Curiously enough, while self-directed teams are often accused of "not scaling", it is the characteristics common to command-and-control hierarchies - asymmetric knowledge distribution and concentrated decision-making - that don't scale.

"Still others have the old idea of training by mass instruction too ingrained in their systems to change." The best way of learning is to do something and not just study it. But on-the-job training can be very expensive, especially if somebody does, and re-does, and re-re-does, and still doesn't get a basic grasp of what it is they're doing. There is a certain allure to separating skill acquisition from skill application, if for no other reason than we can measure the effort spent on skill acquisition in the form of number of hours spent in classroom training and number of people who earn certifications. Things like training and certifications are proxies for competency: if we earn the certifications we will know what we're doing which will jump-start our execution. This makes people feel good about progress toward plan. But giving training to an individual and that individual demonstrating competency are entirely different things. The prior is useless without the latter.

"Also, some by temperament aren't suited to this way of training and are happier in a system other than the Patrol method." Quite a few people can't function without a strong hierarchy. Asking them to perform in a system defined by collaborative teams of peers who are self-directing themselves toward reaching a goal is an unreasonable expectation, and potentially damaging, to somebody who simply can't function that way.

From this, we can glean the characteristics a leader committing to the formation of self-directed teams needs to have if they are to succeed.

Humility - Self-directed teams are about its members, not its leaders.

Patience - A self-directed team will suffer setbacks and disappointments, particularly in the early going. The point isn't to obsess about failure, but to make sure every member of the team learns from it and doesn't make the same mistakes again.

Faith in people - If you believe people need to be told what to do, rather than can figure out what it is they need to do, you'll not get far with self-directed teams.

Belief in the wisdom of crowds - You have to believe that the whole is greater than the sum of the parts, and that a hive-mind will come up with a better solution through execution and continuous feedback than one dictated to them by a small cabal of people.

These prior two points suggest that you have to espouse a professional rather than an industrial mindset. If you can do that, you have the mindset for self-directed teams.

Commitment to the method - Above all else, you have to believe that if faithfully applied, the method will create the conditions - generalized skills, strong leaders, and independently-functioning units - under which a program is far more likely to succeed than if it is delivered in a command-and-control style. If you lack confidence in the values behind the method, you'll be quick to abandon practicing the method.

"The troop that is run as a club, with the Scoutmaster as boss, dies when the boss leaves." Hierarchies with central decision-making can be effective, but they are brittle because of their dependence on a handful of key people. If your goal is to build a resilient, evolutionary, adaptive organization, the price of admission is decentralization. Decentralization requires empowerment; empowerment requires atomic leadership and capability. The history of organizational development teaches us that the process of building an organization that can function this way is a very difficult one indeed.