I consult, write, and speak on running better technology businesses (tech firms and IT captives) and the things that make it possible: good governance behaviors (activist investing in IT), what matters most (results, not effort), how we organize (restructure from the technologically abstract to the business concrete), how we execute and manage (replacing industrial with professional), how we plan (debunking the myth of control), and how we pay the bills (capital-intensive financing and budgeting in an agile world). I am increasingly interested in robustness over optimization.

I work for ThoughtWorks, the global leader in software delivery and consulting.

Friday, November 30, 2018

The Fallacy of Composition

That six month experiment with Agile practices yielded off-the-charts results in time-to-market with quality, as well as sponsor and user satisfaction. It cost a lot, but everybody is smitten, and word has gone round the company. Everyone wants to know: what was different? Well, everything: automated build pipelines, stories, stand-ups, showcases, spikes, desk checks, you name it. Ok, great. Let's make sure everybody in the organization is doing those things everywhere, all the time.

That "Agile pilot" was a deliberate investment in the formation of behaviors designed to yield local (that is, team-level) optimization. Sustained, intense concentration on collaboration among consumers, designers, builders, testers and managers resulted in more experiments, more builds, more deployments, and more frequent feedback. This resulted in not only a better software asset, but an outcome that everybody is emotionally invested in.

The immersive nature in which these new muscle memories were developed came at a cost. You took the flack for using unapproved tools. You paired up your staff, effectively doubling the size of the team and more than doubling the run rate. You had to deal with the daily drama of your employees coping with the muscle confusion of learning new ways of working as well as having to deal with strong consultant personalities. A long-time employee quit - and with him, years of tribal knowledge walked out the door - because he was demoralized by the experience. No one seems to remember that your business partner all but demanded that you to pull the plug on the entire exercise just a few weeks in. With success comes selective amnesia.

Still, it was a success, if a success on a small scale. The experience of success in the small makes us prone to gross misunderstanding of what success looks like in the large. It is easy to fall victim to the fallacy of composition: the inference of the properties of the whole from the properties of its parts. There are two obvious ways this manifests itself: in not being aware of characteristics of success at scale, and in not being aware of the characteristics of what it takes to scale it up.

A small product team with no external dependencies is a closed ecosystem, like a terrarium. A terrarium ecosystem is not particularly diverse, and any deficiencies are easily corrected through minor adjustments a manager can directly make through direct intervention. An enterprise-scale ecosystem has a geometrically larger number of dependencies and interactions; these are not very easily compensated for by management-level employees through direct intervention.

Success at scale requires understanding an organization as a whole, through different lenses. "How do we spread these practices throughout the organization" looks at mechanical activities that yield local (team-level) optimization in a fixed span of time. They make no provision for what happens at an organizational level over a long span of time: different parts of the organization will move at different speeds, held back by legacy constraints that are not so easily undone; inter-team and inter-organizational dependencies are not simply solved through wistful promises of "alignment"; people with scarce skill sets cannot be co-located with every delivery team that needs them; how hiring is done and who is hired will determine whether change is sustained and evolves; procurement practices that buy for unit cost optimization rather than outcomes reward vendors for the wrong behaviors; operating company norms that prioritize efficiency at the expense of learning will render feedback loops inert.

In failing to see the characteristics of the organization that are not present in the parts we are looking at, we are blind to the organizational phenomenon that will impair and ultimately reverse the benefits of change.

It is also important to recognize that the characteristics of change at large scale are not the same as the characteristics of change at small scale.

When answering "what is different", the rigor, discipline and commitment necessary to raise the bar tend not to be part of the answer. This separation of "ends" and "means" invites dilution of skill transfer and subsequently dilution of the skills themselves: coaching replaces immersion, decks and self-service wikis replace coaching. In addition, the implicit reduction of risk - having been "proven out" in a pilot, the practices must no longer require a heavy investment - recasts Agile as an operational problem rather than one of learning and growth. It's just a different process from what we follow today, right?

"What is feasible, or beneficial, in the small may be infeasible, or harmful, in the aggregate."1 Heavy investment in immersive activities at scale - in effect, making an investment to "push" change - is prohibitively expensive at enterprise scale. The sustained period of immersive skill development that was highly beneficial in a single team is infeasible at scale. Diluting change makes it outright harmful, as it can yield false positives of everything from collaboration to quality. That whole "we're going Agile" thing will become a thin veneer over the chronic problems that plague your organization, and once that happens something that could be of genuine benefit is corrupted to a point of uselessness.

The failure to recognize that an enterprise is a geometric rather than a linear degree of difference from a single team dooms a lot of well intentioned change programs. We also see the effect in maturity models and enterprise frameworks that are precise at atomic levels of change but vague or misleading at organizational levels of change. We'll look at the latter in a future post.

1 Kay, John. Other People's Money: The Real Business of Finance.

Wednesday, October 31, 2018

Everyone a Beginner?

I had an exchange with Dan North about the subject I wrote about last month, Beginner's Mind. Dan asked an interesting question: what would it take to make work like this every day for everyone, everywhere?

It's a serious question that deserves serious consideration.

To start, it's worth asking: why isn't work like this today, every day, for everyone, everywhere?

There is a difference between development companies (or divisions within companies) and operating companies. A development company is in the invention or innovation business. Investors and customers place high value on seeing something new, be it features or entirely new products. There is a high tolerance for operational inconsistency from a development company. Recall of how tolerant (or at least, how much less snarky) people were towards Windows when Windows 95 was launched, or the iPhone when the 3GS was launched.

An operating company is in the consistency business. Investors and customers place high value on efficiency and reliability. There is low value in seeing something new as it interferes with set patterns and expectations. There is low tolerance for operational inconsistency. To wit: the ribbon interface that replaced the command menus in Microsoft Office 2007 was not met with enthusiasm.

A company can be both a development company and an operating company. For example, Tesla. Inventing battery technologies and designing cars that are sufficiently different from the cars anybody else has made captures the imagination of financier and customer alike. But they have struggled at the basics of manufacturing things at a modest scale. Manufacturing cars is a path well trod, so nobody places a lot of value on Tesla employees learning things already very well known by people in the auto industry.

There is still room for engaged employees in both development and operating companies, but there is a difference in the nature of their engagement. We want passionate people in the development company, people who will look for ways to deliver value where there was none before. But we don't really want passionate people in utility-type businesses: the passionate accountant who finds ways he believes his clients can dodge the taxman is not beneficial to his clients in particular or society in general. What we do want in operating companies are obsessive people looking for ways to make the core transaction between buyer and seller more efficient or incrementally more delightful without materially altering the nature of the core transaction itself. My unexpectedly very late arrival at the hotel has made it impossible for me to get dinner, but an availability of fresh fruit and healthy snacks free of charge not only satiates my hunger but makes me feel the hotel staff is concerned for their guest's well being. It's still a room with a bed and a bath, but I'm more likely to choose that hotel again in the future because they strike me as being prepared for irregular ops situations experienced by their customers.

There is a fundamental difference between these types of employee engagements. Knowledge of something that customers have never thought to do or thought possible is in the "voyage of discovery" realm. Knowledge of something that customers have long known possible and wondered "why can't / won't they do this" is in the "pay attention" realm. That makes prior questions of skill; the latter questions of will. A lot of things in operating companies fall into that latter category: a decision by a customer service person not to do something is a choice the firm they work for makes, either on the spot (by the employee) or as a matter of policy (by management).

But this shouldn't matter, should it? Either way, the employee is learning. What difference does it make?

It matters from the point of view of both the learning intensity as well as the freedom to act on what has been learned. There is less opportunity for learning and less value placed on it by investors and customers in the operating company realm, hence less opportunity to act on it. And, unfortunately, operating company jobs are the bulk of the jobs people hold today.

The business of software is somewhat unique in that the learning intensity and freedom to act are extremely high, precisely because we get to take voyages of discovery all the time. Sure, learning Pascal was something millions had done before I came along. But my ability to apply that knowledge to do things that had not been done before created a very learning-intense environment: the better my skills in Pascal, the different the class of problem I could solve with it, the more new ground I could clear.

That's great for those of us in the business of software. How do we meet the "for everybody, everywhere" standard?

The opportunity will first present itself by being able to get more people out of rote operating company jobs. But that isn't enough. Yes, in theory I don't need any human interaction at the hotel: I can check-in from my phone and use it as the room key and if I opt out of housekeeping services and order food through Seamless I would never cross paths with any hotel employee. All that does is displace more hotel staff and justify fears of our robotic future and a return to the bad old days of industrialization concentrating wealth and suppressing working classes circa the first half of the 19th century.

There is benefit to having people in opco jobs, and not just to supervise the robots. What if opco employees had tools they could use to solve different classes of problems? Not transactional tools, not canned reports that a back-office accountant can study, not 21st century green screens that clerks can use to sell an upgrade. How about tools that are (a) granular; (b) modular; (c) "programmable" by the user - where the user is not the customer, but an employee; and (d) re-composable. Opcos would be able to provide the operational consistency that investors and customers expect, while enabling a work force with tools allowing them to be less obsessed with efficiency and more passionate about solving customer problems.

What if opco employees could re-program the environment, directly, themselves?

That would certainly bring more people on the same "voyage of discovery" that people in software development enjoy. It would redefine the relationship between developers and consumers as peer solvers of problems of different granularity, not producer-user of a confined operational space. And that would mean less tilt in the playing field of the future, and subsequently less inequality.

How do we make it possible for everyone, everywhere to experience the beginner's mind we get to experience every day in software development? By treating them as problem solvers, not operator-executors.

Sunday, September 30, 2018

Beginner's Mind

A little over a decade ago, I was part of a team introducing new practices to a 200+ person, distributed development team. A lot of people were too quick to understand them. Some participated in stand-up as if it were a one-way status meeting. Others were still creating long-lived branches in Perforce as if they were still using ClearCase. "Stories" was just a new word for the technical tasks they had always assigned. Two people I was working with at the time pointed that everybody would be well served to approach these new practices with "beginner's mind."

* * *

While on vacation in Western Ontario Province a few years ago, my wife suggested we visit an amethyst mine. I had distant memories of doing some lapidary work with my dad back in the 1970s, so I picked up enough small stones for tumbling. Over the years, my dad had grown serious enough about the hobby to build out a considerable workshop of tools, slabs and rough. When I told him that I had picked up a few pounds of amethyst, he gave me a spare rotary tumbler. Five weeks later, I had my first batch of polished stones in nearly 40 years.

I was also disappointed. The polished stone looked good but not great. Why did so many have fractures and chips?

I did some research and tumbled some more stones and learned that to get better results I would have to improve the quality of the rough as well as my technique for tumbling.

To improve the rough, I learned to recognize characteristics of amethyst crystals that were more likely to fracture in tumbling, and excluded these. I did a full inspection after every stage of the tumbling process and removed any stones showing signs of damage so that a shard would not float around the barrel and contaminate other stones in the batch.

I had to improve my technique, too. Stones take quite a pounding in a rotary tumbler. While this action erodes sharp edges off the stones, it also increases the probability of damage. I learned that ceramic media in the barrel can cushion the impact. I realized that the barrels can be difficult to completely clean; even a tiny piece of previous-stage grit can contaminate a batch, so I obtained different barrels for different stages of grit. I also questioned the 7-days-per-each-stage tumbling recipe: what's so special about a week? I sampled stone in the tumblers once per day, comparing it to stone that had been in for a full week, and found I could reduce the amount of time spent in every stage.

Eventually, my dad gave me a lapidary arbor so I could grind the rough into shapes that would tumble more successfully. Later, he gave me a lapidary saw so I could cut fractured stones, allowing me to harvest more from the rough: instead of grinding away one half to save another, I could cut a stone along an obvious fault line and shape each part into a stone that was more likely to tumble successfully.

I quickly realized that the saw and arbor would let me do more than prepare stones for less bad tumbling outcomes. After a bit of research led me to cerium oxide belts and expanding drums, I figured out how I could hand-shape stones from rough to polish without using the tumbler. Hand shaping a stone is time consuming, but far less damaging than tumbling. There is risk: what appears to be innocuous sanding can expose a deep void, and careless handling on the arbor can result in a shattered stone. Still, when I have a very special rough stone, I can shape and finish it by hand, sanding out imperfections while retaining its unique characteristics.

The lapidary world is not too interested in making unique shapes out of small rough on saws and arbors; it is primarily concerned with creating faceted stones or cabochons (domed shapes in fixed sizes). To be serious about lapidary work, I would have to make one or the other or both. I had the tools for cabs, so over the course of several months, I taught myself how to slab, trim, dome and polish cabochons. I had some jewelry findings from my dad and acquired a few more of my own to set the cabs into as finished jewelry pieces.

It occurred to me that what makes lapidary work so satisfying to me is that I approach it with beginner's mind every day that I get to do it. Every stone I work on, every technique I follow, every tool I use is an experiment and therefore a learning opportunity. Through the simple, frequent act of trying lots of different things I have made hundreds of little adjustments to my workshop, my tools, and my methods. The net result is I am more efficient, more accommodative to failure and mistake, have higher throughput and better product quality today than I did at any point in the past.

And I'm still learning. Every day.

The joy of the lapidary work is not the final product. If it ever became that, it would be a chore, a job, and it would lose its allure. The joy of the lapidary work is the never-ending discovery. Some days I work in the workshop, some days I work on the workshop, but no matter which it is I learn something every day, and every day I take action on what I learn.

* * *

In the business of software development, the primary focus is on the product: these features, this timeframe, this cost. When the product disappoints - it costs too much, it takes too long, it isn't reliable, it is difficult to use, it doesn't do what it's supposed to do - the process of how it is made gets as much attention as the product we're trying to make. Those times of focus on the process lead people to investigate things like Agile and Lean and continuous delivery. Sometimes, to people unfamiliar with them, Agile and Lean and continuous delivery appear to be magic.

There is nothing magical about things like Agile and Lean and continuous delivery. The only magic is the (re-)experience of beginner's mind when somebody is first exposed to them. They force us to re-examine the orthodoxy we have long accepted - for reasons we no longer remember - for how we work. They let us break out of the psychic prison that is the modern operating company environment.

Despite containing elements of continuous feedback that drive continuous improvement - build pipelines, retrospectives, experiments - the change to Agile or Lean or continuous delivery does not typically result in a permanent state of beginner's mind. They captivate only for the short period when people care about process: we're broken, we need a fix, this process looks like a fix, let's adopt this process. As important as change may be at the time, product eventually triumphs over process, while change fatigue casts a pall over attitudes and budgets alike. Management wants mastery of whatever mechanical processes they've agreed to; they're not interested in perpetual, indefinite refinement.

Dan North has long argued that first and foremost a company should aspire to be a learning organization. A propensity for organizational learning underpins a lot of well established management theory and more recent instantiations of it, things like autonomous work teams, Agile practices, and Lean startup. If the real objective is not to become a learning organization, these will be little more than mechanical acts that serve only to make software development less bad.

There is merit in being less bad. There is value in fleeting moments of joy from reconnecting with our younger, knowledge-acquisitive selves. But change that does not aspire to rewire people permanently to their beginner's mind does nothing more than make a job less of a slog. It is still a job.

It is not a passion.

Friday, August 31, 2018

Organizing for Innovation, Part VI: Putting It All Together

As we saw in the previous posts in this series, organizations of autonomous teams can scale. Scaling requires different team characteristics (requisite variety, redundancy of function, double-loop learning and minimum critical specification), a different mental model of organization (brain, not machine), a different kind of hierarchy (purpose, not control), and a different style of leadership (guide, not command).

This sounds like it would be chaos in practice on a small scale, let alone enterprise scale. And even if it does work, it sounds like a revolutionary approach to organization. Visualizing it helps explain how all these things combine to create an organization that innovates as well as operates.

The organization of autonomous teams is not chaos. Teams are invested with authority, communication pathways develop along the hierarchy of purpose, the structure adapts itself to the domain, and organizationally-driven objectives plus team incentives align behaviors and outcomes. Of course, this looks nothing like traditional organization, in which hierarchy is the principal means of control, communication, and alignment. Still, while an organization of autonomous teams may be unfamiliar and conceptually unsettling, it isn't chaos.

And, although it may seem revolutionary, it isn't new. Academic research on organizations of autonomous work teams dates to the 1950s, and the early adopters of it were industrial firms. While it may be a way of working that a lot of tech companies happen to adopt for themselves, it is not an organizational phenomenon that has sprung out of tech. So it isn't all that revolutionary, either.

It may not be chaotic or revolutionary, but it is a big leap from how just about every enterprise functions today. The way they work reflects the board's priority for the company. I've written before that companies are financial phenomenon more than they are operating phenomenon. Executives tasked by their boards to prioritize returning cash to investors will look to maximize cash earnings (EBITDA) and free cash flow. In so doing, those executives create an environment where managers must be more concerned with costs than creativity from operations. Managers, in turn, condition employees and contractors to value activity over learning, output over outcomes, and narrow individual independence over broad group autonomy. In this environment, employees, managers and executives are rewarded for every dollar not spent for output; they will not be rewarded for any dollar spent on learning.

With enterprises increasingly feeling the heat from new companies, technologies and products, executives charge managers with extracting more innovations from the business. Managers go about looking at changing ways of working, recognizing that innovation is partially a byproduct of "how" things are done. But as long as the priority of the board is to return cash to investors, the first mission of "how" things are done is to be predictable, because predictability ensures consistent cash flows and consistent cash flows ensure that interest payments, dividends and buybacks can be made while protecting the bond rating. The autonomous team structure is incompatible with predictability.

The autonomous team structure is internally consistent (not chaos) and been around for long enough with enough successes (not revolutionary) that it can succeed, even at scale. Pursuing it is ambitious, and operationalizing it is plenty difficult. But success with it has less to do with operationalizing than it does with the tolerance for it in the capital structure in which it operates. Autonomy - that is, abdication of centralized control - is the price of admission for innovation. Innovation at scale requires autonomy at scale. Autonomy at scale requires the board be committed to innovation, not cash returns, as their top priority.

Tuesday, July 31, 2018

Organizing for Innovation, Part V: The Leadership Challenge

Organizations of autonomous teams require a different set of behaviors than organizations that are run like a machine. People in a self-directed team form their own appreciations for what should be done, prioritize what will be done, and self-determine how it will be done. They are unencumbered by hierarchy, expected to communicate with anybody in the organization they need. They are unencumbered by role definition, as people simply do whatever work is required even when that means acquiring new skills or knowledge. They are unencumbered by organization, as teams form task-forces to solve for problems that are beyond the scope of any single defined team.

This all sounds great. Who wouldn't want to work this way?

The majority of people working in enterprise IT today, that's who.

Enterprise tech labor is highly codified (bounded responsibilities) and stratified (seniority). Aside from the fact that this serves the interests of a multitude of non-tech corporate functions such as human resources, vendor management and finance, it also provides a great deal of comfort to the individual. The employee knows very precisely what is expected of them to earn a salary increase or advance their career, while the contractor knows what they are obliged to do to satisfy the terms of their contract. Over time, jobs become working annuities that require little servicing (such as skill acquisition or excessively long working hours) for comfortable levels of compensation with job security (by e.g., being the only ones familiar with a technology, service or function).

The autonomous team environment is anathema to this. For starters, the autonomous team operates with a lot of ambiguity. New appreciations change the team's priority constantly, meaning there is no deterministic plan. Plus, people adapt themselves to the work that needs to be done as opposed to working in strict swim-lanes. An autonomous team environment implicitly subordinates the traditional goals of the individual: the success of the team is success for the individual. An autonomous team breaks down when its members put individual achievement over team accomplishment.

Of course, working this way is a question of both skill and will. Whether labor can re-orient itself from machine to autonomy is a debate well beyond the scope of any blog post, but suffice to say that some - few, several or most - will not be able to make the transition from doing what they are told to do by management, to figuring out for themselves what needs to be done and doing it. In addition, whether labor willingly chooses to re-orient itself is another matter entirely. Over-exposure to enterprise change programs - or more likely, over-exposure to failed enterprise change programs - won't inspire enthusiasm for yet another one. There is also the suspension of disbelief people need to make to give devolved authority and autonomous team structures a try.

This brings us to the leadership challenge that creating an organization of autonomous teams poses. Obviously, there are institutional barriers that have to be overcome. HR has to be comfortable with ambiguous roles and titles. Vendor contracts for supply of specialist labor have to be replaced with contracts tied to business outcomes. Finance needs an alternative to predictive planning and financial budgeting.

However, the challenge to leadership goes beyond mechanical processes and structural changes. In an organization of autonomous teams the nature of leadership changes from giving commands to guiding actions. The leader does not direct people's performance to achieve a goal (organization-as-machine), but instead projects a goal that people direct themselves toward achieving (organization-as-brain). The leader in the autonomous organization makes use of:

  • Concrete statements of expectations: leaders have to make expectations crystal clear. Saying "all enterprise software is deployed into production every other week" is a clear expectation. Saying "we will be a continuous deployment organization" is a woolly statement: "continuous" will be interpreted relative to the current skills and learned helplessness of the people responsible for enterprise software today. The latter statement also misses the point: how the organization functions (continuous deployment) is less important than describing what it achieves (new software released across the board every other week.) The prescriptive implications of that statement deny the people in the organization the opportunity to figure out for themselves how best to achieve the goal. The leader communicates the expectation; it is up to the people in the organization to figure out how to achieve it.
  • Rewards and recognition: obviously, what gets measured is what gets managed. If the goal is to increase frequency of deployments, reward the teams that find ways to make reliable production releases weekly, then twice weekly, then daily, then multiple times per day. Story-telling is important as well, especially as an organization adopts new behavioral norms and its business partners develop new expectations of it. For example, in its early days, FedEx executives were fond of repeating the story of the junior employee who rented a helicopter on his personal American Express card to fly him to a mountain location to repair snow-damaged phone lines so that they could service remote customers. When you're building a reputation for absolutely, positively getting packages delivered overnight, stories like this go a very long way.
  • Acknowledging and solving constraints: every team, not to mention the organization as a whole, will be short of the capacity, capability and capital to achieve every organizational goal. While constraints force institutional responses such as prioritization and innovation, they are also opportunities for a systemic change. It is the leader's obligation to work with constrained teams to identify the actions they can take today so that they will be less constrained in the future. For example, if there is a capability constraint, what can the team do now to develop skills and knowledge so that it can do more for itself? If a financial constraint, how can the team build a stronger business case or define a set of experiments to explore potential value? In the face of constraints, the leader's responsibility is to develop an organization's ability to learn how to learn.

At the foundation of the organization of autonomous teams is Theory Y. The aspirational leader of an organization of autonomous teams has to believe that people are motivated, responsible, and do not need close supervision. If you do not fundamentally believe this, do not bother pursuing an organization of autonomous teams. Once you revert to command-and-control (Theory X), you have lost the mantle of leadership in a devolved organization.

In the next installment in this series, we will put all of these concepts together to visualize what the autonomous organization looks like at scale.

Thursday, June 28, 2018

Organizing for Innovation Part IV: Autonomy at Scale

It is easy to understand how a small organization of autonomous teams can function. When there are only a few teams, there is a small community, and it is simple for people to communicate with one another in both formal and informal ways.

It is not difficult to see how a large organization of largely independent teams can scale. For nearly 60 years, Gore-Tex has shown that devolved authority can work just fine at scale.

Autonomy scales at Gore-Tex because there is very little overlap between outerwear and dental floss, and subsequently less coordination. What happens when the business becomes complex, with lots of interdependencies among lots of teams?

Dependencies by themselves do not make it difficult to understand how devolved authority can work on a small scale. If there are 4 teams, there are a maximum of 6 communication pathways (n x (n - 1) / 2). Even if there are transitive dependencies, the small size of the community - 4 tech leads, 4 product owners, etc. - makes cross-functional communications relatively easy. But a technology organization of tens of thousands of people will have hundreds of teams - and therefore an extraordinarily large number of communication pathways. Translating a small number of enterprise goals into hundreds of millions of synapses sounds like opacity at best, chaos at worst.

There are four things that an organization of autonomous teams needs if it is to scale.

The first is an implicit hierarchy, but one of purpose rather than control.1 Traditionally, hierarchy is meant to control activity througyh supervisory responsibility and assigned decision rights: the subordinates in one division take direction from superiors in the same division. If, per the initial blog post in this series, decision rights are devolved, "span of control" does not exist in the organization of autonomous teams.

Hierarchy also influences communications. If those "higher up" in the hierarchy use the things produced by those "further down", there is an obvious pattern of communications between producers and consumers. In manufacturing, there might be different teams independently assembling subsystems such as brakes or drivetrains from individual component parts. Each of their subsystems might then be consumed by teams on the line installing them into more comprehensive systems (such as the powertrain), and, ultimately, the finished vehicle itself.

This is called a "hierarchy of purpose." The hierarchy is constructed largely around degrees of granularity. Brakes and drivetrains are smaller assemblies that form larger subsystems that contribute to the finished product. A provider of cloud-based infrastructure such as Amazon can organize in the same way: the "finished product" of a cloud instance consists of more "primitive" components of virtual storage, server and network. Each of those subsystems in turn consists of more finely grained primitive components of network protocol communication, CPU, load balancing, and so forth.

Creating complex higher-order offerings as composites of lower-level capabilities is the “platform effect” of innovation.

It's worth mentioning that enterprise program management has long tried the same structure. It chokes on itself when transitive dependencies that extend several layers deep expose the difference between a boundary in work (a team has exclusive responsibility for producing something used by many other teams) and a boundary in fact (the output is high-touch service, support and maintenance, making those boundaries porous). A primitive component must be consumable in a friction-free manner.

This is a logical transition to the second thing the organization of autonomous teams requires, which is practical patterns of communications. The platform effect scales effectively because consumption patterns are the de-facto communication patterns within the organization. In a hierarchy of purpose, the organization does not have hundreds of millions of communication pathways, because the number of consumers is limited. A cloud instance may consume a load-balancing primitive, but it does so through a more coarsely grained "network" intermediary.

The decision classes introduced in the initial post in this series act as a control system for each individual team. The wider the divergence of type of consumer, the more difficult it is to create patterns of demand and prioritize to form a product strategy. The nested team organization limits the divergence of consumer demand, which creates a narrower range of appreciations, which make cohesiveness of strategy and execution at an individual product level far easier to perform.2

The autonomous-team collective maintains enterprise cohesiveness by virtue of its communication patterns. To understand how, we have to revisit the devolved decision classes. Appreciations (what should be done) act as the organizational system for managerial decisions (what can be done); managerial decisions function as the instrumentation system for appreciations. Managerial decisions, in turn, act as the organizational system for technical decisions (how will it be done); technical decisions, in turn, act as the instrumentation system for managerial decisions.

Scope Nature Organizes Instruments
Appreciations   What should be done?   Managerial    
Managerial What can be done? Technical Appreciations
Technical How will it be done?   Managerial

How this works effectively in practice becomes clear when we look at the characteristics of a single autonomous team that we saw in the last post. The transmission of minimum critical specifications through a hierarchy of purpose limits the noise and confusion. The reception of minimum critical specifications from multiple consumers are interpreted as appreciations through double-loop learning. The relationship of control systems and instrumentation systems of the different decision classes categorizes the information appropriately.

The third thing required for an organization of autonomous teams to function at scale is the ability to handle extraordinary patterns of communication. As there is no hierarchy of control, the organization needs protocols that allow it to adapt itself to a changing problem space, as well as to resolve inter-team conflicts.

A dynamic problem space is addressed through task-forces, which may be short- or long-lived, depending on the nature of the need or opportunity. Task forces are formed organically from members of affected teams to solve for problems that are existential to any one team. For example, suppose three teams conclude that they need a new class of capability that is outside the boundaries of all of them. They could elect to form a "task force" in the form of a long-lived team, staffed by reassigning members of their respective teams to this new one on a full-time basis. Not all task-forces are long-lived and full-time, of course: a task force to address a simple challenge might require each person commit only 1 day each week for a month, allowing them to otherwise remain focused on their line responsibilities.

Another other exceptional form of communication are inter-team conflicts. For example, team A elects not to prioritize something really important to team B, and team B lacks the resources or capability to do it for itself. Without patterns for extraordinary circumstances, it would end with a very grumpy team B. Inter-team conflict is handled through mediation and adjudication protocols. At the first stage of mediation, an acceptable 3rd party - that is, someone who is not a stakeholder in the conflict - is asked to mediate a decision. If the conflict is still not resolved following the initial mediation, a committee of 3rd parties are formed to arbitrate a decision. This provides a fair hearing among peers without the need to resort to authority given through a hierarchy of control.

Finally, the organization of autonomous teams needs mechanisms for aligning the strategic goals of the organization with team and individual execution. Appreciations provide connective tissue among strategic, team and individual goals and objectives, but they still need to be reinforced by executive management. Intermediary (layered) objectives have a tendency to change the interpretation of organizational goals, and therefore the goals of each individual.

Alignment is achieved by telegraphing executive intent throughout the organization. There are techniques popularized by a number of firms - OKRs, V2MOM, and the Big Bets Spreadsheet - and probably many others. Whatever the mechanism, their purpose is to communicate unambiguous goals to give each individual a clear means of reconciling a decision - why, what and how - with an outcome that advances the organizational goals. This creates direct line-of-sight between effort and result - and therefore tactical action with strategic outcome - and allows the organization of autonomous teams to function without an excessive number of people in low-value supervisory roles.

With the right set of characteristics, then, an organization of autonomous teams can reach scale, even in complex environments. But it clearly has a vastly different operating model to the traditional control style imposed over the machine-like organization. Is the labor force equipped for this? Is leadership? We will look at these questions in the next post.

1 Susman, Gerald. Autonomy at Work: A Sociotechnical Analysis of Participative Management Prager Publishers, 1976.

2 There is no guarantee of control, of course. An individual team can still thrash or produce unwanted product.

Thursday, May 31, 2018

Organizing for Innovation, Part III: A New Metaphor

Before looking at autonomy at scale, we need a different understanding of what an organization is. This matters because the way we perceive an organization will determine our interpretation of what "good" and "bad" look like.

Some years ago, I wrote that management thinking is still dominated by the "Freds": Frederick the Great, who's Prussian military structure became the model for the modern organization; and Frederick Taylor, a pioneer of scientific management. Frederick the Great organized his military to function like a machine. Frederick Taylor optimized performance down to the task level. The organization-as-machine metaphor has dominated management thinking for well over a century.

A machine is optimized to require the least amount of labor to produce the highest volume of throughput possible with minimal waste. Machines are designed such that each of its component parts performs specific tasks in a consistent and repetitive fashion. When organized into an orchestrated flow of execution, a machine will yield a high volume of consistent output.

Machines are made for optimal performance within a limited range of environmental variation; they are not adaptive to their environments. If something changes in the internal or external environments, the machine will perform at a less-than-optimal state. An exceptional condition to those the machine was designed to operate in will cause an error in execution. Exceptions are subsequently a source of inefficiency to a machine: the machine's purpose is the consistent and repetitive execution of tasks; exceptions inhibit that execution. An exception must therefore be contained so that the machine returns to efficient execution as quickly as possible.

If a lot of things change in the environment, the machine will completely break down. The machine is not designed to intelligently adapt, it is designed to single-mindedly execute.

The organization managed as a machine may achieve optimal performance, but at the cost of adaptability. We need a different metaphor for the organization: without one, we're doomed to end up with what we've already got. If we see an organization as a machine, we will define it in terms of efficiency: what gets measured is what gets managed, and when we apply old-style thinking we come to rely on old-style managing. If the organizational goal is innovation, we need the organization to have the characteristics described in the previous post in this series: it must be attuned to its environment, and it must be highly adaptable to it.

An more appropriate metaphor for the adaptive organization is to think about the organization as a brain. The brain is a highly adaptable organ:

  • While different regions specialize in different activities, control and execution are not localized - regions are closely independent and capable of acting on behalf of each other when necessary
  • Memory is distributed, not localized
  • Robust connectivity allows for simultaneous processing and awareness of what is going on elsewhere
  • Cross-connectivity creates redundancy that allows the brain to operate in a probabilistic rather than a deterministic manner, allows room to accommodate random error, and creates excess capacity that allows new functions to develop1

The brain is designed to facilitate the process of self-organization where internal structure and function can evolve along with changing circumstances; machines do not do this.

The manager in the organization-as-brain is concerned with redundancy of capability at atomic and coarse levels; recording, curation and accessibility of institutional memory; and unencumbered communications (no strict hierarchy) that are highly contextualized (explain why you want what you want). These are anathema to the manager who's goal is efficiency of execution. Redundancy is paying for something twice. Memory only matters to the most senior people as it affects decisions with long-term ramifications; in the organization-as-machine, those decisions are exclusively their purview. Open communications creates a lot of noise that interferes with orders from management. Having to explain why somebody needs something wastes time.

Traditional organizational thinking inhibits the conditions that create innovation. To have any reasonable expectation of innovation, we need to have the right expectations of how the organization functions if we are to manage in a way that fosters innovation.

Or course, a business is not managed by metaphor. But the way an organization is understood determines the way it is managed. A mindset rooted in learning rather than efficiency provides a set of "first principles" against which measurements and management decisions can be reconciled. It is therefore important to have the right mind-set about the nature of organization to understand how the organization of autonomous teams can exist at scale.

With these goals in mind, the characteristics of an organization of autonomous teams at scale become easier to understand. We'll look at those in the next post.

1 Morgan, Gareth. Images of Organization. Sage Publications, 1986.