I consult, write, and speak on running better technology businesses (tech firms and IT captives) and the things that make it possible: good governance behaviors (activist investing in IT), what matters most (results, not effort), how we organize (restructure from the technologically abstract to the business concrete), how we execute and manage (replacing industrial with professional), how we plan (debunking the myth of control), and how we pay the bills (capital-intensive financing and budgeting in an agile world). I am increasingly interested in robustness over optimization.

I work for ThoughtWorks, the global leader in software delivery and consulting.

Thursday, June 28, 2018

Organizing for Innovation Part IV: Autonomy at Scale

It is easy to understand how a small organization of autonomous teams can function. When there are only a few teams, there is a small community, and it is simple for people to communicate with one another in both formal and informal ways.

It is not difficult to see how a large organization of largely independent teams can scale. For nearly 60 years, Gore-Tex has shown that devolved authority can work just fine at scale.

Autonomy scales at Gore-Tex because there is very little overlap between outerwear and dental floss, and subsequently less coordination. What happens when the business becomes complex, with lots of interdependencies among lots of teams?

Dependencies by themselves do not make it difficult to understand how devolved authority can work on a small scale. If there are 4 teams, there are a maximum of 6 communication pathways (n x (n - 1) / 2). Even if there are transitive dependencies, the small size of the community - 4 tech leads, 4 product owners, etc. - makes cross-functional communications relatively easy. But a technology organization of tens of thousands of people will have hundreds of teams - and therefore an extraordinarily large number of communication pathways. Translating a small number of enterprise goals into hundreds of millions of synapses sounds like opacity at best, chaos at worst.

There are four things that an organization of autonomous teams needs if it is to scale.

The first is an implicit hierarchy, but one of purpose rather than control.1 Traditionally, hierarchy is meant to control activity througyh supervisory responsibility and assigned decision rights: the subordinates in one division take direction from superiors in the same division. If, per the initial blog post in this series, decision rights are devolved, "span of control" does not exist in the organization of autonomous teams.

Hierarchy also influences communications. If those "higher up" in the hierarchy use the things produced by those "further down", there is an obvious pattern of communications between producers and consumers. In manufacturing, there might be different teams independently assembling subsystems such as brakes or drivetrains from individual component parts. Each of their subsystems might then be consumed by teams on the line installing them into more comprehensive systems (such as the powertrain), and, ultimately, the finished vehicle itself.

This is called a "hierarchy of purpose." The hierarchy is constructed largely around degrees of granularity. Brakes and drivetrains are smaller assemblies that form larger subsystems that contribute to the finished product. A provider of cloud-based infrastructure such as Amazon can organize in the same way: the "finished product" of a cloud instance consists of more "primitive" components of virtual storage, server and network. Each of those subsystems in turn consists of more finely grained primitive components of network protocol communication, CPU, load balancing, and so forth.

Creating complex higher-order offerings as composites of lower-level capabilities is the “platform effect” of innovation.

It's worth mentioning that enterprise program management has long tried the same structure. It chokes on itself when transitive dependencies that extend several layers deep expose the difference between a boundary in work (a team has exclusive responsibility for producing something used by many other teams) and a boundary in fact (the output is high-touch service, support and maintenance, making those boundaries porous). A primitive component must be consumable in a friction-free manner.

This is a logical transition to the second thing the organization of autonomous teams requires, which is practical patterns of communications. The platform effect scales effectively because consumption patterns are the de-facto communication patterns within the organization. In a hierarchy of purpose, the organization does not have hundreds of millions of communication pathways, because the number of consumers is limited. A cloud instance may consume a load-balancing primitive, but it does so through a more coarsely grained "network" intermediary.

The decision classes introduced in the initial post in this series act as a control system for each individual team. The wider the divergence of type of consumer, the more difficult it is to create patterns of demand and prioritize to form a product strategy. The nested team organization limits the divergence of consumer demand, which creates a narrower range of appreciations, which make cohesiveness of strategy and execution at an individual product level far easier to perform.2

The autonomous-team collective maintains enterprise cohesiveness by virtue of its communication patterns. To understand how, we have to revisit the devolved decision classes. Appreciations (what should be done) act as the organizational system for managerial decisions (what can be done); managerial decisions function as the instrumentation system for appreciations. Managerial decisions, in turn, act as the organizational system for technical decisions (how will it be done); technical decisions, in turn, act as the instrumentation system for managerial decisions.

Scope Nature Organizes Instruments
Appreciations   What should be done?   Managerial    
Managerial What can be done? Technical Appreciations
Technical How will it be done?   Managerial

How this works effectively in practice becomes clear when we look at the characteristics of a single autonomous team that we saw in the last post. The transmission of minimum critical specifications through a hierarchy of purpose limits the noise and confusion. The reception of minimum critical specifications from multiple consumers are interpreted as appreciations through double-loop learning. The relationship of control systems and instrumentation systems of the different decision classes categorizes the information appropriately.

The third thing required for an organization of autonomous teams to function at scale is the ability to handle extraordinary patterns of communication. As there is no hierarchy of control, the organization needs protocols that allow it to adapt itself to a changing problem space, as well as to resolve inter-team conflicts.

A dynamic problem space is addressed through task-forces, which may be short- or long-lived, depending on the nature of the need or opportunity. Task forces are formed organically from members of affected teams to solve for problems that are existential to any one team. For example, suppose three teams conclude that they need a new class of capability that is outside the boundaries of all of them. They could elect to form a "task force" in the form of a long-lived team, staffed by reassigning members of their respective teams to this new one on a full-time basis. Not all task-forces are long-lived and full-time, of course: a task force to address a simple challenge might require each person commit only 1 day each week for a month, allowing them to otherwise remain focused on their line responsibilities.

Another other exceptional form of communication are inter-team conflicts. For example, team A elects not to prioritize something really important to team B, and team B lacks the resources or capability to do it for itself. Without patterns for extraordinary circumstances, it would end with a very grumpy team B. Inter-team conflict is handled through mediation and adjudication protocols. At the first stage of mediation, an acceptable 3rd party - that is, someone who is not a stakeholder in the conflict - is asked to mediate a decision. If the conflict is still not resolved following the initial mediation, a committee of 3rd parties are formed to arbitrate a decision. This provides a fair hearing among peers without the need to resort to authority given through a hierarchy of control.

Finally, the organization of autonomous teams needs mechanisms for aligning the strategic goals of the organization with team and individual execution. Appreciations provide connective tissue among strategic, team and individual goals and objectives, but they still need to be reinforced by executive management. Intermediary (layered) objectives have a tendency to change the interpretation of organizational goals, and therefore the goals of each individual.

Alignment is achieved by telegraphing executive intent throughout the organization. There are techniques popularized by a number of firms - OKRs, V2MOM, and the Big Bets Spreadsheet - and probably many others. Whatever the mechanism, their purpose is to communicate unambiguous goals to give each individual a clear means of reconciling a decision - why, what and how - with an outcome that advances the organizational goals. This creates direct line-of-sight between effort and result - and therefore tactical action with strategic outcome - and allows the organization of autonomous teams to function without an excessive number of people in low-value supervisory roles.

With the right set of characteristics, then, an organization of autonomous teams can reach scale, even in complex environments. But it clearly has a vastly different operating model to the traditional control style imposed over the machine-like organization. Is the labor force equipped for this? Is leadership? We will look at these questions in the next post.

1 Susman, Gerald. Autonomy at Work: A Sociotechnical Analysis of Participative Management Prager Publishers, 1976.

2 There is no guarantee of control, of course. An individual team can still thrash or produce unwanted product.