I consult, write, and speak on running better technology businesses (tech firms and IT captives) and the things that make it possible: good governance behaviors (activist investing in IT), what matters most (results, not effort), how we organize (restructure from the technologically abstract to the business concrete), how we execute and manage (replacing industrial with professional), how we plan (debunking the myth of control), and how we pay the bills (capital-intensive financing and budgeting in an agile world). I am increasingly interested in robustness over optimization.

I work for ThoughtWorks, the global leader in software delivery and consulting.

Thursday, January 31, 2019

Enterprise Change is Leadership Change

From the COO's perspective, the corporate IT department has hit rock bottom and they keep digging. Very little gets deployed, and the software that does make it live is embarrassingly bad.

IT is not exactly a clean canvas. ERP, CRM, document management and other key systems started life as COTS products but have been configured beyond recognition to a point where they can't be upgraded, and they're so ancient they've been disavowed by their vendors. Adding insult to injury, we're sitting on decades worth of data, but can't point to a single insight we've ever gleaned from it. The COO is not going to be jockeying to get IT reporting up to him any time soon.

The CEO has all but lost patience. She can't understand why she gets the same dead-end answer of "legacy systems" when she asks the CIO "why do we have to spend so much on IT?", "why are our systems not in the cloud?", and "why aren't we mining our data?" The board is asking her these questions, and she's got to have answers, not the same excuse.

* * *

IT knows it has a lot of problems. More than a few believe that the way it works is a big part of reason why. Every delivery team is blocked for reasons of access, authorization, and answers it needs from armies of people spread across a myriad of IT functions. Organizationally, there are silos within silos and shared services within shared services. If teams could work more independently toward a goal they would get a lot more done.

Somebody gets the idea that maybe we can solve this through better process. If we were to adopt Agile we'd have control and predictability and quality and speed-to-market. If we reorganize into cross-functional product teams, we'd get better solutions and some real insights.

A pilot or two here, a consultant or two there, a few slide decks and a site visit to a respected technology firm, and it's decided: we have seen the future, and it works. We're going to change.

We’re a large enterprise, so we need to roll out change at scale, and in a controlled manner. And so we get the enterprise Agile change program...

* * *

Over the years I’ve had the opportunity to examine several enterprise Agile change programs that are well under way. Those that were not living up to their initial fanfare (and quite a few of them were not) share a few common characteristics.

All organizational communication flows still go on one direction. The new "product operating model" bears striking resemblance to the old "program operating model": it moves from analysis to specification to development to deployment, all in a sequence of straight lines going left to right. There are no feedback loops anywhere in this cycle. As a result, there is no tolerance for team-level discovery let alone empowerment to act of its own accord on what it has discovered. Every team is free to build the product they were told to build. People are still bound to the plan, just like always.

Labor is assumed to be interchangeable. The Agile team model assumes there are no silos within a team: once a developer pair is free they take the next highest priority Story. But enterprise IT is loaded with specialist labor: this person only does front-end development, this person Java, this person Tibco, this person ABAP. Creating "cross-functional" teams doesn't cut it: the work distribution will be asymmetric and the team will lurch from blocker to blocker. It doesn't help that enterprise IT is also primarily staffed by contract labor. Contracting firms and their employees have incentives that favor labor specialization. The Agile operating model needs poly-skilled people, but enterprise IT doesn't have many people like that today and there are institutional headwinds against there being more of them tomorrow.

Process is the primary problem we face. No matter how collaborative and encompassing, a different mechanical process will not overcome the rot that plagues enterprise IT organizations: messy and multiple point-to-point integrations, a shortage of systemic knowledge, overloaded data structures, and poor data quality are not problems solved by a change in how work gets done. These problems are much larger than any one delivery team can possibly solve. Worse still, a process that requires a low-friction working environment to be successful doesn't stand much of a chance of taking root when the friction is very high.

As a result, enterprise Agile change programs don't make sense through an Agile lens, but they do make sense through a Waterfall lens. The change is largely cosmetic: Agile has just introduced surrogate terminology for how we’ve always done things. “Intake” and “portfolio management” are new words for “big up front design” and “detailed project plans” and rob the teams of functional discovery. Architecture activity that surfaces technical design before development robs the teams of technical discovery. Cross functional and even co-located teams can’t solve the big problems mentioned above in data, dependencies and system knowledge. Our new process might make us a little better on the margins, but in the main we still have the same integration hell, the same usability shortfalls, the same systemic brittleness and the same quality problems that we always did. The result, as others have written, is Agile in name only.

All right, so top-down change risks getting neither the mechanics nor the values of Agile. Wouldn't a bottom-up change program be more values-centric? A couple of months ago I wrote that bottom-up change - that is, linear scaling of team-level phenomenon - is inadequate for enterprise needs because there are enterprise dynamics and challenges that do not exist at the team level. This means there are classes of need that a focus on team-level process misses entirely.

The answer is not simply "do both top-down and bottom-up at the same time". To change an enterprise requires a change in leadership, not process.

Top-down change must not be charged with trying to answer the questions that enterprise IT has always mistakenly thought it had to solve: those of predictability and control. The only valuable top-down change is cultural. Cultural change is a leadership problem, not a process problem.

If we care about feedback, then by definition we value learning over being right all the time. If we value learning, than our operating model needs to be a picture of knowledge acquisition and application, not a picture of output control. There need to be at least as many (if not more) arrows depicting communication flows going backwards as forwards in the process. More importantly, it must be clear how that feedback is internalized and acted upon in the operating model. In the absence of that, learning is at best an accidental curiosity.

But even cultural change is not enough. Devolving authority and creating a means for genuinely incorporating feedback to team level execution is useless unless people are able to do these things in an unencumbered fashion. That means acknowledging the ground truths of the state of our technology and staff, and taking deliberate action on them. This is not easy to do. Long-tenured people in the organization likely made decisions that contributed to the current state of affairs. It's humiliating for somebody to admit that "past them" contributed to the reality of "present dysfunction."

As hard as that is, acknowledgement only goes so far. As Lisa Simpson pointed out to Homer as he came face to face with the fact that he is a rageoholic, acknowledging the problem is the first step. Dispiriting for Homer, it is not the last step. Acknowledgement has to lead to action that reforms. Without definitive action and lasting commitment to that action, the learned helplessness that plagues organizations with these conditions will erode the will to change.

It's all well and good to champion process, and by looking at process we can understand how ineffective we are and what we can aspire to be. But process will not overcome years of bad choices that have infected our people, assets and data. Process change that ignores these problems or denies they are encumbrances will at best be rendered inert, at worst will result in giving people responsibility without authority.

Leadership is not asking one group of people to make good on another's bad choices. Leadership creates the conditions where the rank and file have a fighting chance. Process is part of that, but in enterprise IT with large legacy estates, process is rarely the primary solution.

Monday, December 31, 2018

I'll be Gone, You'll be Gone

The Financial Times recently ran a long article describing the breakdown of governance over the Crossrail development, the first new underground railway in London in over a century. Good governance is usually only appreciable by its absence, but the Crossrail case allows a before / after contrast between the presence of good and, after the sacking of a key board member, the presence of bad. Early on, "TFL's representative on the Crossrail board ... had kept a close eye on costs, and asked all 'the difficult questions' according to one individual close to the board". Yet after his departure, "It [the board] endorsed everything rather than challenging [management] and asking questions.". As a case study in governance, the FT article is an analysis very much worth reading.

The article also describes a nefarious management phenomenon common to complex, long-term investments: "And the management often doesn't care because they know they won't be there when the project isn't delivered on time and to budget."

I recently read John Kay's book Other People's Money: The Real Business of Finance. In Chapter 4, Dr. Kay describes the trading mentality that replaced that of stewardship across much of the banking sector from the 1980s onward. He introduces the subject using a quote from a senior industry figure: "''We are investment bankers. We don't care what happens in five years.'" The people who make the promises and ink the deal are not the people responsible for fulfilling the promises and seeing the deal through.

The FT article describes this phenomenon within Crossrail: "...like all long-term projects, it suffered from management changeovers at contractors and suppliers. 'You always have a lot of baton changes on large-scale projects like this... So although they will have been suppressing bad news, none of the people who were round the table at the beginning are still there - it's the nature of careers that no one stays for 10 years.'"

Or, as Dr. Kay put it, "Traders need not wait to see when or whether the profits materialise. I'll be gone, you'll be gone."

A little over a decade ago, a collapsing mortgage market in the United States metastasized into a global financial crisis when the value of complex structured products (think collateralized debt obligations backed by sub-prime mortgages) on bank balance sheets suddenly collapsed. Chuck Prince, CEO of Citigroup during the run-up to the global financial crisis of 2008, will forever be enshrined in the annals of finance history for his regrettable statement that 'as long as the music is playing, you've got to get up and dance.' "Soon after," writes Dr. Kay, "Prince was forced from office and Citigroup was bailed out by the US taxpayer."

Earlier this year, Crossrail development, which had appeared to be moving successfully along, fell off a financial cliff, requiring two capital injections in rapid succession, just so they could keep paying the bills. Per the FT: "'It feels as if they clung on to the schedule they had until it became clear that it absolutely couldn't be delivered,' he says. 'But when they dropped it, it seems that they had nothing left to work to. That would explain what looks like a sudden collapse.'" The chairman of Crossrail has been forced to resign (its chief executive having exited just before the bad news broke), and Crossrail has been bailed out by the UK taxpayer.

In the post-collapse analysis of financial crises, Dr. Kay points out that the principal actors are not particularly adept at critical self-assessment. "Rogue traders normally protest that the activity would eventually have been profitable if the bank had not closed its position, just as the gambler dragged home from the casino tells his wife and the world that he would have come out on top only if he had been allowed to stay longer." Additionally, the principal actors in the early stages - who are the primary beneficiaries of "cash borrowed from destiny with some random payback time" - have a tendency to deflect any doubts about the merits of the rewards they reaped in the time before the crash. "With the cognitive dissonance of the tailgater, he [Mozilo] would explain that the considerably larger amount he had received for his services as chief executive of Countrywide was justified by the profits that his company had reported from the sale of mortgages before the borrowers failed to pay them back." While deflection on the grounds that the collapse was episodic rather than systemic may be delusional, it is a convenient pound-the-fist-on-the-table argument to justify individual financial rewards of questionable merit.

In the banking sector, the problem was borne of a combination of the agency problem (the objectives of managers are materially divergent from those of the board) twined with moral hazard (no downside exposure to idiotic risks). It remains to be seen whether or not this proves to be the case with Crossrail. But with a management extracting high compensation derived from other people's money, a large number of contractors each working in a silo, an established pattern of suppressing bad news, and a forced buyer of last resort, Crossrail certainly appears to possess many of the same characteristics.

Per the FT, "People are always adventerous on costs and time because they don't want to tell the truth." The first and last line of defense against the agency problem and moral hazard is good governance. Poor governance plagued the banking sector in the years up to the 2008 financial crisis: few directors of financial institutions had any "sophisticated understanding of banking and risk", consisting as they did of "leading clients, ex-politicos and community leaders". Crossrail appears to have become plagued by the same, with one of it's principal stakeholders having the attitude of "'give him a call when it's time to cut the ribbon.'"

There is no easy fix to poor governance because governance tends to be self-referential among its members and lacking in tools for self-correction and continuous improvement. As Dr. Kay writes, "Their attitude was not the comprehensive immorality of the overt fraudster but the willful blindness of those who do not ask questions when it would be embarrassing, or at least inconvenient, to know the answer. Upton Sinclair's remark is again relevant: 'it is difficult to get a man to understand something, when his salary depends on his not understanding it.'"

Governance is only as effective as each governor, which is why I advocate for everybody in a governing role to study and practice the positive behaviors of activist investors. Given the variable - and generally poor - standard of governance in place over most investments, there needs to be a standard of excellence in governance practice. Activist investing is the closest thing we have to such a standard.

Friday, November 30, 2018

The Fallacy of Composition

That six month experiment with Agile practices yielded off-the-charts results in time-to-market with quality, as well as sponsor and user satisfaction. It cost a lot, but everybody is smitten, and word has gone round the company. Everyone wants to know: what was different? Well, everything: automated build pipelines, stories, stand-ups, showcases, spikes, desk checks, you name it. Ok, great. Let's make sure everybody in the organization is doing those things everywhere, all the time.

That "Agile pilot" was a deliberate investment in the formation of behaviors designed to yield local (that is, team-level) optimization. Sustained, intense concentration on collaboration among consumers, designers, builders, testers and managers resulted in more experiments, more builds, more deployments, and more frequent feedback. This resulted in not only a better software asset, but an outcome that everybody is emotionally invested in.

The immersive nature in which these new muscle memories were developed came at a cost. You took the flack for using unapproved tools. You paired up your staff, effectively doubling the size of the team and more than doubling the run rate. You had to deal with the daily drama of your employees coping with the muscle confusion of learning new ways of working as well as having to deal with strong consultant personalities. A long-time employee quit - and with him, years of tribal knowledge walked out the door - because he was demoralized by the experience. No one seems to remember that your business partner all but demanded that you to pull the plug on the entire exercise just a few weeks in. With success comes selective amnesia.

Still, it was a success, if a success on a small scale. The experience of success in the small makes us prone to gross misunderstanding of what success looks like in the large. It is easy to fall victim to the fallacy of composition: the inference of the properties of the whole from the properties of its parts. There are two obvious ways this manifests itself: in not being aware of characteristics of success at scale, and in not being aware of the characteristics of what it takes to scale it up.

A small product team with no external dependencies is a closed ecosystem, like a terrarium. A terrarium ecosystem is not particularly diverse, and any deficiencies are easily corrected through minor adjustments a manager can directly make through direct intervention. An enterprise-scale ecosystem has a geometrically larger number of dependencies and interactions; these are not very easily compensated for by management-level employees through direct intervention.

Success at scale requires understanding an organization as a whole, through different lenses. "How do we spread these practices throughout the organization" looks at mechanical activities that yield local (team-level) optimization in a fixed span of time. They make no provision for what happens at an organizational level over a long span of time: different parts of the organization will move at different speeds, held back by legacy constraints that are not so easily undone; inter-team and inter-organizational dependencies are not simply solved through wistful promises of "alignment"; people with scarce skill sets cannot be co-located with every delivery team that needs them; how hiring is done and who is hired will determine whether change is sustained and evolves; procurement practices that buy for unit cost optimization rather than outcomes reward vendors for the wrong behaviors; operating company norms that prioritize efficiency at the expense of learning will render feedback loops inert.

In failing to see the characteristics of the organization that are not present in the parts we are looking at, we are blind to the organizational phenomenon that will impair and ultimately reverse the benefits of change.

It is also important to recognize that the characteristics of change at large scale are not the same as the characteristics of change at small scale.

When answering "what is different", the rigor, discipline and commitment necessary to raise the bar tend not to be part of the answer. This separation of "ends" and "means" invites dilution of skill transfer and subsequently dilution of the skills themselves: coaching replaces immersion, decks and self-service wikis replace coaching. In addition, the implicit reduction of risk - having been "proven out" in a pilot, the practices must no longer require a heavy investment - recasts Agile as an operational problem rather than one of learning and growth. It's just a different process from what we follow today, right?

"What is feasible, or beneficial, in the small may be infeasible, or harmful, in the aggregate."1 Heavy investment in immersive activities at scale - in effect, making an investment to "push" change - is prohibitively expensive at enterprise scale. The sustained period of immersive skill development that was highly beneficial in a single team is infeasible at scale. Diluting change makes it outright harmful, as it can yield false positives of everything from collaboration to quality. That whole "we're going Agile" thing will become a thin veneer over the chronic problems that plague your organization, and once that happens something that could be of genuine benefit is corrupted to a point of uselessness.

The failure to recognize that an enterprise is a geometric rather than a linear degree of difference from a single team dooms a lot of well intentioned change programs. We also see the effect in maturity models and enterprise frameworks that are precise at atomic levels of change but vague or misleading at organizational levels of change. We'll look at the latter in a future post.

1 Kay, John. Other People's Money: The Real Business of Finance.

Wednesday, October 31, 2018

Everyone a Beginner?

I had an exchange with Dan North about the subject I wrote about last month, Beginner's Mind. Dan asked an interesting question: what would it take to make work like this every day for everyone, everywhere?

It's a serious question that deserves serious consideration.

To start, it's worth asking: why isn't work like this today, every day, for everyone, everywhere?

There is a difference between development companies (or divisions within companies) and operating companies. A development company is in the invention or innovation business. Investors and customers place high value on seeing something new, be it features or entirely new products. There is a high tolerance for operational inconsistency from a development company. Recall of how tolerant (or at least, how much less snarky) people were towards Windows when Windows 95 was launched, or the iPhone when the 3GS was launched.

An operating company is in the consistency business. Investors and customers place high value on efficiency and reliability. There is low value in seeing something new as it interferes with set patterns and expectations. There is low tolerance for operational inconsistency. To wit: the ribbon interface that replaced the command menus in Microsoft Office 2007 was not met with enthusiasm.

A company can be both a development company and an operating company. For example, Tesla. Inventing battery technologies and designing cars that are sufficiently different from the cars anybody else has made captures the imagination of financier and customer alike. But they have struggled at the basics of manufacturing things at a modest scale. Manufacturing cars is a path well trod, so nobody places a lot of value on Tesla employees learning things already very well known by people in the auto industry.

There is still room for engaged employees in both development and operating companies, but there is a difference in the nature of their engagement. We want passionate people in the development company, people who will look for ways to deliver value where there was none before. But we don't really want passionate people in utility-type businesses: the passionate accountant who finds ways he believes his clients can dodge the taxman is not beneficial to his clients in particular or society in general. What we do want in operating companies are obsessive people looking for ways to make the core transaction between buyer and seller more efficient or incrementally more delightful without materially altering the nature of the core transaction itself. My unexpectedly very late arrival at the hotel has made it impossible for me to get dinner, but an availability of fresh fruit and healthy snacks free of charge not only satiates my hunger but makes me feel the hotel staff is concerned for their guest's well being. It's still a room with a bed and a bath, but I'm more likely to choose that hotel again in the future because they strike me as being prepared for irregular ops situations experienced by their customers.

There is a fundamental difference between these types of employee engagements. Knowledge of something that customers have never thought to do or thought possible is in the "voyage of discovery" realm. Knowledge of something that customers have long known possible and wondered "why can't / won't they do this" is in the "pay attention" realm. That makes prior questions of skill; the latter questions of will. A lot of things in operating companies fall into that latter category: a decision by a customer service person not to do something is a choice the firm they work for makes, either on the spot (by the employee) or as a matter of policy (by management).

But this shouldn't matter, should it? Either way, the employee is learning. What difference does it make?

It matters from the point of view of both the learning intensity as well as the freedom to act on what has been learned. There is less opportunity for learning and less value placed on it by investors and customers in the operating company realm, hence less opportunity to act on it. And, unfortunately, operating company jobs are the bulk of the jobs people hold today.

The business of software is somewhat unique in that the learning intensity and freedom to act are extremely high, precisely because we get to take voyages of discovery all the time. Sure, learning Pascal was something millions had done before I came along. But my ability to apply that knowledge to do things that had not been done before created a very learning-intense environment: the better my skills in Pascal, the different the class of problem I could solve with it, the more new ground I could clear.

That's great for those of us in the business of software. How do we meet the "for everybody, everywhere" standard?

The opportunity will first present itself by being able to get more people out of rote operating company jobs. But that isn't enough. Yes, in theory I don't need any human interaction at the hotel: I can check-in from my phone and use it as the room key and if I opt out of housekeeping services and order food through Seamless I would never cross paths with any hotel employee. All that does is displace more hotel staff and justify fears of our robotic future and a return to the bad old days of industrialization concentrating wealth and suppressing working classes circa the first half of the 19th century.

There is benefit to having people in opco jobs, and not just to supervise the robots. What if opco employees had tools they could use to solve different classes of problems? Not transactional tools, not canned reports that a back-office accountant can study, not 21st century green screens that clerks can use to sell an upgrade. How about tools that are (a) granular; (b) modular; (c) "programmable" by the user - where the user is not the customer, but an employee; and (d) re-composable. Opcos would be able to provide the operational consistency that investors and customers expect, while enabling a work force with tools allowing them to be less obsessed with efficiency and more passionate about solving customer problems.

What if opco employees could re-program the environment, directly, themselves?

That would certainly bring more people on the same "voyage of discovery" that people in software development enjoy. It would redefine the relationship between developers and consumers as peer solvers of problems of different granularity, not producer-user of a confined operational space. And that would mean less tilt in the playing field of the future, and subsequently less inequality.

How do we make it possible for everyone, everywhere to experience the beginner's mind we get to experience every day in software development? By treating them as problem solvers, not operator-executors.

Sunday, September 30, 2018

Beginner's Mind

A little over a decade ago, I was part of a team introducing new practices to a 200+ person, distributed development team. A lot of people were too quick to understand them. Some participated in stand-up as if it were a one-way status meeting. Others were still creating long-lived branches in Perforce as if they were still using ClearCase. "Stories" was just a new word for the technical tasks they had always assigned. Two people I was working with at the time pointed that everybody would be well served to approach these new practices with "beginner's mind."

* * *

While on vacation in Western Ontario Province a few years ago, my wife suggested we visit an amethyst mine. I had distant memories of doing some lapidary work with my dad back in the 1970s, so I picked up enough small stones for tumbling. Over the years, my dad had grown serious enough about the hobby to build out a considerable workshop of tools, slabs and rough. When I told him that I had picked up a few pounds of amethyst, he gave me a spare rotary tumbler. Five weeks later, I had my first batch of polished stones in nearly 40 years.

I was also disappointed. The polished stone looked good but not great. Why did so many have fractures and chips?

I did some research and tumbled some more stones and learned that to get better results I would have to improve the quality of the rough as well as my technique for tumbling.

To improve the rough, I learned to recognize characteristics of amethyst crystals that were more likely to fracture in tumbling, and excluded these. I did a full inspection after every stage of the tumbling process and removed any stones showing signs of damage so that a shard would not float around the barrel and contaminate other stones in the batch.

I had to improve my technique, too. Stones take quite a pounding in a rotary tumbler. While this action erodes sharp edges off the stones, it also increases the probability of damage. I learned that ceramic media in the barrel can cushion the impact. I realized that the barrels can be difficult to completely clean; even a tiny piece of previous-stage grit can contaminate a batch, so I obtained different barrels for different stages of grit. I also questioned the 7-days-per-each-stage tumbling recipe: what's so special about a week? I sampled stone in the tumblers once per day, comparing it to stone that had been in for a full week, and found I could reduce the amount of time spent in every stage.

Eventually, my dad gave me a lapidary arbor so I could grind the rough into shapes that would tumble more successfully. Later, he gave me a lapidary saw so I could cut fractured stones, allowing me to harvest more from the rough: instead of grinding away one half to save another, I could cut a stone along an obvious fault line and shape each part into a stone that was more likely to tumble successfully.

I quickly realized that the saw and arbor would let me do more than prepare stones for less bad tumbling outcomes. After a bit of research led me to cerium oxide belts and expanding drums, I figured out how I could hand-shape stones from rough to polish without using the tumbler. Hand shaping a stone is time consuming, but far less damaging than tumbling. There is risk: what appears to be innocuous sanding can expose a deep void, and careless handling on the arbor can result in a shattered stone. Still, when I have a very special rough stone, I can shape and finish it by hand, sanding out imperfections while retaining its unique characteristics.

The lapidary world is not too interested in making unique shapes out of small rough on saws and arbors; it is primarily concerned with creating faceted stones or cabochons (domed shapes in fixed sizes). To be serious about lapidary work, I would have to make one or the other or both. I had the tools for cabs, so over the course of several months, I taught myself how to slab, trim, dome and polish cabochons. I had some jewelry findings from my dad and acquired a few more of my own to set the cabs into as finished jewelry pieces.

It occurred to me that what makes lapidary work so satisfying to me is that I approach it with beginner's mind every day that I get to do it. Every stone I work on, every technique I follow, every tool I use is an experiment and therefore a learning opportunity. Through the simple, frequent act of trying lots of different things I have made hundreds of little adjustments to my workshop, my tools, and my methods. The net result is I am more efficient, more accommodative to failure and mistake, have higher throughput and better product quality today than I did at any point in the past.

And I'm still learning. Every day.

The joy of the lapidary work is not the final product. If it ever became that, it would be a chore, a job, and it would lose its allure. The joy of the lapidary work is the never-ending discovery. Some days I work in the workshop, some days I work on the workshop, but no matter which it is I learn something every day, and every day I take action on what I learn.

* * *

In the business of software development, the primary focus is on the product: these features, this timeframe, this cost. When the product disappoints - it costs too much, it takes too long, it isn't reliable, it is difficult to use, it doesn't do what it's supposed to do - the process of how it is made gets as much attention as the product we're trying to make. Those times of focus on the process lead people to investigate things like Agile and Lean and continuous delivery. Sometimes, to people unfamiliar with them, Agile and Lean and continuous delivery appear to be magic.

There is nothing magical about things like Agile and Lean and continuous delivery. The only magic is the (re-)experience of beginner's mind when somebody is first exposed to them. They force us to re-examine the orthodoxy we have long accepted - for reasons we no longer remember - for how we work. They let us break out of the psychic prison that is the modern operating company environment.

Despite containing elements of continuous feedback that drive continuous improvement - build pipelines, retrospectives, experiments - the change to Agile or Lean or continuous delivery does not typically result in a permanent state of beginner's mind. They captivate only for the short period when people care about process: we're broken, we need a fix, this process looks like a fix, let's adopt this process. As important as change may be at the time, product eventually triumphs over process, while change fatigue casts a pall over attitudes and budgets alike. Management wants mastery of whatever mechanical processes they've agreed to; they're not interested in perpetual, indefinite refinement.

Dan North has long argued that first and foremost a company should aspire to be a learning organization. A propensity for organizational learning underpins a lot of well established management theory and more recent instantiations of it, things like autonomous work teams, Agile practices, and Lean startup. If the real objective is not to become a learning organization, these will be little more than mechanical acts that serve only to make software development less bad.

There is merit in being less bad. There is value in fleeting moments of joy from reconnecting with our younger, knowledge-acquisitive selves. But change that does not aspire to rewire people permanently to their beginner's mind does nothing more than make a job less of a slog. It is still a job.

It is not a passion.

Friday, August 31, 2018

Organizing for Innovation, Part VI: Putting It All Together

As we saw in the previous posts in this series, organizations of autonomous teams can scale. Scaling requires different team characteristics (requisite variety, redundancy of function, double-loop learning and minimum critical specification), a different mental model of organization (brain, not machine), a different kind of hierarchy (purpose, not control), and a different style of leadership (guide, not command).

This sounds like it would be chaos in practice on a small scale, let alone enterprise scale. And even if it does work, it sounds like a revolutionary approach to organization. Visualizing it helps explain how all these things combine to create an organization that innovates as well as operates.

The organization of autonomous teams is not chaos. Teams are invested with authority, communication pathways develop along the hierarchy of purpose, the structure adapts itself to the domain, and organizationally-driven objectives plus team incentives align behaviors and outcomes. Of course, this looks nothing like traditional organization, in which hierarchy is the principal means of control, communication, and alignment. Still, while an organization of autonomous teams may be unfamiliar and conceptually unsettling, it isn't chaos.

And, although it may seem revolutionary, it isn't new. Academic research on organizations of autonomous work teams dates to the 1950s, and the early adopters of it were industrial firms. While it may be a way of working that a lot of tech companies happen to adopt for themselves, it is not an organizational phenomenon that has sprung out of tech. So it isn't all that revolutionary, either.

It may not be chaotic or revolutionary, but it is a big leap from how just about every enterprise functions today. The way they work reflects the board's priority for the company. I've written before that companies are financial phenomenon more than they are operating phenomenon. Executives tasked by their boards to prioritize returning cash to investors will look to maximize cash earnings (EBITDA) and free cash flow. In so doing, those executives create an environment where managers must be more concerned with costs than creativity from operations. Managers, in turn, condition employees and contractors to value activity over learning, output over outcomes, and narrow individual independence over broad group autonomy. In this environment, employees, managers and executives are rewarded for every dollar not spent for output; they will not be rewarded for any dollar spent on learning.

With enterprises increasingly feeling the heat from new companies, technologies and products, executives charge managers with extracting more innovations from the business. Managers go about looking at changing ways of working, recognizing that innovation is partially a byproduct of "how" things are done. But as long as the priority of the board is to return cash to investors, the first mission of "how" things are done is to be predictable, because predictability ensures consistent cash flows and consistent cash flows ensure that interest payments, dividends and buybacks can be made while protecting the bond rating. The autonomous team structure is incompatible with predictability.

The autonomous team structure is internally consistent (not chaos) and been around for long enough with enough successes (not revolutionary) that it can succeed, even at scale. Pursuing it is ambitious, and operationalizing it is plenty difficult. But success with it has less to do with operationalizing than it does with the tolerance for it in the capital structure in which it operates. Autonomy - that is, abdication of centralized control - is the price of admission for innovation. Innovation at scale requires autonomy at scale. Autonomy at scale requires the board be committed to innovation, not cash returns, as their top priority.

Tuesday, July 31, 2018

Organizing for Innovation, Part V: The Leadership Challenge

Organizations of autonomous teams require a different set of behaviors than organizations that are run like a machine. People in a self-directed team form their own appreciations for what should be done, prioritize what will be done, and self-determine how it will be done. They are unencumbered by hierarchy, expected to communicate with anybody in the organization they need. They are unencumbered by role definition, as people simply do whatever work is required even when that means acquiring new skills or knowledge. They are unencumbered by organization, as teams form task-forces to solve for problems that are beyond the scope of any single defined team.

This all sounds great. Who wouldn't want to work this way?

The majority of people working in enterprise IT today, that's who.

Enterprise tech labor is highly codified (bounded responsibilities) and stratified (seniority). Aside from the fact that this serves the interests of a multitude of non-tech corporate functions such as human resources, vendor management and finance, it also provides a great deal of comfort to the individual. The employee knows very precisely what is expected of them to earn a salary increase or advance their career, while the contractor knows what they are obliged to do to satisfy the terms of their contract. Over time, jobs become working annuities that require little servicing (such as skill acquisition or excessively long working hours) for comfortable levels of compensation with job security (by e.g., being the only ones familiar with a technology, service or function).

The autonomous team environment is anathema to this. For starters, the autonomous team operates with a lot of ambiguity. New appreciations change the team's priority constantly, meaning there is no deterministic plan. Plus, people adapt themselves to the work that needs to be done as opposed to working in strict swim-lanes. An autonomous team environment implicitly subordinates the traditional goals of the individual: the success of the team is success for the individual. An autonomous team breaks down when its members put individual achievement over team accomplishment.

Of course, working this way is a question of both skill and will. Whether labor can re-orient itself from machine to autonomy is a debate well beyond the scope of any blog post, but suffice to say that some - few, several or most - will not be able to make the transition from doing what they are told to do by management, to figuring out for themselves what needs to be done and doing it. In addition, whether labor willingly chooses to re-orient itself is another matter entirely. Over-exposure to enterprise change programs - or more likely, over-exposure to failed enterprise change programs - won't inspire enthusiasm for yet another one. There is also the suspension of disbelief people need to make to give devolved authority and autonomous team structures a try.

This brings us to the leadership challenge that creating an organization of autonomous teams poses. Obviously, there are institutional barriers that have to be overcome. HR has to be comfortable with ambiguous roles and titles. Vendor contracts for supply of specialist labor have to be replaced with contracts tied to business outcomes. Finance needs an alternative to predictive planning and financial budgeting.

However, the challenge to leadership goes beyond mechanical processes and structural changes. In an organization of autonomous teams the nature of leadership changes from giving commands to guiding actions. The leader does not direct people's performance to achieve a goal (organization-as-machine), but instead projects a goal that people direct themselves toward achieving (organization-as-brain). The leader in the autonomous organization makes use of:

  • Concrete statements of expectations: leaders have to make expectations crystal clear. Saying "all enterprise software is deployed into production every other week" is a clear expectation. Saying "we will be a continuous deployment organization" is a woolly statement: "continuous" will be interpreted relative to the current skills and learned helplessness of the people responsible for enterprise software today. The latter statement also misses the point: how the organization functions (continuous deployment) is less important than describing what it achieves (new software released across the board every other week.) The prescriptive implications of that statement deny the people in the organization the opportunity to figure out for themselves how best to achieve the goal. The leader communicates the expectation; it is up to the people in the organization to figure out how to achieve it.
  • Rewards and recognition: obviously, what gets measured is what gets managed. If the goal is to increase frequency of deployments, reward the teams that find ways to make reliable production releases weekly, then twice weekly, then daily, then multiple times per day. Story-telling is important as well, especially as an organization adopts new behavioral norms and its business partners develop new expectations of it. For example, in its early days, FedEx executives were fond of repeating the story of the junior employee who rented a helicopter on his personal American Express card to fly him to a mountain location to repair snow-damaged phone lines so that they could service remote customers. When you're building a reputation for absolutely, positively getting packages delivered overnight, stories like this go a very long way.
  • Acknowledging and solving constraints: every team, not to mention the organization as a whole, will be short of the capacity, capability and capital to achieve every organizational goal. While constraints force institutional responses such as prioritization and innovation, they are also opportunities for a systemic change. It is the leader's obligation to work with constrained teams to identify the actions they can take today so that they will be less constrained in the future. For example, if there is a capability constraint, what can the team do now to develop skills and knowledge so that it can do more for itself? If a financial constraint, how can the team build a stronger business case or define a set of experiments to explore potential value? In the face of constraints, the leader's responsibility is to develop an organization's ability to learn how to learn.

At the foundation of the organization of autonomous teams is Theory Y. The aspirational leader of an organization of autonomous teams has to believe that people are motivated, responsible, and do not need close supervision. If you do not fundamentally believe this, do not bother pursuing an organization of autonomous teams. Once you revert to command-and-control (Theory X), you have lost the mantle of leadership in a devolved organization.

In the next installment in this series, we will put all of these concepts together to visualize what the autonomous organization looks like at scale.