I consult, write, and speak on running better technology businesses (tech firms and IT captives) and the things that make it possible: good governance behaviors (activist investing in IT), what matters most (results, not effort), how we organize (restructure from the technologically abstract to the business concrete), how we execute and manage (replacing industrial with professional), how we plan (debunking the myth of control), and how we pay the bills (capital-intensive financing and budgeting in an agile world). I am increasingly interested in robustness over optimization.

I work for ThoughtWorks, the global leader in software delivery and consulting.

Monday, February 23, 2009

The Agile PMO: Measuring Quality

In the last installment we took a look at the project management information we get from results-based organization and execution, and how that provides an unambiguous status assessment. But project status data doesn't give us the complete picture: we also need to know that a team is delivering solutions in accordance with all of our expectations, such as quality and maintainability. For the PMO, that means looking at technical and functional quality along side project status.

Quality Measures

We have no shortage of quality metrics. For example, in the Java world we can measure all kinds of attributes of code: overcomplicated expressions, wasteful use of resources, ghost code, duplicate code, complexity, and so forth. That’s both good and bad. Good in the sense that code quality doesn’t need to be taken for granted, because we can measure attributes of the asset we’re creating. Bad in the sense that with all this data, it’s hard to separate signal from noise. Quality measurements are great to have, but a meaningful assessment of quality is a different matter entirely.

There are many authorities on code quality, so we’ll not dive into them here. But there are a few metrics worth pointing out that are strong indicators that code will be difficult and expensive to maintain: the extent of code duplication, the presence of highly complex code (i.e., cyclomatic complexity), the presence of many large methods (in terms of number of lines), and code that is poorly encapsulated. If any of these are present to a large degree, we have indication that we’re taking on excessive amounts of technical debt. This is material to maintaining viability of a business case: the higher the technical debt, the more expensive the cost of the solution, the lower the yield on the IT investment.

Quality Tests

In addition to measuring quality, we must also test for quality. Quality comes in many different forms. There is functional quality (does the application perform the tasks that it needs to perform?) and non-functional quality (is it fast? does it scale? is it secure?) Our tests can include unit tests, integration tests, and functional tests. A unit test exercises a specific piece of code, while integration tests validate round trips to other systems, and functional tests validate complete scenarios. We can perform these tests manually, or, better still, we can code them. If we code our tests, we can execute them automatically. We can build a library of automated tests and subject an asset to them at any time, meaning we can get up-to-the-minute information on the functional and non-functional quality of a solution at any time.

We must be cautious. Our quality testing is only as robust as our underlying tests, so we also have to validate that our scenarios are thoughtful and meaningfully complete. Some tests are destructive, and require disciplined environments management. But testing has come along way. Functional testing tools have improved in recent years, making functional tests less fragile and easier to maintain when software changes.

The greater and more comprehensive the automated test coverage, the less assumption there is about technical and functional quality. This provides reassurance to the PMO that both functional and technical quality is being maintained over the life of the project, and that incremental functionality previously gained is sustained. This will have a material impact on achieving our business case.

Assessing Quality Information

Metrics and tests give us a collection of data points about our code, each in a different unit of measure. To turn that data into information, we need to structure it in a meaningful way. We have several options for doing this.

One alternative is a scorecard, which I wrote about in the Agile Journal some time ago. To create a scorecard, we must first normalize metrics into consistent scores using simple rules or heuristics that give us degrees of "great" to "poor." We can compare the code in question against a representative but consistent baseline of quality, so we have an absolute point of reference. We can then collect our metrics into broad categories of “indicators of good” (hygienic measures) and “indicators of bad” (toxic measures). By doing this, we can ascertain how good or bad our current state is, and whether or not it becomes better or worse over time.

Another alternative is a technical balance sheet, which Brad Cross wrote about in the alphaITjournal. In this approach, the business value and technical debt of every package is scrutinized to determine whether we’re right-side-up or under water on the solutions we've created. By drawing the line between “value” and “debt” it also tells us where our real risks lie, and what our priority and urgency should be.

Performance and Quality: The Complete Picture

Structured quality data described here paired with project performance data in the last installment gives the PMO what it needs to assess current and ongoing performance of a project. The PMO can answer for each project the two governance questions: Am I getting value for money? and, Am I receiving solutions in accordance with expectations? (n.b. It is a minimal answer because customer satisfaction and staff capability must also be considered, but that's for another blog post.) But this is a lot of data, and if we don't have an efficient means by which to collect it we'll be crushed under the weight of reporting and collection. In the next installment, we’ll take a look at what we can do to automate data capture so that collection is non-invasive to our organization, a by-product of day-to-day operations, enabling us to be an Agile PMO.