Donald J. Patti

When Quality is Too Costly

In Business, management, project management, Quality, Software Development, Technology on 2 November 2009 at 8:15 am

Throughout his career, my father served as an engineering manager in the aerospace industry, where the parts he and his teams developed were used for missiles and spacecraft. Sometimes, these parts were required to meet specifications within one ten-thousandth (0.0001) of an inch, a level of quality rarely seen in any industry. I remember discussing the day his company announced they would enter the automotive parts market, which used similar components.

“You should be quite successful,” I told my father, “if you’re able to deliver products that are of such high quality to the automotive industry. Who wouldn’t buy a product that meets specifications so precisely?”

“That’s actually the problem,” my father responded. “We can build parts to one ten-thousandth of an inch, but the automotive market doesn’t need anything that precise and isn’t willing to pay for it. It costs an awful lot of money to build something that precise and to verify that it meets that standard.” He continued, “It will take us a while to figure out how to mass-produce products with much lower tolerances that are also competitively priced.”

Many years later, I encountered the same issue in my own career. Educated like many others in the principles of total quality management (TQM) and the concept of “zero defects”, I believed that the key to building excellent software was a clean defect log.  Budgeting time for multiple rounds and types of testing, my team and I dutifully worked to test 100% of all functionality in our systems and didn’t release our software until every known defect was repaired.

The day came to release our software and I was met with an unexpected surprise. Sure enough, not many defects were reported back, but there were defects reported. Why?

It turned out that our large user base was exposing problems that neither our QA team nor our beta testers encountered or anticipated. Only through the rigors of production use did these defects come to the surface. This was my first lesson about the limitations of preaching a “zero-defect” mantra – nothing replaces the “test” of production.

During our project retrospective, an even more important downside to the blind pursuit of “zero defects” surfaced. Over the two extra months our team spent addressing low-severity defects, our client lost roughly $400,000 in new sales because the new system was not available to collect them. (I had done the ROI calculations at the beginning of the project showing a $200K per month of new income, but had completely forgotten that holding on to the system for a couple of months meant the client would be deprived of this money entirely-they were not merely delayed). For the sake of perfection, zero-defects meant withholding key benefits – this was a far more important lesson than the first.

Certainly, few users want to deal with defects in software, particularly software that’s not ready to deliver. And, of course there are software products where a near-zero defect rate is extremely important. For example, I’d be quite upset if a zero-defect standard weren’t set for the key functionality in an air traffic control system or in certain military, aerospace, medical and financial systems.

But now, before I recommend a zero-defect quality target for any project, I make certain that the costs of such a high level of quality are not only beneficial to the client, I make certain I include in my calculations the lost benefits the product will bring to the users by holding it back until this level of quality is achieved. After all, none of us like defects, but would we really want to wait the months or years it might take before we reap the benefits from a new version of our favorite software?

Advertisements
  1. The “release now, fix later” strategy is adopted by major (successful) software companies, including windows. They release software it’s insulting to believe that they don’t know it’s buggy, and they sort out the bugs later.

    Getting constant feedback from the clients is at the heart of agile methods.

    PS: I’ve published an excellent article on meeting client expectations that complements your post, hope you’ll get the chance to read it.

  2. PM Hut:

    Thanks for your input and your insightful article. I particularly appreciate your noting that it is often difficult to identify who is actually the customer, because that is rarely clear in a large or matrix organization.

    I agree that iterative approaches are far better at involving the client throughout the development process. This also reduces the likelihood of missing the mark when interpretting requirements that may not be well written. This is why, in most situations, I prefer to manage an iterative project over waterfall.

    But, an Agile or other iterative approach does not result in zero defects, though it should identify defects earlier and more often. In many cases, defects must be identified and resolved by thoroughly testing or asking the client to conduct some type of user acceptance testing (UAT). Because UAT is often brief and may not involve live data in situations where the data is sensitive or confidential, UAT does not always expose defects.

    In short, there’s always a tension between the desire to release earlier and accept the risk of defects or hold on to the product and reduce the number of defects. Whether the development methodology is iterative or waterfall/traditional, one still has to balance the two. A zero defect mantra, while admirable, is rarely worth pursuing.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: