There’s a great old commercial for Tootsie Roll Tootsie Pops that reminds me a lot of what happens on most software projects. The commercial features a crudely drawn cartoon of a boy trying to figure out a vitally important question.

Boy: Mr. Turtle, how many licks does it take to get to the Tootsie Roll center of a Tootsie Pop?
Mr. Turtle: I’ve never even made it without biting. Ask Mr. Owl.
Boy: Mr. Owl, how many licks does it take to get to the Tootsie Roll center of a Tootsie Pop?
Mr. Owl: Let’s find out. A One… A two-HOO…A three.
(crunch sound effect)
Mr. Owl: Three!

This illustrates two common problems in oral communication. The first is the iterative process of finding someone who might know the answer to your question. In software development, the questions are typically of a more concrete nature, relating to a specific design or implementation choice. (more…)

I’ve worked at two kinds of companies in my career:

1. those with open and honest communication
2. those whose communications better resembled Soviet-style propaganda

As a manager, I’ve always considered accurate information one of my most important assets. I need to know where we are and what it’s going to take to succeed. My team has standing orders to let me know as soon as they figure out that they’re not going to make it so we can adjust the plan.

I’ve always been surprised when the people I have reported to didn’t share this enthusiasm. As managers, our chief responsibility is making effective decisions to achieve the stated goals. How can you possibly do that in an environment that doesn’t encourage people to speak the truth? (more…)

Recent experiences with SlickEdit 2009 have reminded me that there are no small changes. Even when you think you understand the nature of a change, severe side-effects can pop up in complex software.

This was the case when we put out our first release of SlickEdit 2009, version 14.0.0.5. We had completed our usual beta cycle before releasing this version, which included a number of drops to correct reported problems. After the last beta drop, we made a few more changes, all of which seemed totally risk-free.

Within two hours of releasing SlickEdit 2009, we had reports of severe problems with the File tabs and the File list. These problems trace back to one of these simple, risk-free changes. We had done our normal testing after these changes, and we hadn’t discovered any problems. But this problem was so severe we had to pull the release until it could be fixed.

So, what did we learn from this?

First, there are no small changes. Even when you think you know the effect a change will have, you could be wrong.

Should we have tested more? Sure, you can always test more, but that is no guarantee that you will find a problem. Testing suffers from a horizon effect: you can only see as far as you test. The problem you need to find could lay just beyond or miles beyond. You don’t know until you find it.

Second, testing is different than using. When you are testing, you click all the buttons and try all the functions, but you are rarely able to perform all combinations of operations. Some problems depend on the order in which operations are invoked. Others require multiple operations before the problem surfaces, as was the case with the problem we missed.

It should be no surprise that the parts of SlickEdit that work the best are the things we use everyday. But there are many languages we don’t use on a daily basis. Even if we did, with a product like SlickEdit, our testing cannot hope to replicate the variety and complexity of our customers’ use patterns.

We’ve determined that prior to shipping any release, we need help validating it. So, we’ll create a Release Candidate when we have finished our testing. We’ll make the Release Candidate available for early adopters to test, and if all goes well, it will be used for the General Availability release. In this way, we hope to catch more problems before they are set free to roam the countryside and terrorize the populace.

Q&A

Does that mean you’ve given up on testing? No, we plan to continue testing just as we always have. We just need help with final validation.

What if I don’t have time to help you catch problems? Then wait until a release is no longer a release candidate. At that point, others will have helped to validate it, and you will have less likelihood of experiencing issues with a new version.

If an unladen swallow flies west from New York…