Thu 14 May 2009
Recent experiences with SlickEdit 2009 have reminded me that there are no small changes. Even when you think you understand the nature of a change, severe side-effects can pop up in complex software.
This was the case when we put out our first release of SlickEdit 2009, version 126.96.36.199. We had completed our usual beta cycle before releasing this version, which included a number of drops to correct reported problems. After the last beta drop, we made a few more changes, all of which seemed totally risk-free.
Within two hours of releasing SlickEdit 2009, we had reports of severe problems with the File tabs and the File list. These problems trace back to one of these simple, risk-free changes. We had done our normal testing after these changes, and we hadn’t discovered any problems. But this problem was so severe we had to pull the release until it could be fixed.
So, what did we learn from this?
First, there are no small changes. Even when you think you know the effect a change will have, you could be wrong.
Should we have tested more? Sure, you can always test more, but that is no guarantee that you will find a problem. Testing suffers from a horizon effect: you can only see as far as you test. The problem you need to find could lay just beyond or miles beyond. You don’t know until you find it.
Second, testing is different than using. When you are testing, you click all the buttons and try all the functions, but you are rarely able to perform all combinations of operations. Some problems depend on the order in which operations are invoked. Others require multiple operations before the problem surfaces, as was the case with the problem we missed.
It should be no surprise that the parts of SlickEdit that work the best are the things we use everyday. But there are many languages we don’t use on a daily basis. Even if we did, with a product like SlickEdit, our testing cannot hope to replicate the variety and complexity of our customers’ use patterns.
We’ve determined that prior to shipping any release, we need help validating it. So, we’ll create a Release Candidate when we have finished our testing. We’ll make the Release Candidate available for early adopters to test, and if all goes well, it will be used for the General Availability release. In this way, we hope to catch more problems before they are set free to roam the countryside and terrorize the populace.
Does that mean you’ve given up on testing? No, we plan to continue testing just as we always have. We just need help with final validation.
What if I don’t have time to help you catch problems? Then wait until a release is no longer a release candidate. At that point, others will have helped to validate it, and you will have less likelihood of experiencing issues with a new version.
If an unladen swallow flies west from New York…