Thu 14 May 2009
No Small Changes
Posted by Scott Westfall under Dev Management, SlickEdit Products
[3] Comments
Recent experiences with SlickEdit 2009 have reminded me that there are no small changes. Even when you think you understand the nature of a change, severe side-effects can pop up in complex software.
This was the case when we put out our first release of SlickEdit 2009, version 14.0.0.5. We had completed our usual beta cycle before releasing this version, which included a number of drops to correct reported problems. After the last beta drop, we made a few more changes, all of which seemed totally risk-free.
Within two hours of releasing SlickEdit 2009, we had reports of severe problems with the File tabs and the File list. These problems trace back to one of these simple, risk-free changes. We had done our normal testing after these changes, and we hadn’t discovered any problems. But this problem was so severe we had to pull the release until it could be fixed.
So, what did we learn from this?
First, there are no small changes. Even when you think you know the effect a change will have, you could be wrong.
Should we have tested more? Sure, you can always test more, but that is no guarantee that you will find a problem. Testing suffers from a horizon effect: you can only see as far as you test. The problem you need to find could lay just beyond or miles beyond. You don’t know until you find it.
Second, testing is different than using. When you are testing, you click all the buttons and try all the functions, but you are rarely able to perform all combinations of operations. Some problems depend on the order in which operations are invoked. Others require multiple operations before the problem surfaces, as was the case with the problem we missed.
It should be no surprise that the parts of SlickEdit that work the best are the things we use everyday. But there are many languages we don’t use on a daily basis. Even if we did, with a product like SlickEdit, our testing cannot hope to replicate the variety and complexity of our customers’ use patterns.
We’ve determined that prior to shipping any release, we need help validating it. So, we’ll create a Release Candidate when we have finished our testing. We’ll make the Release Candidate available for early adopters to test, and if all goes well, it will be used for the General Availability release. In this way, we hope to catch more problems before they are set free to roam the countryside and terrorize the populace.
Q&A
Does that mean you’ve given up on testing? No, we plan to continue testing just as we always have. We just need help with final validation.
What if I don’t have time to help you catch problems? Then wait until a release is no longer a release candidate. At that point, others will have helped to validate it, and you will have less likelihood of experiencing issues with a new version.
If an unladen swallow flies west from New York…
May 24th, 2009 at 7:34 am
“When you are testing, you click all the buttons and try all the functions”
Does SlickEdit not use unit testing like JUnit then ?
May 27th, 2009 at 12:26 am
I’m also curious as to whether SlickEdit has an automated regression test suite (such as one built using JUnit).
My impression is that automated unit testing has not yet caught on as widely in the C/C++ community as it has in the Java/Ruby/other communities. It is certainly a very valuable and powerful QA tool.
However, it’s quite hard to add unit tests retrospectively to a code base that is not designed for them.
John Hurst
Wellington, New Zealand
May 27th, 2009 at 2:20 pm
It’s true that SlickEdit doesn’t have near as many automated tests as we’d like, but that doesn’t change the nature of the horizon effect. It just extends the horizon.
Even with automated tests, you tend to test operations in the combinations that you expect to have a dependency. You have to conceive of the test case before you implement the test. If you are unaware of an interaction you won’t think of the test case in the first place.
With the particular defect that motivated this article, we had a hard time reproducing it even after our customers reported it. But once we understood what was going on, we could reproduce it easily. While I am a huge fan of automated testing, I don’t think it would have helped in this situation.