Never Again
Did We Leave Something Out? Not!
Here's what happened when a testing team ignored a "minor" change to some SQL code.
Many years ago I worked for Worldcom -- long since bought, bankrupted and bought again. This company had developed a PC-based network-management system that ran at one of its customer's sites. With the developers and the customer in different states, releases were distributed via the network.
During the final testing of a major release, a problem was found in one of the nightly database-cleanup processes that made the process run for more than 10 hours. A review of the SQL code found that the process was extremely convoluted, with some SQL statements having as many as 10 embedded selects and table joins.
The code was rewritten based on simplified SQL using "delete where NOT in" logic. All aspects of the change were unit-tested successfully and the processing time was reduced down to minutes. The modification was provided to the software integration and testing team so they could incorporate it into the release.
About 90 percent of the release testing was completed when the modified code was given to the test team. It was decided that, due to the success of the unit test -- and the fact that it was viewed as only a "minor" change -- the updated code would not be tested as part of the full release. My team didn't know about this.
The full release was staged for distribution via the network that night. Everyone went home for the weekend after some hearty back-patting on a job well done.
On Monday, the customer wanted to know what happened to all of its data. My immediate thoughts were something along the lines of "they left the �0;!@ 'NOT' out of the SQL code," followed by, "but it worked when we unit tested it and again when they did the release testing." After about 30 minutes, however, my gut instincts were confirmed: The "NOT" was not in the released versions of the code.
The end result was the customer was down for about a week while we developed a new process to recover all of the missing data.
From that point on, it didn't matter how small or insignificant a change appeared to be, it was mandated -- meaning, your job was on the line -- that a full test cycle be run, even if it meant going back to square one.
The moral of the story: No matter how insignificant a change may be, it should never be trivialized. Any cost savings may be dwarfed by the loss in your reputation, customer confidence and satisfaction and, in some cases, the bottom line.
About the Author
The submitter of this "Never Again" story wishes to remain anonymous.