You've no doubt seen the phrase "Quality cannot be tested in". Bugs can be eradicated, but the essence of quality is knowing what you are doing every step of the way, and communicating information in a clear and unambiguous way.
Quality software requires a process to insure all the bases are covered. Delivering on that promise depends on the final step in the Parallel Development Process, the QA step. QA involves testing but is very different from the testing programmers usually do. QA is testing done according to a strict script that has been developed independently from the programming and from the exact same source of information, the Formal Requirements Document (FRD). This process insures an independent verification of the program's function, sometimes called Verification and Validation.
The QA script is developed from the FRD. The purpose of the script is to test valid data, the boundaries of valid data and proper handling of invalid data. Most QA will require that data be entered into files or a database before QA can begin. For complete coverage it is necessary to avoid the use of existing data or extracts from that data. The QA database is built with one example of every possible legal combination of data elements.
Creation of this data can be assisted by a simple program or automated tools. Along with each valid combination, the data elements need to have examples of the full range of data values. As a minimum this requires examples of:
Entries in a database are special. Only valid entries are present because when the program runs correctly, only valid values will be created. In cases where the database is not under your control or may have developed invalid entries, a separate program to filter out bad records should be run before entering them in an operational database.
The QA of the user interface is somewhat more complex. In addition to the three data points above, two or three others are needed:
Why do we test out of bounds? Simply to make sure that errors are detected and handled properly without impacting valid entries already in the computer. These tests insure that the program will be robust in the presence of human and computer error, and prevent errors from getting into the database. The out of bounds testing is as important as in bounds testing because it assures that bad data will not be passed to another program, possibly creating more serious problems later.
If it were necessary to test all combinations and permutations, a ten field entry window would require a huge number of tests to be complete. By the time the QA was finished, the product would be obsolete. Fortunately this is not the case. For those fields where the data does not interact, parallel testing can be performed. For example, in the ten field entry window, if only fields one and two interacted, and nine and ten interacted, then fields three through 8 could be tested together.
In the six data point range test, fields 3 to 8 could all be the same test point at one time. Thus six entries suffice to test those fields. For one and two, 6*6 entries are needed, as for nine and ten. Thus the complete coverage test on the ten element window requires 78 test entries in the script.
The QA should be run, and the results documented, by a group that is independent of the programmers and reports at the same level as the programming manager. The QA group needs to have equal footing in the company with the programmers to prevent the programming manager from ignoring the results of the QA process.
It is crucial that the programming manager understand that QA is not an antagonist to make them look bad, but a partner that enables them to deliver top quality. The programmers need to understand this too.
Another crucial point is that QA cannot be done in zero time. When programming delivers a system to QA, this is the beginning of a multiple cycle of QA, document and fix. This time must be added to the planned development schedule, not subtracted from the programming schedule. It is not the same as programmer testing and does not replace it.
QA extends programmer testing to assure full coverage and independent verification of the program function. It is sometimes called Verification and Validation, or V&V. Because of the parallel development of program, documentation and QA, the overall time to delivery may be close to the same as a typical serial development, with better results.
The QA may be performed at any time after the program is functional, though because of the effort involved, it usually begins after the programs can process sample data correctly. A typical QA will test the major components independently until they are clean, then add the cross component testing. If each major component is a phase of QA, then the QA process looks like a program:
For QA Process;
For Each Phase;
Initialize Files & DB;
Enter QA Data;
Observe Results;
Document Errors;
Dump Files & DB;
Reset System;
End Phase;
Analyze Files & DB;
Assemble Error report by function;
End QA Process;
Each pass of the QA process yields documentation which shows the error in context, with before and after records if needed. This detail is key in enabling programming teams to locate and correct a bug quickly. When the correction - QA cycle goes quickly, it makes possible several turns and thorough testing without a long release delay. In many cases the traditional beta testing can be reduced or even eliminated.
Doing QA properly means that steps are not skipped, that areas that test correctly in one pass are still tested in the next pass. This does not preculde the QA group from doing an extra QA on one phase which is causing major problems. Cooperation between programming and QA is what enables choices to be made about using a full or partial pass. In any case, the last pass must be a full one to catch regression errors.
A company which writes software for outside customers will be pressured by marketing to release software with known bugs to meet promised delivery schedules. Sometimes marketing will make delivery committments without any input from programming, which creates a whole set of problems outside the scope of this document. Similar pressures can arise with internal projects. In either case, managing the software release requires a fine balance between customer relationships, schedule commitments, marketing desires and company reputation.
It is for these reasons that I strongly recommend that the QA and programming teams report at the same level as marketing, ideally to a vice president. When demands for early release arise, the decision needs to be made by someone who understands all of the implications to the company. Risking a release with bugs is viewed much differently at the executive level than at the marketing level.
As a former QA manager, I held out for very high quality before release in the face of considerable marketing and programming pressure. The resulting positive feedback from the customers nearly eliminated that pressure in subsequent releases.
The phrase "Quality is Free" doesn't mean QA won't cost you anything. In the software business it means the value of releasing essentially bug free software will more than compensate for the costs involved. It also enhances customer retention. Customers are funny that way - they like not having to fight or avoid bugs. It makes your current customers good references for prospects and enhances your company's reputation. And believe it or not, programmers are proud to be associated with a company which believes in quality.
This wraps up the revised five part (Introduction plus original four parts) 'Building Software in an Organized Fashion' or BSOF as I refer to it. Detailing this approach would take a book and some hands on experience to convey the real difference in a development environment that using BSOF can make. I hope the broad outlines in this series will help you in future development efforts.
Just in case you have something you want to get off your chest, comments good or critical, experiences to share, or thoughts to add; You can reach me via email - see 'Contact the Author' in the links column. I'll be happy to hear from you.