Suppose a test team in a software project is highly motivated, it produces an impressive amount of unit tests, thus creates the most stable system, but the customer is still unsatisfied as the features do not work as he imagines them. Or the team is testing precisely the smallest layout error and the project manager becomes desperate for the escalating costs. Or even worse: a software has been extensively tested, but the core functions have been forgotten or the compatibility to an important target platform was not verified. Or even much worse: the test has not found critical errors because the used methodology was inappropriate.
All these are examples of non-effective testing. Hardly a test procedure will be perfect, but such difficulties can be significantly reduced when some upfront planning is done, which is called a test strategy.
If such a test strategy is composed for the first time, a basic template is useful:
• What (Test subject)
• Acceptance criteria
For some projects, defining a strategy can be easy, usually if a similar project was already completed. However, one should think carefully about the methodologies to be used while creating the strategy, scrutinize what will they result and which fit well for the specific context.
Example: Entity Framework Sync
Entity Framework Sync is a library to enhance Entity Framework with synchronization features. The following test strategy was used for it:
Here you can see number of project specifics. Unlike a website or an application, this library is created in such a way that it can be installed in many applications.
As it is a data access component, it is quite critical for the application, since an error might cause data loss. Therefore, this component must be fundamentally stable, whereas here mainly unit tests are used to test the individual functions.
However, thereby usability is not at all ensured in real applications. For this, test applications must be constructed. If possible, the library should be checked directly with available real applications.
Further to test this library, a risk-based analysis was carried out. This is a very rigid, but immensely structured approach which is well suited for this project and has identified important weaknesses. On the other hand, it also implies a large effort.
For this library, it is important to design the core aspects of the test.
Entity Framework provides several "styles" on how to program it (Designer-driven, Code-driven, Database-first, ...). Therefore, the functionality must be tested effectively for each "style".
Also, it is useless if the functionality runs in a primitive data model, but not in a data model that uses all the possibilities of Entity Framework.
Example: Quality Spy
Quality Spy itself can be considered as a good example for a test strategy. Since it is a desktop application, a completely different test approach is needed than for a data access lib like EF Sync.
Here, the main test is done manually. It is important that the GUI is tested on different operating systems. In Windows 8, even the functionality should be completely tested once again since there the application runs under .NET 4.5 instead of .NET 4, which differs in detail.
Of course, the application should not be full of bugs, but it must not be tested so intensively for stability as in EF sync. Instead, it is important here to check the suitability for different scenarios.
In addition to the actual test strategy, a strategy chart is a possibility to communicate it more visually. It's kind of a cross between test strategy and test plan.
As a whole, a proper planning is important in testing as well as in development. A meaningful test strategy forms the foundation for effective software testing.