The other day I participated in a meeting with the group SNESCM (Scandinavian Network of Excellence in Software Configuration Management) concerning the subject “CMCM: Showstoppers for Continuous Delivery in Small-Scale Projects”. A thesis examining different projects showed that the testing was an Achilles’ heel in implementing Continuous Delivery. Despite Agile principles a sort of Waterfall organization regarding the release and release testing at the end of the projects. Thus an important point for discussion was test automation.
An essential question during the meeting was what a CM responsible can do to facilitate the implementation of Continuous Delivery. According to me, a CM responsible can for example enlighten trends by having a good control of all constructions, the testing of each construction etc. By doing so, all projects receive a foundation for their decisions about how to test.
As a project leader, I want an overview on how the tests are developing over time, additional to the, obvious, update on how the latest construction is working. From the project, this requires a knowledge of what has been constructed, alterations in each construction as well as which tests that have been accomplished and not, the results of these and, possibly the most important part, the outcome compared to earlier results.
With the information above in mind, we can take decisions concerning the second topic during the meeting: which tests should be automatized? According to me, the pyramid consisting of a base of unit tests, an intermediate of integrations tests and a top of GUI tests is a useful figure. This point of view combined with an automatization of the stable parts, whatever that means, of the GUI tests generates the best efficiency. Attempts to automatize the GUI tests in an unstable product (unstable meaning low quality in each unit or frequently changing requirements, as in Agile development), is useless and expensive. Instead, the automatization of the GUI tests should be carried out later. During intense periods of development, manual GUI tests are necessary. Not only due to constant alterations of the GUI, making it impossible to keep the automatic tests up to date, but also due to the automatic tests only running the most obvious scenarios. Automatic tests run the same cases all the time, preventing regression but never detecting eventual new problems. Personally, I would rather use exploratory testing in these cases. Let 20-30 testers look for all sorts of errors when the feature is actually finished.
In due time, when the client is concent with a certain feature, a regression test can be added through GUI test automation, but should still be complemented with manual tests executed as exploratory testing. With critical functions such as payment solutions etc, a number of test scenarios ought to be run manually to ensure the quality; integration with other systems might fail even though the product itself has not changed. I found another article regarding the subject.
Doing Continuous Delivery all the way involves numerous releases each week, and consequently the expenses for enough testers to perform the guarantee of quality can be extensive. However, there are alternative solutions. Send me an email and I will tell you more about my experiences of Community Testing and how it should be integrated with your Continuous Delivery process.