Last night was the occasion for the Belfast Selenium meetup group for a talk by Richard Bradshaw on Flaky Testing and why you need to avoid it. Richard blogs about testing and is involved with the Ministry Of Testing. There was a good turnout for the Flaking Test event and once again we were hosted by Puppet in Linenhall Street. Using a pnemonic “SCARED” – Richard took us through some of the issues, pitfalls and potential approaches to avoid creating tests that either fail intermittently or that are not fit for purpose. Hence the Flaky reference – Flaky Good, Flaky Bad ? . He was also focusing on the automation of testing and the potential for having the knowledge of how to operate a test tool without thinking about the application and tests required for it
State
Codified Oracles
Algorithm
Reporting
Execution
Deterministic
Taking each of those briefly.
State – ensuring that the application state can be controlled for the purpose of setting up testing.
Codified Oracle(s) – the use of one or more oracles to inform the testing.
Algorithm – the method of defining the tests 9not the implementation.
Reporting – ensuring that the appropriate level of reporting and logging are present throughout – rather than trying to capture when a bug or issue is identified.
Execution – the method by which the tests will be run and at which level (UI versus API for example).
Deterministic – ensure that the testing is deterministic by design
Ministry of Testing
I thought that Richard’s explanation of the current common approach to testing and the flaws that it raises was quite interesting. The feeling that tests and code are “manipulated” to deliver a “green” result rather than the best test coverage. As usual there were a few questions from the floor and from the types of questions I felt that the talk had alt least provoked a number of questions especially in areas where conventional testing may be proving hard to use. Plenty of food for thought from this meetup and especially in the area of tools that may be of benefit – such as Swagger, Wraith, Mocky and Mockable