The state of automated user acceptance testing
The subject of user acceptance testing is pretty crucial for agile development teams. Its certainly an issue which we have given quite a lot of thought to in our XP team in the past few months. There are two points I want to make in regard to this, the first concerning our overall approach to UAT.
To break it down for those people who are not aware of the debates around automated UAT in agile there are basically two schools of thought. One says that the developers and the customers should sit down before the coding starts for a particular feature and put the tests together for it. The rationale behind this is that this will initiate a discussion between these two parties during which a better understanding of the requirements will emerge. This ongoing discussion between developers and customers is particularly important in agile as there is no formal requirements spec adopted by most of the main methods.
The other way of thinking suggests that the tests should not be written way before the production code is done because, in that state, they are not actually doing what they are supposed to namely documenting the code. This view is informed by a Lean perspective which says that we should cut down on artefacts which don’t contribute to our overall goal. In other words tests which are not documenting code are ‘inventory’ and should be cut out. This second approach is one of the ways in which Lean thinking is influencing the day-to-day approach to agile. I think we can expect to see more and more of this in the near future, which, as I’ve recently written is a good thing.
Whichever approach is adopted for testing it still remains crucial for agile teams. Given the increasing popularity of agile and the importance of UAT for it one would expect there to be many different frameworks to allow automation of the process. Not quite. The ones we have tried in our team all have problems and while we have settled on using Selenium we often speak of it as being the ‘best of a bad bunch’ rather than ‘best of breed’. Now ours is a very small team and it got me wondering how larger teams cope with these flaky frameworks. Do they use them at all or are they in fact more reliant on the manual variey of UAT. I don’t know of any studies which investigate this issue but they would make for interesting reading.