In my opinion, the most difficult part of automated user acceptance testing is maintaining the tests rather than writing them (We are talking about serious automated testing here, hundreds or more runing every day).

My rules of user acceptance testing

• Customers shall understand the test scripts easily.
• Each line of test script shall match (or at least very very closely) one step of operation as if the test is executed manually
• A test case can be executed individually or as a part of suite or all.
• If a test fails, the console output show error/failure stack trace, including the line number
• After the test is run whether success or failure, it shall not affect other tests’ result (it is quite hard)
• If a test fails, the system is left with the state the tester can check manually.

Preferably

• Users/Customers write the test scripts (It’s actually much easier than it sounds, :-)), developers write helper tier undereath the tests. For example,

// In test script (java):
assertEquals(getTransactionDate(), today())

// In test helper (parent class):
protected String getTransactionDate() {
// retrieve the transaction date on the screen
}
protected String today() {
// return today in defined format
}

• The tests are run everyday (obviously), and with (funky) test report
• The test case name clearly indicates its intention, such as test_guest_cannot_access_member_area_redirect_to_login
• The testing framework is in some kind of scripting language (no compiling), they are ‘test scripts’ after all.
• Only one configuration parameter: env. For example, to test staging environment, just pass the -env=staging in execution parameter

A common mistake: It is easy for developers to put her in dominant position of choosing the framework and writing automated functional tests ; Sadly it is even easier for customers/testers to accept that.

Tags: functional testing