May SPIN – Transition From Manual to Automated Testing

Tim Leach heads a test automation group at ACS as a Systems Consulting Manager in Quality Engineering. Tim is one of those guys who knows his stuff so well that the presentation sort of got in his way. He could have single-handedly led a discussion on transition to automated testing without any preparation and we could have soaked it all in all night. That became obvious as we neared the end of the evening when Tim drilled into what he calls Keyword Scripts.

Until that point, Tim ran through some pretty straight-forward scenarios of the planning and expectation setting involved in trying to transition an organization away from manual testing processes. With all real business process re-engineering, no silver bullet exists to make the transition to full test automation easy. As business progresses at lightening speed, we all find ourselvesTim Leach with less time to do more work. With developers having the latest cool tools they’re cranking out more than testers can hope to test on a good day. And management expects that a five-minute coding change should only require five minutes of testing.

So where does an organization start? First, evaluate and take an inventory. What tools do you have already? Then set a baseline here as a point from which you can measure progress and modifications to the process as you move forward in experience and you add tools to the testing environment. Then find a good pilot project. Not all projects lend themselves to automated testing, and large, mission-critical projects don’t lend themselves well to pilot projects. Finally, plan the move so that you can execute. This is no different than a development project as you’ll need the same foundation and tool sets to support your efforts that developers have to support their development efforts.

As you begin your efforts, know that you will hit road-bumps and issues along the way. So don’t choose a project with a critical path when making your first efforts here. And ensure you set management expectations correctly along the way. This is a long-term delivery effort. Probably a year or more.

When selecting tools, don’t accept vendor sales stories. They all claim to work well together, but Tim made it clear that he’s never seen two tools that truly integrate without issue. Also, testing tools require quite a bit of overhead so be ready to upgrade machines to handle the necessary horsepower. In the end, an automated testing effort will add cost, and experiencing the claims of automated testing reducing manpower and overall cost simply don’t pan out. Testing environments and equipment are expensive. Automation, itself, requires a true developer and not just an analyst experienced in manual testing. You need to develop the tests and test environment the same way that you develop the code or the product itself.

Okay, so when you set these kinds of expectations management will put your efforts under a microscope. So you need to report both the good and the bad in a constant stream of communication so that everyone involved is on the same page and no ambiguity exists.

When developing a plan, align your equipment and your personnel, and prepare your project team. Shoot low, take small steps, and report that you’ve hit your real goals successfully. You’ll develop more confidence, and management will be more likely to support your efforts going forward. Execution will reveal obstacles and you’ll face the same obstacles that development efforts face, so be sure to keep a good eye on scope creep.

For bang for the buck, start with load and stress testing because this can be handled with much less manpower than automated functional testing. What automation guarantees is that the tests run the same way every time. With manual testing this is not the case. No human will really test the same way every time no matter how well the test plans are written. The implication of automation here is that you’ll be able to point to some part of the process where something has changed, as the tests, themselves, don’t change, and they will run the same way every single time. So if the development manager says, “We didn’t change anything in the code,” you’ll know that’s not true. Something must have changed, whether it’s in the code or the environment.

Keys to success include setting correct expectations, not piloting in the critical path because you’ll fail and you won’t get a second chance, keeping everyone informed, and working in small cycles. Sounds like agile development, eh? Keep in mind the transition is a long one. Tim has not completed a transition in under a year.

After talking through the transition issues, Tim took some questions.

Where should the tests reside? Should they be kept with the code in the configuration management tool? It really depends on the team and the choices they make. Some tools integrate better with version control tools than others. Choose the company standard because you’re going to need to get help for release management and change management, and it’s better to have them on your side than not.

You can use record and playback to populate data from one application to another.

Automation keeps you off the manual treadmill. You will eventually run out of space and money to continue adding people. You will not have enough resources to keep up. Especially in an agile environment. If your development methodology is agile, you need an agile testing methodology with automation at its core in order to match the pace.

Automation requires specialized skills and there is no silver bullet. The tools do some of the work. Humans are still a necessary part of the process. A tool gives you the ability to automate. Still, manual testers are not going anywhere, they are still necessary to test the changes while automation allows coverage and security around the functions that don’t change. Automation will not find new bugs. It will find bugs that have existed as you continue to add details to the test scripts. In fact, keep the regression tests that validate bugs in the scripts so that you don’t introduce the same bugs again. And, finally, automation is not the end, but the beginning. You’ll continue to build out your automated test processes once you’ve won the battle to make the transition happen.

Like a planned architecture in the software development space, test automation planning requires thinking ahead for reuse and migration of tests from project to project. Creating data is a difficult barrier to cross when creating a test environment. To get to automated testing quickly use keyword testing. Keyword testing allows you to create a framework around testing scripts that can be used both manually and with an automated tool.

The process essentially works like this: A driver script loads keyword scripts and runs them against the application. The concept is to develop individual actions against a UI, i.e. put the username in the username field. Define an action by screen, test step, and the data to enter. Stick these in columns in a spreadsheet. Then run them. For instance, an action might place username data AErickson into the username field as a test step on the login screen. Chain a number of these actions together, feed it into a driver, and you’ve got some tests to get things running quickly. The steps in a login screen might be 1) put the username in the username field, 2) put the password in the password field, and 3) press the login button. Each one of these steps is placed on a row of the spreadsheet.

The beauty of this spreadsheet process is that if automation is down then these scripts work for manual testing. A manual tester can follow the spreadsheet just as easily as the driver script can.

You’ll find an example of the spreadsheets and Keyword Scripts at the end of Tim’s presentation. You’ll find them here once they are posted for everyone.



~ by Andy on May 15, 2008.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: