Similar questions

Explain broad criteria of test automation tool evaluation

Marks: 10

Year: June 2012,Dec 2012 , June 2013, June 2015

Evaluation And Selection Of Test Automation Tools

• A test automation is a software application that assists in the automation of test cases that would otherwise run manually.

• The broad criteria for evaluating test automation tools have been classified into the following eight categories :

1.Test Development Criteria

• An automation test tool should provide a high level, preferably non-proprietary, easy to use test scripting language such as Tcl.

• It should have the ability to interface and drive modules that can be easily written in, for example C, Tcl, Pearl or Visual Basic.

• The tool must provide facility to directly access, read, modify and control the internals of the automated test scripts.

• The tool should have built in templates of test scripts, test cases, tutorials and demo application examples to show how to develop automated test cases.

2.Test Maintenance Criteria

• The tool should possess a rich set of features, such as version control capability on test cases, test data, and migration of test cases across different platforms.

• The tool must provide powerful, easy- to- use facilities to browse, navigate, modify and reuse the test suites.

• The tool should have the ability to select a subset of test cases to form a group for a particular test run based on one or more distinguishing characteristic.

• A tool needs to have features to allow modification and replication of test cases, easy addition of new test cases, and import from another.

3. Test Execution Criteria

• An automation tool should allow test cases to be executed individually, as a group, or in a predefined sequence.

• The user should have the ability to check the interim results during the execution of a group of tests.

• The user should have the option to pause and resume the execution of a test suite.

• The tool should have the facility to execute the test suite over the Internet. The tool should allow simultaneous execution of several test suites that can be distributed across multiple machines for parallel execution.

4. Test Results Criteria

• The test tool must provide a flexible, comprehensive logging process during execution of the test suite, which may include detailed records of each test case, test results, time and date.

• The test result log can be archived in a industry standard data format and the tool should have an effective way to access and browse the archived test results.

• Finally, the tool should have the capability to collect and analyse response time and throughput as an aid to performance testing.

5. Test Management Criteria

• A tool should have the ability to provide a test structure, or hierarchy, that allows test cases to be stored and retrieved in a manner that the test organization wants to organize.

• The tool should have the capability to allocate tests or group of tests to specific test engineers and compare the work status with the plan through graphic display.

• A tool needs to have authorization features. For example, a test script developer may be authorized to create and update the test scripts while the test executer can only access them in the run mode.

6.GUI Testing Capability Criteria

• An automated GUI test tool should include a record/ playback feature which allows the test engineers to create, modify and run automated tests across many environments.
• These tools should have a capability to recognize and deal with all types of GUI objects, such as list boxes, radio buttons, icons, joysticks, and bitmap images with changes in color shades and presentation fonts.
• The performance of a GUI test tool needs to be evaluated. One may consider the question: How fast can the tool record and playback a complex test scenario or a group of test scenarios?

7.Vendor Qualification Criteria

• Many questions need to be asked about the vendor’s financial stability, age of the vendor company, and its capability to support the tool.

• The vendor must be willing to fix problems that arise with the tool. The maturity and market share of the product must be evaluated.

8.Pricing Criteria

• Pricing is an important aspect of the product evaluation criteria.

• One can ask a number of questions : Is the price competitive? Is it within the estimated price range for an initial tool purchase?

• For a large number of licenses, a pricing discount can be negotiated with the vendor. Finally, the license must explicitly cap the maintenance cost of the test tool from year to year.