This question appears in Mumbai University > Software Testing & Quality Assurance Subject

Marks: 8M,10M

Year: Dec 2012, May 14,May 2014,Dec 2013

Test oracle :

• A test oracle is a mechanism that verifies the correctness of program outputs. The term test oracle was coined by William E Howden.

• An oracle provides a method to :

  1. Generate expected results for test inputs

  2. Compare the expected results with the actual results of execution of the implementation under test (IUT)

• It consists of two parts : a result generator to obtain the expected results and a comparator to compare predicted and obtained results.

• It is used by comparing the outputs of the system under test for a given test case input to the outputs that the oracle determines that product should have.

• Common types of oracle include Perfect Oracle, Gold standard Oracle, Parametric Oracle, Statistical Oracle.

• A test oracle might specify correct output for all possible input or only for specific input. It might not specify actual output values but only constraints on them.

Parametric Oracle :

• An algorithm is used to extract some parameters from the actual outputs and compare them with the expected parameter values as shown in the figure :

• It is a special case of a parametric oracle.

• In a statistical oracle, statistical characteristics of the actual test results are verified.

• Statistical Oracle is a Heuristic Oracle using statistical methods, especially statistical tests. The Statistical Oracle is applicable in case there are explicit formulae for the mean, the distribution, and so on, of characteristics computable from the test result.

• The above figure shows the principal structure of the Statistical Oracle in case of random test input generation. It consists of a statistical analyzer and a comparator.

• The statistical analyzer computes various characteristics that may be modelled as random variables.

• The comparator computes the empirical sample mean and the empirical sample variance of its inputs.

• Furthermore, expected values and properties of the characteristics are computed by the comparator (based on the distributional parameters of the random test input).

• The decision of a Statistical Oracle is not always correct—in contrast to usual oracles. At best, the probability for a correct decision can be given.

• An important restriction of the Statistical Oracle is that is cannot decide whether a single test case passes or not. It can only make this decision for a couple of test cases. If a failure occurs, no single test case can be identified that detected the bug—it is the couple of test cases as a whole.

• The Statistical Oracle does not check the actual output but only some characteristics of it.