0
904views
What is a Paired T Test (Paired Samples T Test / Dependent Samples T Test)?

Imagine you designed two classification algorithms, labeled A and B. The table below shows the accuracy that was observed for 10-fold cross-validation. Using a paired t-test, can you determine whether the performance differences are statistically similar? Extremely statistically similar? show all of your work.

A B
73 93
81 83
82 83
72 90
85 89
86 78
87 93
89 90
93 95
81 90

Assume Alpha Level is 0.05 (5%)

1 Answer
0
23views

Paired T-Test

  • A paired t-test is also called a Correlated Paired t-test, a Paired-Samples t-test, or Dependent Samples t-test.
  • Paired T-Test is used when two samples are somehow connected or dependent or perform tests on the same things.
  • In the given scenario, two classification algorithms called A and B test the accuracy on the same 10-fold cross-validation samples.
  • Therefore, here paired t-test is applicable to determine the performance differences of the two classification algorithms A and B are statistically similar or not.

Step 1 –

The null hypothesis in the paired t-test assumes the means are equal.

Therefore,

Null Hypothesis:

H0 - There is no difference between classification algorithms A and B. The expected accuracies of classification algorithms A and B are the same.

$$H_0: µ_d = 0$$


Step 2 –

Find out Differences by subtracting the accuracies of algorithm B from the accuracies of algorithm A.

A B Difference (A - B)
73 93 -20
81 83 -2
82 83 -1
72 90 -18
85 89 -4
86 78 8
87 93 -6
89 90 -1
93 95 -2
81 90 -9

Step 3 –

Find out the square of all the differences generated in step 1.

A B Difference (A - B) Square of the Difference
73 93 -20 400
81 83 -2 4
82 83 -1 1
72 90 -18 324
85 89 -4 16
86 78 8 64
87 93 -6 36
89 90 -1 1
93 95 -2 4
81 90 -9 81

Step 4 –

Find out the sum of differences and the sum of the square of differences.

Therefore,

Sum of Differences = ∑ D = - 55

Sum of Square of Differences = ∑ D2 = 931


Step 5 –

Calculate the value of t using the formula of t-test.

T-test Formula

Where,

Sample Size = N = 10

Sum of Differences ∑ D = - 55

Sum of Square of the Differences ∑ D2 = 931

Therefore,

t-test calculation

Now, we get the value of t = - 2.081


Step 6 –

Find out the Degree of Freedom (DF) = Sample Size (N) - 1

Therefore,

Degree of Freedom (DF) = Sample Size (N) - 1 = 10 - 1 = 9


Step 7 –

Find out the p-value in the t-table, using the Degrees of Freedom (DF) calculated in step 6.

Also the value of the Alpha Level is given as 0.05 (5%).

Therefore,

p-value = 2.262 .....(for DF = 9 and Alpha level = 0.05 using t-table)


Step 8 –

Compare the p-value = 2.262 with calculated t-value = - 2.081

The calculated t-value is less than the t-table value at an Alpha Level of 0.05 (5%)

The p-value is greater than the Alpha Level: p > 0.05.

Hence, We can ACCEPT the Null Hypothesis (H0) that there is no difference between the classification Algorithm A and B.

That means Classification Algorithms A and B are Extremely Similar.

Please log in to add an answer.