Paired t-test in Machine Learning

When you’re starting out in machine learning, you’ll quickly find that building a model isn’t enough. You need to know if your new, fancy model is genuinely better than the old one. This is where statistical tests, like the paired t-test, come into play. They help you make data-driven decisions instead of relying on a gut feeling.

What is a Paired t-test?

Imagine you want to know if a new diet pill works. You would weigh a group of people before they start taking the pill and then weigh them again after a few weeks. You’re collecting two pieces of data from the same group of people. A paired t-test is designed for exactly this scenario: when you have two measurements from the same subjects or related units.

In machine learning, the “subjects” are usually datasets. You use the same data folds to train and test two different models. The test then analyzes the difference in their performance scores (like accuracy) to tell you if the observed improvement is consistent and statistically significant or just due to random chance.

Comparing Two Classification Models

Let’s say you have a dataset for classifying emails as ‘spam’ or ‘not spam’. You’ve built a Logistic Regression model and a Support Vector Machine (SVM) model. You want to know if the SVM is better.

1. Run both models: You evaluate each model on the exact same 10 validation sets (or test folds). This gives you two lists of 10 accuracy scores.
2. Calculate the differences: For each of the 10 validation sets, subtract the Logistic Regression score from the SVM score. This gives you a list of 10 difference scores.
3. Perform the test: The paired t-test analyzes these differences. It calculates the average difference and checks how spread out the differences are. The core question it answers is: “Is the average difference in scores significantly different from zero?”
4. Get the result: The test outputs a p-value. A small p-value (typically less than 0.05) suggests that the average improvement from using the SVM model is real and not a fluke. A large p-value means the difference isn’t statistically significant.

The K-Fold Cross-Validation Paired t-test

In the example above, we used 10 validation sets. A more robust and common way to do this is with K-Fold Cross-Validation (CV). Here’s how it works:

1. You randomly split your dataset into K equally sized folds (e.g., K=10).
2. You train both Model A and Model B on K-1 folds and test them on the held-out fold. You repeat this process K times, each time with a different fold as the test set.
3. This results in K performance scores for each model, all from the same data splits.
4. You now have paired results (the scores from the same test fold) and can perform the paired t-test on these K pairs of scores.

This method is powerful because it maximizes the use of your data and ensures the comparison between the two models is fair and based on the exact same testing conditions.

t-test vs. K-Fold CV Paired t-test

Aspect	Standard Paired t-test	K-Fold CV Paired t-test
Data Usage	Typically uses a single, static training/test split or pre-defined pairs.	Uses K iterations of training/test splits on the same dataset, making more efficient use of the data.
Result Robustness	Results can be highly sensitive to how the single train/test split was made.	Results are more robust and reliable as they are averaged over multiple splits, reducing variance.
Common Application	Comparing any two sets of paired measurements (e.g., pre-test vs. post-test scores).	Almost exclusively used in machine learning for comparing model performance fairly.
Output	A single p-value from one set of paired comparisons.	A single p-value derived from K paired comparisons (the scores from each test fold).