Test Oracles
Test Oracles
A test oracle is a mechanism used in software testing to determine whether a system under test behaves correctly for a given input. It helps testers verify if the actual output matches the expected result, acting as a “source of truth” during validation.
Capabilities of a Good Test Oracle
- Validate results accurately across diverse scenarios
- Handle both expected and unexpected inputs effectively
- Detect inconsistencies in outputs efficiently
- Provide clear pass/fail criteria for tests
- Adapt to system changes with minimal maintenance
A good test oracle should have the following:
Generator
Creates test input data or predicts expected outcomes based on specifications. Example: Automatically generating valid credit card numbers for payment processing tests.
Comparator
Compares actual system outputs with expected results. Example: Checking if a calculated total matches pre-computed values after a discount application.
Evaluator
Assesses whether test results meet acceptance criteria. Example: Verifying response times fall within specified thresholds for performance tests.
Examples of Test Oracles
- Predefined Outputs: Expected results stored in test scripts (e.g., login success/failure messages)
- Reference Implementation: A legacy system used as a benchmark for new software
- Heuristic Oracle: Rules-based checks (e.g., “Total should never decrease when adding items”)
- Statistical Oracle: Validates output patterns against historical data trends
- Human Oracle: Manual verification by domain experts for complex scenarios
Example Scenario
Test Input: User submits form with valid email Oracle Action: Check if confirmation message matches "Success! Check your inbox"
