A model with 99% accuracy can still be useless. If only 0.5% of transactions are fraudulent, a model that predicts “legitimate” every time can look excellent while catching no fraud. The metric you choose determines what “good” means.
Start With the Confusion Matrix
Classification metrics begin with four outcomes. In this guide, “positive” means the event we want to detect, such as fraud.
| Predicted: Positive | Predicted: Negative | |
|---|---|---|
| Actually positive | True Positive (TP) | False Negative (FN) |
| Actually negative | False Positive (FP) | True Negative (TN) |
- TP: positive event correctly detected.
- TN: negative event correctly rejected.
- FP: false alarm; a negative case flagged as positive.
- FN: miss; a positive case predicted negative.
Remember: the second word (positive/negative) is the prediction; the first (true/false) says whether it was correct.
Example: a processor evaluates 1,000 transactions: TP=15, FN=5, FP=10, TN=970.
Accuracy: The Overall Hit Rate
Accuracy = (TP + TN) / (TP + TN + FP + FN)
Here, (15 + 970) / 1,000 = 98.5%. Accuracy is useful when classes are reasonably balanced and error costs are similar. With rare events, it can mislead: a model that predicts “legitimate” for every transaction gets 980/1,000 = 98% accuracy while detecting zero fraud.
Exam clue: fraud, rare disease, equipment failure, or security threats signal class imbalance. Do not trust accuracy alone.
Precision: Trustworthiness of Positive Predictions
Precision = TP / (TP + FP)
15 / (15 + 10) = 60%. Of the transactions flagged as fraud, 60% really were fraud. Prioritize precision when false positives are costly—for example, when a spam filter must avoid hiding legitimate business email or an automated system must avoid banning innocent users.
Recall: Coverage of Actual Positives
Recall = TP / (TP + FN)
15 / (15 + 5) = 75%. The model catches three-quarters of actual fraud. Recall is also called sensitivity or true positive rate. Prioritize it when false negatives are costly, such as missing a safety-critical defect.
Precision vs. Recall: The Trade-Off
| Threshold | Precision | Recall |
|---|---|---|
| Lower: flag more cases | Usually falls | Usually rises |
| Higher: flag fewer cases | Usually rises | Usually falls |
Most classifiers turn a score or probability into a decision using a threshold. The business must decide which error costs more.
| Scenario | Costlier error | Metric to emphasize |
|---|---|---|
| Spam filter | Legitimate email blocked (FP) | Precision |
| Safety defect detection | Defect missed (FN) | Recall |
| Fraud with analyst review | Missed fraud, subject to review capacity | Recall and precision |
| Automatic account bans | Innocent user banned (FP) | Precision |
Memory aid: precision protects against false alarms; recall protects against misses.
F1 Score: One Number for Precision and Recall
F1 = 2 × (Precision × Recall) / (Precision + Recall)
For our example: 2 × (0.60 × 0.75) / 1.35 ≈ 0.667. F1 is the harmonic mean: it stays low when either precision or recall is low. Use it when both matter and you need one comparison value, particularly when accuracy is misleading on imbalanced data. F1 does not tell you which error dominates; inspect precision and recall separately when costs differ.
AUC-ROC: Across Classification Thresholds
The ROC curve plots true positive rate (recall) against false positive rate across thresholds. AUC summarizes the curve: 0.5 is approximately random ranking, while 1.0 indicates perfect separation. AUC helps compare ranking performance across thresholds, but it does not choose the operating threshold and can look deceptively strong on highly imbalanced data. For rare positives, also consider the precision-recall curve and business costs.
Metrics for Regression
| Metric | Meaning | Characteristic |
|---|---|---|
| MAE | Mean absolute error | Average absolute error, in target units |
| MSE | Mean squared error | Squares errors; large errors weigh more |
| RMSE | Square root of MSE | Same units as target; emphasizes large errors |
| R² | Explained variance relative to a baseline | Higher is generally better; can be negative on test data |
For a numeric prediction such as house price or energy demand, use regression metrics—not precision and recall.
Evaluating Generative AI Output
| Metric | Typical use | Core idea |
|---|---|---|
| ROUGE | Summarization | Overlap with reference summaries, often recall-oriented |
| BLEU | Machine translation | Precision-oriented n-gram overlap with reference translations |
| BERTScore | Semantic similarity | Compares contextual embedding representations |
| Perplexity | Language-model prediction | How well a model predicts text; lower is generally better on comparable setups |
| LLM-as-a-judge | Open-ended quality assessment | Another model scores outputs against criteria; judge bias and inconsistency remain possible |
Automatic metrics do not prove truthfulness or usefulness. Human evaluation remains important for qualities such as helpfulness, tone, and factual accuracy. Amazon Bedrock supports automatic and human-based evaluations; current AWS guidance also discusses LLM-as-a-judge. See the Bedrock evaluation guide and SageMaker model evaluation documentation.
Overfitting, Underfitting, and Held-Out Data
- Overfitting: excellent training performance but poor performance on unseen data—the model memorized patterns.
- Underfitting: poor performance even on training data—the model has not captured the underlying pattern.
Evaluate on representative validation and test data that was not used to fit the model. Training scores alone do not show how well it generalizes.
A Practical Decision Framework
- Category or number? Classification or regression.
- Generated text? Consider ROUGE, BLEU, BERTScore, LLM-as-a-judge, and human review.
- Imbalanced classes? Do not rely on accuracy alone.
- Which error costs more? False positives → precision; false negatives → recall.
- Both errors matter? Compare precision and recall and consider F1.
- Comparing ranking across thresholds? Consider ROC-AUC and, for rare positives, precision-recall analysis.
Business Walkthrough: Predicting Customer Churn
A meal-kit company predicts which 5% of customers will cancel. If discounts go to customers who would have stayed anyway, false positives waste money—precision matters. If likely cancellers are missed, revenue is lost—recall matters. If a discount is cheap compared with losing a customer, the company may lower the threshold to favor recall, while monitoring precision and F1. The metric encodes a business decision.
Common Mistakes
- Trusting accuracy on imbalanced data.
- Confusing precision (predicted positives) with recall (actual positives).
- Treating F1 as universally best.
- Evaluating only on training data.
- Using classification metrics for numeric predictions.
- Relying on text-overlap metrics alone to judge factuality.
Exam-Focused Insights for AIF-C01
The current AIF-C01 exam guide explicitly names ROUGE, BLEU, BERTScore, and LLM-as-a-judge as examples of foundation-model performance metrics. Practice matching the task to the metric, translating business risk into false-positive/false-negative costs, and recognizing the limits of automated scoring. Read the official AIF-C01 exam guide.
Key Takeaways
- Classification metrics derive from TP, FP, FN, and TN.
- Accuracy can mislead when positives are rare.
- Precision measures the reliability of positive predictions; recall measures how many actual positives are found.
- F1 balances precision and recall; it is not automatically the best metric.
- Regression uses MAE, MSE, RMSE, and R².
- Generative AI evaluation can combine ROUGE, BLEU, BERTScore, LLM-as-a-judge, and human review.
Original AIF-C01 Practice Questions
These are original, unofficial educational questions written by CloudExamPro. They are not actual AWS exam questions.
Question 1 — Rare fraud and misleading accuracy
A bank’s fraud dataset contains 0.5% fraud. A model reports 99.5% accuracy but catches almost no fraud. Which metric best reveals the problem?
- Accuracy
- Recall
- Perplexity
- BLEU
Correct: B. Recall. It measures the proportion of actual fraud detected. A model that misses almost all fraud has very low recall. Accuracy is distorted by the dominant legitimate class; perplexity and BLEU concern language tasks.
Question 2 — Spam filter false positives
An email provider says wrongly blocking a legitimate email is much worse than allowing occasional spam. Which metric should it prioritize?
- Recall
- Precision
- MAE
- ROUGE
Correct: B. Precision. The expensive error is a false positive. Higher precision means a greater share of flagged spam is truly spam. Recall prioritizes catching spam, even if more legitimate messages are flagged.
Question 3 — One score balancing two errors
A team detects rare equipment failures and cares about both false alarms and missed failures. Which single metric balances precision and recall?
- Accuracy
- R²
- F1 score
- Perplexity
Correct: C. F1 score. F1 is the harmonic mean of precision and recall. Accuracy can hide failure on rare classes; R² is for regression; perplexity is for language models.
Question 4 — Calculate precision
A model flags 50 tickets as urgent. Forty are truly urgent; 10 are not. What is precision?
- 40%
- 67%
- 80%
- 83%
Correct: C. 80%. Precision = TP/(TP+FP) = 40/50. The 40/60 ≈ 67% figure is recall, not precision.
Question 5 — Summary overlap
A publisher compares generated summaries with human-written reference summaries. Which metric is a common fit?
- RMSE
- Precision
- ROUGE
- AUC
Correct: C. ROUGE. ROUGE compares generated summaries with references, commonly emphasizing recall-oriented overlap. The other options are regression or classification metrics.
Frequently Asked Questions
Can a model have high accuracy and still be bad?
Yes. On imbalanced data, predicting only the majority class can produce high accuracy while missing every rare positive.
What is the difference between recall and sensitivity?
They are the same measure: TP/(TP+FN), also known as the true positive rate.
Is a higher F1 score always better?
Higher is generally better when precision and recall matter similarly, but inspect each metric if one error is more costly.
Why test on unseen data?
To estimate generalization. Training data performance can hide overfitting.
Are ROUGE and BLEU enough for generative AI evaluation?
No. They measure similarity to reference text, not factual correctness or usefulness. Add task-specific checks and human evaluation.
Official AWS References
- AWS Certified AI Practitioner exam guide
- Amazon Bedrock model and knowledge base evaluations
- Amazon SageMaker AI model evaluation
- Machine learning concepts on AWS
Metrics make the most sense when you apply them. Try our free AIF-C01 diagnostic exam. For more scenario-based practice with detailed explanations, explore the AIF-C01 Premium practice exams. Original Practice Questions — No Dumps. No Shortcuts.
Keep reading: Responsible AI: bias, fairness, and human oversight, AWS AI services compared, and AI vs. ML vs. deep learning vs. generative AI.
Continue your AIF-C01 preparation: For exam-focused examples that connect evaluation metrics to the wider AWS certification objectives, see the AWS Certified AI Practitioner study guide.