Evaluating a foundation model requires more than asking whether an answer sounds good. For text-generation tasks such as summarization and translation, automatic metrics compare model output with reference text, but each metric captures a different kind of similarity. ROUGE, BLEU, and BERTScore are useful to recognize because they measure different aspects of overlap or semantic similarity.
ROUGE: overlap with a reference
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures overlap between a generated summary and one or more reference summaries. ROUGE-N measures n-gram overlap: ROUGE-1 compares unigrams (individual tokens), while ROUGE-2 compares bigrams (two-token sequences). ROUGE-L uses the longest common subsequence. These scores can help assess summarization, but they may penalize valid paraphrases that use different wording.
BLEU: n-gram precision for translation
BLEU (Bilingual Evaluation Understudy) was developed for machine-translation evaluation. It measures how much of the generated text’s n-gram content matches reference translations, with a brevity penalty to discourage overly short output. BLEU is useful for comparing translation systems on a dataset, but it does not fully capture meaning, fluency, or whether a different valid translation expresses the same idea.
BERTScore: contextual semantic similarity
BERTScore uses contextual token embeddings from a pretrained language model to compare generated text with reference text. It aims to capture semantic similarity beyond exact word overlap. Two sentences can express similar meaning with different wording and still receive a meaningful similarity score. However, BERTScore depends on the embedding model and can miss important factual changes, especially in short outputs.
Side-by-side comparison
| Metric | What it compares | Common use / limitation |
|---|---|---|
| ROUGE | N-gram and sequence overlap with reference text | Common for summarization; may miss valid paraphrases |
| BLEU | N-gram precision against reference translations, with brevity penalty | Common for translation; does not fully measure meaning |
| BERTScore | Contextual embedding similarity between generated and reference text | Captures semantic similarity better than exact overlap in many cases; still not a factuality guarantee |
How AWS uses evaluation metrics
Amazon SageMaker Foundation Model Evaluations (FMEval) supports task-specific evaluation dimensions and metrics. For summarization, AWS documentation describes ROUGE-N, METEOR, and BERTScore as accuracy-related measures. For question answering, evaluation can use exact match, quasi-exact match, and word-level F1. The metric must match the task: a translation metric is not automatically the right measure for open-ended generation.
Why one score is never enough
- Automatic overlap metrics can reward matching words even when a response contains a factual error.
- Semantic metrics can recognize paraphrases but may overlook a small, critical change such as a negation or numeric value.
- Reference-based metrics depend on the quality and coverage of the reference answers.
- Human evaluation can assess dimensions such as helpfulness, coherence, relevance, and factual accuracy that are difficult to reduce to one automatic score.
Exam traps
- ROUGE is strongly associated with summarization and n-gram overlap.
- BLEU is strongly associated with machine translation and n-gram precision.
- BERTScore uses contextual embeddings to compare semantic similarity.
- Do not interpret a high automatic similarity score as proof that an answer is factually correct or safe.
Test yourself
Question 1
A team is evaluating generated summaries against reference summaries and wants a metric based on unigram and bigram overlap. Which metric family should it consider?
- ROUGE
- BLEU only
- ROC AUC
- Silhouette score
Reveal answer
Correct answer: A. ROUGE-N measures n-gram overlap and is commonly used for summarization evaluation.
Question 2
A generated summary uses different wording from the reference but preserves similar meaning. Which metric uses contextual embeddings to assess semantic similarity?
- BLEU
- BERTScore
- Exact match
- Accuracy only
Reveal answer
Correct answer: B. BERTScore compares contextual token embeddings and can better recognize semantically similar paraphrases.
Key takeaway
ROUGE focuses on reference overlap for summarization, BLEU measures n-gram precision for translation, and BERTScore uses contextual embeddings for semantic similarity. Combine automatic metrics with human review and task-specific validation.
Related AWS topics
Precision, Recall, F1 and AUC · Foundation Model Hallucinations · Tokens, Embeddings and Vector Databases
✦ CloudExamPro Premium
Ready for the AWS Certified AI Practitioner (AIF-C01)?
Move beyond memorizing terms. Practice with realistic, scenario-based questions and detailed explanations that help you understand the AWS AI services and concepts behind each answer.
Original practice questions. No dumps. No shortcuts.