Evaluating a foundation model requires more than asking whether an answer sounds good. For text-generation tasks such as summarization and translation, automatic metrics compare model output with reference text, but each metric captures a different kind of similarity. ROUGE, BLEU, and BERTScore are useful to recognize because they measure different aspects of overlap or semantic similarity.

ROUGE: overlap with a reference

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures overlap between a generated summary and one or more reference summaries. ROUGE-N measures n-gram overlap: ROUGE-1 compares unigrams (individual tokens), while ROUGE-2 compares bigrams (two-token sequences). ROUGE-L uses the longest common subsequence. These scores can help assess summarization, but they may penalize valid paraphrases that use different wording.

BLEU: n-gram precision for translation

BLEU (Bilingual Evaluation Understudy) was developed for machine-translation evaluation. It measures how much of the generated text’s n-gram content matches reference translations, with a brevity penalty to discourage overly short output. BLEU is useful for comparing translation systems on a dataset, but it does not fully capture meaning, fluency, or whether a different valid translation expresses the same idea.

BERTScore: contextual semantic similarity

BERTScore uses contextual token embeddings from a pretrained language model to compare generated text with reference text. It aims to capture semantic similarity beyond exact word overlap. Two sentences can express similar meaning with different wording and still receive a meaningful similarity score. However, BERTScore depends on the embedding model and can miss important factual changes, especially in short outputs.

Side-by-side comparison

MetricWhat it comparesCommon use / limitation
ROUGEN-gram and sequence overlap with reference textCommon for summarization; may miss valid paraphrases
BLEUN-gram precision against reference translations, with brevity penaltyCommon for translation; does not fully measure meaning
BERTScoreContextual embedding similarity between generated and reference textCaptures semantic similarity better than exact overlap in many cases; still not a factuality guarantee

How AWS uses evaluation metrics

Amazon SageMaker Foundation Model Evaluations (FMEval) supports task-specific evaluation dimensions and metrics. For summarization, AWS documentation describes ROUGE-N, METEOR, and BERTScore as accuracy-related measures. For question answering, evaluation can use exact match, quasi-exact match, and word-level F1. The metric must match the task: a translation metric is not automatically the right measure for open-ended generation.

Why one score is never enough

Exam traps

Test yourself

Question 1

A team is evaluating generated summaries against reference summaries and wants a metric based on unigram and bigram overlap. Which metric family should it consider?

  1. ROUGE
  2. BLEU only
  3. ROC AUC
  4. Silhouette score
Reveal answer

Correct answer: A. ROUGE-N measures n-gram overlap and is commonly used for summarization evaluation.

Question 2

A generated summary uses different wording from the reference but preserves similar meaning. Which metric uses contextual embeddings to assess semantic similarity?

  1. BLEU
  2. BERTScore
  3. Exact match
  4. Accuracy only
Reveal answer

Correct answer: B. BERTScore compares contextual token embeddings and can better recognize semantically similar paraphrases.

Key takeaway

ROUGE focuses on reference overlap for summarization, BLEU measures n-gram precision for translation, and BERTScore uses contextual embeddings for semantic similarity. Combine automatic metrics with human review and task-specific validation.

Related AWS topics

Precision, Recall, F1 and AUC · Foundation Model Hallucinations · Tokens, Embeddings and Vector Databases

✦ CloudExamPro Premium

Ready for the AWS Certified AI Practitioner (AIF-C01)?

Move beyond memorizing terms. Practice with realistic, scenario-based questions and detailed explanations that help you understand the AWS AI services and concepts behind each answer.

6 Full-Length Exams390 practice questions
Detailed ExplanationsLearn why answers are right or wrong
Scenario-Based PracticeOriginal questions, not dumps
6 Months of AccessReview at your own pace
Get the AIF-C01 Practice Exams →Try the free AIF-C01 exam first ↗

Original practice questions. No dumps. No shortcuts.

Limited-time offer · ends in -- Days : -- Hrs : -- Min : -- Sec
Get Instant Access