BLEU Score Explained for Translation Agencies (and Its Limits)

Your vendor sends over a benchmark report: their customized MT engine for EN→DE scored 48.3 BLEU on a technical documentation test set. The previous version scored 43.7. Does this mean the new engine is meaningfully better? Is 48.3 actually good? And should you be using this number to decide whether to route production content through their engine?

Those are the right questions. The fact that many agencies can’t answer them is why BLEU causes so much confusion in vendor evaluation.

What BLEU actually measures¶

BLEU - Bilingual Evaluation Understudy - was developed at IBM by Kishore Papineni and colleagues in 2002 and became the standard automatic MT evaluation metric almost immediately. The reason it took over: it was fast, cheap, and correlated reasonably well with human judgments at the corpus level, at a time when human evaluation was the only alternative and it was slow and expensive.

The algorithm is conceptually simple. Take the MT output. Take one or more human reference translations of the same source. Count how much the MT output overlaps with the reference translations, using sequences of 1-4 consecutive words (called n-grams). More overlap = higher score.

To prevent a system from gaming the score by repeating high-frequency words, BLEU uses “clipped” counts - each word in the MT output can only be matched against a reference as many times as it appears in that reference. It also applies a brevity penalty to prevent very short translations from scoring artificially high.

The result is a number between 0 and 100 (or 0 and 1 in some implementations - the same scale, just expressed differently).

As Microsoft’s Custom Translator documentation explains:

BLEU averages out individual sentence judgment errors over a test corpus, rather than attempting to devise the exact human judgment for every sentence.

This corpus-level averaging is both BLEU’s strength and its primary limitation - which we’ll get to.

How to read the numbers¶

The most widely referenced interpretation table, based on Google’s guidance:

A few practical calibrations:

The >60 case is suspicious. If you see a score above 60, it usually means the test set leaked into training - the model was trained on examples that overlap with what it’s being tested on. In practice, well-calibrated production systems for high-resource language pairs typically land in the 40-55 range.

Score differences under 4-5 points are noise. A jump from 43.7 to 48.3 is meaningful. A jump from 43.7 to 45.1 may not be - especially if the test set is small. The variance in BLEU scoring means small differences don’t necessarily indicate real quality improvements.

Human translators don’t score 100. When two professional human translators translate the same source text, their translations will differ enough that each one, measured against the other as a “reference,” typically scores 60-80. This is fundamental to understanding what BLEU is measuring: not absolute quality, but similarity to one specific reference.

Where BLEU is genuinely useful for agencies¶

BLEU has a well-defined set of legitimate uses. Outside these, it misleads.

Tracking MT engine development over successive versions. If you’re working with an MT vendor who is customizing an engine on your domain content, BLEU is the right tool to verify that each new model version is actually improving. Same test set, same language pair, same domain - BLEU reliably tells you direction of change.

As the CEO of Iconic Translation Machines noted in an interview with Slator:

We use BLEU scores to give an intuitive feel of where an engine is at the beginning, and then to benchmark different versions as we improve it.

That’s the appropriate use case: relative comparison of the same system over time, not absolute quality judgment.

Deciding whether a domain adaptation experiment improved things. You have a generic engine and a fine-tuned version trained on your specific content. Run both on a representative test set of your own domain. If the fine-tuned version scores 8+ points higher, the adaptation worked. If it’s within 3 points, the gain isn’t worth the cost.

Initial screening of new MT engine vendors. A vendor who can’t provide a BLEU score on a representative test set for your language pair and domain - using a test set you control, not theirs - is a red flag. The score itself is secondary; the ability to produce one on your test data indicates methodological seriousness.

BLEU’s limits - what the number can’t tell you¶

This is where agencies get into trouble: treating BLEU as a quality gate for production content.

Semantic blindness. BLEU counts word matches. It cannot tell whether two translations convey the same meaning. A translation that inverts the meaning of a sentence by adding a negation - “the system does not require authentication” vs. “the system requires authentication” - can score identically to the correct translation if the surrounding word matches are similar enough.

Synonym deafness. “Begin,” “start,” “commence,” and “initiate” all mean the same thing. If the reference translation uses “begin” and the MT output uses “initiate,” BLEU penalizes the MT output even though the translation is correct. This is especially problematic for domains with rich specialist vocabulary where multiple correct terms exist.

Sentence-level unreliability. BLEU was designed and validated at the corpus level - hundreds or thousands of sentences. Applied to individual sentences, it produces results that don’t correlate well with human judgment. An agency that uses BLEU to decide whether individual segments need post-editing is misapplying the tool.

Grammaticality blindness. BLEU doesn’t evaluate whether the translation reads grammatically. A grammatically broken sentence that happens to share many words with the reference can score higher than a fluent one that paraphrases slightly differently.

University of Edinburgh’s Rico Sennrich, one of the leading researchers in neural MT, has noted that BLEU “systematically underestimates quality of some translation systems, particularly rule-based systems” - and by extension, systems that produce correct but non-literal translations.

CSA Research was more direct in their practitioner-focused assessment:

These metrics are artificial and irrelevant for translation production environments.

That doesn’t mean BLEU is useless - it means it belongs in engine development, not in the go/no-go decision for whether a translated document can ship.

Test set contamination. If the vendor controls the test set, they control the score. Benchmark numbers in marketing materials are frequently measured on test data the vendor’s system was implicitly or explicitly trained on. Always insist on evaluating against a test set you provide and control.

The specific problems for agency use cases¶

Translation agencies face a particular set of challenges where BLEU’s limitations are most acute.

Legal and compliance content. A BLEU score of 45 on a translated contract tells you nothing about whether jurisdictional terms are correctly rendered, whether the obligations in clause 7 are accurately conveyed, or whether the terminological choices have legal consequences. A single wrong term can invalidate a translation that scores perfectly. Human expert review is the only reliable gate here.

Marketing and brand content. BLEU penalizes creative, culturally adapted translations - which are exactly the good ones for this content type. A transcreation that renders an English idiom in a culturally appropriate equivalent in the target language will score lower than a literal (and bad) translation. For this content, BLEU is actively misleading.

Multi-reference test sets. BLEU improves when you have multiple reference translations, because there are more n-grams to potentially match against. But most agency test sets have a single reference - the resource cost of generating multiple expert translations of a test set is prohibitive. Single-reference BLEU is noisier and more prone to the synonym problem.

What to use instead (or alongside)¶

The evaluation landscape has matured significantly since BLEU was developed. For agencies that need more reliable quality signals:

COMET (Crosslingual Optimized Metric for Evaluation of Translation) is a neural metric trained on human quality judgments. It considers source sentence meaning, not just surface overlap with the reference. Multiple independent studies have shown COMET correlates better with human evaluation than BLEU, especially for high-quality MT systems where BLEU’s ceiling effects become a problem.

MQM (Multidimensional Quality Metrics) is a structured human evaluation framework that categorizes errors by type (accuracy, fluency, terminology, style, locale convention) and severity (critical, major, minor). It’s the standard in professional quality evaluation and is used by the EU Translation Centre, major LSPs, and global enterprises. MQM doesn’t replace automatic metrics - it provides the human ground truth that those metrics are trying to approximate.

TER (Translation Error Rate) measures how many edits a human reviewer would need to make to the MT output to produce acceptable translation. Lower is better. It’s more interpretable than BLEU for post-editing cost estimation because it correlates directly with post-editing effort.

BERTScore uses contextual embeddings to measure semantic similarity rather than surface overlap, making it more robust to synonyms and paraphrases.

For most agency workflows in 2026, the recommended approach is: use BLEU for MT engine benchmarking and development tracking, use COMET for automated quality estimation at the segment level, and use MQM-based human review as the final gate for client-facing or high-stakes content.

Practical guidelines for agency PMs¶

A few rules that prevent the most common BLEU misuse patterns:

Never compare BLEU scores from different test sets. A vendor quoting BLEU from their internal benchmark versus your score on your own test set are not comparable numbers, even for the same engine and language pair.

Set a minimum score threshold - but not as a hard quality gate. A reasonable floor for production-quality MT content is 35+ for most language pairs, understanding that this needs validation per domain. Below 30, the MT output is likely requiring near-retranslation effort in post-editing.

Use it for relative comparison, not absolute judgment. “Version B scores 6 points higher than Version A on our test set” is a useful signal. “This engine scores 47” in isolation tells you little without context about the test set, language pair, and domain.

Always specify the test set conditions when requesting vendor benchmarks. Require: test set size (minimum 1,000 segments), language pair, domain, and that the test set was not used in training or fine-tuning.

For new language pairs, run your own test. Take 500-1,000 representative segments from your actual content, have a senior linguist translate them as a reference, then run candidate engines against this set. Your own BLEU scores on your own content are always more predictive than vendor benchmarks.

FAQ¶

What does BLEU stand for?¶

BLEU stands for Bilingual Evaluation Understudy. “Understudy” refers to the metric standing in for a human evaluator - automatically approximating what a human reviewer would assess.

Is BLEU scored 0-1 or 0-100?¶

Both representations are used. The underlying calculation produces a number between 0 and 1, but it’s commonly multiplied by 100 and expressed as a percentage (so 0.45 = BLEU 45). When comparing scores, make sure you’re on the same scale.

Can you use BLEU to evaluate translation of legal documents?¶

You can run it - but the results are not meaningful as a quality gate for legal content. BLEU cannot detect the meaning-level errors that matter most in legal translation: incorrect jurisdictional terms, missed obligations, wrong numeric values, incorrect proper nouns. Human expert review by a specialist translator is the only reliable quality method for legal documents.

How many reference translations does BLEU need?¶

Technically one, but more references improve reliability. With a single reference, BLEU penalizes synonyms and paraphrases more severely. With four references (the original BLEU paper used four human translations), the metric becomes more forgiving of valid variation. Most practical deployments use one reference due to cost constraints - this is one reason BLEU scores are noisier than they could be.

What BLEU score should I require from an MT vendor?¶

This depends entirely on language pair and domain. For a high-resource pair like EN→DE in technical documentation with a good domain-adapted engine, 45+ is reasonable to expect. For a lower-resource pair or specialized domain, 30-35 may be strong performance. Don’t set universal thresholds - test on your own content and calibrate against what you consider acceptable post-editing effort.

Is BLEU still used in 2026?¶

Yes, though its limitations are increasingly well-understood in the industry. BLEU remains the default benchmarking metric because it’s free, fast, and reproducible. Neural metrics like COMET have better correlation with human judgment and are growing in use, but BLEU hasn’t been replaced - it’s been supplemented. Most serious MT development pipelines report both.

Sources¶

• Slator - How BLEU Measures Translation and Why It Matters
• ModernMT - Understanding MT Quality: BLEU Scores
• Microsoft Learn - What is a BLEU score?
• Google Cloud - The BLEU translation quality metric
• Translated.com - MT Quality Evaluation in the Age of LLM-based MT
• ACL Anthology - Re-evaluating the Role of BLEU in Machine Translation Research (Callison-Burch et al., 2006)