MT Quality Metrics: Edit Distance vs BLEU vs Human Evaluation

Your MT system scored 42 BLEU. A competitor’s system scores 38. So yours is better, right? Not necessarily - and that uncertainty is exactly what makes MT quality evaluation one of the most contested problems in computational linguistics.

Three metrics dominate the conversation: edit distance (and its variants TER, HTER), BLEU, and human evaluation. They measure completely different things, have completely different blind spots, and are useful in completely different situations. Here’s what each actually measures - and where each falls apart.

Why measuring MT quality is harder than it looks¶

A translation can be accurate but unnatural. Or natural but inaccurate. Or accurate and natural but culturally wrong. These three failure modes require completely different detection methods.

The core problem: to measure quality, you need a benchmark. And the only truly reliable benchmark is human judgment - which is expensive, slow, and inconsistent between raters. Automated metrics approximate that judgment cheaply and quickly. All of them sacrifice something in the process.

MT quality evaluation research has been a subfield of NLP for 25 years, and the verdict is still contested. The WMT 2024 Metrics Shared Task - the annual benchmark that tests how well automated metrics correlate with human judgments - found that neural metrics like COMET significantly outperform BLEU, but still fall short of human reliability for specific error types.

Edit distance: counting the changes¶

Edit distance (also called Levenshtein distance) is the conceptually simplest MT quality metric: how many operations do you need to transform the MT output into a reference translation?

The standard operations: - Insertions - adding a word that’s missing - Deletions - removing a word that shouldn’t be there - Substitutions - replacing a wrong word with the correct one - Shifts - moving a phrase to the right position

At character level, edit distance counts individual character changes. At word level - word changes. The result is usually normalized by text length to get a rate (percentage) rather than a raw count.

As Slator explains, “MT output is the starting point and the final version of the target language is the end point” - edit distance measures the gap between them. A score of 0% means no changes needed. A score of 80% means 80% of the words required editing.

Where it’s useful: Edit distance is how CAT tools like memoQ and Phrase track how much a translator modified an MT draft. It underpins MTPE pricing - more edits needed means more work means higher rate.

The critical limitation: Edit distance tracks the number of changes, not the quality of the result. You can make minimal edits and still produce a bad translation. And a translator who stylistically rewrites an MT draft that was actually correct will show high edit distance for work that didn’t need doing. The metric is blind to WHY changes were made.

BLEU: the n-gram overlap machine¶

BLEU (Bilingual Evaluation Understudy) was introduced in 2002 by Papineni et al. at IBM Research and became the dominant MT evaluation metric for two decades. The core idea: compare MT output to one or more reference translations by counting how many word sequences (n-grams) they share.

Here’s the calculation: 1. Count how many 1-word sequences (unigrams) in the MT output appear in the reference 2. Do the same for 2-word (bigrams), 3-word (trigrams), and 4-word sequences (4-grams) 3. Calculate precision for each n-gram size 4. Compute a geometric mean of those precisions 5. Apply a brevity penalty if the MT output is shorter than the reference (to prevent gaming with very short outputs)

Result: a score between 0 and 1, usually expressed as 0-100. Higher = more overlap with the reference = presumably better.

Typical BLEU score ranges:

These ranges vary dramatically by language pair. English-French BLEU scores run higher than English-Arabic, which run higher than English-Japanese - because of word order differences and morphological complexity.

What BLEU gets right: It’s fast, completely automated, and reproducible. You can run BLEU on 10 million sentence pairs in minutes. It correlates reasonably well with human judgment at corpus level - comparing two MT systems on the same test set, BLEU usually ranks them in the right order.

What BLEU gets wrong: A lot. A detailed analysis of BLEU’s flaws catalogued the core problems: semantic blindness (it can’t tell “big” from “large” or distinguish the two meanings of “bank”), reference dependency (you need perfect human reference translations), insensitivity to negation (a translation that reverses meaning with “not” can score almost as high as the correct version), and poor performance on morphologically rich languages.

As one NLP researcher noted when comparing evaluation frameworks:

Neural MT systems with lower BLEU scores are sometimes preferred by humans over statistical MT systems with higher BLEU scores. The metric was calibrated for an older era of MT.

This disconnect became critical with the rise of neural MT after 2016. NMT systems produce fluent, natural translations that differ from references in wording while preserving meaning better. BLEU penalizes that divergence even when the translation is excellent.

TER and HTER: edit distance meets post-editing¶

TER (Translation Edit Rate) is edit distance packaged as a standardized MT metric. Published in 2006 by Snover et al., TER measures the minimum number of edits needed to transform MT output into a reference translation, divided by the length of the reference.

TER = (edits needed) / (reference length)

A TER of 0 means the MT output matches the reference exactly. A TER of 1.0 means you’d need to rewrite 100% of the reference length. In practice, good general-domain MT achieves TER between 0.2 and 0.5.

Unlike BLEU, TER handles word order through “phrase shifts” - moving a correctly translated phrase to the right position counts as one edit, not multiple individual errors. This makes TER more forgiving than BLEU for translations that get the words right but in the wrong order.

HTER (Human-Targeted TER) is TER’s more practical cousin: instead of comparing MT output against a pre-written reference, it compares against the human post-edited version. A trained post-editor actually fixes the MT output, then you measure how many edits they made.

This matters because standard TER uses reference translations written independently of the MT output - they represent an ideal but not the minimal corrections needed. HTER uses the actual post-editing work as the benchmark, which better reflects real effort.

According to research on HTER, HTER “correlates more closely with individual human judgments than inter-judge correlations” - it’s actually more consistent than having two human raters agree with each other.

Where TER/HTER shine: Measuring post-editing effort for pricing and workflow optimization. If your HTER is 0.12, translators are making minimal corrections - light MTPE territory. If it’s 0.45, they’re essentially rewriting. This is actionable data for MTPE rate negotiations.

Where they fall short: Same fundamental problem as raw edit distance - they count changes, not quality. High HTER could mean the MT was bad, or it could mean the translator made unnecessary stylistic changes. Without observing the post-editor’s decision process, you can’t distinguish the two.

Human evaluation: MQM and Direct Assessment¶

Human evaluation is slow, expensive, and variable between raters. It’s also the only method that reliably catches the failure modes that automated metrics miss.

Two frameworks dominate:

Direct Assessment (DA)¶

The evaluator reads the MT output and rates it on a scale (usually 0-100) for adequacy or fluency. Simple, fast per annotation, easy to aggregate. WMT used DA as the primary human evaluation protocol for years.

The problems: scores are absolute, not comparative. Inter-rater agreement is moderate. Raters tend to be lenient with fluent but inaccurate translations - they read smoothly, so they feel correct. And DA doesn’t identify what is wrong - only how bad things are overall.

MQM (Multidimensional Quality Metrics)¶

MQM is the current gold standard for professional MT evaluation. Trained annotators identify specific error spans, categorize the error type, and assign severity:

• Accuracy errors: meaning changes, omissions, additions, mistranslations
• Fluency errors: grammar, spelling, style, register
• Terminology errors: wrong domain-specific terms
• Locale convention errors: wrong date formats, currency symbols, etc.

Severity levels: minor, major, critical.

A critical accuracy error (meaning completely reversed) penalizes far more than a minor fluency issue (slightly awkward phrasing). The final MQM score aggregates these penalties into a numeric quality signal.

The WMT 2024 Metrics Task used MQM-based human evaluation as the primary gold standard against which all automated metrics were measured. MQM inter-rater agreement (around 0.58 for English-German) is substantially higher than DA.

The catch: you need trained linguists, it takes hours per document, and costs scale with text volume. It’s not practical for evaluating MT systems during development or monitoring production pipelines in real time.

COMET and neural metrics: the generation after BLEU¶

Since 2019, a new class of metrics - learned reference-based metrics - has consistently outperformed BLEU in correlation with human judgment.

COMET (Crosslingual Optimized Metric for Evaluation of Translation) is the most widely adopted. It uses a pre-trained multilingual encoder (XLM-RoBERTa) fine-tuned on human quality scores. Instead of counting word overlaps, COMET encodes the source, reference, and MT output into semantic vectors and computes similarity in that space.

The result: COMET catches semantic equivalence that BLEU misses. “Automobile” and “car” differ in BLEU; COMET recognizes they mean the same thing.

At the WMT 2024 Metrics Task, COMET-family metrics achieved the highest system-level correlations with human judgment - significantly above BLEU, ChrF, and TER. Recent extensions like xCOMET add fine-grained error detection, moving closer to automated MQM.

But COMET has its own limits. Research at Translated.com found COMET can fall short for LLM-based MT outputs, which differ stylistically from the traditional MT outputs it was trained on. It’s also computationally heavier than BLEU, and reference-based COMET still requires reference translations for each segment.

Which metric to use and when¶

No single metric is right for all situations:

A practical approach used by large-scale MT teams: run COMET as the primary automated signal, keep BLEU for historical comparisons, and do periodic MQM audits on sampled output (500 sentences per month, for example) to catch drift that automated metrics miss.

The core lesson from 25 years of MT evaluation research: any single metric gives you a partial picture. BLEU tells you something. HTER tells you something else. Neither tells you what MQM tells you. For anything customer-facing, periodic human evaluation isn’t optional - it’s the only way to know what’s actually being shipped.

If you’re running an MTPE workflow, the practical starting point is HTER to price the work, COMET to benchmark the MT engine, and MQM spot-checks for domains where accuracy errors have real consequences.

FAQ¶

What is a good BLEU score for machine translation?¶

It depends on the language pair and domain. Rough benchmarks: under 15 is poor, 15-30 is roughly understandable with high error rates, 30-50 is usable for narrow technical domains, above 50 approaches human-level for constrained topics. These ranges aren’t comparable across language pairs - English-French 40 BLEU is not the same as English-Japanese 40 BLEU. Always compare BLEU scores within the same language pair and the same test set.

Is BLEU still relevant in 2025?¶

Yes, widely used for quick benchmarks and because existing systems have years of BLEU history. The research community now uses COMET as the primary metric, and WMT 2024 used MQM as the gold standard. For new evaluation setups without legacy constraints, COMET is the better default. But BLEU isn’t going away - too many workflows depend on it.

What does a TER of 0.3 mean in practice?¶

It means 30% of reference-length words needed editing to transform the MT output into the reference. For comparison: good general-domain MT typically achieves TER between 0.25 and 0.45. Below 0.2 is excellent. Above 0.6 means substantial rework - at that point you’re close to translating from scratch.

Can I use BLEU to compare two different MT systems?¶

Yes, but only if you use the same test set, the same reference translations, and the same tokenization scheme. BLEU scores are not comparable across different test sets - a system scoring 35 on one corpus might score 50 on a different, easier corpus. This is one of the most common ways BLEU is misused in industry.

What’s the difference between TER and HTER?¶

TER compares MT output against a pre-written reference translation. HTER compares MT output against the actual post-edited version - a real translator fixed the MT, and then you measure how much they changed. HTER is more meaningful for MTPE workflows because it measures real editing effort rather than distance from an idealized reference that might have been written differently anyway.

Why doesn’t BLEU detect meaning errors?¶

BLEU only counts n-gram overlap at the surface level - it sees words, not meanings. “The bank approved the loan” and “The loan approved the bank” contain identical words and would score almost identically against each other as references. Negation flips (“the patient should take the medication” vs “the patient should not take the medication”) barely affect BLEU if most other words match. This is why BLEU is dangerous as the sole quality signal for medical, legal, or safety-critical translations.