You run 40,000 words through your preferred MT engine on Monday. By Wednesday, your post-editor flags you: “The output quality on this batch is rougher than usual - I’m taking twice as long per segment.” You’re already committed to the timeline and the MTPE rate in the client quote.
This is what happens when MT quality goes unmeasured until mid-project. Knowing how to assess quality before you commit - and having a system to catch degradation when it happens - is what separates agencies that run MTPE profitably from those that absorb the losses.
Why MT quality measurement is a business decision, not an academic exercise¶
Translation quality measurement exists in two worlds. Academic researchers track metric correlations to hundredths of decimal points. Production agencies need to answer three practical questions:
- Is this MT output good enough to run through MTPE at the rate I’m quoting?
- Which engine performs better for this specific content type and language pair?
- Has quality degraded mid-project, and by how much?
Automated metrics like BLEU get a lot of criticism in research circles - and much of it is deserved. But for production agencies, their value isn’t perfect correlation with human judgment. It’s being fast, free, and consistent enough to catch big quality drops before they become expensive problems.
The framework that follows is built around what agencies actually need in production, not what’s theoretically optimal.
Automated metrics: what they measure and when to use each¶
Automated metrics compare MT output against a reference translation - usually a human post-edited version - and produce a score. No human reviewer needed, results in seconds. The tradeoff is that all automated metrics measure surface-level agreement with the reference, not actual translation quality.
BLEU - the industry default you need to understand¶
BLEU (Bilingual Evaluation Understudy) was published by Papineni et al. at IBM Research in 2002 and has been the default MT evaluation metric ever since. It measures how many n-grams (word sequences) in the MT output also appear in the reference translation. The score runs from 0 to 100 - higher means more overlap with the reference.
Why agencies still use it: - Fast and free (sacrebleu is the standard open-source implementation) - Reproducible - given the same data, two people get the same score - Comparable across engines if you use the same test set and tokenization
Why you can’t rely on BLEU alone: - Measures word matching, not meaning. A sentence that translates correctly but uses different words scores lower than one that matches the reference but is subtly wrong. - Sensitive to the reference translation’s style choices - if your reference reflects one translator’s phrasing, MT that makes equally valid but different choices scores poorly. - Correlates poorly with human quality judgments, especially for creative content, flexible-word-order languages, or low-resource language pairs.
Rough working guide for technical content in major language pairs:
| BLEU range | What it suggests | Likely post-editing level |
|---|---|---|
| < 25 | Poor quality, extensive rework expected | FPE or retranslation |
| 25-40 | Moderate quality | FPE required |
| 40-50 | Good quality | LPE likely sufficient |
| > 50 | High quality (domain-specific content) | LPE or light review |
These ranges shift dramatically by language pair and domain. EN-FR technical at BLEU 40 is very different from EN-AR legal content at BLEU 40.
TER - directly useful for estimating post-editing effort¶
Translation Edit Rate (TER) measures how many edits - insertions, deletions, substitutions, and shifts - are needed to turn the MT output into the reference, divided by reference length. TER of 0.3 means roughly 30% of words need editing. Lower is better.
TER maps naturally to post-editing workflows because it directly models editing effort. Agencies that track TER across projects can build empirical data on what threshold predicts acceptable post-editing speed for their specific editors and content types.
Practical thresholds based on industry benchmarks:
| TER range | What it means | Action |
|---|---|---|
| < 0.3 | High quality MT output | LPE is viable, quote accordingly |
| 0.3-0.5 | Moderate quality | FPE required - budget editor time accordingly |
| > 0.5 | Poor quality | Post-editing effort approaches retranslation - reprice or redirect |
Caveat: TER doesn’t account for the difficulty of individual edits. Fixing a terminology error that required 10 minutes of research counts the same as adding a missing article. That’s why human pilot data always beats automated TER estimates.
chrF - better for morphologically rich languages¶
Character n-gram F-score (chrF) works at the character level rather than the word level, making it more sensitive to morphological variation. For languages where word endings change significantly by grammatical context - Ukrainian, Russian, Finnish, Hungarian, Turkish, Polish - chrF often correlates with quality better than BLEU.
If your agency works heavily in Eastern European or agglutinative language pairs, adding chrF to your toolkit costs nothing (it’s in sacrebleu alongside BLEU) and gives a more reliable signal for those pairs.
COMET - the best automated metric available now¶
COMET (Crosslingual Optimized Metric for Evaluation of Translation), developed by Unbabel, uses a neural network trained on human quality judgments from WMT evaluation campaigns. It’s been the top-ranked automated metric in WMT Metrics Shared Tasks consistently since 2020.
The practical difference over BLEU: COMET captures whether the translation is semantically correct, not just whether it uses the same words as the reference. Two translations that convey the same meaning but phrase it differently get very different BLEU scores but similar COMET scores.
When to use COMET: - Engine selection - comparing DeepL vs Google Translate vs Microsoft Translator vs a custom engine on your content types - Regression testing - detecting quality drops when an engine updates or a new domain model is deployed - Language pair benchmarking - understanding which pairs have reliably high MT quality for your workflows
COMET requires Python setup (via the Unbabel repository on GitHub). For agencies without engineering resources, some TMS platforms now integrate COMET scores directly into their QA pipelines.
Metric summary:
| Metric | Reference needed? | Best for | Main limitation |
|---|---|---|---|
| BLEU | Yes | Quick signal, engine comparison | Poor correlation with meaning |
| TER | Yes | Estimating editing effort | Doesn’t capture edit difficulty |
| chrF | Yes | Morphologically rich languages | Still surface-level |
| COMET | Yes (DA version) | Best automated quality signal | Requires Python setup |
MQM: the professional standard for human quality evaluation¶
Automated metrics tell you how similar your MT output is to a reference. MQM (Multidimensional Quality Metrics) tells you what’s actually wrong with it.
MQM is an error taxonomy framework - a structured way for human reviewers to categorize and score translation errors. It’s used by Google Translate’s quality team, Microsoft, and government translation bodies for formal quality audits. The error categories give you actionable data, not just a score.
Core MQM error categories:
Accuracy errors (does the translation say what the source says?): - Mistranslation: wrong meaning conveyed - Undertranslation: source content omitted - Overtranslation: content added that isn’t in the source - Untranslated: source text left as-is
Fluency errors (does the translation read naturally?): - Grammar: incorrect morphology, syntax, agreement - Typography: punctuation, capitalization, spacing errors - Register: wrong formality level for the context
Terminology errors: - Wrong technical term used against a specified glossary - Inconsistent term usage across the document
Style errors: - Doesn’t match brand voice or style guide - Unnatural phrasing that’s technically correct but wouldn’t be written that way by a native expert
Each error gets a severity rating: Critical (causes harm or legal problems), Major (significantly affects quality), Minor (small errors that don’t affect understanding). The MQM score for a document is calculated from error counts weighted by severity, against thresholds you define per content type.
When agencies need MQM: - Client contracts that specify formal quality metrics (increasingly common with enterprise clients) - ISO 17100 or 18587 compliance workflows - Formal engine selection between two competitive options - Spot-checking ongoing MTPE output quality against an established baseline
When agencies don’t need full MQM: - Internal quick-turn content with no formal QA requirement - High-volume low-stakes MTPE where editing rate is the proxy for quality
Even if you’re not running formal MQM audits, the error categories are a useful mental checklist for human reviewers. Training reviewers to classify what they find - accuracy vs fluency vs terminology - generates data that helps you identify systematic MT weaknesses in specific domains or language pairs. That data informs engine selection and post-editor briefing far better than a single BLEU score.
Quality Estimation: scoring MT quality without a reference¶
Reference-based metrics (BLEU, TER, COMET-DA) require a human post-edited version to compare against. That’s the problem: you need quality data before the post-editing to decide how to handle it.
Quality Estimation (QE) solves this by predicting translation quality directly from the source and the MT output, without any reference. It’s the practical tool for production routing.
QE models are trained on human quality judgments and learn to predict quality scores from source-MT pairs alone. They can flag individual segments likely to need heavy editing before the post-editor sees them, enabling automated routing decisions at scale.
QE tools available in 2026:
COMET-Kiwi (open-source, from Unbabel): the current top-performing QE model from WMT shared tasks. Predicts segment-level quality scores without a reference. Requires Python but runs locally.
ModelFront: commercial QE tool used by agencies and LSPs specifically for automated workflow routing. Integrates with major TMS platforms. Provides per-segment confidence scores.
DeepL Quality Check: DeepL’s proprietary confidence scoring, accessible via API alongside translations. Useful if you’re already using DeepL as your MT engine.
Phrase (formerly Memsource): has built-in QE from multiple providers, accessible without custom tooling.
Practical QE use cases:
Automated routing: Segments above a confidence threshold go to LPE. Segments below threshold get flagged for FPE or human retranslation. In high-volume workflows, this can reduce post-editing effort by 20-30% by focusing human attention where it’s actually needed.
Pre-project risk assessment: Run the source file through QE before you commit to a quote. If average QE scores are low, this project will require heavier post-editing than typical, and your quote needs to reflect that.
Engine comparison without a reference set: Compare QE scores between two engines on the same source. Less reliable than reference-based metrics, but useful when you don’t have a gold standard reference for new content types.
Running a pilot: the one check that actually works¶
Automated metrics give you signals. QE gives you predictions. Neither replaces what you get from a human pilot on real content.
Before committing any significant project to MTPE - especially for a new language pair, new content type, or new MT engine - run a human pilot. It takes a few hours. It can prevent weeks of losses.
Pilot protocol:
Step 1 - Select the sample. Choose 3,000-5,000 words representative of the actual project. Don’t cherry-pick easy segments. If the project has tables, legal clauses, and technical specs, include all three in the sample.
Step 2 - Generate MT output. Use the exact engine and settings you’d use for the full project. Don’t test on a different engine version or with different pre-processing.
Step 3 - Post-edit with tracking. Have a trusted linguist post-edit the sample while tracking: time per segment (most CAT tools log this), edit distance (how much they changed vs the MT output), and any segments they had to fully rewrite.
Step 4 - Calculate and interpret. From the pilot data: - Words/hour: if below 500, the project is approaching retranslation effort - your MTPE quote is likely unviable - Average TER: gives a reliable estimate of effort for the full project - Percentage of segments fully rewritten: above 20% signals the MT engine is failing on this content type
Step 5 - Decide. If pilot data shows MTPE is viable, quote accordingly. If not, either reprice to FPE rates, recommend human translation to the client, or flag it before you’re committed.
As SwissGlobal documented in their 2026 productivity analysis, MT performance varies dramatically by language pair - they saw +130% post-editing speed gain for EN-FR vs. a -7% gain for EN-SE (slower than human translation). That gap is only discoverable through pilot testing per pair and per content type.
One practical note on the pilot: when surveyed about MT output quality in 2025, 66% of freelance translators described MT as “acceptable but requiring significant edits” - with only 12.74% calling it high quality. That’s your baseline expectation for major language pairs with current neural MT systems. If your pilot comes in above that average - good engine for your content. Below it - re-evaluate.
Tracking quality over time: baselines and degradation detection¶
One-time assessment before a project isn’t enough. MT engines update without notice. Content types shift. Post-editors change. Quality can degrade gradually over months without anyone noticing until a client complaint arrives.
Build a quality baseline and measure against it regularly.
What to track: - BLEU or COMET scores on a held-out test set (same segments, always - never change the test set) - TER across projects in the same language pair and content type - Post-editing speed (words/hour) per linguist per content type, tracked in your TMS - Error density from human reviews (errors per 1,000 words by MQM category)
When to run a check: - After any MT engine update or switch - When a post-editor reports that quality feels different - When post-editing time per project is trending up - Quarterly spot-check on your highest-volume language pairs
A held-out test set of 1,000-2,000 segments, post-edited once by a trusted linguist and locked permanently, gives you a reproducible benchmark. Running COMET on fresh MT output against this baseline takes minutes and immediately shows whether quality has shifted.
The Nimdzi MTPE Efficiency Gap report documents exactly this problem: agencies that route content to MTPE without tracking quality often pay near-human-translation effort at MTPE rates, burning translator relationships in the process. Regular quality monitoring is what closes that gap.
MT quality metrics vs translation quality (they’re not the same thing)¶
One distinction that gets conflated: MT quality metrics evaluate how good the raw MT output is. Translation quality assessment (per ISO 18587:2017 and ISO 17100) evaluates the final post-edited output.
For client contracts and quality guarantees, what matters is final output quality - not the raw MT quality. You can have poor raw MT that a skilled post-editor brings to excellent final quality. You can have good raw MT that a careless post-editor leaves with errors.
This means MT quality metrics should drive your internal workflow decisions - routing, pricing, engine selection - while human review of post-edited output drives client-facing quality assurance.
The exception: if a client requires a certification that no AI tools were used, or that translation was “human-authored” - those are legal/contractual terms that MT quality metrics can’t satisfy regardless of scores. Those require full human translation or specific disclosure agreements.
Tools that make this work at scale¶
Manual quality evaluation only scales so far. What agencies running MTPE at volume actually use:
For automated metric calculation: sacrebleu handles BLEU, TER, and chrF from the command line. It standardizes tokenization to make scores reproducible across teams. COMET runs via Python and gives segment-level scores you can average per project.
For integrated QA in CAT workflows: Trados Studio and MemoQ both log editing distance and time-per-segment automatically. Over 6 months of projects, that data builds into a solid empirical baseline for each language pair and content type. You don’t need external tooling - the data is already there in your TMS, it just needs to be exported and analyzed.
For production routing: ModelFront integrates with several TMS platforms and provides API-based QE scoring. It’s designed specifically for the use case of automatically routing segments to different editing tiers based on confidence scores.
For document workflows with formatted output: platforms like ChatsControl handle document upload and formatted draft generation (DOCX, PDF with layout preserved), then route the result to your post-editor. The linguist gets a formatted draft to review rather than raw text; MT quality assessment still applies to decide routing - the platform handles the file handling layer, not the quality judgment.
FAQ¶
How do I evaluate two MT engines to decide which one to use for my agency?¶
Run both on the same test set of 500-1,000 segments representative of your actual content. Calculate BLEU and COMET on each output using sacrebleu and the COMET library. Then have one post-editor work through 100-200 segments from each engine, tracking editing time separately. Automated scores tell you which engine performs better on paper. The human pilot tells you whether the better-scoring engine is actually meaningfully faster to post-edit for your workflow.
Don’t run the evaluation on generic internet test sets - they won’t reflect your content domain, and the scores won’t predict your real workflow performance.
Do I need to build my own reference translations, or can I reuse existing ones?¶
You need your own domain-specific references. Generic references from WMT news datasets don’t reflect technical or legal translation quality and produce misleading scores.
Build your reference set from post-edited work you’re already doing. After any significant project, save a sample of 200-500 post-edited segments with their sources as a permanent test set for that language pair and content type. Over 6 months, you’ll have solid evaluation data for your most common pairs without any extra work.
How do I measure MT quality in production when I don’t have reference translations ready?¶
Use Quality Estimation tools. COMET-Kiwi, ModelFront, or the built-in QE features in Phrase can score MT output without a reference. They predict quality rather than measuring against a gold standard, but for routing decisions - deciding which segments need heavy vs light editing - QE scores work well in practice.
You can also use the post-editing process itself as quality data: track editing distance per segment as post-editors work. After accumulating 10,000+ segments of data, you have a real empirical baseline for what good vs poor MT output looks like in your specific workflows.
What’s the actual difference in post-editing workload between a BLEU of 35 and 45?¶
For technical EN-DE content: BLEU 35 typically means 35-45% of segments need significant editing, while BLEU 45 might mean 20-30%. That translates roughly to 600-700 words/hour at BLEU 35 vs 800-1,000 words/hour at BLEU 45 for full post-editing. Whether that 10-point BLEU difference changes your project margin depends on your client rate and editor pay.
For creative or marketing content, the same BLEU scores are largely meaningless as effort predictors - MT output can be fluent and still require near-retranslation because it’s culturally wrong.
Can MT quality metrics be written into client contracts?¶
Yes, but carefully. If you specify a minimum BLEU or error-rate threshold in a contract, you and the client need to agree on the methodology upfront - same test set, same reference, same scoring tool. MQM-based contracts are more defensible because the error categories are interpretable: “no critical accuracy errors per MQM taxonomy” is clearer and more auditable than “BLEU above 45.” Enterprise clients increasingly require formal quality metrics in translation contracts, especially for regulated industries.
How often should we re-evaluate MT engines?¶
At minimum, after any engine update or switch. In practice: run a quick COMET check on your held-out test set every quarter for high-volume language pairs. More often if post-editing times are trending up or post-editors flag a change. Google Translate and DeepL push updates without version notices - quality can shift between updates, and you won’t know unless you measure.
Sources¶
- Papineni et al. (2002) - BLEU: A Method for Automatic Evaluation of Machine Translation
- Unbabel/COMET - Crosslingual Optimized Metric for Evaluation of Translation
- ISO 18587:2017 - Post-editing of machine translation output
- GTS Translation - The State of MTPE in 2025: What Translators Think
- SwissGlobal - Translation productivity 2026: human translation vs MTPE
- Slator - How Fast Should You Post-edit Machine Translation?
- Nimdzi - The MTPE Efficiency Gap
- Hunnect - Fair Pricing Strategies for MTPE