How to Run a Human Evaluation of MT Output the Right Way

Your vendor just sent over a glowing benchmark: their customized EN→PL engine scored 47.2 BLEU on the technical documentation test set, up from 39.8 six months ago. You route 30% of your content volume through it. Three weeks later a major client flags a batch of translated user manuals - wrong units, inverted safety instructions, one clause that means the exact opposite of the source.

BLEU didn’t catch any of it. Your vendor’s benchmark didn’t catch any of it. A human evaluator reading the output as a user would catch all of it in ten minutes.

This is the problem human evaluation of MT exists to solve - and the reason most agencies run it wrong.

Why automatic metrics aren’t enough¶

BLEU, COMET, TER - these metrics are useful for tracking MT engine development over successive versions. They measure things that are fast and cheap to compute: n-gram overlap, neural similarity scores, edit distance. They do not measure whether a translated sentence means the same thing as the source. They do not catch inverted negations, wrong domain terms, missing safety disclaimers, or outputs that are grammatically fluent but factually incorrect.

As CSA Research put it in their practitioner assessment: automatic MT metrics are “artificial and irrelevant for translation production environments.” That’s harsh, but the core point stands: no automatic metric currently tells you whether translated content is fit for its intended purpose with its intended audience.

Human evaluation does. The catch is that running it badly gives you results that are equally misleading - just expensive and slow instead of cheap and fast.

The three main frameworks¶

Before you design your evaluation, pick the right tool for your question.

Direct Assessment (DA)¶

DA is WMT’s segment-level evaluation protocol. Raters see a source segment and a single translation (they don’t see the reference or which system produced it), then score overall quality on a continuous 0-100 slider. Multiple raters score each segment; scores are z-score normalized per rater to remove scale bias (one rater’s 70 might be another’s 85 if their internal scales differ).

What it answers: “How does System A rank against System B on this content?” Fast to administer, good for comparing multiple systems when you need a relative ranking without deep error analysis.

What it doesn’t tell you: Where the errors are, what type they are, or whether the output meets a specific quality threshold. A score of 72 doesn’t tell you if that 28-point gap is one critical mistranslation or a hundred minor punctuation issues.

MQM (Multidimensional Quality Metrics)¶

MQM is the standard for production-grade evaluation. Raters don’t score holistically - they read the translation against the source and mark specific error spans, then classify each one by type and severity.

Error types under MQM:

Severity weights (from themqm.org):

MQM score formula: (Critical × 25 + Major × 5 + Minor × 1) ÷ word count × 1,000

Lower is better. Most production quality thresholds are set at under 5-10 MQM points per 1,000 words for publishable content; legal and medical content is typically set at under 2.

What MQM answers: Exactly where quality breaks down, in what categories, and at what severity. If your engine is producing accurate but unnatural-sounding output, MQM tells you it’s a Fluency problem, not an Accuracy problem. That determines whether you fix it with style guides, TM cleanup, or post-editing workflows.

Error Span Annotation¶

A lighter version that sits between DA and full MQM. Raters mark spans that contain errors without requiring full categorization. Easier to train raters on, faster to run, and the 2024 paper on Error Span Annotation showed it correlates well with MQM while taking 30-40% less time per segment.

Useful when you need more signal than DA but don’t have the budget or time for full MQM annotation.

Rater selection - the decision that determines everything¶

Every other choice you make in evaluation design is secondary to this one. The wrong raters will give you confident-looking numbers that don’t reflect real quality.

What makes a rater qualified¶

Native speaker of the target language, no exceptions. A highly fluent non-native speaker will miss register issues and naturalness problems that any native speaker would catch.

Professional translator with domain experience. Research published in TACL (MIT Press) on “Experts, Errors, and Context” confirmed what practitioners already suspected: professional translators catch significantly more errors than bilingual non-translators, particularly subtle ones - incorrect domain terms, cohesion breaks across segments, pragmatic errors. Crowd workers are “more accepting of MT output” and systematically give it higher scores, which skews your results in the direction of overestimating quality.

Domain match matters. A literary translator rating medical device manuals will miss terminology errors that are obvious to a medical translator. If your content is legal, your raters need legal translation experience. For technical documentation, technical translators.

As WMT noted when revising its evaluation procedures, crowd workers “were less capable of detecting subtle MT errors” than professional translators - which is why WMT moved away from crowd-sourcing toward professional translation pools for high-quality evaluation.

How many raters¶

Minimum 3 per segment for MQM. Most serious evaluations use 2-3 annotators per segment with an arbitration protocol for disagreements. Below 3 raters, your inter-annotator agreement calculation becomes unreliable.

For Direct Assessment, WMT uses multiple ratings per segment (typically 15-25 ratings per segment in shared task evaluations, though that’s for research-grade confidence - production evaluations typically use 3-5).

Rater training - non-negotiable 2 hours minimum¶

Raters need to be calibrated before they touch real data. At minimum: 1. Walk them through your annotation guidelines (not just hand them a PDF) 2. Have them annotate 20-30 pre-labeled training segments where you know the correct answers 3. Review their annotations together, discuss disagreements 4. Only start counting ratings after this calibration phase

Research guidelines recommend “at least 2 hours of training per rater.” In practice, for MQM on specialized content, plan for a full half-day calibration session.

Designing the test set¶

Bad test set design is the second most common way an evaluation fails.

Size¶

The MQM sampling guidelines at themqm.org are clear: evaluating the entire translated content is the gold standard; when sampling is used for cost reasons, minimum usable sample is around 500-1,000 words. For comparing two production systems with enough statistical power to be confident in the result, target 1,000-5,000 segments.

Below 500 segments, you don’t have enough data to distinguish real quality differences from variance in the content - a few unusually easy or unusually hard segments can flip your results.

For practical production use: - Quick MT engine screening: 500-1,000 words - Vendor comparison before contract: 1,000-2,000 segments - Production quality gate validation: 2,000-5,000 segments representative of your real content

Content representativeness¶

Your test set should look like your actual production content. Don’t use an easy, well-formatted sample for evaluation if your production content includes tables, footnotes, legal cross-references, and domain-specific acronyms - those are exactly where MT fails most often.

If your content has multiple sub-types (user manuals + marketing copy + support FAQs), stratify your sample to include all of them. One number for a mixed test set will hide the fact that the engine is excellent on manuals and terrible on marketing.

Test set isolation¶

The test set must not have been seen by the MT engine during training or fine-tuning. This sounds obvious, but it’s violated constantly in vendor benchmarks. If a vendor is providing their own test results, require that they’re measured against segments you control, not theirs. Ask directly: “Was any portion of this test set used in fine-tuning or adaptation?” If they can’t confirm it wasn’t - or won’t confirm - treat the numbers as marketing, not evaluation.

Source text quality¶

Evaluate the MT output against the actual source. Don’t translate back-translated content (text that was originally in the target language, translated to the source, then used as MT input) - back-translated sources are simpler and easier for MT, which inflates quality scores.

Running the evaluation¶

Blind setup¶

Every rater sees: - The source segment - A single translation - (For MQM) A reference translation, if using one

What raters must not see: - Which system produced which translation - Other raters’ annotations on the same segment - Results from other systems side-by-side

Present systems in different random orders to each rater. Annotate multiple systems in interleaved fashion so raters don’t settle into a mental model of “this looks like System A.” This is the same principle as blind tasting in food evaluation - and it matters as much here.

As confirmed by research on WMT evaluation, “although assessments are blind, researchers have been shown to slightly favor translations produced by their own system” - which means even supposedly objective raters have biases when they know provenance. Remove the opportunity.

Quality control segments¶

Embed pre-annotated control segments throughout the evaluation set - segments where you already know the correct annotation. These let you flag raters who are performing unreliably without telling them which segments are controls.

eBay’s human MT evaluation team described this approach: professional judgments are “randomly inserted throughout the evaluation job” and used to remove judges who perform below threshold on the controls. It’s the same principle as attention checks in survey research.

If a rater fails more than 20-25% of your control segments, their annotations should be flagged and reviewed before inclusion.

Annotation guidelines¶

Write annotation guidelines before you recruit raters - not after, and not during. The guidelines should cover:

• How to handle ambiguous source text (don’t penalize MT for ambiguity that’s already in the source)
• How to handle multiple valid translations (synonyms that mean the same thing are not errors)
• Scope: are you evaluating raw MT output or MT+light post-editing?
• Which MQM error categories apply to your content type (legal content doesn’t need “Style” evaluated the same way as marketing copy)
• Examples for each severity level drawn from your actual content domain

Have at least one pilot run where you annotate a sample together with your raters and revise the guidelines based on questions and edge cases they surface. Guidelines are living documents; plan to update them.

Calculating inter-annotator agreement¶

If you have 3 raters and skip this step, you’re trusting a number that might be driven by one outlier rater.

Cohen’s kappa (for two raters) and Fleiss’ kappa (for 3+) measure how much raters agree beyond what chance would predict. Scale:

For MQM error annotation, the research average is around kappa 0.51, which is in the “moderate” range. Studies with well-trained specialist annotators have reached 0.92-0.95, but that requires careful calibration and domain-matched expertise.

If your kappa comes back below 0.3: stop before analyzing results, review your guidelines with the raters, re-annotate a sample together, fix the disagreements, then continue.

A low kappa doesn’t mean your raters are wrong - it means your guidelines leave too much room for interpretation. Find where the disagreements are clustering and write explicit rules for those cases.

Common mistakes that invalidate results¶

Evaluating segments out of document context. A segment that says “Do not press this button” could mean something very different in context. Sentence-level evaluation is blind to cohesion and coherence problems. For anything beyond a quick ranking, evaluate at document level or at minimum show raters preceding and following segments.

Not controlling for source text difficulty. Easy content (short segments, common vocabulary, no ambiguity) inflates MT scores relative to hard content. If you’re comparing a generic MT engine on easy content with a domain-adapted engine on hard content, you’ve measured the content, not the engines.

Mixing rater expertise levels. One professional translator and two bilingual non-translators doesn’t give you three equivalent data points. Either use all professionals or be explicit about which evaluations are expert and which aren’t, and analyze them separately.

Using fluency as a proxy for accuracy. MT systems are much better at producing fluent output than accurate output - this is one of the most consistent findings in MT research. A segment that reads smoothly can still be a mistranslation. Raters, especially non-translators, systematically mistake fluency for quality. MQM prevents this by separating the two dimensions; DA encourages it.

Evaluating without domain reference. Raters who don’t know the domain will miss terminology errors. “Myocardial infarction” and “heart attack” might look interchangeable to a general translator but have different implications in a clinical trial protocol.

Letting raters know which system is which. Even subtle cues - “System A is our new engine” - introduce bias. Blind evaluation is not optional if you want defensible results.

What to do with the results¶

MQM results tell you where to intervene¶

If your MQM output shows: - High accuracy errors: The engine doesn’t understand source meaning well enough. Fine-tune on domain-parallel data, or the content category isn’t suitable for raw MT and needs full human translation. - High fluency errors: The output is accurate but unnatural. A good style guide + post-editing workflow can fix this at lower cost than retranslation. - High terminology errors: The engine is missing domain vocabulary. Fix with glossary injection, terminology fine-tuning, or term enforcement tools. - High locale convention errors: Configuration problem - date formats, unit systems, currency. These are usually fixable in post-processing or through engine configuration.

This diagnosis is only possible with MQM. A DA score of 65 tells you the quality is suboptimal; MQM tells you whether to invest in fine-tuning, post-editing, or glossary work.

Setting pass/fail thresholds¶

Common production thresholds by content type (MQM score per 1,000 words):

These numbers aren’t universal - calibrate against your own content and client requirements. But they give you a starting framework.

Feed results back to vendors¶

A human evaluation that produces only a number is a missed opportunity. Share the MQM error distribution with your MT vendor - they can use it to prioritize fine-tuning. “Your engine has 3x more critical accuracy errors on subordinate clauses” is actionable engineering information that a BLEU score delta can’t provide.

FAQ¶

How long does a human MT evaluation take?¶

For MQM annotation, an experienced professional translator can annotate roughly 400-600 words per hour (factoring in marking error spans, categorizing, and noting severity). A 2,000-segment evaluation with 3 raters takes approximately 30-40 person-hours, plus 4-6 hours of training and calibration. Budget 1-2 weeks for scheduling and turnaround.

Can I use the same raters for multiple evaluations over time?¶

Yes, and it has real advantages - trained raters are calibrated to your guidelines, which improves consistency. The risk is rater fatigue and “anchoring” to previous evaluations. If evaluating the same content type over time, rotate at least one rater every few evaluations to detect drift.

What’s the difference between quality evaluation and quality estimation?¶

Human evaluation (MQM, DA) measures actual quality by humans reading the output. Quality estimation (QE) is automated - it predicts quality scores without a reference translation, using models trained on human evaluation data. COMET-QE, for example, predicts MQM-style scores at the segment level. QE is much cheaper and faster; human evaluation is the ground truth QE is trying to approximate.

Should I evaluate raw MT or post-edited MT?¶

Both, but for different purposes. Raw MT evaluation tells you whether the engine is suitable for a content type and helps you set post-editing requirements. Post-edited MT evaluation tells you whether your post-editing workflow is producing acceptable final quality. Conflating the two gives you a score that’s really measuring your post-editors, not the engine.

Is there a standard tool for running MQM evaluations?¶

Alconost’s MQM Annotation Tool and Unbabel’s annotation platform support MQM workflows. For research-grade evaluation, Appraise (used by WMT) is open-source. For agencies without annotation infrastructure, running evaluations through a structured spreadsheet with explicit columns for error span, category, and severity is workable for smaller datasets.

Sources¶

• MQM Framework - themqm.org
• MQM Sampling Guidelines
• Slator - How to Fix the 5 Flaws in Evaluating Machine Translation
• Experts, Errors, and Context - TACL/MIT Press
• eBay - Human Evaluation of Machine Translation
• Machine Translate - Human evaluation metrics
• Language IO - How to Perform Machine Translation Evaluation
• Error Span Annotation: A Balanced Approach (2024)