How to Run an MT Engine Bake-Off: A Step-by-Step Comparison Guide

Your team just spent three days “evaluating” MT engines. One PM pasted a product description into each web UI, one linguist compared two segments from a legal contract, and somebody ran a marketing slogan through all five candidates. Now you’re being asked which engine to use.

You don’t know. Nobody actually knows. What you ran wasn’t an evaluation - it was a collection of impressions.

Here’s how to run a real bake-off: structured, repeatable, and defensible when the finance team asks why you committed to a six-figure annual API spend.

Why informal MT comparisons fail¶

Three failure modes show up in almost every informal MT evaluation:

Cherry-picked samples. Someone picks sentences where a favorite engine looks good, and that impression sticks. But what looks impressive in five hand-selected segments doesn’t generalize to 50,000 production segments - and this is the most common bake-off failure mode in agency settings.

UI testing vs. API deployment. The engine you evaluated in a free web UI may behave differently at API scale: context handling, batch processing, glossary enforcement, and segment boundary treatment all vary between the consumer product and the production API. What you test in the UI isn’t what you deploy in production.

No baseline. Comparing Engine A to Engine B tells you which is relatively better. It doesn’t tell you whether either one actually saves post-editing time compared to translating from scratch - which is the entire economic case for using MT at all.

A structured bake-off fixes all three: fixed corpus, defined metrics, blind evaluation, and a comparison against a from-scratch baseline.

What a bake-off is actually measuring¶

Before picking metrics, decide what question you’re answering. Three scenarios lead to very different evaluation designs:

Scenario A - deciding whether to adopt MT at all. Your primary question is whether MT + post-editing is faster than translation from scratch on your content. Run a time-comparison pilot: give your translators a 2,000-word batch translated from scratch vs. a 2,000-word batch of MT output. Measure words per hour. If the MT batch doesn’t show at least 30% throughput improvement, the economics don’t work.

Scenario B - choosing between 2-3 specific engines. This is the classic bake-off. You’re comparing known candidates on the same corpus using automatic metrics and human post-editing evaluation.

Scenario C - evaluating whether a current engine should be replaced. You’re measuring whether a new entrant or a new model version beats your production system on your content.

Most agency bake-offs are Scenario B. The rest of this guide focuses there.

Step 1: Define your evaluation criteria before touching any engine¶

Write down what “winning” looks like before you see any results. This is the single most important procedural rule in MT evaluation - without it, the evaluation will unconsciously validate whatever impression someone formed during setup.

Typical agency evaluation criteria, with example weights:

Your weights will differ based on your business. An agency doing mostly EU language pairs with heavy CAT tool use should weight integration higher. An agency doing rare language pairs should weight coverage higher. A legal or medical LSP should weight data security higher.

Write these weights down before you see any results, and lock them in. Changing weights after seeing the results turns evaluation into rationalization.

Step 2: Build a representative test corpus¶

The test corpus is the most consequential input to a bake-off. Get this wrong and every downstream metric is meaningless.

Corpus size¶

The practical minimum is 2,000 segments for a single language pair and domain. For results you can defend:

• Single language pair, uniform domain: 2,000 segments minimum
• Multiple domains (e.g., technical + marketing): 500-1,000 per domain, minimum 2,500 total
• Multiple language pairs: 1,000 segments per language pair minimum

Under 2,000 segments, quality differences between engines may not be statistically meaningful - a run of difficult or easy segments can skew results in either direction.

Corpus composition¶

Your test corpus should reflect what your agency actually translates, in the proportions you do it. If 60% of your volume is software UI strings, 30% technical documentation, and 10% marketing copy, your corpus should match that split.

Don’t use your TM as your test set without scrubbing it first. Segments already in translation memory may show inflated scores because some engines partially memorize content from public parallel corpora. Use recently translated content that isn’t in any public TM.

Include edge cases deliberately: - Sentences with numbers, dates, and units of measure - MT errors here are often the most expensive to miss - Named entities (product names, company names, people) - engine behavior on these varies significantly between vendors - Long sentences (40+ words) - MT quality degrades on long segments in some engines - Domain-specific terminology - the segments where your glossary matters most

Reference translations¶

You need human reference translations for every segment in your corpus. These are your gold standard for all automatic metrics (BLEU, COMET, TER). Use existing translations from your TM that you’re confident in, or commission new ones specifically for the bake-off.

A common mistake: using TM translations as references without reviewing them. If your references contain errors, every automatic metric is measuring against a wrong target. Spot-check 5-10% of your reference translations before running metrics.

Step 3: Get trial access and run all engines on the same corpus¶

Set up API access for each candidate. Most major engines offer free trials:

Run all engines on the same segments in the same order, as close in time as possible. This controls for model updates - major engines update their weights 2-4 times per year, and an update mid-bake-off would invalidate your comparison.

Anonymize outputs before human evaluation. Label files as Engine 1, Engine 2, Engine 3 - not DeepL, Google, Azure. Assign engine labels only after all human evaluations are complete. This step is non-negotiable. Evaluators who know which engine produced an output will unconsciously score it differently based on brand expectations.

Step 4: Automatic metrics - what they tell you and what they don’t¶

Run at least two automatic metrics. Don’t rely on just one.

BLEU (Bilingual Evaluation Understudy)¶

BLEU measures how many word sequences (n-grams of length 1-4) in the MT output appear in the reference translation. Score range 0-100; higher is better.

Typical BLEU scores for modern neural MT: - Major European language pairs (EN↔DE, EN↔FR, EN↔ES): 30-45 - Less common European pairs: 20-35 - Distant language pairs (EN↔ZH, EN↔JA): 15-30 - Low-resource languages: often below 15

BLEU is useful for catching obvious quality gaps between engines and tracking regressions over time. What it misses: it doesn’t penalize fluent-sounding but semantically wrong output, and it doesn’t reward correct output that uses different phrasing than the reference. As WMT 2023 findings show, BLEU correlates poorly with human judgment in fluency-sensitive domains like marketing and literary translation.

COMET (Crosslingual Optimized Metric for Evaluation of Translation)¶

COMET is a neural metric trained on human evaluation data, designed specifically to correlate with human judgment where BLEU fails. It takes the source, MT output, and reference as inputs and produces a score typically between 0.6 and 1.0. Higher is better.

The reference model wmt22-comet-da (available on Hugging Face) is the current standard. For production bake-offs, COMET is the more reliable automatic metric - it catches register mismatches and meaning shifts that BLEU misses entirely.

TER (Translation Edit Rate)¶

TER measures how many edits (insertions, deletions, substitutions, reorderings) are needed to turn the MT output into the reference. Lower is better; 0 means identical to the reference. Typical MT TER values: 0.35-0.65 for major language pairs.

TER is particularly useful in MTPE contexts because it proxies post-editing effort. An engine with TER 0.35 theoretically requires fewer edits per segment than one with TER 0.55. In practice, TER is a rough proxy - real PE effort also depends on error distribution, segment type, and individual translator speed.

What automatic metrics can’t tell you¶

None of BLEU, TER, or COMET will tell you: - Whether the MT output is good enough to publish without editing vs. needs heavy rewriting - Whether a specific error type matters more in your context (a wrong date in a legal document is more critical than an awkward adjective in a product description) - How long a post-editor will actually spend, which depends on their domain familiarity and typical working speed

This is why automatic metrics are the first filter, not the final answer.

Step 5: Human post-editing evaluation - the metric that determines your ROI¶

Two human evaluation methods work reliably at agency scale.

Method A: Blind direct assessment¶

Evaluators see each MT output segment and rate it on a 0-100 scale. Two dimensions: adequacy (is the meaning correct and complete?) and fluency (does it read naturally in the target language?). No reference translation is shown - evaluators judge the MT output on its own merits.

Scores above 80 generally indicate output suitable for light post-editing. Scores below 60 often indicate segments faster to retranslate from scratch than to fix.

Run at least two evaluators per engine and calculate inter-annotator agreement (IAA). If IAA falls below 0.7 (Cohen’s kappa), your evaluation protocol needs tightening - evaluators may be applying different standards.

Method B: Timed post-editing (the metric that predicts your economics)¶

Give post-editors a 2,000-word batch of MT output - without revealing which engine produced it - and time them completing the task. Track: - Words edited per hour for each engine - Edit distance per segment (automated via Levenshtein distance, comparing MT output to the finished post-edited translation)

This is the only metric that directly translates to business value. If Engine A enables 8,500 PE words per hour and Engine B enables 6,200 PE words per hour, Engine A saves roughly 27% of post-editing labor cost - regardless of how either engine scored on BLEU or COMET.

As Nimdzi research on the MTPE efficiency gap shows, the spread between best and worst-performing engines on post-editing throughput is typically 20-40% - far larger than most people expect before running the numbers.

MQM for structured error analysis¶

When you need to know not just which engine is worse but where each engine systematically fails, use MQM (Multidimensional Quality Metrics). MQM categorizes errors by type:

• Accuracy - mistranslation, omission, addition, untranslated content
• Fluency - grammar, punctuation, spelling, register
• Style - awkward phrasing, terminology inconsistency
• Locale convention - date format, number format, currency, address format

Each error is assigned a severity: minor (1 point), major (5 points), critical (25 points). Final MQM score = total weighted error count per 1,000 words. Lower is better.

MQM is the evaluation framework used in professional procurement and by large enterprise MT buyers. For a standard LSP bake-off you don’t need the full MQM framework, but the error taxonomy is valuable for understanding where each engine fails - and whether those failure modes matter for your specific content type.

Step 6: Test integration in your actual TMS and CAT tool¶

An engine that scores well in isolation but creates integration headaches in your workflow is not a production-viable choice.

What to verify¶

• Native plugin vs. API connector: Does the engine have an official plugin for your CAT tool (memoQ, Trados, Phrase, Smartcat)? Native plugins handle segment context better and are maintained by the vendor. API connectors via middleware work but add a dependency that can break on CAT tool updates.
• Glossary enforcement: Upload a client termbase of 20-30 terms and verify they appear correctly in the MT output. Test with both common and edge-case terms.
• Context handling: Does the engine receive surrounding sentences as context, or translate each segment in isolation? Context-aware translation (sentence window) improves consistency on connected text - particularly for UI strings and structured documents.
• Batch throughput: How many characters per second does the API process for your typical batch size? For overnight processing workflows, a 2-hour vs. 20-minute job completion time has real operational impact.

Minimal integration test¶

Set up a mock project: a real client file (or one that mirrors your typical format), pre-translated by each engine, imported into your TMS, assigned to a post-editor, and completed. Time the full cycle. This surfaces issues you’d never catch in segment-level evaluation: file formatting problems, tag handling, segment boundary errors, and TM match conflicts with MT output.

Step 7: Calculate total cost of ownership¶

API pricing is the most visible number, but it’s rarely the largest component of TCO for agencies doing MTPE at volume.

TCO formula¶

Annual TCO = (Monthly char volume × API price/char × 12)
           + (Annual PE words ÷ PE words/hour × hourly PE rate)
           + (Integration setup cost ÷ 3)      [amortized over 3 years]
           + (Annual maintenance: plugin updates, glossary sync, TM management)

Example with real numbers¶

Agency processing 5 million characters per month, EN→DE, post-editing at 7,500 words/hour at a cost of €35/hour. Approximate character-to-word ratio: 1 word ≈ 5.5 characters.

(PE cost = annual PE words ÷ throughput × hourly rate. At 5M chars/month: annual chars = 60M, at 5.5 chars/word = ~10.9M words/year)

At this scale, a 700-word/hour PE throughput difference translates to €5,000-7,000/year in labor cost - more than the spread in API pricing. This is why the PE throughput pilot from Step 5 is the most important number in your TCO model.

Bake-off timeline and roles¶

A minimal two-engine bake-off takes 3-4 weeks:

Who does what: - Localization engineer / project lead: corpus preparation, API setup, anonymization, metric scoring - 2-3 senior post-editors: blind direct assessment and timed PE pilot - Finance or operations: TCO model and API contract review

The person who managed the API setup should not do the human evaluation - they’ve already seen which engine is which.

Common shortcuts that invalidate a bake-off¶

• Showing evaluators the engine names before evaluation is complete
• Using a test corpus taken from vendor-published benchmarks (curated to show favorable results)
• Using TM segments the engine may have seen during training
• Skipping the timed PE pilot because it “takes too long” - this eliminates the only directly business-relevant measurement

Scoring and decision template¶

Fill this in after all evaluations are complete, before discussing results with the team:

MT Engine Bake-Off Scorecard
Language pair: _______________   Domain: _______________   Date: ___________

                           Weight    Engine A    Engine B    Engine C
COMET score                 25%        ___         ___         ___
MQM penalty score           10%        ___         ___         ___
PE throughput (words/hour)  30%        ___         ___         ___
CAT integration fit         15%        ___         ___         ___
TCO year 1                  10%        ___         ___         ___
Language coverage            5%        ___         ___         ___
Data security compliance     5%        ___         ___         ___
                            ---
Weighted total              100%       ___         ___         ___

Present the weighted totals before discussing individual scores. This prevents the highest-paid person in the room from anchoring the decision before the data is on the table.

FAQ¶

What’s the difference between a bake-off and a pilot?¶

A bake-off is a structured evaluation for selecting an engine. A pilot is a real production run (typically 15,000-25,000 words) with the winning engine, used to validate business economics before full rollout. They’re sequential: bake-off first, pilot second.

Should we include LLM-based translation (GPT-4, Claude) in the bake-off?¶

Worth testing for your content type. LLM translation handles context and nuance well and is competitive with dedicated MT engines on short-form, marketing, and context-dependent content. Practical limitations for agency workflows: unpredictable latency, higher cost per character, no native CAT tool plugins, and no support for CAT-format file processing without custom integration. If your workflow is CAT-centric, LLM translation currently requires significant development work to integrate.

How do we evaluate rare language pairs where we don’t have in-house linguists to review?¶

Two options: (a) contract with a freelance linguist in that language specifically for the bake-off evaluation - the only fully reliable option; (b) use COMET-QE (quality estimation mode, no reference required) as a proxy, but validate with at least a small human sample. Don’t rely solely on automatic metrics for languages where you can’t validate.

Is a bake-off worth running for lower-volume agencies?¶

It becomes worth the effort once your MT volume exceeds roughly 1 million characters per month. Below that, API cost differences are in the hundreds of dollars annually and don’t justify 4 weeks of evaluation work. Use vendor free tiers to test on your content, consult independent benchmarks like WMT annual findings, and start with whichever engine your most experienced linguists find easiest to post-edit.

What about MT Quality Estimation tools - do they replace human evaluation?¶

MT Quality Estimation (QE) tools predict segment-level quality without a reference translation. They’re useful for routing: sending high-confidence MT output to light PE and low-confidence output to full human translation. They’re not a replacement for bake-off evaluation because they tell you how an engine scores on its own output distribution, not how it compares to your specific quality standard.

How often should you re-run the bake-off after choosing an engine?¶

Major engines update their models 2-4 times per year. Run a quick auto-metric check on a fixed held-out set quarterly. Run a full bake-off when: a competing engine releases a major version claiming to beat your current choice, your language mix changes significantly, or you’re about to renew an annual API contract.