A single missing “not” can turn a safety instruction into its opposite, while a polished contract clause can quietly make the wrong party responsible. Spelling checks and terminology checks may pass both translations because the words look familiar and the sentences read smoothly.
Those checks still matter. They catch real problems, but they don’t prove that the translation preserves who must do what, what is allowed, or which action is prohibited. Semantic translation QA needs a separate pass focused on the proposition: the basic claim or instruction the sentence communicates.
This distinction matters in contracts, medical and safety material, financial instructions, and any document where a changed obligation or actor can affect a decision. The practical fix isn’t to discard automated QA. It’s to give human reviewers a short, repeatable way to check the parts of meaning that ordinary mechanical checks may miss.
What semantic translation QA checks¶
Semantic translation QA checks whether a target sentence communicates the same proposition as its source. A proposition includes more than the words in a sentence: it includes the polarity of a claim, the strength of an instruction, the people or things involved, and the relationships between them.
The MQM framework groups translation-quality issues into seven top-level dimensions: Accuracy, Terminology, Fluency, Style, Locale conventions, Audience appropriateness, and Design and markup. That structure makes a useful distinction: a text can be fluent and correctly formatted while still being inaccurate. MQM’s error typology defines Accuracy errors as target content that doesn’t correspond to the source proposition because the message has been distorted, omitted or added to.
A proofreader looking only for awkward phrasing may never notice that the translation has changed who receives a payment. A terminology checker may confirm that every defined term appears consistently while missing that a clause now says a party may act where the source says it must. Different checks answer different questions.
A useful working split looks like this:
| QA check | What it can help identify | What it doesn’t establish on its own |
|---|---|---|
| Spellcheck | Misspellings and some typing mistakes | Whether the sentence makes the same claim as the source |
| Formatting check | Missing or changed layout elements | Whether a warning or condition has been reversed |
| Terminology check | Inconsistent or unapproved terms | Whether the right party is attached to the right action |
| Number and name check | Some changed figures or named entities | Whether a sentence’s logic, scope or modality is correct |
| Bilingual semantic review | Changes to proposition, roles, conditions and exceptions | Every possible stylistic or layout issue |
The distinction also helps teams describe findings clearly. “The sentence sounds wrong” doesn’t tell a reviser what changed. “The target removes the source prohibition” identifies a polarity problem. “The target assigns the payment obligation to the recipient instead of the sender” points to a participant-role reversal.
MQM separates mistranslation from addition and omission. Its subordinate mistranslation categories include false friends, misrepresentation of technical relationships and machine-translation hallucinations. That terminology is helpful when a translation looks plausible but gives the reader a different relationship or claim.
Reviewers don’t need to label every issue with a large taxonomy. The MQM Full typology says teams should choose a level of detail that fits their working environment. A practical project can use a focused checklist for polarity, modality and participants without applying every available error subtype.
Why negation, modality and swapped parties slip through¶
Meaning errors pass QA tools when a check measures a nearby property rather than the proposition itself. A sentence can have correct spelling, familiar terminology and fluent grammar even as its meaning changes.
Negation changes whether a proposition is affirmed or denied. Modality changes how strongly a statement expresses obligation, permission, possibility, ability or recommendation. Participant reversals change which actor performs an action, which party receives it, or what the action affects. Those features can be small in form and large in consequence.
Negation: one cue can reverse the instruction¶
Negation words such as “not,” “never” and “without” can determine whether an action is permitted, required or prohibited. A missing negative cue may leave a grammatical sentence behind. A reviewer who reads for fluency can pass over the exact word that reverses the instruction.
The MQM Full typology gives a clear example: the source says a medicine should not be administered above 200 mg, while the translation says it should be administered above 200 mg. The translated wording can be perfectly readable; the instruction is still the opposite.
A negation review has to look beyond whether a visible negative word appears. Reviewers also need to check its scope: which action, amount, person or condition does the negative apply to? A sentence can contain a negative cue and still attach it to the wrong part of the proposition.
A manual analysis of Chinese-to-English statistical MT published in 2015 categorized negation errors through semantic elements and string-level operations. Scope reordering was the most frequent category in that paper’s task, showing why a search for a visible negative word can miss a change in what the negation applies to (study). The finding is specific to that task and system; it’s a useful warning about review method, not a general error rate.
Research also shows why individual results need context. A study published in 2020 tested negation translation across 17 directions and reported that downstream quality reductions exceeded 60% in some cases (study). That result describes the study’s tested systems, data and directions. It doesn’t mean that current translation systems have a general negation error rate above 60%.
The same study distinguishes omission of a negative cue from semantic reversal. Both errors can make a sentence that reads fluently communicate the opposite proposition. A reviewer should therefore record what changed, not just whether the target contains a negative word.
Modality: must, may and should aren’t interchangeable¶
Modality tells a reader how to interpret an action or statement. “Must” can express an obligation; “may” can express permission or possibility; “should” can express a recommendation or expectation. “Can” can express ability or possibility. The exact relationship varies by context, so word matching alone isn’t enough.
A contract clause that changes “the supplier must notify the customer” to “the supplier may notify the customer” weakens an obligation into permission. A safety instruction that changes a recommendation into a command changes the force of the guidance. A policy statement that changes “may be suspended” into “will be suspended” can make a possible outcome sound certain.
The reviewer should identify the source’s force before judging the target. Ask whether the source requires, permits, recommends, predicts or describes an ability. Then ask whether the target says the same thing in the context of the document. A dictionary match isn’t enough if the translated modal verb has a different force in that sentence.
Conditions matter too. “The provider may suspend service if payment is late” doesn’t make the same claim as “the provider must suspend service if payment is late.” The conditional event stays the same, but the action’s force changes. A QA check that finds both the provider and the payment reference can still miss that shift.
Swapped parties: the right words can describe the wrong relationship¶
A participant-role reversal happens when a translation changes who acts, who receives an action, or what the action affects. The parties and verbs may all appear in the target, which can make a segment look complete during a quick review.
Consider a source clause in which a buyer pays a seller. A target clause that says the seller pays the buyer contains the same parties and the same basic action, but the relationship has flipped. Similar reversals can occur in instructions, insurance language, technical descriptions and customer correspondence.
The MQM Full typology describes technical-relationship mistranslation as content that can sound plausible while conveying the wrong relationship between technical components. The same idea helps reviewers classify swapped parties: the problem is an accuracy error, not just a fluency defect.
A translation can also introduce ambiguity that isn’t present in its source, which MQM treats as an accuracy problem. Word-level overlap doesn’t guarantee that a reader will interpret the target in the same way. Reviewers should check the relationship in the full sentence and compare its participants with defined terms elsewhere in the document.
How automated QA and human review fit together¶
Automated QA and bilingual revision solve different parts of the same quality problem. Automated checks can screen for patterns a team has decided to track. Bilingual revision compares the source and target propositions and judges whether the meaning carries across.
A tool that checks numbers, names, terminology or omissions can catch useful, specific issues. It can also help reviewers focus their attention by flagging a difference. A clean report, though, doesn’t prove that negation, modality and participant roles are correct unless the check has been designed and tested for those particular tasks.
Automatic translation metrics have a similar boundary. A metric compares translation output using a defined method; it doesn’t automatically provide a reliable, complete account of every critical meaning error. Benchmark performance on a test set can help teams understand a metric’s behavior, but it isn’t a production guarantee.
A WMT 2022 study evaluated 22 automatic MT metrics using a challenge set designed to test synonym handling and the detection of catastrophic word- and sentence-level errors (study). In that evaluation, embedding-based metrics performed relatively well at distinguishing sentence-level negation and affirmation errors, but poorly at relating synonyms. Some metrics were also sensitive to text style, which limited how broadly the results could be applied.
The study’s authors write: “The results show that although embedding-based metrics perform relatively well on discerning sentence-level negation/affirmation errors, their performances on relating synonyms are poor.” (WMT 2022 metric study)
That result isn’t a reason to trust every score as a semantic verdict. The study tested a particular metric set and challenge set. A reviewer should treat an automatic score as evidence about the tested behavior, not as proof that a high-scoring translation preserves every obligation or participant relationship.
Negation findings also depend on model, language pair and direction. A neural MT study published in 2021 reported manual negation-evaluation accuracies of 95.7% for EN→DE, 94.8% for DE→EN, 93.4% for EN→ZH and 91.7% for ZH→EN (study). Those figures describe the models and data evaluated in that study, not an assurance that production output is error-free.
The same paper identified under-translation as its most significant negation error type and reported variation across language pairs and directions. The authors put it this way: “In addition, we show that under-translation is the most significant error type in NMT” (study). For reviewers, the practical message is to check what the target leaves out as well as what it adds or reverses.
The WMT 2020 robustness task defined catastrophic errors as errors that critically change a segment’s meaning and can create misleading translations with religious, health, safety, legal or financial implications (task findings). Its categories included reverse negation, reverse sentiment or polarity, mistranslated named entities, and changes to units, time, dates or numbers.
Those categories give teams useful prompts for a semantic review, but no single review step replaces the others. A number check won’t identify a reversed obligation. A fluency review won’t confirm an actor’s role. A bilingual proposition check complements these tools by asking whether the source and target say the same thing.
For another way to think about prioritization, see post-editing accuracy vs fluency. A reviewer who has to choose what to fix first should distinguish a meaning change from a preference about style.
A practical bilingual review protocol¶
A semantic review works best when the reviewer checks the same elements in a consistent order. The protocol below synthesizes MQM Accuracy and the critical-error categories in the cited research; it is a practical editorial procedure, not a published claim about measured effectiveness.
Use it on individual clauses, but also compare the roles and defined terms across the document. A sentence can make sense in isolation and conflict with an earlier definition or another clause.
- Mark the proposition. Write a short, plain-language summary of what the source says. Keep the action and its participants visible: “the supplier must notify the buyer before shipment.”
- Check polarity. Find negative cues such as “not,” “never” and “without.” Confirm whether the target affirms or denies the same action, and check what the negative applies to.
- Check modality. Mark whether the source expresses obligation, permission, possibility, recommendation or ability. Compare the force of the target in context.
- Map participant roles. Identify the actor, recipient or affected party, action and object. Check which participant performs and receives the action in both languages.
- Compare conditions and exceptions. Match the “if,” “unless,” “only when” and similar elements. Confirm that each condition attaches to the right action and party.
- Read the target independently. Ask what a reader who sees only the translation would understand. A target that sounds natural can still communicate a different proposition.
- Check consistency across the document. Compare actors, defined terms and recurring relationships in related clauses. MQM treats terminology as a distinct dimension and identifies inconsistency as a possible terminology problem when consistent naming is required.
- Record the change precisely. Note whether the issue is a polarity reversal, a weakened or strengthened modal force, an omission, an addition, a role reversal or a different relationship. Give the reviser enough information to correct the meaning without guessing.
A compact review worksheet can make the process easier to apply:
| Element | Source check | Target check |
|---|---|---|
| Polarity | Is the claim affirmed, denied or restricted? | Does the target affirm, deny or restrict the same claim? |
| Modality | Is the action required, permitted, possible or recommended? | Does the target express the same force? |
| Actor | Who performs the action? | Is the same party acting? |
| Recipient or affected party | Who receives or is affected by the action? | Does the same party receive or experience it? |
| Action and object | What happens, and to what? | Does the target describe the same action and object? |
| Conditions | What must be true for the claim to apply? | Are the same conditions attached to the same claim? |
| Exceptions | What is excluded or exempted? | Does the target preserve the same exception? |
Take a hypothetical contract sentence: “The buyer must not release the deposit until the seller confirms receipt of the goods.” A reviewer can break it down into a prohibition, the buyer as actor, release as the action, the deposit as its object, and the seller’s confirmation as the condition that ends the prohibition.
A target that drops “not” reverses the instruction. A target that changes “must not” to “may not” could change the force depending on the target language and context. A target that assigns confirmation to the buyer instead of the seller changes the condition’s actor. Each problem needs a different correction, even if the sentence reads smoothly.
A source sentence can also contain nested conditions or exceptions. For example, a clause may prohibit an action except under a specific condition. Reviewers should preserve the exception’s scope: an exception applying to one action shouldn’t accidentally apply to a neighboring action. Split a long sentence into separate propositions when that makes the relationships easier to check.
Bilingual reviewers can coordinate the process by recording the source segment, target segment, error category and a short explanation of the changed proposition. A second reviewer can then verify the interpretation without needing to reconstruct the entire discussion. The goal isn’t to create extra paperwork; it’s to make a high-impact correction clear and traceable.
For documents with recurring technical or legal roles, a glossary alone won’t carry the full burden. A glossary can stabilize names and terms, while a relationship check verifies who owes what, who can act and who receives an action. Reviewers can use both: terminology controls for consistency and the proposition checklist for accuracy.
Teams also need a clear escalation rule. If reviewers disagree about whether a modal verb expresses obligation or permission, they should consult the surrounding clause and the document’s definitions rather than settle the question by choosing the more fluent phrase. If the source itself is ambiguous, a reviewer shouldn’t silently resolve that ambiguity by inventing a clearer obligation in the target.
When semantic review is needed - and when it isn’t¶
A focused semantic pass earns its time when a mistranslation could change a decision, responsibility, permission, warning or safety instruction. Contracts, medical instructions, policies, financial communications and operational guidance are obvious candidates because a small change in polarity or participant can affect how a reader acts.
A review is especially useful when:
- A sentence contains a prohibition, exception or condition.
- A modal verb determines whether a party must, may, should or can act.
- Two or more parties appear in a clause and their responsibilities differ.
- The text describes a technical relationship, sequence or transfer.
- A translation has gone through automated processing or post-editing and a fluent surface could hide a meaning change.
- A reviewer sees a number, date, unit, named entity or polarity change that could alter the instruction.
Not every sentence needs the same level of scrutiny. A low-impact marketing line may not call for a clause-by-clause role map if no decision or obligation depends on its exact force. Teams can set review depth according to the document’s purpose and risk, then reserve detailed proposition checks for the segments where a changed meaning matters most.
A semantic checklist also isn’t a replacement for proofreading, terminology review, formatting checks or expert review in a specialized domain. A correct proposition can still use the wrong required term, contain a misspelling, break a table or fail a domain-specific requirement. The best QA process assigns each check a clear job and doesn’t treat one clean result as evidence that every other job is done.
The same principle applies to automatic metrics. A metric can help compare outputs or screen for behavior it has been designed to measure. A metric shouldn’t be treated as the sole arbiter of whether a contract clause preserves the parties’ obligations unless the team has established that the metric is appropriate for that use and the actual text.
For related document-level risks, legal translation mistakes that cost your clients money covers why meaning errors matter beyond an individual segment. Teams handling errors in source files can also refer to what to do if there’s an error in a document.
Common semantic QA mistakes¶
Reviewers often catch obvious mistranslations but miss the narrow point where a proposition changes. A few habits can make that blind spot worse.
Searching for “not” instead of checking scope¶
A search for a negative word can reveal a missing cue, but it can’t confirm that the cue applies to the right action or condition. The 2015 Chinese-to-English analysis found scope reordering was the most frequent negation-error category in its specific task (study). Review the complete clause and identify the exact proposition under negation.
Treating grammatical fluency as evidence of accuracy¶
Fluent target text is easy to read, which can make a semantic change harder to spot. MQM’s distinction between Accuracy and Fluency helps reviewers avoid treating a polished sentence as proof that its content corresponds to the source (MQM typology).
Checking for familiar terms but not their relationships¶
A segment can contain every expected name and still reverse the relationship between the parties. Check who performs each action, who receives it and what the action affects. For defined terms, compare the clause with the rest of the document rather than reviewing it in isolation.
Treating an automatic score as a certificate¶
A benchmark result applies to the evaluation that produced it. The WMT 2022 metric study found different strengths and weaknesses across its tested metrics and challenge set; it doesn’t establish that all metrics, languages or document types behave the same way (study). Use a metric to inform review, not to declare the meaning verified.
Correcting an ambiguous source by guessing¶
A reviewer shouldn’t add an obligation, permission or exception just because one reading seems more likely. Flag unclear source wording for clarification, and preserve the ambiguity in the target where clarification isn’t available. MQM also treats ambiguity introduced in the target, when absent from the source, as an accuracy concern (MQM Full typology).
The MQM Full typology gives teams room to choose the granularity that suits their work. A small, consistent set of categories can be more useful than a long taxonomy reviewers apply inconsistently. The key is to record enough detail to show which part of the proposition changed.
Before the final handoff, ask one direct question about each high-risk segment: would a reader who saw only the target understand the same prohibition, permission, obligation and participant relationship as a reader of the source? A “yes” should come from comparison, not from fluency or a green status indicator.
FAQ¶
Why do translation QA tools miss negation and swapped parties?¶
Many automated checks focus on spelling, terminology, numbers or text similarity rather than who did what to whom. A fluent translation can preserve many words while reversing the proposition, so a clean check on one feature doesn’t establish semantic accuracy.
How do I check negation in a translation?¶
Compare the source and target for negative cues such as “not,” “never” and “without,” then check what each cue applies to. Read the full proposition in both languages rather than searching only for a matching negative word.
How do I check must, may and should in a translation?¶
Compare the force of each instruction or permission in context: obligation, possibility, recommendation and ability aren’t interchangeable. Record the source and target modality side by side, and check whether a condition or exception changes its scope.
How can I detect a subject-object reversal in a translated contract?¶
Map each clause to its actor, action, recipient or affected party, object, conditions and exceptions. Check defined parties across the document as well as within individual clauses, because a locally plausible sentence can still conflict with another clause’s roles.
Can automatic translation metrics identify critical semantic errors?¶
Some metrics can detect some errors on specific test sets, but benchmark results don’t establish that every current tool will catch a critical error in production. Use metrics as a screening signal and pair them with bilingual review for high-impact meaning.
Are negation errors the only critical semantic errors to check?¶
No. The WMT 2020 robustness task included reverse negation, reverse sentiment or polarity, mistranslated named entities, and changes to units, time, dates or numbers among its catastrophic-error categories (task findings). A semantic checklist should cover the error types that could mislead readers in the document at hand.
Does bilingual review replace automated QA?¶
No. Automated checks can flag specific patterns, while bilingual revision compares what the source and target communicate. A sound workflow uses each check for its own purpose and doesn’t treat one passing result as proof that every other risk has been resolved.
Should reviewers apply the full MQM typology to every project?¶
No. MQM Full says evaluation teams should choose error-type detail suited to their environment. A project can use a compact checklist for polarity, modality and participant roles without applying every subtype in the typology.