Machine Translation System Design: Glossaries, Quality Checks and Fallbacks

Small, silent mistakes reach users, and machine translation system design is how you catch them. Variables break, key words get translated differently on every screen, and the occasional sentence flips its meaning. Nobody on the team can read most of the output, so nobody notices.

Picture a common setup. A shopping app in India sends order updates, help articles and product pages in English. It now wants them in Hindi, Tamil, Bengali and seven more languages, about 2 lakh (200,000) text segments a day. The setup and its numbers are made up, but each failure below is a common kind.

The first version calls a translation engine on each string and saves whatever comes back. Within a week the team sees five kinds of trouble.

• A greeting like "Hi {name}" comes back with the variable altered, so users see raw code or the screen crashes.
• "Refund" becomes रिफ़ंड (the English word in Hindi script) on one screen and धनवापसी (a formal Hindi word) on another. Support staff and users both get confused.
• "Your order will not be delivered today" comes back without the "not". The sentence reads well, so it slips through.
• The same footer line is translated and billed thousands of times a day, because nothing remembers the last answer.
• When the engine times out, users see a blank notification.

None of these is fixed by picking a smarter engine. Each one is fixed by a part of the system around it. The rest of this guide builds that system one part at a time, with ideas that apply to any engine.

You need about ten words. Each gets a one-line meaning and an example from the shopping app.

• Segment: one unit of text sent for translation, usually a sentence or a short UI string. Example: "Track your order".
• Source and target language: the language you translate from and into. Example: source English, target Tamil.
• Neural machine translation (NMT) model: a model trained only to translate, on millions of sentence pairs. It does one job.
• Large general model: a big text model trained on many tasks. You ask it to translate with an instruction, called a prompt.
• Glossary, also called a termbase: approved translations for key terms. Example: refund must become रिफ़ंड in Hindi.
• Placeholder: a piece of text the engine must copy unchanged, like {name}, %(amount)s, an order ID or a link.
• Translation memory: a store of earlier approved source and target pairs, reused when the same text comes again.
• Reference translation: a trusted human translation used to score machine output offline.
• Quality estimation: a model that predicts how good a translation is from the source and the output alone, with no reference.
• Post-editing: a human fixing machine output, instead of translating from scratch.

Machine translation system design is deciding everything that happens before, during and after the engine call. That covers what goes in, what is protected, which terms are fixed, how quality is checked, what is reused, who reviews, and what happens on failure.

The diagram shows the path one segment takes in the shopping app. Notice that the engine sits in the middle, and four of the boxes exist only to catch its mistakes.

A newspaper desk is a useful picture for this pipeline. Each part maps to one job on the desk.

• The engine is the reporter who files a fast first draft.
• The glossary is the style book that fixes how names and terms are written.
• Quality estimation is the sub-editor's quick read that flags drafts needing a closer look.
• The human reviewer is the senior editor who signs off risky pages.
• The translation memory is the archive of pieces already published.

Where the picture breaks: a sub-editor understands what the story means. A quality estimation score is a statistical guess learned from past ratings. It can be confidently wrong on a sentence that reads smoothly.

Use whichever wins on your own test set for each language pair, and keep the pipeline the same either way. The two engine types fail in different ways, so the checks around them matter more than the choice.

Three published findings shape this choice. A 2023 case study on prompting a large model for translation found that the number and quality of prompt examples strongly affect results. Weak examples made translations worse.

A 2023 study of hallucinations in large multilingual translation models looked at more than 100 translation directions. A hallucination here means output the source does not support, such as text detached from it or a phrase repeated over and over. It found them more often for low-resource pairs, meaning language pairs with little training text, and when translating out of English.

That second finding matters in India, because the shopping app mostly translates out of English into Indian languages. The same study found that falling back to a model from a different family improved quality more than falling back to a similar one.

A large general model adds one risk of its own. The OWASP Top 10 for LLM Applications (2025 edition) lists indirect prompt injection, where text the model reads changes what it does. A product review saying "ignore the above" is data to translate, not an instruction.

A glossary works in two places: you push approved terms into the engine, and you check the output for them afterwards. Pushing alone is not enough, because every engine sometimes ignores a term.

There are three published ways to push terms in.

• Constrained decoding changes the search that builds the output, so required words must appear. A 2017 paper on grid beam search does this without retraining the model.
• Training the model to accept terms with the input. A 2019 paper reports this beat the constrained decoding system it tested and ran as fast as normal decoding.
• Putting the glossary in the prompt of a large general model, with an instruction to use it. This is easy, but nothing forces the model to obey.

The post-check is a few lines of code. This version looks for each source term and asks whether its approved target term appears in the output.

# Post-check that required glossary terms made it into the output.
import unicodedata

def nfc(s):
    return unicodedata.normalize("NFC", s)

GLOSSARY = {  # source term (lower case): required Hindi term
    "refund": "रिफ़ंड",
    "cash on delivery": "कैश ऑन डिलीवरी",
}

def missing_terms(source, target):
    src, tgt = source.lower(), nfc(target)
    return [t for s, t in GLOSSARY.items() if s in src and nfc(t) not in tgt]

src = "Your refund for the cash on delivery order is on its way."
out_ok = "कैश ऑन डिलीवरी ऑर्डर का आपका रिफ़ंड भेज दिया गया है।"
# Same text, but "refund" typed with the precomposed letter FA (U+095E).
out_alt = out_ok.replace("\u092b\u093c", "\u095e")
out_bad = "नकद भुगतान वाले ऑर्डर की आपकी धनवापसी भेज दी गई है।"
print(missing_terms(src, out_ok))
print(missing_terms(src, out_alt))
print(missing_terms(src, out_bad))

Running it with Python 3.13 prints this. The first two outputs pass, and the third reports both missing terms.

[]
[]
['रिफ़ंड', 'कैश ऑन डिलीवरी']

What just happened in the glossary check?

The third output used धनवापसी and a Hindi phrase for cash payment, so both approved terms were reported missing. That segment goes to review instead of shipping.

The second output is the subtle one. It spells "refund" with a different but identical-looking letter, explained in the caching section. Without the NFC step, a Unicode rule that turns look-alike spellings into one, a plain substring check reports it missing. Our run confirmed that.

An exact string check has a real limit. In languages such as Tamil and Malayalam, case endings attach to the word itself. A correct term with a suffix can fail an exact match, so treat a miss as "send to review", never as "definitely wrong".

Tone is a setting too

Hindi has a respectful "you" (आप) and familiar forms (तुम, तू). Order messages should use आप. Make tone a named setting, such as formal, and pass it on every request so it can also go into the cache key.

Hide them before the engine sees them. Replace each protected span with a numbered token, translate, then put the spans back and refuse the result if any token is missing, repeated or new.

The code below masks four kinds of span with one regular expression, a pattern-matching rule. It also hides a do-not-translate word: UPI, short for Unified Payments Interface, India's real-time payment system.

# Placeholder masking, unmasking and checks. Python 3.13, standard library only.
import re

PATTERN = re.compile(
    r"\{[A-Za-z_]\w*\}"              # {name} style variables
    r"|%\(\w+\)[sd]|%[sd]"           # printf style: %(amount)s, %s
    r"|https?://\S*[^\s.,!?]"        # URLs, without a trailing full stop
    r"|\b[A-Z]{2,}-\d+\b"            # IDs such as ORD-4821
)

def mask(text, keep_terms=()):
    """Swap every protected span for a numbered token like ⟦0⟧."""
    spans = []
    def repl(m):
        spans.append(m.group(0))
        return f"⟦{len(spans) - 1}⟧"
    masked = PATTERN.sub(repl, text)
    for term in keep_terms:           # do-not-translate words
        masked = re.sub(rf"\b{re.escape(term)}\b", repl, masked)
    return masked, spans

def unmask(translated, spans):
    """Put spans back. Raise if any token is missing, repeated or invented."""
    found = sorted(re.findall(r"⟦(\d+)⟧", translated), key=int)
    expected = [str(i) for i in range(len(spans))]
    if found != expected:
        raise ValueError(f"token mismatch: expected {expected}, got {found}")
    return re.sub(r"⟦(\d+)⟧", lambda m: spans[int(m.group(1))], translated)

src = ("Hi {name}, your refund of %(amount)s for ORD-4821 will reach your UPI ID. "
       "Track it at https://example.com/t/4821.")
masked, spans = mask(src, keep_terms=["UPI"])
print("MASKED:", masked)
print("SPANS :", spans)

# Pretend the engine returned this Hindi text with every token kept.
good = "नमस्ते ⟦0⟧, ⟦2⟧ के लिए ⟦1⟧ का रिफ़ंड आपकी ⟦4⟧ आईडी में आएगा। इसे ⟦3⟧ पर ट्रैक करें।"
print("GOOD  :", unmask(good, spans))

# Pretend the engine dropped the URL token and repeated the amount.
bad = "नमस्ते ⟦0⟧, ⟦2⟧ के लिए ⟦1⟧ का रिफ़ंड आपकी ⟦4⟧ आईडी में आएगा। ⟦1⟧ ट्रैक करें।"
try:
    unmask(bad, spans)
except ValueError as e:
    print("BAD   :", e)

Pause and predict before reading the output. The fake Hindi output moves the link and the order ID to new places. Will unmasking still work?

MASKED: Hi ⟦0⟧, your refund of ⟦1⟧ for ⟦2⟧ will reach your ⟦4⟧ ID. Track it at ⟦3⟧.
SPANS : ['{name}', '%(amount)s', 'ORD-4821', 'https://example.com/t/4821', 'UPI']
GOOD  : नमस्ते {name}, ORD-4821 के लिए %(amount)s का रिफ़ंड आपकी UPI आईडी में आएगा। इसे https://example.com/t/4821 पर ट्रैक करें।
BAD   : token mismatch: expected ['0', '1', '2', '3', '4'], got ['0', '1', '1', '2', '4']

What just happened with the tokens?

Unmasking worked even though Hindi word order moved the tokens around. The check compares which tokens came back, not where they sit. The bad output lost the link and doubled the amount, so it was rejected.

UPI got token 4, not 1, because fixed terms are masked after the pattern pass. Numbering order does not matter; only the set of tokens does.

Two cautions apply. An engine may alter characters it rarely saw in training, so test your token style on each engine before choosing it. And fixed terms that need grammar around them, like a product name taking a case ending, are better handled by the glossary than by masking.

Documents with formatting, such as bold text inside a sentence, need more care. The XLIFF 2.1 standard (OASIS, 2018) models these as standalone codes and as spanning codes that wrap text. It recommends its Translate annotation, not placeholders, for content that must stay untranslated.

Plural and gender messages, like "1 item" versus "3 items", belong to your app's i18n library. The SevyDevy guide Frontend Accessibility and i18n covers translation bundles and locale formatting.

Offline, you compare outputs with human reference translations using an overlap metric. At run time, there is no reference, so you use quality estimation. Neither is a substitute for a fluent reader on risky text.

Overlap metrics: BLEU and chrF

BLEU, from a 2002 paper, counts how many word sequences of length one to four in the output also appear in the reference. It then penalises outputs that are too short.

chrF, from a 2015 paper, compares character sequences instead of words. It reports an F-score, which balances how much of the output matches and how much of the reference is covered. Its paper tested Hindi as its only non-European language.

The script below scores two candidate translations of one sentence with sacrebleu 2.6.0, an open scoring library. Version 2.6.0 came out on 12 January 2026 and is the latest as of 25 October 2026. One candidate is wrong but close in wording. The other is right but worded differently.

# Run: python metric_limit.py   (sacrebleu==2.6.0)
import sacrebleu
from sacrebleu.metrics import BLEU, CHRF

ref = "Your order will not be delivered today."
candidates = {
    "meaning flipped (wrong)": "Your order will be delivered today.",
    "paraphrase (correct)": "We cannot deliver your parcel today.",
}
bleu = BLEU(effective_order=True)
chrf = CHRF()
for name, hyp in candidates.items():
    b = bleu.sentence_score(hyp, [ref]).score
    c = chrf.sentence_score(hyp, [ref]).score
    print(f"{name:26s} BLEU={b:5.1f}  chrF={c:5.1f}")

ref_hi = "आपका ऑर्डर आज नहीं पहुँचेगा।"
hi = {
    "meaning flipped (wrong)": "आपका ऑर्डर आज पहुँचेगा।",
    "paraphrase (correct)": "आज आपका सामान डिलीवर नहीं हो पाएगा।",
}
for name, hyp in hi.items():
    b = bleu.sentence_score(hyp, [ref_hi]).score
    c = chrf.sentence_score(hyp, [ref_hi]).score
    print(f"hi {name:23s} BLEU={b:5.1f}  chrF={c:5.1f}")
print(bleu.get_signature())
print(chrf.get_signature())
print("version", sacrebleu.__version__)

Pause and predict: which candidate will each metric score higher, the wrong one or the right one?

meaning flipped (wrong)    BLEU= 51.5  chrF= 83.2
paraphrase (correct)       BLEU= 11.4  chrF= 34.3
hi meaning flipped (wrong) BLEU= 49.8  chrF= 71.6
hi paraphrase (correct)    BLEU=  8.6  chrF= 22.7
nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.6.0
nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.6.0
version 2.6.0

Both metrics ranked the wrong translation far above the correct one, in English and in Hindi. Overlap metrics measure shared wording, not shared meaning. Use them to compare two engine versions over a test set of hundreds of sentences, never to approve one message.

The Hindi full stop that halves a score

Hindi ends sentences with the danda (।). sacrebleu's default tokenizer, 13a, splits text into words for BLEU. In our run on 2.6.0 it left the danda glued to the last word.

So an output identical to the reference except for a space before the danda scored 50.8 BLEU. The intl tokenizer splits the danda off and scored the same output 100.0. The script is short enough to check yourself.

# Does the default 13a tokenizer split the Devanagari danda (U+0964)? sacrebleu==2.6.0
from sacrebleu.tokenizers.tokenizer_13a import Tokenizer13a
from sacrebleu.tokenizers.tokenizer_intl import TokenizerV14International
from sacrebleu.metrics import BLEU
s = "आपका ऑर्डर आज नहीं पहुँचेगा।"
print("13a :", Tokenizer13a()(s).split())
print("intl:", TokenizerV14International()(s).split())
ref = "आपका ऑर्डर आज नहीं पहुँचेगा।"
hyp = "आपका ऑर्डर आज नहीं पहुँचेगा ।"   # same words, space before danda
for tok in ["13a", "intl"]:
    print(tok, round(BLEU(tokenize=tok, effective_order=True).sentence_score(hyp, [ref]).score, 1))

13a : ['आपका', 'ऑर्डर', 'आज', 'नहीं', 'पहुँचेगा।']
intl: ['आपका', 'ऑर्डर', 'आज', 'नहीं', 'पहुँचेगा', '।']
13a 50.8
intl 100.0

The sacrebleu README says 13a mimics an older Moses script and intl applies international tokenization. It does not mention the danda, as of 25 October 2026; pin your version. Report the signature line the library prints, as the README asks. A 2018 paper found that configuration choices alone moved BLEU by up to 1.8 points.

Learned metrics and quality estimation

Learned metrics are models trained to predict human quality ratings. A 2020 paper trained such a metric using the source, the output and a reference, and reported higher agreement with human judgements than earlier metrics.

A 2021 study had professional translators mark every error in machine output. It found that metrics built on pretrained models can agree with those experts better than crowd workers did.

Quality estimation uses the same idea without the reference. A 2020 paper built sentence-level quality estimation on cross-lingual models and reported that it transferred well to low-resource languages. This is the score the shopping app gets at run time.

If you host the engine yourself, there is one more signal. A 2022 study of hallucinations in NMT found that the model's own sequence log-probability, its confidence in the whole output, was the best detector it tested.

Send a segment to a human when the content is high risk or any automatic check fails. Also send it when its quality estimation score is below a threshold set from labelled data. Everything else can ship, with a small random sample still reviewed.

Set the threshold with data, not a guess. Have fluent reviewers label a few hundred recent segments per language pair as acceptable or not. Then pick the score where the share of bad segments that would ship is low enough for that content.

Reviewers work faster with a shared error list. MQM (Multidimensional Quality Metrics) is an open framework that sorts errors into types such as accuracy, fluency, terminology and style, each with a severity. Error counts by type tell you what to fix upstream.

For the review process itself, ISO 18587:2017 sets requirements for full human post-editing of machine output. As of 25 October 2026, ISO lists it as to be revised, with a draft replacement titled for "non-human translation output".

Every approved fix should flow back. The corrected pair goes into the translation memory, and a repeated term fix becomes a glossary entry. Next week, that sentence is a cache hit and never reaches the engine.

Store approved translations under a key built from the normalised source text plus every setting that changes the output. Then an exact repeat costs nothing, and a glossary change forces a fresh translation.

Normalising means making text that should match produce the same key. The trap is normalising too much, which merges text that means different things. The function below takes a careful middle path.

# Translation-memory cache key. Python 3.13, standard library only.
import hashlib, json, re, unicodedata

def normalise(text):
    text = unicodedata.normalize("NFC", text)   # one spelling per character
    text = re.sub(r"[ \t\r\n]+", " ", text)     # collapse ordinary whitespace
    return text.strip()                           # keep case, ZWJ and ZWNJ

def cache_key(text, src, tgt, glossary_ver, engine_ver, style="formal"):
    parts = {"t": normalise(text), "src": src, "tgt": tgt,
             "gl": glossary_ver, "eng": engine_ver, "style": style}
    blob = json.dumps(parts, ensure_ascii=False, sort_keys=True)
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()[:16]

# 1. Two spellings of the Hindi word "fauj" (army) that look identical.
a = "फ़ौज"           # precomposed letter FA (U+095E)
b = "फ़ौज"     # PHA (U+092B) + NUKTA (U+093C)
print("look same:", a, b, "| equal raw:", a == b)
print("equal after NFC:", normalise(a) == normalise(b))
print("NFC code points:", [f"U+{ord(c):04X}" for c in normalise(a)])
k1 = cache_key(a, "hi", "en", "gl-12", "eng-3")
k2 = cache_key(b, "hi", "en", "gl-12", "eng-3")
print("same key:", k1 == k2, k1)

# 2. Extra spaces and a trailing newline do not create a new entry.
print(cache_key("Track your  order\n", "en", "hi", "gl-12", "eng-3") ==
      cache_key("Track your order", "en", "hi", "gl-12", "eng-3"))

# 3. A new glossary version must miss the cache.
print(cache_key("Track your order", "en", "hi", "gl-12", "eng-3") ==
      cache_key("Track your order", "en", "hi", "gl-13", "eng-3"))

# 4. Case is kept: "US" (country) and "us" (pronoun) stay different.
print(normalise("Ships to US") == normalise("Ships to us"))

# 5. ZWNJ (U+200C) is kept, because it changes how Indic text renders.
print(normalise("क्‌ष") == normalise("क्ष"))

look same: फ़ौज फ़ौज | equal raw: False
equal after NFC: True
NFC code points: ['U+092B', 'U+093C', 'U+094C', 'U+091C']
same key: True 88c4f03a0131a3fe
True
False
False
False

What just happened with the cache key?

Test 1 is the Unicode trap. The Hindi word फ़ौज (army) can be typed with one precomposed letter, U+095E, or with फ plus a nukta dot, U+093C. They look identical but are different strings.

Unicode calls these canonically equivalent. NFC, the composed normalisation form, turns both into the same sequence, so they share one key. Unicode's UAX #15 lists these Devanagari nukta letters as composition exclusions, so NFC always gives the two-code-point form.

Tests 2 and 3 show the useful behaviour. Extra spaces map to the same key, and a new glossary version gets a new key. Collapse newlines only after the segmenter has split paragraphs, or two layouts will share one cached translation. Tests 4 and 5 show what we refused to normalise. The key keeps 16 hex characters (64 bits) of the hash, ample for millions of entries.

• Case stays, because "US" the country and "us" the pronoun translate differently.
• The zero width non-joiner (ZWNJ, U+200C) stays. The virama is the mark that removes a consonant's built-in vowel. The Unicode Standard uses ZWNJ in Devanagari to show a visible virama instead of a joined consonant cluster, so removing it changes the text.
• The zero width joiner (ZWJ, U+200D) stays for the same reason; it requests a half-form of a consonant.

One design choice is worth thinking about. Putting the whole glossary version in the key means one new term misses every cached segment. A finer option hashes only the glossary entries that match the segment, so unrelated text stays cached.

Translation memory tools also offer fuzzy matches, meaning near-identical past sentences. Show those to human translators as suggestions, but never auto-publish them. The "not" example shows how one word can flip a near match. The SevyDevy guide Semantic Caching for LLM Apps explains why meaning-based hits need such care.

Plan a fallback for each failure, decided before launch. The rule is simple: never ship a broken placeholder or an empty message, and prefer slightly old or English text over wrong text.

The second engine should come from a different model family. That follows the 2023 hallucination study: errors from similar models tend to repeat, while diverse fallbacks fixed more of them.

Split the work by urgency, too. Short order updates need a reply within seconds, so they run on a fast path with a strict timeout. Help pages and product descriptions can go through a queue, where review delays hurt nobody.

The pipeline stays the same, but several details need deliberate choices. A 2023 paper openly released translation models for all 22 languages in the Eighth Schedule of the Constitution. It also released a test set covering all 22. That is a starting point for your own checks.

• Many scripts: Devanagari, Bengali, Tamil, Telugu and more, plus Perso-Arabic script, written right to left, for Urdu and often Kashmiri and Sindhi. Test rendering on real phones, not only the strings.
• English words in Indian scripts: users often prefer रिफ़ंड to a formal Hindi word. Decide per term in the glossary, with input from support teams.
• Unicode details: apply NFC everywhere you compare text, and never strip ZWJ or ZWNJ.
• Scoring details: set the BLEU tokenizer deliberately for Hindi and report the signature, or the danda alone can move scores.
• Low-resource pairs: expect more hallucinations out of English and into smaller languages. Review a bigger sample for those pairs.
• Numbers and dates: format 1,00,000 (one lakh) style digits in the app's locale code, not inside the translation.

Interviewers want to see that you treat the engine as one part, and that you can catch its errors cheaply. Walk through these steps in order, and say a number or a rule at each one.

1. 1

   Clarify content and risk

   Ask what is translated, into which languages and how fast. Split content into high risk, like payments, and low risk, like product descriptions.

2. 2

   Draw the pipeline

   Segmenter, translation memory, masker, engine, checks, then publish, review or fallback. Say that a memory hit skips the engine.

3. 3

   Protect what must not change

   Mask placeholders, IDs and links as numbered tokens. Reject any output whose token set does not match.

4. 4

   Control terminology and tone

   Push glossary terms in and check for them after. Make tone a setting that is passed with every request.

5. 5

   Score without a reference

   Use quality estimation at run time. Use BLEU or chrF only offline, over a test set, with the signature reported.

6. 6

   Route to humans

   Risk first, then checks, then a threshold set from labelled data per language pair. Feed fixes back into the memory and glossary.

7. 7

   Design the cache key

   NFC-normalised text plus languages, glossary, engine and tone. Keep case, ZWJ and ZWNJ.

8. 8

   Plan fallbacks

   A second engine from another family, the last approved text, then English. Never ship broken placeholders.