When the Transcriber Does Not Know Your Language

Anseya is our AI phone receptionist. It answers real calls for small businesses whose callers switch between English, Urdu and Punjabi, sometimes inside one sentence. This is the story of the day we noticed the Urdu in our call transcripts was stored in the wrong alphabet, why the obvious explanation and the obvious fix were both wrong, and the measured bake-off that ended the argument.

The symptom: right words, wrong alphabet

A caller said, in Urdu, that their boiler had broken. The transcript read:

मेरा boiler खराब हो गया है

Semantically perfect. Every word is what the caller said. But Urdu is written in the Arabic-derived Nastaliq script, and this is Devanagari, the script of Hindi. To a business owner reading the call summary, and to any downstream system matching against Urdu phrases, the words are unusable as written. A brief that comes back in the wrong script is a brief the business cannot use.

The transcriber at the time was Deepgram nova-3 with language: multi, chosen because it was the only configuration that handled callers who code-switch mid-sentence. The explanation turned out to be simple and is worth knowing before you sell a language: nova-3's multilingual set does not include Urdu or Punjabi. Spoken Urdu and Hindi are close enough that a multilingual model resolves what it hears to the largest neighbour it does know. So it heard correctly and wrote in the neighbour's alphabet. The words were right; the model never had our language in the first place.

The wrong fix, and what comparing failures taught us

The obvious fix was to switch to Whisper, which writes Urdu in Urdu script. We did. A live Urdu call then came back as, verbatim from the transcript log: The grass is green, Richcombe, Sure. Mistake. Auto language detection settles on the wrong language on short utterances and then transcribes phonetically into it.

We reverted, and the comparison became the most useful artifact of the whole incident. Deepgram produced the right meaning in the wrong alphabet. Whisper produced the wrong meaning in the right alphabet. Those are not equivalent defects: a business acts on what the caller meant, and the alphabet is, at worst, a display problem you can teach the downstream model about. We wrote the rule into the system prompt, that Devanagari from a caller means Urdu, and the assistant kept working. It even handled a real gas-leak emergency through a wrong-alphabet transcript: the model recognised the hazard in Devanagari, read the evacuation script verbatim, alerted the owner and ended the call.

But a workaround with a known wrong output is a standing debt, so we measured our way to a real answer.

The bake-off: five engines at telephone quality

The test rig synthesized reference clips (an English control, two Urdu, one Punjabi, one code-switched English-Urdu), then degraded them to what a phone line actually carries: 8 kHz mono through a 300 to 3400 Hz band. Word error rate was scored against the reference after punctuation stripping. One scoring property matters: output in the wrong script scores near 1.0 by construction, which is the point. The words may be semantically right, but as written they are unusable, and the metric should say so.

Selected rows, copied from the results table:

Two things in that table repay attention. The booking clip shows that on short Urdu utterances, every engine on auto-detection fell into Devanagari; only prompting recovered the script. And the Punjabi row shows the failure is not binary: one engine answered in Gurmukhi, the script of Indian Punjabi, a third alphabet entirely. Script selection under code-switching is genuinely unstable, and if your evaluation only checks word accuracy, you will never see it.

The prompt lesson: an example beats an instruction

gpt-4o-transcribe accepts a text prompt, and the first attempt was the obvious one, an instruction: callers speak English, Urdu or Punjabi; write Urdu and Punjabi in Urdu script, never Devanagari. It fixed every Urdu clip, and then it translated the English control clip into Urdu. The reference was Hi, I need a plumber to come out tomorrow morning; the output began میں چاہتا ہوں کہ ایک پلمبر کل صبح آئے. A transcriber that obeys instructions will also obey them off a cliff, and an English caller whose words reach the model translated into Urdu is a worse failure than the one being fixed.

What shipped instead was a prompt containing no instruction at all, just one example of what a transcript on this line looks like: a greeting in both languages, Urdu written in Urdu script. English stayed English at 0.0 WER, Urdu came back in Urdu script, Punjabi in Shahmukhi, and the code-switched clip kept both alphabets. Fifteen repeat runs: correct alphabet in all fifteen. An example shows the model the distribution it is in; an instruction gives it a rule to over-apply.

What shipped, and the compromises worth naming

The production transcriber is now gpt-4o-transcribe, with two deliberate absences in its configuration. There is no language pin, because pinning urwould transcribe English callers as Urdu on a line whose whole premise is that callers switch, and the platform's language enum has Urdu but not Punjabi anyway. And the winning example prompt is not in the config, because the telephony platform we run on exposes no prompt field for this transcriber. The residual script wobble on short utterances therefore still exists, and the system prompt's clause teaching the model that Devanagari means Urdu stays, no longer as a workaround but as a documented second layer.

On the same clips, the shipped engine got the words right on every run: English perfect, Urdu near-perfect, Punjabi usable, code-switched speech followed across the switch. Meaning first. This engine is the first one that always delivered it.

If you run speech recognition for a bilingual audience

• Read the language list before you sell the language. Multi does not mean all. Our live model never claimed Urdu; we assumed it.
• Score the script, not only the words. Cross-script output can be semantically perfect and operationally useless. Make your metric punish it, or your evaluation will call the broken engine fine.
• Degrade your test audio to the channel. Telephone audio is 8 kHz through a narrow band. An engine ranked on studio clips is ranked for a product you do not run.
• Distinguish wrong-meaning from wrong-rendering. They are different severities with different fixes, and conflating them is how we nearly shipped the worse engine because it looked better.
• Prefer examples to instructions in transcription prompts. The failure mode of an obeyed instruction is worse than the failure mode of an ignored one.