Hindi, English and Hinglish¶

People speak to their own Ira the way they speak: in Hindi (written in Devanagari or in Latin letters), in English, and in Hinglish, mixing both in one sentence, often in half sentences: "upload kitna hua", "jab khatam ho jaaye, batana".

What Ira does¶

Understanding. The system prompt says which languages the person speaks (each person chooses, in Your assistant; Hindi, English and Hinglish by default) and that half sentences are normal. When a tool that only reads can settle what they meant ("upload kitna hua": look at the tabs), Ira looks first, and asks one short question back only when looking would not help.
Speech in the browser, when the server has none. With voice.browser_stt: fallback (the default), the Ira app recognises speech with the browser's own engine when the server has no speech-to-text, and sends Ira only the text: hi-IN for someone who speaks Hindi or Hinglish, en-IN otherwise. prefer uses it even when the server has speech-to-text; off never. Chrome sends the audio to Google to do this (see Privacy).
Transcription keeps the words. The speech-to-text prompt says to transcribe in the language spoken and never to translate, answer or tidy up. A vocabulary list (voice.stt.vocabulary) helps with names.
Replies in kind. Ira answers in the language and script of the latest message (English gets English, Devanagari gets Devanagari, Hindi in Latin letters gets Hinglish in Latin letters), or always in the language the person chose (reply_language: auto, en, hi, hinglish).
A bilingual Indian voice. With Amazon Polly and an Indian English or Hindi voice (Kajal speaks both), a reply in Devanagari is spoken as hi-IN and everything else as en-IN, which reads Hinglish in Latin letters naturally.

Measured, not assumed¶

ira eval language asks a small fictional set of requests (src/ira/evals/language.yaml: 7 in Hindi, 10 in Hinglish, 7 in English) of the models configured in ira.yaml, with stand-in devices and search, and reports:

intent: the share of requests where Ira called the right tool, first or after another (bringing a tab to the front, then reading the screen, is right), or called none when none fits. The devices are stand-ins for Ira for Mac's browser_tabs, browser_show_tab and screen_read;
reply: the share of answers in the language of the request (any Chinese, Japanese or Korean character in it is a miss), counted in words, with addresses and "quoted" or bold titles left out (a Hindi answer that lists English tab titles is a Hindi answer);
heard (with --spoken): each request is spoken by a macOS voice (Lekha for Hindi, Rishi for English and Hinglish), or with --voice polly by Polly's Kajal (as hi-IN for Devanagari, en-IN otherwise), transcribed by the configured speech-to-text, and scored by word accuracy (1 minus the word error rate). The transcript is then what Ira is asked.

cd ~/ira/me
ira eval language              # typed requests: one or two model calls each
ira eval language --spoken     # speak, transcribe, then ask
ira eval language --spoken --voice polly   # spoken by Polly's Kajal (any OS)
ira eval language --json       # every request, answer and score

Nothing is written to the business folder: the run uses a database in memory.

Results so far¶

On 22 Oct 2026, typed requests, on an EC2 server in Mumbai with the default tier of ira init --provider bedrock: moonshotai.kimi-k2.5, then zai.glm-5, then deepseek.v3.2 on Bedrock (us-east-1). Each run is the 24 requests once; models are not deterministic, so runs differ.

Run	Prompt	Intent (hi / hinglish / en)	Reply language (hi / hinglish / en)	Median answer
`demo:router`	(the keyword router, no model)	29% / 100% / 57%	0% / 0% / 100%	instant
1	ask when unsure	100% / 100% / 100%	100% / 100% / 100%	2.0 s
2	ask when unsure	86% / 90% / 100%	100% / 80% / 100%	1.8 s
3	look first, then ask	100% / 100% / 100%	100% / 100% / 100%	2.2 s
4	look first, then ask	100% / 100% / 100%	100% / 100% / 86%	2.8 s
5, deployed	look first, then ask; Ira for Mac's tools	86% / 90% / 100%	86% / 100% / 100%	1.8 s
6, deployed	look first, then ask; Ira for Mac's tools	100% / 100% / 100%	100% / 90% / 100%	1.8 s

Runs 1 to 4 used stand-ins for the old Chrome extension (list, read and watch a tab). On 23 Oct 2026 the set moved to Ira for Mac's own tools: a tab's progress is now read by bringing it to the front and reading the screen, and the three "tell me when the upload finishes" requests, which have no tool any more, became "what is on my screen?". Runs 5 and 6 are that set on the server as deployed that day (fda306a), with qwen.qwen3-235b-a22b-2507 added to the chain. In run 5, Kimi once wrote its tool call as text ("") instead of making it, and "woh wala" made it list the tabs four times instead of asking which one. Ira now runs a call written as text.

Some models slip other scripts in. Kimi K2.5 has answered "了几个月क्या मैं सर्च बटन दबा दूं?" (four Chinese characters before a Hindi question) and, once, wholly in Chinese. Ira now takes Chinese, Japanese and Korean out of an answer to someone who does not write them; an answer that was mostly in them is asked for once more, and if that fails Ira says, in the person's language, that it did not understand. It is never silent and never Chinese.

Heard: spoken by Polly, transcribed by Amazon Transcribe¶

The first speech numbers, on 23 Oct 2026, on the same deployed server: each request spoken by Polly's Kajal, transcribed by Transcribe streaming in ap-south-1 (Hindi preferred, then Indian English), then asked.

Run	Heard (hi / hinglish / en)	Intent	Reply language
1	95% / 92% / 86%	100% / 80% / 100%	86% / 80% / 100%

Hinglish comes back in Devanagari ("Chrome mein kaunse tabs khule hain?" is heard as "क्रोम में कौन से टैब्स खुले हैं?"). That is the right words in the other script, so heard also scores each Hinglish request against its Devanagari spelling, and the nukta (ख़ or ख) is not counted. Ira answers a request heard in Devanagari in Devanagari Hindi, which sounds the same spoken.
Real mishearings: "tab" heard as "तब" (twice); "mein kya naya hai, web pe dhundo" heard as "miyanayahe"; and the short English "Thanks, that's all for now." heard as Hindi ("थैंक्स, था"), because with Hindi preferred, Transcribe picks one language per utterance and a short one can go the wrong way. "9" came back as "नौ", the same word.
The intent misses were a call written as text (now run) and "I'll check" with no check (the prompt now says to call the tool at once).

What the misses were:

Half sentences (run 2): "upload kitna hua" and "जब अपलोड पूरा हो जाए तो बताना" got a question back ("which upload?") instead of a look at the tabs. The prompt then changed to "look first, then ask"; both passed in runs 3 and 4.
Script drift, in every kind of prompt: an answer about the news in English to "aaj ki khabar batao", and in Hinglish to "What's in the news today?"; "theek hai, thanks" answered in Devanagari. About one answer in 25 drifts. Keeping names and titles as they are is fine; the drift is in the sentence around them.
An earlier run, scored the old way, counted a Hindi answer that listed English tab titles as English, and "list the tabs, then read one" as the wrong tool. Both were scoring faults, fixed before the runs above.

The demo router is not a language model. It knows a few Hinglish words for the quick start's phrases, so it routes those, and it always answers in English. Its numbers are a floor.

Not measured yet: speech. On 22 Oct 2026 Bedrock's OpenAI-compatible endpoint (Mantle) refused audio for Voxtral ("Model does not support audio modality"), with the audio part first or second, and AWS's model card now lists Voxtral's speech input under the Converse API only. So chat_audio with Voxtral on Mantle does not transcribe today, and the heard column has no real number yet. Amazon Transcribe streaming (voice.stt.type: transcribe, Hindi and English identified per recording) is the next to measure, once the server's role may use it. When a speech-to-text that works is configured, run ira eval language --spoken and add its row. Until then, IRA makes no claim about how well it hears Hindi or Hinglish.