HYDERABAD: Artificial intelligence systems can struggle to understand spoken Telugu, with speech-recognition and translation errors sometimes changing the meaning of questions and leading to incorrect answers, a study by researchers from the International Institute of Information Technology-Hyderabad (IIIT-H) has found.
The researchers developed VakQA, which they describe as the first benchmark for evaluating spoken question-answering in Telugu. It contains 2,001 factoid question-answer pairs covering science, general knowledge, politics, history, culture and geography, along with 2.53 hours of speech recordings.
Tests found that translating Telugu questions into English reduced Gemini’s performance on 18.8% of questions, while switching from Telugu text to speech reduced performance on 21.2%.
Gemini’s average score fell from 3.63 for Telugu text to 3.52 after translation into English, and to 3.28 when questions were provided as speech. Performance declined on 18.8% of questions after translation, improved on 16.5% and remained unchanged on 64.7%. With spoken Telugu, performance declined on 21.2% and improved on 13.1%.
In one example, a Telugu question referring to “our country’s first satellite” prompted Gemini to answer Aryabhata. After translation, the reference became ambiguous and it answered Sputnik.
Another question correctly produced mango as Telangana’s state fruit in text form, but a speech-recognition error between the Telugu words for “fruit” and “festival” led Gemini to answer Bathukamma.
A similar error occurred with a question about ringworm. The correct answer was fungus when the question was provided as Telugu text, but a recognition error after spoken Telugu was converted into text changed the question, prompting the AI to answer heat.
Speech recognition produced substantial errors, with word error rates of 30.25% and 35.23% in systems tested. Translation compounded the problem, with Gemini’s score falling to 2.74 after Telugu text was translated into English and to 2.44 in the most error-prone speech-and-translation setup.
The study also found differences among AI models. On Telugu text, Gemini-2.5-Flash scored 3.63 out of five, while the best-performing open-weight model tested, Gemma-3 27B, scored 2.55.
The researchers said the findings show how errors can accumulate across a voice-AI pipeline and underline the need to evaluate AI directly in Indian languages rather than assume that performance in English will translate to Telugu.
VakQA was developed by IIIT-H researchers and Santosh Kesiraju of Brno University of Technology. The benchmark has been publicly released and was accepted for presentation at IEEE Spoken Language Technology 2026.
