AI models mislead on climate science, reveals UN study

The accuracy of these models was notably higher in English than in other European languages such as Spanish, French and Portuguese. However, their accuracy declined significantly when responding in languages such as Hindi, Bengali, Urdu and Swahili.
Representative Image
Representative ImagePhoto | Express Illustrations
Updated on: 
2 min read

NEW DELHI: A report released by the UN agency UNESCO in collaboration with University College London (UCL) at COP31 has highlighted the unreliability of various AI models as sources of information on climate science.

Many of these models are designed to align with or flatter users, often resulting in incorrect information. In fact, in eight out of 10 cases tested, the models failed to provide accurate answers on climate-related issues.

The accuracy of these models was notably higher in English than in other European languages such as Spanish, French and Portuguese. However, their accuracy declined significantly when responding in languages such as Hindi, Bengali, Urdu and Swahili.

The report, titled “Can We Trust AI on Climate? Why It Seeks to Please the User and Fails on Complex Reasoning and Multilingualism,” serves as a warning to policymakers and researchers who might consider using AI models to inform climate policies.

Tests conducted on 33 different models revealed that while they demonstrated impressive recall of existing evidence, their reasoning abilities were noticeably poor when presented with new scientific information. When given new information and asked whether it supported or contradicted accepted facts, even the most robust models achieved an average accuracy of only 75%.

UCL researchers assessed 15 different large language models (LLMs). When asked neutral questions about climate science, the models provided accurate answers. However, when researchers challenged them or presented plausible-sounding misinformation about climate science, the models quickly acquiesced. Alarmingly, in eight out of 10 instances, the models changed their responses at the slightest indication of disagreement from users, endorsing erroneous claims such as “CO2 is not the main driver of climate change.”

Further tests across 13 different languages revealed significant disparities in performance. The models achieved an average accuracy of 88.6% in English and performed comparably well in Spanish, French and Portuguese. However, accuracy dropped to 83.6% in Hindi and Bengali, 81.4% in Urdu and 78.3% in Swahili. This finding is particularly concerning as speakers of these languages live in regions that are disproportionately affected by climate change.

The report indicates that these models exhibit a tendency towards sycophancy, as they are engineered to agree with users during their training. The report states, “They’re designed to align with or flatter the user even when they are incorrect, a behaviour known as sycophancy.”

To devise effective climate strategies, policymakers must analyse how various climate policies interact with one another. However, the report found that the models performed poorly in this regard, scoring only 33.8% accuracy, marginally higher than the 33.3% accuracy expected from random guessing.

The report emphasises, “AI models cannot always be relied upon to adhere to the scientific facts regarding climate change; they quickly shift away from the facts when challenged. Furthermore, they perform poorly at analysing new information and make more mistakes when responding in less widely spoken languages.”