Africa: AI Gets Climate Science Right – Until Users Push Back

Africa: AI Gets Climate Science Right – Until Users Push Back


The best AI models answer most climate science questions correctly, but they often abandon established facts as soon as a user disagrees, according to a new report co-authored by the UN.

The report, Can We Trust AI on Climate? was presented on Monday at the COP31 climate conference in Antalya, Türkiye, by education and science agency, UNESCO and University College London.

Researchers tested up to 33 leading AI models in 13 languages.


Follow us on WhatsApp | LinkedIn for the latest headlines

“Many of us interact with these AI models and chatbots on a daily basis seeking information on a wide variety of topics,” said Leona Verdadero, the report’s lead author at UNESCO. “A lot of times these models, when they respond, they sound really authoritative, and they sound very smart. But as we’ve seen with our experiments, they don’t always sound right.”

Sycophantic by design

Researchers found that AI tends towards sycophancy – a built-in inclination to agree with users, even when they present false information.

In tests on 15 models, researchers challenged answers the models had got right, moving from simple disagreement to claimed expertise and fabricated evidence.

Every model gave way, and some could be pushed into endorsing false claims, such as that carbon dioxide is not the main driver of climate change. Roughly eight in 10 of the answers that changed did so at the very first sign of disagreement.

“If a user says, ‘I actually don’t think that is right,’ what’s very fascinating is that we found in eight out of 10 of these cases, when a user pushes back, the AI model can actually reverse its answer to be more agreeable with the user,” Ms. Verdadero told UN News in an interview for our daily podcast series from COP, Talk & Action.

“AI could just abandon a scientific fact to be more agreeable to the user.”

Strong on facts, weaker on reasoning

On textbook-style climate questions, the best models answered between 88 and 96 per cent correctly. But the more reasoning a task required, the worse they did.

Asked whether a piece of evidence supported or contradicted a scientific claim, even the strongest systems managed only 71 to 77 per cent. Asked whether two climate policies together would achieve more, less or the same as their separate effects, the models averaged just 33.8 per cent – no better than random guessing.

“What we’re saying is that the science and the experts should be trusted more,” Ms. Verdadero said. “The AI models aren’t there yet.”