Forecasting the success of environmental and sustainability activities in international development using language models
type: CreativeWork
description: International aid and cooperation improve the lives of the poorest in developing countries and help safeguard the environment and promote sustainability. However, international aid activities sometimes fail to achieve their objectives. Few attempts have been made in the literature to create models that forecast the success of international aid activities, and none focus solely on environmental outcomes. This thesis produces a forecasting system for various metrics measuring the success of international aid activities at the time of evaluation using data from the International Aid Transparency Initiative (IATI) database, combining classical statistical methods with modern language model techniques. Novel techniques were applied to improve forecasting skill, including using the reasoning and information-gathering abilities of large language models (LLMs) to improve forecasts, introducing LLM summaries of various dimensions of activity documents, and defining a novel narrative similarity grade benchmark. Forecasting success ratings on a scale from 1 to 6, narrative forecasts, cost-effectiveness forecasts, and finally true/false outcome tag forecasts were assessed. While no methods showed consistently high accuracy in forecasting overall success, statistical models could reliably rank above-chance which activities were more likely to succeed for activities with the same reporting organization and start year (“within-group pairwise ranking”). The full forecasting system outperformed what could be extrapolated from the stated risks in activity documents alone for overall evaluations in tests against a held-out test set of 200 latest-starting evaluated activities in a dataset of 1181 environmental and sustainability-improving activities restricted to 4 reporting organizations. Compared to the baseline of 50.7% pairwise ranking for the risks extrapolation, the chosen statistical model with all input features reached 59.7% [95% CI: 51 %, 66 %] on the test set. Across 14 bina…
abstract: International aid and cooperation improve the lives of the poorest in developing countries and help safeguard the environment and promote sustainability. However, international aid activities sometimes fail to achieve their objectives. Few attempts have been made in the literature to create models that forecast the success of international aid activities, and none focus solely on environmental outcomes. This thesis produces a forecasting system for various metrics measuring the success of international aid activities at the time of evaluation using data from the International Aid Transparency Initiative (IATI) database, combining classical statistical methods with modern language model techniques. Novel techniques were applied to improve forecasting skill, including using the reasoning and information-gathering abilities of large language models (LLMs) to improve forecasts, introducing LLM summaries of various dimensions of activity documents, and defining a novel narrative similarity grade benchmark. Forecasting success ratings on a scale from 1 to 6, narrative forecasts, cost-effectiveness forecasts, and finally true/false outcome tag forecasts were assessed. While no methods showed consistently high accuracy in forecasting overall success, statistical models could reliably rank above-chance which activities were more likely to succeed for activities with the same reporting organization and start year (“within-group pairwise ranking”). The full forecasting system outperformed what could be extrapolated from the stated risks in activity documents alone for overall evaluations in tests against a held-out test set of 200 latest-starting evaluated activities in a dataset of 1181 environmental and sustainability-improving activities restricted to 4 reporting organizations. Compared to the baseline of 50.7% pairwise ranking for the risks extrapolation, the chosen statistical model with all input features reached 59.7% [95% CI: 51 %, 66 %] on the test set. Across 14 bina…
author: Rivers, Morgan
keywords: sustainable development, international aid effectiveness, decision support systems, forecasting, large language models, International Aid Transparency Initiative (IATI), Entscheidungsunterstützungssysteme, Prognose, Wirksamkeit internationaler Entwicklungshilfe, große Sprachmodelle, nachhaltige Entwicklung
date published: 2026-07-02
date modified: 2026-07-24T03:52:15Z
provider: Universität Potsdam - Thes
publisher: Networked Digital Library of Theses and Dissertations
type: WebPage
source organization: Universität Potsdam