{"id":{"repo_id":"potsdam-thes","oai_identifier":"oai:kobv.de-opus4-uni-potsdam:70771"},"canonical_url":"https://search.dev.ndltd.org/etd/potsdam-thes/oai:kobv.de-opus4-uni-potsdam:70771","repository":{"repo_id":"potsdam-thes","name":"Universität Potsdam - Thes","base_url":"https://publishup.uni-potsdam.de/opus4-ubp/oai"},"display":{"title":"Forecasting the success of environmental and sustainability activities in international development using language models","abstract":"International aid and cooperation improve the lives of the poorest in developing countries and help safeguard the environment and promote sustainability. However, international aid activities sometimes fail to achieve their objectives. Few attempts have been made in the literature to create models that forecast the success of international aid activities, and none focus solely on environmental outcomes. This thesis produces a forecasting system for various metrics measuring the success of international aid activities at the time of evaluation using data from the International Aid Transparency Initiative (IATI) database, combining classical statistical methods with modern language model techniques. Novel techniques were applied to improve forecasting skill, including using the reasoning and information-gathering abilities of large language models (LLMs) to improve forecasts, introducing LLM summaries of various dimensions of activity documents, and defining a novel narrative similarity grade benchmark. Forecasting success ratings on a scale from 1 to 6, narrative forecasts, cost-effectiveness forecasts, and finally true/false outcome tag forecasts were assessed. While no methods showed consistently high accuracy in forecasting overall success, statistical models could reliably rank above-chance which activities were more likely to succeed for activities with the same reporting organization and start year (“within-group pairwise ranking”). The full forecasting system outperformed what could be extrapolated from the stated risks in activity documents alone for overall evaluations in tests against a held-out test set of 200 latest-starting evaluated activities in a dataset of 1181 environmental and sustainability-improving activities restricted to 4 reporting organizations. Compared to the baseline of 50.7% pairwise ranking for the risks extrapolation, the chosen statistical model with all input features reached 59.7% [95% CI: 51 %, 66 %] on the test set. Across 14 binary outcome tags, statistical models achieved an average pairwise ranking of 60 % (range: 48 %–77 %) for the 45% of pairs with differing outcomes averaging two chosen methods on the test set. This work lays the foundation to improve decision making wherever there is a large collection of pre-intervention description documents and matched post-activity evaluation documents.","abstract_html":"International aid and cooperation improve the lives of the poorest in developing countries and help safeguard the environment and promote sustainability. However, international aid activities sometimes fail to achieve their objectives. Few attempts have been made in the literature to create models that forecast the success of international aid activities, and none focus solely on environmental outcomes. This thesis produces a forecasting system for various metrics measuring the success of international aid activities at the time of evaluation using data from the International Aid Transparency Initiative (IATI) database, combining classical statistical methods with modern language model techniques. Novel techniques were applied to improve forecasting skill, including using the reasoning and information-gathering abilities of large language models (LLMs) to improve forecasts, introducing LLM summaries of various dimensions of activity documents, and defining a novel narrative similarity grade benchmark. Forecasting success ratings on a scale from 1 to 6, narrative forecasts, cost-effectiveness forecasts, and finally true/false outcome tag forecasts were assessed. While no methods showed consistently high accuracy in forecasting overall success, statistical models could reliably rank above-chance which activities were more likely to succeed for activities with the same reporting organization and start year (“within-group pairwise ranking”). The full forecasting system outperformed what could be extrapolated from the stated risks in activity documents alone for overall evaluations in tests against a held-out test set of 200 latest-starting evaluated activities in a dataset of 1181 environmental and sustainability-improving activities restricted to 4 reporting organizations. Compared to the baseline of 50.7% pairwise ranking for the risks extrapolation, the chosen statistical model with all input features reached 59.7% [95% CI: 51 %, 66 %] on the test set. Across 14 binary outcome tags, statistical models achieved an average pairwise ranking of 60 % (range: 48 %–77 %) for the 45% of pairs with differing outcomes averaging two chosen methods on the test set. This work lays the foundation to improve decision making wherever there is a large collection of pre-intervention description documents and matched post-activity evaluation documents.","abstract_has_math":false,"creators":["Rivers, Morgan"],"institution":"Universität Potsdam","degree_name":null,"degree_level":"master","degree_discipline":null,"degree_department":null,"school":null,"contributors":["Kuhlicke, Christian","Kuznetsov, Ivan"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2026,"date_issued":"2026-07-02","date_published":"2026-07-02","updated_at":"2026-07-24T03:52:15Z","subjects":["sustainable development","international aid effectiveness","decision support systems","forecasting","large language models","International Aid Transparency Initiative (IATI)","Entscheidungsunterstützungssysteme","Prognose","Wirksamkeit internationaler Entwicklungshilfe","große Sprachmodelle","nachhaltige Entwicklung"],"languages":[],"rights":["CC-BY - Namensnennung 4.0 International"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://publishup.uni-potsdam.de/frontdoor/index/index/docId/70771","outbound_label":"Repository record","outbound_source":"source_url"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Kuhlicke, Christian","Kuznetsov, Ivan"]},{"key":"dc:creator","label":"Author","values":["Rivers, Morgan"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:publisher","label":"Institution","values":["Universität Potsdam"]},{"key":"dc:type","label":"Dc Type","values":["masterThesis"]},{"key":"thesis:degree_level","label":"Degree Level","values":["master"]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["Universität Potsdam"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["sustainable development","international aid effectiveness","decision support systems","forecasting","large language models","International Aid Transparency Initiative (IATI)","Entscheidungsunterstützungssysteme","Prognose","Wirksamkeit internationaler Entwicklungshilfe","große Sprachmodelle","nachhaltige Entwicklung"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:rights","label":"Dc Rights","values":["CC-BY - Namensnennung 4.0 International"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["International aid and cooperation improve the lives of the poorest in developing countries and help safeguard the environment and promote sustainability. However, international aid activities sometimes fail to achieve their objectives. Few attempts have been made in the literature to create models that forecast the success of international aid activities, and none focus solely on environmental outcomes. This thesis produces a forecasting system for various metrics measuring the success of international aid activities at the time of evaluation using data from the International Aid Transparency Initiative (IATI) database, combining classical statistical methods with modern language model techniques. Novel techniques were applied to improve forecasting skill, including using the reasoning and information-gathering abilities of large language models (LLMs) to improve forecasts, introducing LLM summaries of various dimensions of activity documents, and defining a novel narrative similarity grade benchmark. Forecasting success ratings on a scale from 1 to 6, narrative forecasts, cost-effectiveness forecasts, and finally true/false outcome tag forecasts were assessed. While no methods showed consistently high accuracy in forecasting overall success, statistical models could reliably rank above-chance which activities were more likely to succeed for activities with the same reporting organization and start year (“within-group pairwise ranking”). The full forecasting system outperformed what could be extrapolated from the stated risks in activity documents alone for overall evaluations in tests against a held-out test set of 200 latest-starting evaluated activities in a dataset of 1181 environmental and sustainability-improving activities restricted to 4 reporting organizations. Compared to the baseline of 50.7% pairwise ranking for the risks extrapolation, the chosen statistical model with all input features reached 59.7% [95% CI: 51 %, 66 %] on the test set. Across 14 binary outcome tags, statistical models achieved an average pairwise ranking of 60 % (range: 48 %–77 %) for the 45% of pairs with differing outcomes averaging two chosen methods on the test set. This work lays the foundation to improve decision making wherever there is a large collection of pre-intervention description documents and matched post-activity evaluation documents.","Internationale Entwicklungszusammenarbeit verbessert das Leben der Ärmsten in Entwicklungsländern und trägt zum Schutz der Umwelt sowie zur Förderung der Nachhaltigkeit bei. Allerdings verfehlen internationale Hilfsmaßnahmen mitunter ihre Ziele. In der Fachliteratur finden sich bisher nur wenige Ansätze zur Entwicklung von Modellen zur Vorhersage der Auswirkungen internationaler Hilfsmaßnahmen, und keiner davon konzentriert sich ausschließlich auf Umweltwirkungen. Diese Arbeit entwickelt auf Grundlage der Datenbank der International Aid Transparency Initiative (IATI) ein Prognosesystem für verschiedene Metriken zur Messung des Erfolgs internationaler Hilfsmaßnahmen zum Zeitpunkt der Evaluierung. Dabei werden klassische statistische Methoden mit modernen Techniken großer Sprachmodelle (Large Language Models, LLMs) kombiniert. Zur Verbesserung der Vorhersagefähigkeit kommen neuartige Techniken zum Einsatz, darunter die Nutzung der Schlussfolgerungs- und Informationsbeschaffungsfähigkeiten großer Sprachmodelle, die Erstellung von LLM-Zusammenfassungen verschiedener Dimensionen von Projektdokumenten sowie die Definition eines neuartigen Benchmarks zur Messung narrativer Ähnlichkeit. Untersucht werden Prognosen von Erfolgsevaluierungen auf einer Skala von 1 bis 6, narrative Prognosen, Prognosen zur Kosteneffizienz sowie Vorhersagen binärer Outcome-Tags (wahr/falsch). Während keine der Methoden eine durchgehend hohe Genauigkeit bei der Vorhersage des Gesamterfolgs erreicht, zeigen statistische Modelle zuverlässig eine über dem Zufall liegende Fähigkeit, innerhalb von Maßnahmen derselben berichtenden Organisation und desselben Startjahres (sogenanntes „Within-Group Pairwise Ranking“) erfolgreichere Maßnahmen zu identifizieren. Das vollständige Prognosesystem übertrifft bei der Evaluierung des Gesamterfolgs die Ergebnisse, die allein aus den in den Projektdokumenten angegebenen Risiken extrapoliert werden können. Dies zeigt sich in Tests mit einem zurückgehaltenen Testdatensatz von 200 der zuletzt gestarteten und evaluierten Maßnahmen aus einem Datensatz von 1181 umwelt- und nachhaltigkeitsbezogenen Maßnahmen, beschränkt auf vier Organisationen, die Berichte erstellen. Im Vergleich zur Baseline von 50.7% Pairwise Ranking für die Risikoextrapolation erreicht das ausgewählte statistische Modell unter Einbeziehung aller Eingangsmerkmale 59.7% [95%-Konfidenzintervall: 51 %, 66 %] auf dem Testdatensatz. Über 14 binäre Outcome-Tags hinweg erzielen statistische Modelle ein durchschnittliches Pairwise Ranking von 60 % (Spanne: 48 % bis 77 %) für die 45% der Paare mit unterschiedlichen Ergebnissen, gemittelt über zwei ausgewählte Methoden auf dem Testdatensatz. Diese Arbeit legt den Grundstein für eine Methode zur Verbesserung der Entscheidungsfindung, die in Fällen angewendet werden kann, in denen umfangreiche Sammlungen von Beschreibungsdokumenten vor einer Intervention sowie zugehörige Evaluierungsdokumente nach Abschluss vorliegen."]},{"key":"dc:format.medium","label":"Dc Format Medium","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Forecasting the success of environmental and sustainability activities in international development using language models","Vorhersage des Erfolgs von Umwelt- und Nachhaltigkeitsmaßnahmen in der internationalen Entwicklungszusammenarbeit mithilfe von Sprachmodellen"]}]}],"canonical_facts":{"dc:contributor":["Kuhlicke, Christian","Kuznetsov, Ivan"],"dc:creator":["Rivers, Morgan"],"dc:description.abstract":["International aid and cooperation improve the lives of the poorest in developing countries and help safeguard the environment and promote sustainability. However, international aid activities sometimes fail to achieve their objectives. Few attempts have been made in the literature to create models that forecast the success of international aid activities, and none focus solely on environmental outcomes. This thesis produces a forecasting system for various metrics measuring the success of international aid activities at the time of evaluation using data from the International Aid Transparency Initiative (IATI) database, combining classical statistical methods with modern language model techniques. Novel techniques were applied to improve forecasting skill, including using the reasoning and information-gathering abilities of large language models (LLMs) to improve forecasts, introducing LLM summaries of various dimensions of activity documents, and defining a novel narrative similarity grade benchmark. Forecasting success ratings on a scale from 1 to 6, narrative forecasts, cost-effectiveness forecasts, and finally true/false outcome tag forecasts were assessed. While no methods showed consistently high accuracy in forecasting overall success, statistical models could reliably rank above-chance which activities were more likely to succeed for activities with the same reporting organization and start year (“within-group pairwise ranking”). The full forecasting system outperformed what could be extrapolated from the stated risks in activity documents alone for overall evaluations in tests against a held-out test set of 200 latest-starting evaluated activities in a dataset of 1181 environmental and sustainability-improving activities restricted to 4 reporting organizations. Compared to the baseline of 50.7% pairwise ranking for the risks extrapolation, the chosen statistical model with all input features reached 59.7% [95% CI: 51 %, 66 %] on the test set. Across 14 binary outcome tags, statistical models achieved an average pairwise ranking of 60 % (range: 48 %–77 %) for the 45% of pairs with differing outcomes averaging two chosen methods on the test set. This work lays the foundation to improve decision making wherever there is a large collection of pre-intervention description documents and matched post-activity evaluation documents.","Internationale Entwicklungszusammenarbeit verbessert das Leben der Ärmsten in Entwicklungsländern und trägt zum Schutz der Umwelt sowie zur Förderung der Nachhaltigkeit bei. Allerdings verfehlen internationale Hilfsmaßnahmen mitunter ihre Ziele. In der Fachliteratur finden sich bisher nur wenige Ansätze zur Entwicklung von Modellen zur Vorhersage der Auswirkungen internationaler Hilfsmaßnahmen, und keiner davon konzentriert sich ausschließlich auf Umweltwirkungen. Diese Arbeit entwickelt auf Grundlage der Datenbank der International Aid Transparency Initiative (IATI) ein Prognosesystem für verschiedene Metriken zur Messung des Erfolgs internationaler Hilfsmaßnahmen zum Zeitpunkt der Evaluierung. Dabei werden klassische statistische Methoden mit modernen Techniken großer Sprachmodelle (Large Language Models, LLMs) kombiniert. Zur Verbesserung der Vorhersagefähigkeit kommen neuartige Techniken zum Einsatz, darunter die Nutzung der Schlussfolgerungs- und Informationsbeschaffungsfähigkeiten großer Sprachmodelle, die Erstellung von LLM-Zusammenfassungen verschiedener Dimensionen von Projektdokumenten sowie die Definition eines neuartigen Benchmarks zur Messung narrativer Ähnlichkeit. Untersucht werden Prognosen von Erfolgsevaluierungen auf einer Skala von 1 bis 6, narrative Prognosen, Prognosen zur Kosteneffizienz sowie Vorhersagen binärer Outcome-Tags (wahr/falsch). Während keine der Methoden eine durchgehend hohe Genauigkeit bei der Vorhersage des Gesamterfolgs erreicht, zeigen statistische Modelle zuverlässig eine über dem Zufall liegende Fähigkeit, innerhalb von Maßnahmen derselben berichtenden Organisation und desselben Startjahres (sogenanntes „Within-Group Pairwise Ranking“) erfolgreichere Maßnahmen zu identifizieren. Das vollständige Prognosesystem übertrifft bei der Evaluierung des Gesamterfolgs die Ergebnisse, die allein aus den in den Projektdokumenten angegebenen Risiken extrapoliert werden können. Dies zeigt sich in Tests mit einem zurückgehaltenen Testdatensatz von 200 der zuletzt gestarteten und evaluierten Maßnahmen aus einem Datensatz von 1181 umwelt- und nachhaltigkeitsbezogenen Maßnahmen, beschränkt auf vier Organisationen, die Berichte erstellen. Im Vergleich zur Baseline von 50.7% Pairwise Ranking für die Risikoextrapolation erreicht das ausgewählte statistische Modell unter Einbeziehung aller Eingangsmerkmale 59.7% [95%-Konfidenzintervall: 51 %, 66 %] auf dem Testdatensatz. Über 14 binäre Outcome-Tags hinweg erzielen statistische Modelle ein durchschnittliches Pairwise Ranking von 60 % (Spanne: 48 % bis 77 %) für die 45% der Paare mit unterschiedlichen Ergebnissen, gemittelt über zwei ausgewählte Methoden auf dem Testdatensatz. Diese Arbeit legt den Grundstein für eine Methode zur Verbesserung der Entscheidungsfindung, die in Fällen angewendet werden kann, in denen umfangreiche Sammlungen von Beschreibungsdokumenten vor einer Intervention sowie zugehörige Evaluierungsdokumente nach Abschluss vorliegen."],"dc:format.medium":["application/pdf"],"dc:publisher":["Universität Potsdam"],"dc:rights":["CC-BY - Namensnennung 4.0 International"],"dc:subject":["sustainable development","international aid effectiveness","decision support systems","forecasting","large language models","International Aid Transparency Initiative (IATI)","Entscheidungsunterstützungssysteme","Prognose","Wirksamkeit internationaler Entwicklungshilfe","große Sprachmodelle","nachhaltige Entwicklung"],"dc:title":["Forecasting the success of environmental and sustainability activities in international development using language models","Vorhersage des Erfolgs von Umwelt- und Nachhaltigkeitsmaßnahmen in der internationalen Entwicklungszusammenarbeit mithilfe von Sprachmodellen"],"dc:type":["masterThesis"],"thesis:degree_level":["master"],"thesis:institution_name":["Universität Potsdam"]},"updated_at":"2026-07-24T03:52:15Z"}