Entity-based long document summarization using LLMs

Recent studies have found that the summaries generated by Large Language Models (LLMs) such as OpenAI's Generative Pre-trained Transformer (GPT) tend to be ranked as the most fluent abstractive summaries. Existing long document summarization research has focused on changing model architecture (such as different attention modules) but since LLMs (especially now that recent models have very large context windows recently) seem to be the best at outputting fluent summaries, we seek to understand if we can augment LLMs with information so that it produces the most accurate summary. Specifically, in this project, we aim to investigate if we can use a tandem approach of entity extraction and LLM prompting to generate the highest quality summary possible for scientific papers (long documents). We compare summarization using GPT only, using GPT and an entity extraction approach, and using a GPT Chain-of-Density based approach with the extracted entities and find that providing the entities improves the summary quality. Despite long documents containing over 6000 tokens on average, we find that we can generate an adequate to good summary in over half the cases using our chain-of-density method (nearly 80\% of inputs in two of the datasets). We also show how our entity extraction method is better in this setting than some contemporary approaches and experiment with some variations of early stopping and entity decay on the Chain-of-Density based prompting. While this still leaves significant room for improvement, our results are promising first steps towards a new methodology for long document summarization of scientific papers.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Potluri, Abhilash

Contributors dc:contributor

• Han, Jiawei

• Summarization
• Long Documents
• Nlp
• Llm
• Entity Extraction
• Chain-of-density

Statement dc:rights

• Copyright 2024 Abhilash Potluri

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/124661

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F124661