A Study on the Application of Natural Language Processing Methods to Scientific Text

The massive volume of scientific papers published in journals or deposited in preprint servers each day makes it difficult for scientists to stay on top of their respective areas of study, leading to a state of "information overload''. This trend is accelerating, with the total number of published papers climbing by ~9% year-over-year for the past several decades; in biomedicine alone, articles are deposited in PubMed at a rate greater than 2 per minute. Therefore, automated methods are necessary to manage the rapidly growing volume of literature and maximize the pace of scientific discovery. The explosive increase in the volume of scientific literature has coincided with a breakneck pace of progress in natural language processing (NLP). The application of NLP to scientific literature (sometimes called "scholarly document processing'' or SDP) has enabled literature-scale information extraction, precise search across 100s of millions of papers, and automatic summarization. In this dissertation, I study the methods behind these applications to (1) understand failure modes, (2) reduce the dependence on labelled training data, and (3) develop novel and performant solutions. I make several contributions to the burgeoning field of SDP. First, in the area of information extraction (IE), I quantify the poor generalization of an existing state-of-the-art approach and propose three complementary solutions to close the train-test performance gap. I then introduce a novel architecture that leverages pre-trained language models (PLMs) to improve performance while reducing training time and labelled data requirements. Additionally, I demonstrate that a generative approach to IE can match or exceed the performance of existing discriminative methods while being more flexible. Second, I make fundamental contributions to dense text representations and similarity by developing one of the first unsupervised pre-training strategies for learning high-quality sentence and paragraph embeddings. Finally, I name and study a new framework for query-based, multi-document summarization (MDS) and explore its potential for automatic literature review. I release all research artifacts, including trained models and code for reproducing our results to enable future work on these important tasks.

Department dc:contributor.department

Computer Science

Year dc:date.issued

2024

Author dc:creator

• Giorgi, John Michael

Advisors dc:contributor.advisor

• Bader, Gary D
• Wang, Bo

• Artificial intelligence
• information extraction
• language models
• Machine learning
• Natural language processing
• scholarly document processing

Statement dc:rights

• Attribution 4.0 International

Licence dc:rights.uri

• http://creativecommons.org/licenses/by/4.0/

Handle dc:identifier.uri

http://hdl.handle.net/1807/140924

OAI identifier oai:identifier

oai:utoronto.scholaris.ca:1807/140924