Ranking techniques for keyphrase extraction

This thesis focuses on the task of extracting keyphrases from research papers. Keyphrases are short phrases that summarize and characterize the contents of documents. They help users explore sets of documents and quickly understand the contents of individual documents. Most academic papers do not have keyphrases assigned to them, and manual keyphrase assignment is highly laborious. As such, there is a strong demand for automatic keyphrase extraction systems. The task of automatic keyphrase extraction presents a number of challenges. Human indexers are heavily informed by domain knowledge and comprehension of the contents of the papers. Keyphrase extraction is an intrinsically noisy and ambiguous task, as different human indexers select different keyphrases for the same paper. Training data is limited in both quality and quantity. In this thesis, we present a number of advancements to the ranking methods and features used to automatically extract keyphrases. We demonstrate that, through the reweighing of training examples, the quality of the learned bagged decision trees can be improved with negligible runtime cost. We use reranking to improve accuracy and explore several extensions thereof. We propose a number of new features, including augmented domain keyphraseness and average word length. Augmented domain keyphraseness incorporates information from a hierarchical document clustering to improve the handling of multi-domain corpora. We explore the technique of per-document feature scaling and discuss the impact of feature removal. Over three diverse corpora, these advancements substantially improve accuracy and runtime. Combined, they give keyphrase assignments that are competitive with those produced by human indexers.

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science.

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2014

Author dc:creator

• Field, David, M. Eng. (David M.). Massachusetts Institute of Technology

Advisor dc:contributor.advisor

• Regina Barzilay.

• Electrical Engineering and Computer Science.

Statement dc:rights

• M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission.

Licence dc:rights.uri

• http://dspace.mit.edu/handle/1721.1/7582

Language dc:language.iso

eng

Handle dc:identifier.uri

http://hdl.handle.net/1721.1/100660

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/100660