Microbial named entity recognition using BERT models

Bacteria are critical subjects of microbiological research that span many rapidly-growing fields of study. The development and widespread application of high-throughput sequencing has led to more microbial data being collected in recent years than ever before. This study investigates the capabilities of the popular Natural Language Processing (NLP) model Bidirectional Encoder Representations from Transformers (BERT) on the relatively understudied text mining domain of microbiology. This is done by fine-tuning a variety of BERT models (BERT, DistilBERT, SciBERT, BioBERT, PubMedBERT) on the Bacteria Biotope 2019 Open Shared Task (BB2019-OST) corpus of annotated microbial research text and evaluating the best performing models on the Named Entity Recognition (NER) task. Following this, an in-depth error analysis was conducted to gain insights into BERT’s entity recognition capabilities. Finally, to investigate performance capabilities further, learning rate and batch size hyperparameters were tuned to increase F1-score. The best BERT model in the comparison was BioBERT, earning an F1-score of 73.82 (±1.04) with default hyperparameters, and 75.35 (±0.62) with tuned hyperparameters. BioBERT had better F1-scores and entity-level statistics, despite PubMedBERT ranking high in biomedical NLP benchmarks. This suggests that the generality of the pretraining corpora of BERT models is particularly important for text mining in the microbial domain.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Bioinformatics

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2022

Author dc:creator

• Rao, Brian K

Contributors dc:contributor

• Kilicoglu, Halil

• Bioinformatics

Statement dc:rights

• Copyright 2022 Brian Rao

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/115955

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F115955