{"id":{"repo_id":"mit","oai_identifier":"oai:dspace.mit.edu:1721.1/87941"},"canonical_url":"https://search.dev.ndltd.org/etd/mit/oai:dspace.mit.edu:1721.1/87941","repository":{"repo_id":"mit","name":"MIT","base_url":"https://dspace.mit.edu/oai/request"},"display":{"title":"Lexical and Language Modeling of Diacritics and Morphemes in Arabic Automatic Speech Recognition","abstract":"Arabic is a morphologically rich language which rarely displays diacritics. These two features of the language pose challenges when building Automatic Speech Recognition (ASR) systems. Morphological complexity leads to many possible combinations of stems and affixes to form words, and produces texts with high Out Of Vocabulary (OOV) rates. In addition, texts rarely display diacritics which informs the reader about short vowels, geminates, and nunnations (word ending /n/). A lack of diacritics means that 30% of textual information is missing, causing ambiguities in lexical and language modeling when attempting to model pronunciations, and the context of a particular pronunciation. Intuitively, from an English centric view, the phrase th'wrtr wrt n thwrt with 'morphological decomposition' is realized as, th wrtr wrt n th wrt. Including 'diacritics' produces, the writer wrote in the writ. Thus our investigations in this thesis are twofold. Firstly, we show the benefits and interactions between modeling all classes of diacritics (short vowels, geminates, nunnations) in the lexicon. On a Modern Standard Arabic (MSA) corpus of broadcast news, this provides a 1.9% absolute improvement in Word Error Rate (WER) (p < 0.001). We also extend this graphemic lexicon with pronunciation rules, yielding a significant improvement over a lexicon that does not explicitly nodel diacritics. This results in a of 2.4% absolute improvement in WER (p < 0.001). Secondly, we show the benefits of language modeling at the morphemic level with diacritics, over the commonly available, word-based, nondiacratized text. This yields an absolute WER improvement of 1.0% (p < 0.001).","abstract_html":"Arabic is a morphologically rich language which rarely displays diacritics. These two features of the language pose challenges when building Automatic Speech Recognition (ASR) systems. Morphological complexity leads to many possible combinations of stems and affixes to form words, and produces texts with high Out Of Vocabulary (OOV) rates. In addition, texts rarely display diacritics which informs the reader about short vowels, geminates, and nunnations (word ending /n/). A lack of diacritics means that 30% of textual information is missing, causing ambiguities in lexical and language modeling when attempting to model pronunciations, and the context of a particular pronunciation. Intuitively, from an English centric view, the phrase th&#x27;wrtr wrt n thwrt with &#x27;morphological decomposition&#x27; is realized as, th wrtr wrt n th wrt. Including &#x27;diacritics&#x27; produces, the writer wrote in the writ. Thus our investigations in this thesis are twofold. Firstly, we show the benefits and interactions between modeling all classes of diacritics (short vowels, geminates, nunnations) in the lexicon. On a Modern Standard Arabic (MSA) corpus of broadcast news, this provides a 1.9% absolute improvement in Word Error Rate (WER) (p &lt; 0.001). We also extend this graphemic lexicon with pronunciation rules, yielding a significant improvement over a lexicon that does not explicitly nodel diacritics. This results in a of 2.4% absolute improvement in WER (p &lt; 0.001). Secondly, we show the benefits of language modeling at the morphemic level with diacritics, over the commonly available, word-based, nondiacratized text. This yields an absolute WER improvement of 1.0% (p &lt; 0.001).","abstract_has_math":false,"creators":["Alhanai, Tuka (Tuka Waddah Talib Ali Al Hanai)"],"institution":"Massachusetts Institute of Technology","degree_name":null,"degree_level":null,"degree_discipline":null,"degree_department":"Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science","school":null,"contributors":[],"advisors":["James R. Glass."],"committee_chairs":[],"committee_members":[],"year":2014,"date_issued":"2014","date_published":"2014","updated_at":"2026-07-22T22:20:51Z","subjects":["Electrical Engineering and Computer Science."],"languages":["eng"],"rights":["M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission."],"rights_urls":["http://dspace.mit.edu/handle/1721.1/7582"],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/1721.1/87941","outbound_label":"Handle","outbound_source":"dc:identifier.uri"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.advisor","label":"Advisor","values":["James R. Glass."]},{"key":"dc:contributor.department","label":"Department","values":["Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science"]},{"key":"dc:contributor.other","label":"Dc Contributor Other","values":["Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science."]},{"key":"dc:creator","label":"Author","values":["Alhanai, Tuka (Tuka Waddah Talib Ali Al Hanai)"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.accessioned","label":"Dc Date Accessioned","values":["2014-06-13T22:34:48Z"]},{"key":"dc:date.available","label":"Dc Date Available","values":["2014-06-13T22:34:48Z"]},{"key":"dc:date.issued","label":"Date","values":["2014"]},{"key":"dc:publisher","label":"Institution","values":["Massachusetts Institute of Technology"]},{"key":"dc:type","label":"Dc Type","values":["Thesis"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Electrical Engineering and Computer Science."]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language.iso","label":"Language (ISO)","values":["eng"]},{"key":"dc:rights","label":"Dc Rights","values":["M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission."]},{"key":"dc:rights.uri","label":"Rights URI","values":["http://dspace.mit.edu/handle/1721.1/7582"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.uri","label":"Identifier URI","values":["http://hdl.handle.net/1721.1/87941"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Thesis: S.M., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2014.","Cataloged from PDF version of thesis.","Includes bibliographical references (pages 69-72)."]},{"key":"dc:description.abstract","label":"Abstract","values":["Arabic is a morphologically rich language which rarely displays diacritics. These two features of the language pose challenges when building Automatic Speech Recognition (ASR) systems. Morphological complexity leads to many possible combinations of stems and affixes to form words, and produces texts with high Out Of Vocabulary (OOV) rates. In addition, texts rarely display diacritics which informs the reader about short vowels, geminates, and nunnations (word ending /n/). A lack of diacritics means that 30% of textual information is missing, causing ambiguities in lexical and language modeling when attempting to model pronunciations, and the context of a particular pronunciation. Intuitively, from an English centric view, the phrase th'wrtr wrt n thwrt with 'morphological decomposition' is realized as, th wrtr wrt n th wrt. Including 'diacritics' produces, the writer wrote in the writ. Thus our investigations in this thesis are twofold. Firstly, we show the benefits and interactions between modeling all classes of diacritics (short vowels, geminates, nunnations) in the lexicon. On a Modern Standard Arabic (MSA) corpus of broadcast news, this provides a 1.9% absolute improvement in Word Error Rate (WER) (p < 0.001). We also extend this graphemic lexicon with pronunciation rules, yielding a significant improvement over a lexicon that does not explicitly nodel diacritics. This results in a of 2.4% absolute improvement in WER (p < 0.001). Secondly, we show the benefits of language modeling at the morphemic level with diacritics, over the commonly available, word-based, nondiacratized text. This yields an absolute WER improvement of 1.0% (p < 0.001)."]},{"key":"dc:description.degree","label":"Dc Description Degree","values":["S.M."]},{"key":"dc:title","label":"Title","values":["Lexical and Language Modeling of Diacritics and Morphemes in Arabic Automatic Speech Recognition"]}]}],"canonical_facts":{"dc:contributor.advisor":["James R. Glass."],"dc:contributor.department":["Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science"],"dc:contributor.other":["Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science."],"dc:creator":["Alhanai, Tuka (Tuka Waddah Talib Ali Al Hanai)"],"dc:date.accessioned":["2014-06-13T22:34:48Z"],"dc:date.available":["2014-06-13T22:34:48Z"],"dc:date.issued":["2014"],"dc:description":["Thesis: S.M., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2014.","Cataloged from PDF version of thesis.","Includes bibliographical references (pages 69-72)."],"dc:description.abstract":["Arabic is a morphologically rich language which rarely displays diacritics. These two features of the language pose challenges when building Automatic Speech Recognition (ASR) systems. Morphological complexity leads to many possible combinations of stems and affixes to form words, and produces texts with high Out Of Vocabulary (OOV) rates. In addition, texts rarely display diacritics which informs the reader about short vowels, geminates, and nunnations (word ending /n/). A lack of diacritics means that 30% of textual information is missing, causing ambiguities in lexical and language modeling when attempting to model pronunciations, and the context of a particular pronunciation. Intuitively, from an English centric view, the phrase th'wrtr wrt n thwrt with 'morphological decomposition' is realized as, th wrtr wrt n th wrt. Including 'diacritics' produces, the writer wrote in the writ. Thus our investigations in this thesis are twofold. Firstly, we show the benefits and interactions between modeling all classes of diacritics (short vowels, geminates, nunnations) in the lexicon. On a Modern Standard Arabic (MSA) corpus of broadcast news, this provides a 1.9% absolute improvement in Word Error Rate (WER) (p < 0.001). We also extend this graphemic lexicon with pronunciation rules, yielding a significant improvement over a lexicon that does not explicitly nodel diacritics. This results in a of 2.4% absolute improvement in WER (p < 0.001). Secondly, we show the benefits of language modeling at the morphemic level with diacritics, over the commonly available, word-based, nondiacratized text. This yields an absolute WER improvement of 1.0% (p < 0.001)."],"dc:description.degree":["S.M."],"dc:identifier.uri":["http://hdl.handle.net/1721.1/87941"],"dc:language.iso":["eng"],"dc:publisher":["Massachusetts Institute of Technology"],"dc:rights":["M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission."],"dc:rights.uri":["http://dspace.mit.edu/handle/1721.1/7582"],"dc:subject":["Electrical Engineering and Computer Science."],"dc:title":["Lexical and Language Modeling of Diacritics and Morphemes in Arabic Automatic Speech Recognition"],"dc:type":["Thesis"]},"updated_at":"2026-07-22T22:20:51Z"}