Web robots also known as crawlers or spiders are used by search engines, hackers and spammers to gather information about web pages. Timely detection and prevention of unwanted crawlers increases privacy and security of websites. In this research, a novel method to identify web crawlers is proposed to prevent unwanted crawler to access websites. The proposed method suggests a five-factor identification process to detect unwanted crawlers. This study provides the pretest and posttest results along with a systematic evaluation of web pages with the proposed identification technique versus web pages without the proposed identification process. An experiment was performed with repeated measures for two groups with each group containing ninety web pages. The outputs of the logistic regression analysis of treatment and control groups confirm the novel five-factor identification process as an effective mechanism to prevent unwanted web crawlers. This study concluded that the proposed five distinct identifier process is a very effective technique as demonstrated by a successful outcome.
","abstract_html":"<p>Web robots also known as crawlers or spiders are used by search engines, hackers and spammers to gather information about web pages. Timely detection and prevention of unwanted crawlers increases privacy and security of websites. In this research, a novel method to identify web crawlers is proposed to prevent unwanted crawler to access websites. The proposed method suggests a five-factor identification process to detect unwanted crawlers. This study provides the pretest and posttest results along with a systematic evaluation of web pages with the proposed identification technique versus web pages without the proposed identification process. An experiment was performed with repeated measures for two groups with each group containing ninety web pages. The outputs of the logistic regression analysis of treatment and control groups confirm the novel five-factor identification process as an effective mechanism to prevent unwanted web crawlers. This study concluded that the proposed five distinct identifier process is a very effective technique as demonstrated by a successful outcome.</p>","abstract_has_math":false,"creators":["Aghamohammadi, Alireza"],"institution":null,"degree_name":"Doctor of Philosophy (PhD)","degree_level":"Open Access Dissertation","degree_discipline":"College of Technology","degree_department":null,"school":null,"contributors":["Dr. Ali Eydgahi, Ph.D., (Chair)","Dr. Daniel Fields, Ph.D.","Dr. Huei Lee, Ph.D."],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2013,"date_issued":"2013-11-05T08:00:00Z","date_published":"2013-11-05T08:00:00Z","updated_at":"2026-07-24T02:16:57Z","subjects":["privacy","computer crimes","security systems","world wide web","hackers","Digital Communications and Networking","Information Security"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://commons.emich.edu/theses/544","outbound_label":"Repository record","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Dr. Ali Eydgahi, Ph.D., (Chair)","Dr. Daniel Fields, Ph.D.","Dr. Huei Lee, Ph.D."]},{"key":"dc:creator","label":"Author","values":["Aghamohammadi, Alireza"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.available","label":"Dc Date Available","values":["2014-04-04T07:00:00Z"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["College of Technology"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Open Access Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Doctor of Philosophy (PhD)"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["privacy","computer crimes","security systems","world wide web","hackers","Digital Communications and Networking","Information Security"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://commons.emich.edu/theses/544"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["Web robots also known as crawlers or spiders are used by search engines, hackers and spammers to gather information about web pages. Timely detection and prevention of unwanted crawlers increases privacy and security of websites. In this research, a novel method to identify web crawlers is proposed to prevent unwanted crawler to access websites. The proposed method suggests a five-factor identification process to detect unwanted crawlers. This study provides the pretest and posttest results along with a systematic evaluation of web pages with the proposed identification technique versus web pages without the proposed identification process. An experiment was performed with repeated measures for two groups with each group containing ninety web pages. The outputs of the logistic regression analysis of treatment and control groups confirm the novel five-factor identification process as an effective mechanism to prevent unwanted web crawlers. This study concluded that the proposed five distinct identifier process is a very effective technique as demonstrated by a successful outcome.
"]},{"key":"dc:title","label":"Title","values":["A novel defense mechanism against web crawler intrusion"]}]}],"canonical_facts":{"dc:contributor":["Dr. Ali Eydgahi, Ph.D., (Chair)","Dr. Daniel Fields, Ph.D.","Dr. Huei Lee, Ph.D."],"dc:creator":["Aghamohammadi, Alireza"],"dc:date.available":["2014-04-04T07:00:00Z"],"dc:description.abstract":["Web robots also known as crawlers or spiders are used by search engines, hackers and spammers to gather information about web pages. Timely detection and prevention of unwanted crawlers increases privacy and security of websites. In this research, a novel method to identify web crawlers is proposed to prevent unwanted crawler to access websites. The proposed method suggests a five-factor identification process to detect unwanted crawlers. This study provides the pretest and posttest results along with a systematic evaluation of web pages with the proposed identification technique versus web pages without the proposed identification process. An experiment was performed with repeated measures for two groups with each group containing ninety web pages. The outputs of the logistic regression analysis of treatment and control groups confirm the novel five-factor identification process as an effective mechanism to prevent unwanted web crawlers. This study concluded that the proposed five distinct identifier process is a very effective technique as demonstrated by a successful outcome.
"],"dc:identifier":["https://commons.emich.edu/theses/544"],"dc:subject":["privacy","computer crimes","security systems","world wide web","hackers","Digital Communications and Networking","Information Security"],"dc:title":["A novel defense mechanism against web crawler intrusion"],"thesis:degree_discipline":["College of Technology"],"thesis:degree_level":["Open Access Dissertation"],"thesis:degree_name":["Doctor of Philosophy (PhD)"]},"updated_at":"2026-07-24T02:16:57Z"}