When conducting a statistical test the initial risk that must be considered is a Type I error, also known as a false positive. The Type I error rate is set by nominal alpha, assuming all underlying conditions of the statistic are met. Experiment-wise Type I error inflation occurs when multiple tests are conducted overall for a single experiment. There is a growing trend in the social and behavioral sciences utilizing nested designs. A Monte Carlo study was conducted using a two layer design. Five theoretical distributions and four real datasets taken from Micceri (1989) were used, each with five different samples sizes and conducted with nominal alpha set to 0.05 and 0.01. These were conducted both unconditionally and conditionally. All permutations were executed for 1,000,000 repetitions. It was found that when conducted unconditionally, the experiment-wise Type I error rate increases from alpha = 0.05 to 0.10 and 0.01 increases to 0.02. Conditionally, it is extremely unlikely to ever find results for the factor, as it requires a statistically significant nest as a precursor, which leads to extremely reduced power. Hence, caution should be used when interpreting nested designs.
","abstract_html":"<p>When conducting a statistical test the initial risk that must be considered is a Type I error, also known as a false positive. The Type I error rate is set by nominal alpha, assuming all underlying conditions of the statistic are met. Experiment-wise Type I error inflation occurs when multiple tests are conducted overall for a single experiment. There is a growing trend in the social and behavioral sciences utilizing nested designs. A Monte Carlo study was conducted using a two layer design. Five theoretical distributions and four real datasets taken from Micceri (1989) were used, each with five different samples sizes and conducted with nominal alpha set to 0.05 and 0.01. These were conducted both unconditionally and conditionally. All permutations were executed for 1,000,000 repetitions. It was found that when conducted unconditionally, the experiment-wise Type I error rate increases from alpha = 0.05 to 0.10 and 0.01 increases to 0.02. Conditionally, it is extremely unlikely to ever find results for the factor, as it requires a statistically significant nest as a precursor, which leads to extremely reduced power. Hence, caution should be used when interpreting nested designs.</p>","abstract_has_math":false,"creators":["Sawilowsky, Jack"],"institution":null,"degree_name":"Ph.D.","degree_level":"Open Access Dissertation","degree_discipline":"Education Evaluation and Research","degree_department":null,"school":null,"contributors":["Barry Markman"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2014,"date_issued":"2014-01-01T08:00:00Z","date_published":"2014-01-01T08:00:00Z","updated_at":"2026-07-24T05:59:39Z","subjects":["Bonferroni-Dunn","Experiment-wise Type I error inflation","Hierarchical linear modeling","Monte Carlo simulation","Nested testing","Educational Assessment, Evaluation, and Research","Statistics and Probability"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://digitalcommons.wayne.edu/oa_dissertations/914","outbound_label":"Repository record","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Barry Markman"]},{"key":"dc:creator","label":"Author","values":["Sawilowsky, Jack"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.available","label":"Dc Date Available","values":["2014-01-01T08:00:00Z"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Education Evaluation and Research"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Open Access Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Bonferroni-Dunn","Experiment-wise Type I error inflation","Hierarchical linear modeling","Monte Carlo simulation","Nested testing","Educational Assessment, Evaluation, and Research","Statistics and Probability"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://digitalcommons.wayne.edu/oa_dissertations/914"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["When conducting a statistical test the initial risk that must be considered is a Type I error, also known as a false positive. The Type I error rate is set by nominal alpha, assuming all underlying conditions of the statistic are met. Experiment-wise Type I error inflation occurs when multiple tests are conducted overall for a single experiment. There is a growing trend in the social and behavioral sciences utilizing nested designs. A Monte Carlo study was conducted using a two layer design. Five theoretical distributions and four real datasets taken from Micceri (1989) were used, each with five different samples sizes and conducted with nominal alpha set to 0.05 and 0.01. These were conducted both unconditionally and conditionally. All permutations were executed for 1,000,000 repetitions. It was found that when conducted unconditionally, the experiment-wise Type I error rate increases from alpha = 0.05 to 0.10 and 0.01 increases to 0.02. Conditionally, it is extremely unlikely to ever find results for the factor, as it requires a statistically significant nest as a precursor, which leads to extremely reduced power. Hence, caution should be used when interpreting nested designs.
"]},{"key":"dc:title","label":"Title","values":["The Impact Of Nested Testing On Experiment-Wise Type I Error Rate"]}]}],"canonical_facts":{"dc:contributor":["Barry Markman"],"dc:creator":["Sawilowsky, Jack"],"dc:date.available":["2014-01-01T08:00:00Z"],"dc:description.abstract":["When conducting a statistical test the initial risk that must be considered is a Type I error, also known as a false positive. The Type I error rate is set by nominal alpha, assuming all underlying conditions of the statistic are met. Experiment-wise Type I error inflation occurs when multiple tests are conducted overall for a single experiment. There is a growing trend in the social and behavioral sciences utilizing nested designs. A Monte Carlo study was conducted using a two layer design. Five theoretical distributions and four real datasets taken from Micceri (1989) were used, each with five different samples sizes and conducted with nominal alpha set to 0.05 and 0.01. These were conducted both unconditionally and conditionally. All permutations were executed for 1,000,000 repetitions. It was found that when conducted unconditionally, the experiment-wise Type I error rate increases from alpha = 0.05 to 0.10 and 0.01 increases to 0.02. Conditionally, it is extremely unlikely to ever find results for the factor, as it requires a statistically significant nest as a precursor, which leads to extremely reduced power. Hence, caution should be used when interpreting nested designs.
"],"dc:identifier":["https://digitalcommons.wayne.edu/oa_dissertations/914"],"dc:subject":["Bonferroni-Dunn","Experiment-wise Type I error inflation","Hierarchical linear modeling","Monte Carlo simulation","Nested testing","Educational Assessment, Evaluation, and Research","Statistics and Probability"],"dc:title":["The Impact Of Nested Testing On Experiment-Wise Type I Error Rate"],"thesis:degree_discipline":["Education Evaluation and Research"],"thesis:degree_level":["Open Access Dissertation"],"thesis:degree_name":["Ph.D."]},"updated_at":"2026-07-24T05:59:39Z"}