{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/46636"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/46636","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Guidelines for coverage-based comparisons of non-adequate test suites","abstract":"A fundamental question in software testing research is how to compare test suites, often as a means for comparing test-generation techniques that produce those test suites. Researchers frequently compare test suites by measuring their coverage. A coverage criterion C provides a set of test requirements and measures how many requirements a given suite satisfies. A suite that satisfies 100% of the (feasible) requirements is called C-adequate. Previous rigorous evaluations of coverage criteria mostly focused on such adequate test suites: given two criteria C and C′, are C-adequate suites (on average) more effective than C′-adequate suites? However, in many realistic cases, producing adequate suites is impractical or even impossible. This thesis presents the first extensive study that evaluates coverage criteria for the com- mon case of non-adequate test suites: given two criteria C and C′, which one is better to use to compare test suites? Namely, if suites T1, T2,...,Tn have coverage values c1,c2,...,cn for C and c′1,c2,...,c′n for C′, is it better to compare suites based on c1,c2,...,cn or based on c′1,c2,...,c′n? This thesis evaluates a large set of plausible criteria, including basic criteria such as statement and branch coverage, as well as stronger criteria used in recent studies, including criteria based on program paths, equivalence classes of covered statements, and predicate states. The criteria are evaluated on a set of Java and C programs with both manually written and automatically generated test suites. The evaluation uses three correlation measures. Based on these experiments, two criteria perform best: branch coverage and an intra-procedural acyclic path coverage. We provide guidelines for testing researchers aiming to evaluate test suites using coverage criteria as well as for other researchers evaluating coverage criteria for research use.","abstract_html":"A fundamental question in software testing research is how to compare test suites, often as a means for comparing test-generation techniques that produce those test suites. Researchers frequently compare test suites by measuring their coverage. A coverage criterion C provides a set of test requirements and measures how many requirements a given suite satisfies. A suite that satisfies 100% of the (feasible) requirements is called C-adequate. Previous rigorous evaluations of coverage criteria mostly focused on such adequate test suites: given two criteria C and C′, are C-adequate suites (on average) more effective than C′-adequate suites? However, in many realistic cases, producing adequate suites is impractical or even impossible. This thesis presents the first extensive study that evaluates coverage criteria for the com- mon case of non-adequate test suites: given two criteria C and C′, which one is better to use to compare test suites? Namely, if suites T1, T2,...,Tn have coverage values c1,c2,...,cn for C and c′1,c2,...,c′n for C′, is it better to compare suites based on c1,c2,...,cn or based on c′1,c2,...,c′n? This thesis evaluates a large set of plausible criteria, including basic criteria such as statement and branch coverage, as well as stronger criteria used in recent studies, including criteria based on program paths, equivalence classes of covered statements, and predicate states. The criteria are evaluated on a set of Java and C programs with both manually written and automatically generated test suites. The evaluation uses three correlation measures. Based on these experiments, two criteria perform best: branch coverage and an intra-procedural acyclic path coverage. We provide guidelines for testing researchers aiming to evaluate test suites using coverage criteria as well as for other researchers evaluating coverage criteria for research use.","abstract_has_math":false,"creators":["Sharma, Rohan"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Marinov, Darko","Caesar, Matthew C."],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2014,"date_issued":"2014-01-16T17:56:58Z","date_published":"2014-01-16T17:56:58Z","updated_at":"2026-07-22T22:25:36Z","subjects":["Coverage criteria","non-adequate test suites"],"languages":["en"],"rights":["Copyright 2013 Rohan Sharma"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/46636","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Marinov, Darko","Caesar, Matthew C."]},{"key":"dc:creator","label":"Author","values":["Sharma, Rohan"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2014-01-16T17:56:58Z","2013-12"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Coverage criteria","non-adequate test suites"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2013 Rohan Sharma"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/46636"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["A fundamental question in software testing research is how to compare test suites, often as a means for comparing test-generation techniques that produce those test suites. Researchers frequently compare test suites by measuring their coverage. A coverage criterion C provides a set of test requirements and measures how many requirements a given suite satisfies. A suite that satisfies 100% of the (feasible) requirements is called C-adequate. Previous rigorous evaluations of coverage criteria mostly focused on such adequate test suites: given two criteria C and C′, are C-adequate suites (on average) more effective than C′-adequate suites? However, in many realistic cases, producing adequate suites is impractical or even impossible. This thesis presents the first extensive study that evaluates coverage criteria for the com- mon case of non-adequate test suites: given two criteria C and C′, which one is better to use to compare test suites? Namely, if suites T1, T2,...,Tn have coverage values c1,c2,...,cn for C and c′1,c2,...,c′n for C′, is it better to compare suites based on c1,c2,...,cn or based on c′1,c2,...,c′n? This thesis evaluates a large set of plausible criteria, including basic criteria such as statement and branch coverage, as well as stronger criteria used in recent studies, including criteria based on program paths, equivalence classes of covered statements, and predicate states. The criteria are evaluated on a set of Java and C programs with both manually written and automatically generated test suites. The evaluation uses three correlation measures. Based on these experiments, two criteria perform best: branch coverage and an intra-procedural acyclic path coverage. We provide guidelines for testing researchers aiming to evaluate test suites using coverage criteria as well as for other researchers evaluating coverage criteria for research use.","Item withdrawn by Mark Zulauf (zulauf@illinois.edu) on 2013-12-09T14:31:21Z Item was in collections: University of Illinois Theses & Dissertations (ID: 1) No. of bitstreams: 1 Sharma_Rohan.pdf: 500722 bytes, checksum: e3dc6abdee9a3cbb6999f6a0aac73802 (MD5)","Made available in DSpace on 2014-01-16T17:56:58Z (GMT). No. of bitstreams: 2 Rohan_Sharma.pdf: 500722 bytes, checksum: e3dc6abdee9a3cbb6999f6a0aac73802 (MD5) license.txt: 4062 bytes, checksum: 7afaba17392b199a0e5963a5968e5675 (MD5)"]},{"key":"dc:title","label":"Title","values":["Guidelines for coverage-based comparisons of non-adequate test suites"]}]}],"canonical_facts":{"dc:contributor":["Marinov, Darko","Caesar, Matthew C."],"dc:creator":["Sharma, Rohan"],"dc:date":["2014-01-16T17:56:58Z","2013-12"],"dc:description":["A fundamental question in software testing research is how to compare test suites, often as a means for comparing test-generation techniques that produce those test suites. Researchers frequently compare test suites by measuring their coverage. A coverage criterion C provides a set of test requirements and measures how many requirements a given suite satisfies. A suite that satisfies 100% of the (feasible) requirements is called C-adequate. Previous rigorous evaluations of coverage criteria mostly focused on such adequate test suites: given two criteria C and C′, are C-adequate suites (on average) more effective than C′-adequate suites? However, in many realistic cases, producing adequate suites is impractical or even impossible. This thesis presents the first extensive study that evaluates coverage criteria for the com- mon case of non-adequate test suites: given two criteria C and C′, which one is better to use to compare test suites? Namely, if suites T1, T2,...,Tn have coverage values c1,c2,...,cn for C and c′1,c2,...,c′n for C′, is it better to compare suites based on c1,c2,...,cn or based on c′1,c2,...,c′n? This thesis evaluates a large set of plausible criteria, including basic criteria such as statement and branch coverage, as well as stronger criteria used in recent studies, including criteria based on program paths, equivalence classes of covered statements, and predicate states. The criteria are evaluated on a set of Java and C programs with both manually written and automatically generated test suites. The evaluation uses three correlation measures. Based on these experiments, two criteria perform best: branch coverage and an intra-procedural acyclic path coverage. We provide guidelines for testing researchers aiming to evaluate test suites using coverage criteria as well as for other researchers evaluating coverage criteria for research use.","Item withdrawn by Mark Zulauf (zulauf@illinois.edu) on 2013-12-09T14:31:21Z Item was in collections: University of Illinois Theses & Dissertations (ID: 1) No. of bitstreams: 1 Sharma_Rohan.pdf: 500722 bytes, checksum: e3dc6abdee9a3cbb6999f6a0aac73802 (MD5)","Made available in DSpace on 2014-01-16T17:56:58Z (GMT). No. of bitstreams: 2 Rohan_Sharma.pdf: 500722 bytes, checksum: e3dc6abdee9a3cbb6999f6a0aac73802 (MD5) license.txt: 4062 bytes, checksum: 7afaba17392b199a0e5963a5968e5675 (MD5)"],"dc:identifier":["http://hdl.handle.net/2142/46636"],"dc:language":["en"],"dc:rights":["Copyright 2013 Rohan Sharma"],"dc:subject":["Coverage criteria","non-adequate test suites"],"dc:title":["Guidelines for coverage-based comparisons of non-adequate test suites"],"dc:type":["text"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:36Z"}