{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/105272"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/105272","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Empirically revisiting and enhancing IR-based test-case prioritization","abstract":"Regression testing is widely used to check code changes during software evolution. How- ever, regression testing can also incur huge cost since real-world software systems can accumulate huge and time-consuming test suites. Therefore, a large number of approaches have been proposed to speed up regression testing, including test-case prioritization, regression test selection, and test-suite minimization/reduction. Test-case prioritization (TCP) aims to detect regression bugs faster via reordering regression tests. To date, various TCP techniques have been proposed in the literature, including both change-aware ones that consider program changes between the old and new versions for better prioritization and change-unaware ones that simply use the dynamic or static in- formation from the old version. Although various studies have investigated the effectiveness of the existing TCP techniques, they suffer from the following threats: (1) they are usually evaluated on a small dataset; (2) they are usually performed on seeded artificial or real bugs, and not on real evolution with real bugs; (3) they are usually evaluated using cost-unaware metrics. In this work, we study the recent program changes based information retrieval (IR) approach for TCP, which has been claimed to perform better than state-of-the-art coverage based techniques but only evaluated on a small dataset by cost-unaware metrics. We reduce the threats of and further enhance the prior work by conducting a better evaluation and proposing potential improvements. To do so, we evaluate the original technique on a large-scale, real-world software-evolution dataset containing 123 projects and 2,980 program builds using both cost-aware and cost-unaware metrics under various configurations, and we design and evaluate several hybrid techniques combining the IR features and historical test execution times and failure frequencies. As a result, we confirm the effectiveness of program changes based IR approach for TCP and find an ideal configuration to maximize the effectiveness. Also, we successfully improve the performance of the original technique with our hybrid techniques. Moreover, we show that flaky tests have a substantial impact on the program changes based TCP techniques.","abstract_html":"Regression testing is widely used to check code changes during software evolution. How- ever, regression testing can also incur huge cost since real-world software systems can accumulate huge and time-consuming test suites. Therefore, a large number of approaches have been proposed to speed up regression testing, including test-case prioritization, regression test selection, and test-suite minimization/reduction. Test-case prioritization (TCP) aims to detect regression bugs faster via reordering regression tests. To date, various TCP techniques have been proposed in the literature, including both change-aware ones that consider program changes between the old and new versions for better prioritization and change-unaware ones that simply use the dynamic or static in- formation from the old version. Although various studies have investigated the effectiveness of the existing TCP techniques, they suffer from the following threats: (1) they are usually evaluated on a small dataset; (2) they are usually performed on seeded artificial or real bugs, and not on real evolution with real bugs; (3) they are usually evaluated using cost-unaware metrics. In this work, we study the recent program changes based information retrieval (IR) approach for TCP, which has been claimed to perform better than state-of-the-art coverage based techniques but only evaluated on a small dataset by cost-unaware metrics. We reduce the threats of and further enhance the prior work by conducting a better evaluation and proposing potential improvements. To do so, we evaluate the original technique on a large-scale, real-world software-evolution dataset containing 123 projects and 2,980 program builds using both cost-aware and cost-unaware metrics under various configurations, and we design and evaluate several hybrid techniques combining the IR features and historical test execution times and failure frequencies. As a result, we confirm the effectiveness of program changes based IR approach for TCP and find an ideal configuration to maximize the effectiveness. Also, we successfully improve the performance of the original technique with our hybrid techniques. Moreover, we show that flaky tests have a substantial impact on the program changes based TCP techniques.","abstract_has_math":false,"creators":["Peng, Qianyang"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Marinov, Darko","Zhang, Lingming"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2019,"date_issued":"2019-08-23T20:48:27Z","date_published":"2019-08-23T20:48:27Z","updated_at":"2026-07-22T22:24:44Z","subjects":["Software Testing","Test Case Prioritization","Flaky Test","Information Retrieval"],"languages":["en"],"rights":["Copyright 2019 Qianyang Peng"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/105272","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Marinov, Darko","Zhang, Lingming"]},{"key":"dc:creator","label":"Author","values":["Peng, Qianyang"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2019-08-23T20:48:27Z","2021-08-24T09:15:20Z","2019-04-26","2019-05"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Software Testing","Test Case Prioritization","Flaky Test","Information Retrieval"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2019 Qianyang Peng"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/105272"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Regression testing is widely used to check code changes during software evolution. How- ever, regression testing can also incur huge cost since real-world software systems can accumulate huge and time-consuming test suites. Therefore, a large number of approaches have been proposed to speed up regression testing, including test-case prioritization, regression test selection, and test-suite minimization/reduction. Test-case prioritization (TCP) aims to detect regression bugs faster via reordering regression tests. To date, various TCP techniques have been proposed in the literature, including both change-aware ones that consider program changes between the old and new versions for better prioritization and change-unaware ones that simply use the dynamic or static in- formation from the old version. Although various studies have investigated the effectiveness of the existing TCP techniques, they suffer from the following threats: (1) they are usually evaluated on a small dataset; (2) they are usually performed on seeded artificial or real bugs, and not on real evolution with real bugs; (3) they are usually evaluated using cost-unaware metrics. In this work, we study the recent program changes based information retrieval (IR) approach for TCP, which has been claimed to perform better than state-of-the-art coverage based techniques but only evaluated on a small dataset by cost-unaware metrics. We reduce the threats of and further enhance the prior work by conducting a better evaluation and proposing potential improvements. To do so, we evaluate the original technique on a large-scale, real-world software-evolution dataset containing 123 projects and 2,980 program builds using both cost-aware and cost-unaware metrics under various configurations, and we design and evaluate several hybrid techniques combining the IR features and historical test execution times and failure frequencies. As a result, we confirm the effectiveness of program changes based IR approach for TCP and find an ideal configuration to maximize the effectiveness. Also, we successfully improve the performance of the original technique with our hybrid techniques. Moreover, we show that flaky tests have a substantial impact on the program changes based TCP techniques.","Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2021-05-01","The student, Qianyang Peng, accepted the attached license on 2019-04-25 at 20:31.","The student, Qianyang Peng, submitted this Thesis for approval on 2019-04-25 at 20:42.","This Thesis was approved for publication on 2019-04-26 at 11:00.","DSpace SAF Submission Ingestion Package generated from Vireo submission #13937 on 2019-08-22 at 16:24:01","Made available in DSpace on 2019-08-23T20:48:27Z (GMT). No. of bitstreams: 2 PENG-THESIS-2019.pdf: 713997 bytes, checksum: a26746e823ce876dc4b4a8f1f6d7b4f1 (MD5) LICENSE.txt: 4210 bytes, checksum: 414ea6767b803244823afdc361815c30 (MD5) Previous issue date: 2019-04-26","Embargo set by: Seth Robbins for item 112394 Lift date: 2021-08-23T20:48:32Z Reason: Author requested closed access (OA after 2yrs) in Vireo ETD system","Limited Restriction Lifted for Item 112394 on 2021-08-24T09:15:20Z."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Empirically revisiting and enhancing IR-based test-case prioritization"]}]}],"canonical_facts":{"dc:contributor":["Marinov, Darko","Zhang, Lingming"],"dc:creator":["Peng, Qianyang"],"dc:date":["2019-08-23T20:48:27Z","2021-08-24T09:15:20Z","2019-04-26","2019-05"],"dc:description":["Regression testing is widely used to check code changes during software evolution. How- ever, regression testing can also incur huge cost since real-world software systems can accumulate huge and time-consuming test suites. Therefore, a large number of approaches have been proposed to speed up regression testing, including test-case prioritization, regression test selection, and test-suite minimization/reduction. Test-case prioritization (TCP) aims to detect regression bugs faster via reordering regression tests. To date, various TCP techniques have been proposed in the literature, including both change-aware ones that consider program changes between the old and new versions for better prioritization and change-unaware ones that simply use the dynamic or static in- formation from the old version. Although various studies have investigated the effectiveness of the existing TCP techniques, they suffer from the following threats: (1) they are usually evaluated on a small dataset; (2) they are usually performed on seeded artificial or real bugs, and not on real evolution with real bugs; (3) they are usually evaluated using cost-unaware metrics. In this work, we study the recent program changes based information retrieval (IR) approach for TCP, which has been claimed to perform better than state-of-the-art coverage based techniques but only evaluated on a small dataset by cost-unaware metrics. We reduce the threats of and further enhance the prior work by conducting a better evaluation and proposing potential improvements. To do so, we evaluate the original technique on a large-scale, real-world software-evolution dataset containing 123 projects and 2,980 program builds using both cost-aware and cost-unaware metrics under various configurations, and we design and evaluate several hybrid techniques combining the IR features and historical test execution times and failure frequencies. As a result, we confirm the effectiveness of program changes based IR approach for TCP and find an ideal configuration to maximize the effectiveness. Also, we successfully improve the performance of the original technique with our hybrid techniques. Moreover, we show that flaky tests have a substantial impact on the program changes based TCP techniques.","Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2021-05-01","The student, Qianyang Peng, accepted the attached license on 2019-04-25 at 20:31.","The student, Qianyang Peng, submitted this Thesis for approval on 2019-04-25 at 20:42.","This Thesis was approved for publication on 2019-04-26 at 11:00.","DSpace SAF Submission Ingestion Package generated from Vireo submission #13937 on 2019-08-22 at 16:24:01","Made available in DSpace on 2019-08-23T20:48:27Z (GMT). No. of bitstreams: 2 PENG-THESIS-2019.pdf: 713997 bytes, checksum: a26746e823ce876dc4b4a8f1f6d7b4f1 (MD5) LICENSE.txt: 4210 bytes, checksum: 414ea6767b803244823afdc361815c30 (MD5) Previous issue date: 2019-04-26","Embargo set by: Seth Robbins for item 112394 Lift date: 2021-08-23T20:48:32Z Reason: Author requested closed access (OA after 2yrs) in Vireo ETD system","Limited Restriction Lifted for Item 112394 on 2021-08-24T09:15:20Z."],"dc:format":["application/pdf"],"dc:identifier":["http://hdl.handle.net/2142/105272"],"dc:language":["en"],"dc:rights":["Copyright 2019 Qianyang Peng"],"dc:subject":["Software Testing","Test Case Prioritization","Flaky Test","Information Retrieval"],"dc:title":["Empirically revisiting and enhancing IR-based test-case prioritization"],"dc:type":["text"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:44Z"}