{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/108468"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/108468","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Trust, but verify: An investigation of methods of verification and dissemination of computational research artifacts for transparency and reproducibility","abstract":"In this study, I investigate how research communities are addressing concerns about the quality and rigor of computational research. I focus specifically on initiatives to expand the peer review and publication process to include new requirements for the assessment and dissemination of computational research artifacts. I report the results of a multiple-case analysis of two primary (American Journal of Political Science, ACM/IEEE Supercomputing) and five supplemental cases in political science, computer science, economics, mathematics, and statistics. Cases were developed through qualitative analysis based on interviews with key stakeholders including editors, reviewers, and verifiers; a sample of verified artifacts; and documentary evidence including policies, guidelines, and workflows. The central argument of this dissertation is that these reproducibility initiatives represent a set of experiments across the sciences exploring how changes to the incentives and information requirements of authors impact the quality, rigor, reproducibility, and trustworthiness of published research. These initiatives are part of a broader effort to change community norms with respect to the dissemination of the results of research that involves computation, elevating the importance of computational artifacts and clearly signaling that authors cannot be trusted to provide this information voluntarily. Expanding peer review and increasing the information required of authors through publication reproducibility audits is just one approach -- and a costly one -- to improving research quality and trustworthiness. The effect of these changes on research quality has yet to be demonstrated or studied. Based on the cases, I identify key factors that influence the operationalization of policies and workflows; the elements that each community considers important to the assessment of computational transparency and reproducibility; as well as the tools and infrastructure that they leverage to aid in the creation, assessment and dissemination of reproducible research artifacts. I develop a framework to analyze the reproducibility initiatives and a conceptual model of reproducible research artifacts. I relate my findings to recommendations from the recent National Academies of Science, Engineering and Medicine (NASEM) report on \\emph{Reproducibility and Replicability in Science} and provide a set of normative guidelines for communities interested in pursuing similar initiatives with implications for journal and conference leadership; tool and infrastructure developers; and funding bodies. I conclude that, while promising, further efforts should be made to increase our understanding of the effect of initiative policies and technological advancements on research quality.","abstract_html":"In this study, I investigate how research communities are addressing concerns about the quality and rigor of computational research. I focus specifically on initiatives to expand the peer review and publication process to include new requirements for the assessment and dissemination of computational research artifacts. I report the results of a multiple-case analysis of two primary (American Journal of Political Science, ACM/IEEE Supercomputing) and five supplemental cases in political science, computer science, economics, mathematics, and statistics. Cases were developed through qualitative analysis based on interviews with key stakeholders including editors, reviewers, and verifiers; a sample of verified artifacts; and documentary evidence including policies, guidelines, and workflows. The central argument of this dissertation is that these reproducibility initiatives represent a set of experiments across the sciences exploring how changes to the incentives and information requirements of authors impact the quality, rigor, reproducibility, and trustworthiness of published research. These initiatives are part of a broader effort to change community norms with respect to the dissemination of the results of research that involves computation, elevating the importance of computational artifacts and clearly signaling that authors cannot be trusted to provide this information voluntarily. Expanding peer review and increasing the information required of authors through publication reproducibility audits is just one approach -- and a costly one -- to improving research quality and trustworthiness. The effect of these changes on research quality has yet to be demonstrated or studied. Based on the cases, I identify key factors that influence the operationalization of policies and workflows; the elements that each community considers important to the assessment of computational transparency and reproducibility; as well as the tools and infrastructure that they leverage to aid in the creation, assessment and dissemination of reproducible research artifacts. I develop a framework to analyze the reproducibility initiatives and a conceptual model of reproducible research artifacts. I relate my findings to recommendations from the recent National Academies of Science, Engineering and Medicine (NASEM) report on \\emph{Reproducibility and Replicability in Science} and provide a set of normative guidelines for communities interested in pursuing similar initiatives with implications for journal and conference leadership; tool and infrastructure developers; and funding bodies. I conclude that, while promising, further efforts should be made to increase our understanding of the effect of initiative policies and technological advancements on research quality.","abstract_has_math":false,"creators":["Willis, Craig Alan"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Library & Information Science","degree_department":null,"school":null,"contributors":["Stodden, Victoria","Darch, Peter","Ludaescher, Bertram","Taufer, Michela"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2020,"date_issued":"2020-10-07T20:59:43Z","date_published":"2020-10-07T20:59:43Z","updated_at":"2026-07-22T22:24:48Z","subjects":["Computational reproducibility","Peer review","Reproducibility"],"languages":["en"],"rights":["Copyright 2020 Craig Willis"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/108468","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Stodden, Victoria","Darch, Peter","Ludaescher, Bertram","Taufer, Michela"]},{"key":"dc:creator","label":"Author","values":["Willis, Craig Alan"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2020-10-07T20:59:43Z","2020-07-13","2020-08"]},{"key":"dc:type","label":"Dc Type","values":["text","Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Library & Information Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Computational reproducibility","Peer review","Reproducibility"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2020 Craig Willis"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/108468"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["In this study, I investigate how research communities are addressing concerns about the quality and rigor of computational research. I focus specifically on initiatives to expand the peer review and publication process to include new requirements for the assessment and dissemination of computational research artifacts. I report the results of a multiple-case analysis of two primary (American Journal of Political Science, ACM/IEEE Supercomputing) and five supplemental cases in political science, computer science, economics, mathematics, and statistics. Cases were developed through qualitative analysis based on interviews with key stakeholders including editors, reviewers, and verifiers; a sample of verified artifacts; and documentary evidence including policies, guidelines, and workflows. The central argument of this dissertation is that these reproducibility initiatives represent a set of experiments across the sciences exploring how changes to the incentives and information requirements of authors impact the quality, rigor, reproducibility, and trustworthiness of published research. These initiatives are part of a broader effort to change community norms with respect to the dissemination of the results of research that involves computation, elevating the importance of computational artifacts and clearly signaling that authors cannot be trusted to provide this information voluntarily. Expanding peer review and increasing the information required of authors through publication reproducibility audits is just one approach -- and a costly one -- to improving research quality and trustworthiness. The effect of these changes on research quality has yet to be demonstrated or studied. Based on the cases, I identify key factors that influence the operationalization of policies and workflows; the elements that each community considers important to the assessment of computational transparency and reproducibility; as well as the tools and infrastructure that they leverage to aid in the creation, assessment and dissemination of reproducible research artifacts. I develop a framework to analyze the reproducibility initiatives and a conceptual model of reproducible research artifacts. I relate my findings to recommendations from the recent National Academies of Science, Engineering and Medicine (NASEM) report on \\emph{Reproducibility and Replicability in Science} and provide a set of normative guidelines for communities interested in pursuing similar initiatives with implications for journal and conference leadership; tool and infrastructure developers; and funding bodies. I conclude that, while promising, further efforts should be made to increase our understanding of the effect of initiative policies and technological advancements on research quality.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2020-10-02 without embargo terms","The student, Craig Willis, accepted the attached license on 2020-07-09 at 14:16.","The student, Craig Willis, submitted this Dissertation for approval on 2020-07-09 at 14:21.","This Dissertation was approved for publication on 2020-07-13 at 11:57.","DSpace SAF Submission Ingestion Package generated from Vireo submission #15549 on 2020-10-02 at 15:12:51","Made available in DSpace on 2020-10-07T20:59:43Z (GMT). No. of bitstreams: 3 WILLIS-DISSERTATION-2020.pdf: 2236628 bytes, checksum: 146ff96dee17f187fb7ccea60c47bb39 (MD5) LICENSE.txt: 4209 bytes, checksum: 1b5162bf45e404aacb0f89c346a7ba15 (MD5) PROQUEST_LICENSE.txt: 4555 bytes, checksum: a1f8c13d540977967d05fb4a5cc71b2e (MD5) Previous issue date: 2020-07-13"]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Trust, but verify: An investigation of methods of verification and dissemination of computational research artifacts for transparency and reproducibility"]}]}],"canonical_facts":{"dc:contributor":["Stodden, Victoria","Darch, Peter","Ludaescher, Bertram","Taufer, Michela"],"dc:creator":["Willis, Craig Alan"],"dc:date":["2020-10-07T20:59:43Z","2020-07-13","2020-08"],"dc:description":["In this study, I investigate how research communities are addressing concerns about the quality and rigor of computational research. I focus specifically on initiatives to expand the peer review and publication process to include new requirements for the assessment and dissemination of computational research artifacts. I report the results of a multiple-case analysis of two primary (American Journal of Political Science, ACM/IEEE Supercomputing) and five supplemental cases in political science, computer science, economics, mathematics, and statistics. Cases were developed through qualitative analysis based on interviews with key stakeholders including editors, reviewers, and verifiers; a sample of verified artifacts; and documentary evidence including policies, guidelines, and workflows. The central argument of this dissertation is that these reproducibility initiatives represent a set of experiments across the sciences exploring how changes to the incentives and information requirements of authors impact the quality, rigor, reproducibility, and trustworthiness of published research. These initiatives are part of a broader effort to change community norms with respect to the dissemination of the results of research that involves computation, elevating the importance of computational artifacts and clearly signaling that authors cannot be trusted to provide this information voluntarily. Expanding peer review and increasing the information required of authors through publication reproducibility audits is just one approach -- and a costly one -- to improving research quality and trustworthiness. The effect of these changes on research quality has yet to be demonstrated or studied. Based on the cases, I identify key factors that influence the operationalization of policies and workflows; the elements that each community considers important to the assessment of computational transparency and reproducibility; as well as the tools and infrastructure that they leverage to aid in the creation, assessment and dissemination of reproducible research artifacts. I develop a framework to analyze the reproducibility initiatives and a conceptual model of reproducible research artifacts. I relate my findings to recommendations from the recent National Academies of Science, Engineering and Medicine (NASEM) report on \\emph{Reproducibility and Replicability in Science} and provide a set of normative guidelines for communities interested in pursuing similar initiatives with implications for journal and conference leadership; tool and infrastructure developers; and funding bodies. I conclude that, while promising, further efforts should be made to increase our understanding of the effect of initiative policies and technological advancements on research quality.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2020-10-02 without embargo terms","The student, Craig Willis, accepted the attached license on 2020-07-09 at 14:16.","The student, Craig Willis, submitted this Dissertation for approval on 2020-07-09 at 14:21.","This Dissertation was approved for publication on 2020-07-13 at 11:57.","DSpace SAF Submission Ingestion Package generated from Vireo submission #15549 on 2020-10-02 at 15:12:51","Made available in DSpace on 2020-10-07T20:59:43Z (GMT). No. of bitstreams: 3 WILLIS-DISSERTATION-2020.pdf: 2236628 bytes, checksum: 146ff96dee17f187fb7ccea60c47bb39 (MD5) LICENSE.txt: 4209 bytes, checksum: 1b5162bf45e404aacb0f89c346a7ba15 (MD5) PROQUEST_LICENSE.txt: 4555 bytes, checksum: a1f8c13d540977967d05fb4a5cc71b2e (MD5) Previous issue date: 2020-07-13"],"dc:format":["application/pdf"],"dc:identifier":["http://hdl.handle.net/2142/108468"],"dc:language":["en"],"dc:rights":["Copyright 2020 Craig Willis"],"dc:subject":["Computational reproducibility","Peer review","Reproducibility"],"dc:title":["Trust, but verify: An investigation of methods of verification and dissemination of computational research artifacts for transparency and reproducibility"],"dc:type":["text","Thesis"],"thesis:degree_discipline":["Library & Information Science"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:48Z"}