{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/78421"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/78421","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Optimizing I/O performance for high performance computing applications: from auto-tuning to a feedback-driven approach","abstract":"The 2014 TOP500 supercomputer list includes over 40 deployed petascale systems, and the high performance computing (HPC) community is working toward developing the first exaflop system by 2023. Scientific applications on such large-scale computers often read and write a lot of data. With such rapid growth in computing power and data intensity, I/O continues to be a challenging factor in determining the overall performance of HPC applications. We address the problem of optimizing I/O performance for HPC applications by firstly examining the I/O behavior of thousands of supercomputing applications. We analyzed the high-level I/O logs of over a million jobs representing a combined total of six years of I/O behavior across three leading high-performance computing platforms. Our analysis provides a broad portrait of the state of HPC I/O usage. We proposed a simple and effective analysis and visualization procedure to help scientists who do not have I/O expertise to quickly locate the bottlenecks and inefficiencies in their I/O approach. We proposed several filtering criteria for system administrators to find application candidates that are consuming system I/O resources inefficiently. Overall, our analysis techniques can help both application users and platform administrators improve I/O performance and I/O system utilization. In the second part, we develop a framework that can hide the complexity of the I/O stack from scientists without penalizing performance. This framework will allow application developers to issue I/O calls without modification and rely on an intelligent runtime system to transparently determine and execute an I/O strategy that takes all the levels of the I/O stack into account. Lastly, we develop a multi-level tracing framework that provides a much more detailed feedback for application’s I/O runtime behavior. These details are needed for in-depth application’s performance analysis and tuning.","abstract_html":"The 2014 TOP500 supercomputer list includes over 40 deployed petascale systems, and the high performance computing (HPC) community is working toward developing the first exaflop system by 2023. Scientific applications on such large-scale computers often read and write a lot of data. With such rapid growth in computing power and data intensity, I/O continues to be a challenging factor in determining the overall performance of HPC applications. We address the problem of optimizing I/O performance for HPC applications by firstly examining the I/O behavior of thousands of supercomputing applications. We analyzed the high-level I/O logs of over a million jobs representing a combined total of six years of I/O behavior across three leading high-performance computing platforms. Our analysis provides a broad portrait of the state of HPC I/O usage. We proposed a simple and effective analysis and visualization procedure to help scientists who do not have I/O expertise to quickly locate the bottlenecks and inefficiencies in their I/O approach. We proposed several filtering criteria for system administrators to find application candidates that are consuming system I/O resources inefficiently. Overall, our analysis techniques can help both application users and platform administrators improve I/O performance and I/O system utilization. In the second part, we develop a framework that can hide the complexity of the I/O stack from scientists without penalizing performance. This framework will allow application developers to issue I/O calls without modification and rely on an intelligent runtime system to transparently determine and execute an I/O strategy that takes all the levels of the I/O stack into account. Lastly, we develop a multi-level tracing framework that provides a much more detailed feedback for application’s I/O runtime behavior. These details are needed for in-depth application’s performance analysis and tuning.","abstract_has_math":false,"creators":["Luu, Huong Vu Thanh"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Winslett, Marianne","Gropp, William","Snir, Marc","Ross, Robert"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2015,"date_issued":"2015-07-22T22:17:08Z","date_published":"2015-07-22T22:17:08Z","updated_at":"2026-07-22T22:26:11Z","subjects":["High Performance Computing (HPC)","Input/Output (I/O)","Parallel I/O","Performance Analysis"],"languages":["en"],"rights":["Copyright 2015 Huong Vu Thanh Luu"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/78421","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Winslett, Marianne","Gropp, William","Snir, Marc","Ross, Robert"]},{"key":"dc:creator","label":"Author","values":["Luu, Huong Vu Thanh"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2015-07-22T22:17:08Z","2015-05","2015-04-22","2015-5"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["High Performance Computing (HPC)","Input/Output (I/O)","Parallel I/O","Performance Analysis"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2015 Huong Vu Thanh Luu"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/78421"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["The 2014 TOP500 supercomputer list includes over 40 deployed petascale systems, and the high performance computing (HPC) community is working toward developing the first exaflop system by 2023. Scientific applications on such large-scale computers often read and write a lot of data. With such rapid growth in computing power and data intensity, I/O continues to be a challenging factor in determining the overall performance of HPC applications. We address the problem of optimizing I/O performance for HPC applications by firstly examining the I/O behavior of thousands of supercomputing applications. We analyzed the high-level I/O logs of over a million jobs representing a combined total of six years of I/O behavior across three leading high-performance computing platforms. Our analysis provides a broad portrait of the state of HPC I/O usage. We proposed a simple and effective analysis and visualization procedure to help scientists who do not have I/O expertise to quickly locate the bottlenecks and inefficiencies in their I/O approach. We proposed several filtering criteria for system administrators to find application candidates that are consuming system I/O resources inefficiently. Overall, our analysis techniques can help both application users and platform administrators improve I/O performance and I/O system utilization. In the second part, we develop a framework that can hide the complexity of the I/O stack from scientists without penalizing performance. This framework will allow application developers to issue I/O calls without modification and rely on an intelligent runtime system to transparently determine and execute an I/O strategy that takes all the levels of the I/O stack into account. Lastly, we develop a multi-level tracing framework that provides a much more detailed feedback for application’s I/O runtime behavior. These details are needed for in-depth application’s performance analysis and tuning.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2015-07-22 without embargo terms","The student, Huong Luu, accepted the attached license on 2015-04-20 at 10:10.","The student, Huong Luu, submitted this Dissertation for approval on 2015-04-20 at 10:29.","This Dissertation was approved for publication on 2015-04-22 at 10:43.","DSpace SAF Submission Ingestion Package generated from Vireo submission #7944 on 2015-07-22 at 10:32:50","Made available in DSpace on 2015-07-22T22:17:08Z (GMT). No. of bitstreams: 2 LUU-DISSERTATION-2015.pdf: 11868379 bytes, checksum: 168ccd35428ce27374a23b0e6e43d2e2 (MD5) LICENSE.txt: 4206 bytes, checksum: 05f860f2f8f3c2bc094bfb2a409d5760 (MD5) Previous issue date: 2015-04-22"]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Optimizing I/O performance for high performance computing applications: from auto-tuning to a feedback-driven approach"]}]}],"canonical_facts":{"dc:contributor":["Winslett, Marianne","Gropp, William","Snir, Marc","Ross, Robert"],"dc:creator":["Luu, Huong Vu Thanh"],"dc:date":["2015-07-22T22:17:08Z","2015-05","2015-04-22","2015-5"],"dc:description":["The 2014 TOP500 supercomputer list includes over 40 deployed petascale systems, and the high performance computing (HPC) community is working toward developing the first exaflop system by 2023. Scientific applications on such large-scale computers often read and write a lot of data. With such rapid growth in computing power and data intensity, I/O continues to be a challenging factor in determining the overall performance of HPC applications. We address the problem of optimizing I/O performance for HPC applications by firstly examining the I/O behavior of thousands of supercomputing applications. We analyzed the high-level I/O logs of over a million jobs representing a combined total of six years of I/O behavior across three leading high-performance computing platforms. Our analysis provides a broad portrait of the state of HPC I/O usage. We proposed a simple and effective analysis and visualization procedure to help scientists who do not have I/O expertise to quickly locate the bottlenecks and inefficiencies in their I/O approach. We proposed several filtering criteria for system administrators to find application candidates that are consuming system I/O resources inefficiently. Overall, our analysis techniques can help both application users and platform administrators improve I/O performance and I/O system utilization. In the second part, we develop a framework that can hide the complexity of the I/O stack from scientists without penalizing performance. This framework will allow application developers to issue I/O calls without modification and rely on an intelligent runtime system to transparently determine and execute an I/O strategy that takes all the levels of the I/O stack into account. Lastly, we develop a multi-level tracing framework that provides a much more detailed feedback for application’s I/O runtime behavior. These details are needed for in-depth application’s performance analysis and tuning.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2015-07-22 without embargo terms","The student, Huong Luu, accepted the attached license on 2015-04-20 at 10:10.","The student, Huong Luu, submitted this Dissertation for approval on 2015-04-20 at 10:29.","This Dissertation was approved for publication on 2015-04-22 at 10:43.","DSpace SAF Submission Ingestion Package generated from Vireo submission #7944 on 2015-07-22 at 10:32:50","Made available in DSpace on 2015-07-22T22:17:08Z (GMT). No. of bitstreams: 2 LUU-DISSERTATION-2015.pdf: 11868379 bytes, checksum: 168ccd35428ce27374a23b0e6e43d2e2 (MD5) LICENSE.txt: 4206 bytes, checksum: 05f860f2f8f3c2bc094bfb2a409d5760 (MD5) Previous issue date: 2015-04-22"],"dc:format":["application/pdf"],"dc:identifier":["http://hdl.handle.net/2142/78421"],"dc:language":["en"],"dc:rights":["Copyright 2015 Huong Vu Thanh Luu"],"dc:subject":["High Performance Computing (HPC)","Input/Output (I/O)","Parallel I/O","Performance Analysis"],"dc:title":["Optimizing I/O performance for high performance computing applications: from auto-tuning to a feedback-driven approach"],"dc:type":["text"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:26:11Z"}