{"id":{"repo_id":"mit","oai_identifier":"oai:dspace.mit.edu:1721.1/76492"},"canonical_url":"https://search.dev.ndltd.org/etd/mit/oai:dspace.mit.edu:1721.1/76492","repository":{"repo_id":"mit","name":"MIT","base_url":"https://dspace.mit.edu/oai/request"},"display":{"title":"Key challenges to model-based design : distinguishing model confidence from model validation","abstract":"Model-based design is becoming more prevalent in industry due to increasing complexities in technology while schedules shorten and budgets tighten. Model-based design is a means to substantiate good design under these circumstances. Despite this, organizations often have a lack of confidence in the use of models to make critical decisions. As a consequence they often invest heavily in expensive test activities that may not yield substantially new or better information. On the other hand, models are often used beyond the bounds within which they had been previously calibrated and validated and their predictions in the new regime may be substantially in error and this can add substantial risk to a program. This thesis seeks to identify factors that cause either of these behaviors. Eight factors emerged as the key variables to misaligned model confidence. These were found by studying three case studies to setup the problem space. This was followed by a review of the literature with emphasis on model validation and assessment processes to identify remaining gaps. These gaps include proper model validation processes, limited research from the perspective of the decision-maker, and lack of understanding of the impact of contextual variables surrounding a decision. The impact these eight factors have on model confidence and credibility was tested using a web-based experiment that included a simple model of a catapult and varying contextual details representing the factors. In total 252 respondents interacted with the model and made a binary decision on a design problem to provide a measure for model confidence. Results from the testing showed several factors proved to cause an outright change in model confidence. One factor, a representation of model uncertainty, did not result in any differences to model confidence despite support from the literature suggesting otherwise. Findings such as these were used to gain additional insights and recommendations to address the problem of misaligned model confidence. Recommendations included system-level approaches, improved quality of communication, and use of decision analysis techniques. Applying focus in these areas can help to alleviate pressures from the contextual factors involved in the decision-making process. This will allow models to be used more effectively thereby supporting model-based design efforts.","abstract_html":"Model-based design is becoming more prevalent in industry due to increasing complexities in technology while schedules shorten and budgets tighten. Model-based design is a means to substantiate good design under these circumstances. Despite this, organizations often have a lack of confidence in the use of models to make critical decisions. As a consequence they often invest heavily in expensive test activities that may not yield substantially new or better information. On the other hand, models are often used beyond the bounds within which they had been previously calibrated and validated and their predictions in the new regime may be substantially in error and this can add substantial risk to a program. This thesis seeks to identify factors that cause either of these behaviors. Eight factors emerged as the key variables to misaligned model confidence. These were found by studying three case studies to setup the problem space. This was followed by a review of the literature with emphasis on model validation and assessment processes to identify remaining gaps. These gaps include proper model validation processes, limited research from the perspective of the decision-maker, and lack of understanding of the impact of contextual variables surrounding a decision. The impact these eight factors have on model confidence and credibility was tested using a web-based experiment that included a simple model of a catapult and varying contextual details representing the factors. In total 252 respondents interacted with the model and made a binary decision on a design problem to provide a measure for model confidence. Results from the testing showed several factors proved to cause an outright change in model confidence. One factor, a representation of model uncertainty, did not result in any differences to model confidence despite support from the literature suggesting otherwise. Findings such as these were used to gain additional insights and recommendations to address the problem of misaligned model confidence. Recommendations included system-level approaches, improved quality of communication, and use of decision analysis techniques. Applying focus in these areas can help to alleviate pressures from the contextual factors involved in the decision-making process. This will allow models to be used more effectively thereby supporting model-based design efforts.","abstract_has_math":false,"creators":["Flanagan, Genevieve (Genevieve Elise Cregar)"],"institution":"Massachusetts Institute of Technology","degree_name":null,"degree_level":null,"degree_discipline":null,"degree_department":"System Design and Management Program.","school":null,"contributors":[],"advisors":["Olivier L. de Weck and Noelle Eckley Selin."],"committee_chairs":[],"committee_members":[],"year":2012,"date_issued":"2012","date_published":"2012","updated_at":"2026-07-22T22:20:50Z","subjects":["System Design and Management Program.","Engineering Systems Division."],"languages":["eng"],"rights":["M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission."],"rights_urls":["http://dspace.mit.edu/handle/1721.1/7582"],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/1721.1/76492","outbound_label":"Handle","outbound_source":"dc:identifier.uri"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.advisor","label":"Advisor","values":["Olivier L. de Weck and Noelle Eckley Selin."]},{"key":"dc:contributor.department","label":"Department","values":["System Design and Management Program.","Massachusetts Institute of Technology. Engineering Systems Division."]},{"key":"dc:contributor.other","label":"Dc Contributor Other","values":["Massachusetts Institute of Technology. Engineering Systems Division."]},{"key":"dc:creator","label":"Author","values":["Flanagan, Genevieve (Genevieve Elise Cregar)"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.accessioned","label":"Dc Date Accessioned","values":["2013-01-23T19:44:02Z"]},{"key":"dc:date.available","label":"Dc Date Available","values":["2013-01-23T19:44:02Z"]},{"key":"dc:date.issued","label":"Date","values":["2012"]},{"key":"dc:publisher","label":"Institution","values":["Massachusetts Institute of Technology"]},{"key":"dc:type","label":"Dc Type","values":["Thesis"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["System Design and Management Program.","Engineering Systems Division."]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language.iso","label":"Language (ISO)","values":["eng"]},{"key":"dc:rights","label":"Dc Rights","values":["M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission."]},{"key":"dc:rights.uri","label":"Rights URI","values":["http://dspace.mit.edu/handle/1721.1/7582"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.uri","label":"Identifier URI","values":["http://hdl.handle.net/1721.1/76492"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Thesis (S.M. in Engineering and Management)--Massachusetts Institute of Technology, Engineering Systems Division, System Design and Management Program, 2012.","Cataloged from PDF version of thesis.","Includes bibliographical references (p. 93-97)."]},{"key":"dc:description.abstract","label":"Abstract","values":["Model-based design is becoming more prevalent in industry due to increasing complexities in technology while schedules shorten and budgets tighten. Model-based design is a means to substantiate good design under these circumstances. Despite this, organizations often have a lack of confidence in the use of models to make critical decisions. As a consequence they often invest heavily in expensive test activities that may not yield substantially new or better information. On the other hand, models are often used beyond the bounds within which they had been previously calibrated and validated and their predictions in the new regime may be substantially in error and this can add substantial risk to a program. This thesis seeks to identify factors that cause either of these behaviors. Eight factors emerged as the key variables to misaligned model confidence. These were found by studying three case studies to setup the problem space. This was followed by a review of the literature with emphasis on model validation and assessment processes to identify remaining gaps. These gaps include proper model validation processes, limited research from the perspective of the decision-maker, and lack of understanding of the impact of contextual variables surrounding a decision. The impact these eight factors have on model confidence and credibility was tested using a web-based experiment that included a simple model of a catapult and varying contextual details representing the factors. In total 252 respondents interacted with the model and made a binary decision on a design problem to provide a measure for model confidence. Results from the testing showed several factors proved to cause an outright change in model confidence. One factor, a representation of model uncertainty, did not result in any differences to model confidence despite support from the literature suggesting otherwise. Findings such as these were used to gain additional insights and recommendations to address the problem of misaligned model confidence. Recommendations included system-level approaches, improved quality of communication, and use of decision analysis techniques. Applying focus in these areas can help to alleviate pressures from the contextual factors involved in the decision-making process. This will allow models to be used more effectively thereby supporting model-based design efforts."]},{"key":"dc:description.degree","label":"Dc Description Degree","values":["S.M.in Engineering and Management"]},{"key":"dc:title","label":"Title","values":["Key challenges to model-based design : distinguishing model confidence from model validation"]}]}],"canonical_facts":{"dc:contributor.advisor":["Olivier L. de Weck and Noelle Eckley Selin."],"dc:contributor.department":["System Design and Management Program.","Massachusetts Institute of Technology. Engineering Systems Division."],"dc:contributor.other":["Massachusetts Institute of Technology. Engineering Systems Division."],"dc:creator":["Flanagan, Genevieve (Genevieve Elise Cregar)"],"dc:date.accessioned":["2013-01-23T19:44:02Z"],"dc:date.available":["2013-01-23T19:44:02Z"],"dc:date.issued":["2012"],"dc:description":["Thesis (S.M. in Engineering and Management)--Massachusetts Institute of Technology, Engineering Systems Division, System Design and Management Program, 2012.","Cataloged from PDF version of thesis.","Includes bibliographical references (p. 93-97)."],"dc:description.abstract":["Model-based design is becoming more prevalent in industry due to increasing complexities in technology while schedules shorten and budgets tighten. Model-based design is a means to substantiate good design under these circumstances. Despite this, organizations often have a lack of confidence in the use of models to make critical decisions. As a consequence they often invest heavily in expensive test activities that may not yield substantially new or better information. On the other hand, models are often used beyond the bounds within which they had been previously calibrated and validated and their predictions in the new regime may be substantially in error and this can add substantial risk to a program. This thesis seeks to identify factors that cause either of these behaviors. Eight factors emerged as the key variables to misaligned model confidence. These were found by studying three case studies to setup the problem space. This was followed by a review of the literature with emphasis on model validation and assessment processes to identify remaining gaps. These gaps include proper model validation processes, limited research from the perspective of the decision-maker, and lack of understanding of the impact of contextual variables surrounding a decision. The impact these eight factors have on model confidence and credibility was tested using a web-based experiment that included a simple model of a catapult and varying contextual details representing the factors. In total 252 respondents interacted with the model and made a binary decision on a design problem to provide a measure for model confidence. Results from the testing showed several factors proved to cause an outright change in model confidence. One factor, a representation of model uncertainty, did not result in any differences to model confidence despite support from the literature suggesting otherwise. Findings such as these were used to gain additional insights and recommendations to address the problem of misaligned model confidence. Recommendations included system-level approaches, improved quality of communication, and use of decision analysis techniques. Applying focus in these areas can help to alleviate pressures from the contextual factors involved in the decision-making process. This will allow models to be used more effectively thereby supporting model-based design efforts."],"dc:description.degree":["S.M.in Engineering and Management"],"dc:identifier.uri":["http://hdl.handle.net/1721.1/76492"],"dc:language.iso":["eng"],"dc:publisher":["Massachusetts Institute of Technology"],"dc:rights":["M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission."],"dc:rights.uri":["http://dspace.mit.edu/handle/1721.1/7582"],"dc:subject":["System Design and Management Program.","Engineering Systems Division."],"dc:title":["Key challenges to model-based design : distinguishing model confidence from model validation"],"dc:type":["Thesis"]},"updated_at":"2026-07-22T22:20:50Z"}