A method for analysis of expert committee decision-making applied to FDA medical device panels

Committees of experts are critical for decision-making in engineering systems. This is because the complexity of these systems requires that information is pooled from across multiple specialties and domains of knowledge. The social elements of technical decision-making are not well understood, particularly among expert committees. This is largely due to a lack of methodology for directly studying such interactions in real-world situations. This thesis presents a method for the analysis of transcripts of expert committee meetings, with an eye towards understanding the process by which information is communicated in order to reach a decision. In particular, we focus on medical device advisory panels in the US Food and Drug Administration. The method is based upon natural language processing tools, and is designed to extract social networks in the form of directed graphs from the meeting transcripts which are representative of the flow of information and communication on the panel. Application of this method to a set of 37 meetings from the FDA's Circulatory Systems Devices Panel shows the presence of numerous effects. Prominent among these is the propensity for panel members from similar medical specialties to use similar language. Furthermore, panel members who use similar language tend to vote similarly. We find that these propensities are correlated - i.e., as panel members' language converges by medical specialty, panel members' votes also converge. This suggests that voting behavior is mediated by membership in a medical specialty and supports the notion that voting outcome is, to some extent, dependent on an interpretation of the data associated with training, particularly when a small number of interpretations of the data are possible. Furthermore, there is some preliminary evidence to suggest that as clinical trial data ambiguity and difficulty of decisionmaking increases, the strength of the mediating effect of medical specialty decreases. Assuming a common decision is reached, this might indicate that committee members are able to overcome their specialty perspective as the committee jointly deals with hard problems over longer periods of time. In cases where the panel's vote is split, a lack of linguistic coherence among members of the same medical specialty correlates with a lack of linguistic coherence among members who vote the same way. This could be due to the presence of multiple interpretations of the data, leading to idiosyncratic or value-based choice. We also find that voting outcome is associated with the order in which panel members ask questions - a sequence set by the committee chair. Members in the voting minority are more likely to ask questions later than are members in the voting majority. Voting minority members are also more likely to be graph sinks (i.e., nodes in a social network that have no outflow) than are voting majority members. This suggests an influence mechanism on these panels that might be associated with framing - i.e., later speakers seem to be less able to convince other panel members to discuss their topics of interest contributing to these members' minority status. These results may have some relation to FDA panel procedures and structure. Finally, we present a computational model that embodies a theory of panel voting procedures. Model results are compared to empirical results and implications are drawn for the design of expert committees and their associated procedures in engineering systems.

Department dc:contributor.department

Massachusetts Institute of Technology. Engineering Systems Division.

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2010

Author dc:creator

• Broniatowski, David André, 1982-

Advisor dc:contributor.advisor

• Christopher L. Magee.

• Engineering Systems Division.

Statement dc:rights

• M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission.

Licence dc:rights.uri

• http://dspace.mit.edu/handle/1721.1/7582

Language dc:language.iso

eng

Handle dc:identifier.uri

http://hdl.handle.net/1721.1/62617

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/62617