The design of composite indicators of healthcare quality: a multi-method analysis

High profile composite indicators seeking to summarise aspects of the quality of healthcare organisations often have important impacts: they may be reputationally consequential for healthcare organisations, may affect prioritisation for inspection or action by regulators, and may be responsible for effecting changes in the way services are delivered. Use of composite indicators, which include examples such as the Hospital Compare Star Ratings in the US and the Quality and Outcomes Framework in the UK, is proliferating in healthcare. But to date there is limited understand of how such indicators should be designed and reported. This thesis seeks to address this gap, characterising relevant challenges in the design of composite indicators and exploring their reporting might be improved. My examination of composite indicators is formed from two principal components. The first comprises two quantitative studies, which are conceptually linked. These studies investigate the sensitivity of the ratings and rankings of hospitals to the exact technical methods by which composite indicators are produced. I show that composite indicators can be highly sensitive to technical decisions, and that different composite indicators are affected differently by these, rarely examined, technical issues. The second component is a qualitative interview study, for which I interviewed experts in performance measurement about the process of developing a composite indicator. This study highlighted two crucial aspects of the development of composite indicators. First, the purpose of a composite indicator needs to drive every aspect of technical decisions relating to their specification. Second, the development process should be iterative: developers must be willing and able to revise the purpose of the indicator and restart the development process if it subsequently becomes apparent that the composite indicator does not match the purpose for which it is being developed. The results of the quantitative and qualitative studies in this thesis revealed a number of challenges for the design of composite indicators, but also helped to identify possible ways for improving their design. The qualitative study catalogues important decision points which could be used as a framework for design choices for composite indicators. In combination with the results of the quantitative studies, which inter alia show the usefulness of applications of Monte Carlo sensitivity analysis as part of the development and evaluation of composites, this catalogue of decisions holds promise for identifying and addressing problems with composite indicator in the design stage and for offering critiques of existing indicators. Together these considerations point toward designing-in an iterative approach to processes used in the development of composite indicators, in which sensitivity analysis is used to assess the implications of each design decision. Finally, my results lay the groundwork for future development of a reporting guideline for the development and reporting of composite indicators of healthcare quality.

Name dc:type.qualificationname

Doctor of Philosophy (PhD)

Level dc:type.qualificationlevel

Doctoral

Grantor dc:publisher.institution

University of Cambridge

Year dc:date.issued

2021

Author dc:creator

• Barclay, Matthew

Advisors dc:contributor.advisor

• Dixon-Woods, Mary
• Lyratzopoulos, Georgios

• health services research
• composite indicators
• performance measurement
• statistics

Licence

• https://apollo8-f-pro.lib.cam.ac.uk/bitstreams/36b8db80-0cf3-4b59-9819-13fda86a3e54/download
• https://creativecommons.org/licenses/by/4.0/

Language dc:language

eng

Author Identifier

0000-0003-1148-1922

OAI identifier oai:identifier

oai:www.repository.cam.ac.uk:1810/328778