Assessing quality in systematic reviews of the effectiveness of health promotion: areas of consensus and dissension

Systematic reviews have played an increasingly important role in health promotion in recent<br/>years. Yet there are debates about how they should be conducted, particularly about how the<br/>quality of evidence should be assessed. The aim of this research was to assess current<br/>approaches to, and general views on, the use of quality assessment in systematic reviews of<br/>effectiveness in health promotion, and to identify areas of consensus and dissension around the<br/>choice of techniques, methods and criteria employed.<br/><br/>There were two stages of data collection. The first was a structured mapping of a random<br/>sample of 30 systematic reviews of the effectiveness of health promotion to identify and explain<br/>trends and themes in methods and approaches to quality assessment. During the second stage<br/>semi-structured interviews were conducted with a purposive sample of 17 systematic reviewers<br/>who had conducted at least one review of a health promotion topic, to investigate some of these<br/>trends and approaches in greater detail.<br/><br/>The mapping found that the majority of systematic reviews had assessed the quality of the<br/>included studies, to varying degrees. However, procedures were not always explicitly reported<br/>or consistent. There was some degree of consensus over criteria, with experimental evaluation<br/>methods commonly favoured. Most frequently used quality assessment criteria included<br/>participant attrition, the validity and reliability of data collection and analysis methods, and<br/>adequacy of sample sizes. External validity was commonly assessed, primarily in terms of<br/>generalisability and replicability, but less so in terms of intervention quality.<br/><br/>The interviews revealed some of the barriers to effective systematic reviewing, including: lack<br/>of time and resources, complexity of some health promotion interventions, inclusion of<br/>observational evaluation designs, and poor reporting of primary studies. Systematic reviewing<br/>was commonly done in small teams, mostly comprising academics, sometimes with<br/>practitioners. Interviewees learned systematic review skills through a combination of training,<br/>support from colleagues and mentors, literature and a strong emphasis on hands-on practical<br/>learning. Subjective judgement was often required, contra to the popular belief that systematic<br/>reviews are wholly objective.<br/><br/>The overall conclusions of this study are that systematic reviewing in health promotion is often<br/>challenging due the complexity of interventions and evaluation designs. This places additional<br/>demands on reviewers in terms of knowledge and skills required, often exacerbated by finite<br/>time scales and limited funding. Initiatives are in place to foster shared ways of working,<br/>although the extent to which complete consensus is achievable in a multi- disciplinary area such<br/>as health promotion is questionable.

Name dc:type.qualificationname

Ph.D.

Level dc:type.qualificationlevel

doctoral

Grantor dc:publisher.institution

University of Southampton

Year dc:date.issued

2009

Author dc:creator

• Shepherd, Jonathan Paul

Advisor dc:contributor.advisor

• Weare, Katherine