HTA Home Page | Details page for this publication | Publications listing | Publications search

Health Technology Assessment 2001; Vol. 5: No. 8

Executive summary 

View/Download full monograph in Adobe Acrobat format (347 kbytes)
View/Download this 4-page summary in Adobe Acrobat format  (suitable for printing)

Issues in methodological research: perspectives from researchers and commissioners

RJ Lilford¹
A Richardson²
A Stevens^1*
R Fitzpatrick³
S Edwards⁴
F Rock⁵
JL Hutton⁶

¹ Department of Public Health and Epidemiology, University of Birmingham, UK
² Research Consultant, London, UK
³ Institute of Health Sciences, University of Oxford, UK
⁴ Centre for Ethics in Medicine, University of Bristol, UK
⁵ English Department, University of Birmingham, UK
⁶ Department of Statistics, University of Warwick, UK

^* Corresponding author

Objectives

• Methodological research has few well-defined tools and processes analogous to those available for reviews and data collection in substantive health technology assessment.

• This project was set up to obtain researchers’ and others’ views on the innovative projects on research methodology under the NHS Health Technology Assessment Programme and the usefulness of the research.

• The study was intended to span both epistemological and management issues.

• The following issues were explored:
  • the degree to which researchers would feel constrained by the “Cochrane” approach to systematic reviews when undertaking reviews of a methodological nature
  • whether methodological projects may require exceptional design and management arrangements, in view of their novelty, subjectivity and complexity
  • whether researchers would seek out other methods, in addition to undertaking reviews of argument, as a means of extending their understanding of methodological issues (there may be three categories of research methods in methodology: reviews of methodulogical argument, studies that use the literature as a source of data, and research that collects new primary data)
  • whether the Methodology Programme overall can be considered a “success”.

  Methods

• Telephone interviews were carried out on researchers (one senior and one junior per project), resulting in 35 interviews from 19 of the 20 target projects.

• A qualitative postal survey was sent to 12 people who had played a key role in the development of the Methodology Programme; replies were received from six of them.

• Analysis was undertaken of the hit rates for 29 projects on the NCCHTA website by the end of February and the end of May 1999, comparing those concerned with methodology (n = 10) and those concerned with other issues (n = 19).

Results

Undertaking methodological research: views of researchers
This section summarises the views of 35 researchers who were interviewed by telephone.

The nature of methodological reviews

• There was a reluctance among researchers to use the term “systematic review” in the methodological context.

• Practical problems in undertaking methodological reviews were found at every stage of the research process.

  – In the initial search stage, preplanned strategies were difficult to maintain, owing to the need to respond to the problems of too few or too many references.

  – At the analysis stage, most studies were not formally weighted, but there was implicit weighting in researchers’ views of their merits or relevance.

  – It was often only at the synthesis stage that researchers could see clearly what their study was able to do; iteration was frequently necessary at this point.

  – It was difficult to form simple conclusions and recommendations beyond summaries of what was known in the field.

  – Dissemination activities were most often directed to other health service researchers, with some attention to NHS policy makers and research commissioners.

The need for flexibility

• Few researchers had amended their topic or methods once their research was under way, although some had made minor changes to their original plan, generally to refine the topic to fit the time or data available.

• Changing a topic was seen as inappropriate unless checked with funders, but changes in research methods were viewed as reasonable because questions might be refined in the light of information gained or early thinking.

The question of bias

• Few researchers considered that this kind of research could be undertaken or presented in a wholly unbiased way because of the need to assess the research studied.

• Objectivity was nonetheless seen as something that researchers should strive towards. Efforts to do so included presenting data clearly, separating findings from discussion, covering all points of view, setting out their own assumptions and values, and testing their ideas on others known to have differing views.

• The formal peer-review process was not seen to have made a difference here, primarily because of the stage at which referees become involved.

Project management

Timing and time management

• A majority of projects were completed within 3 months of their due date. Those studies completed roughly on time were considered to have efficient junior researchers and good project management, including clear deadlines for different stages of the research.

• Some studies had severe problems of time management. Too much time tended to be spent on collecting and reading the literature and the writing stage was not always well planned. Referees’ comments were also slow in coming.

Day-to-day management

• Most projects were considered to have been well managed, but some had serious problems.

• Good management consisted of a clarity of roles between senior and junior researchers, active supervision of the latter and a set timetable. Careful time management was seen as essential, including building in a sufficient amount of a senior person’s time.

• Problems noted in the projects included staffing difficulties (both unanticipated departures and researchers’ unsuitability for the work), being under-resourced and the project not being well planned from the outset.

Involvement of others

• Grantholders were not always actively involved, but this was not generally seen as problematic. There was no consensus on whether payments to senior researchers should be built into projects.

• Only a few projects had a steering or advisory group beyond the grantholders. Such groups were generally viewed as helpful because they served as a sounding board and brought in additional expertise.

• Most researchers thought that the HTA Programme itself had exercised an appropriate level of involvement.

Collaboration

• Many projects overlapped with other HTA funded projects and entailed some collaboration among researchers, which was generally seen as helpful in saving time and in stimulating ideas.

• Some questioned the desirability of collaboration across similar projects because separate projects could develop independent corroboration of results; there were also problems of academic rivalry.

• There was much less collaboration with groups outside the HTA Programme.

• The Methodology Projects Group, an informal meeting of grantholders and research fellows, was widely seen as helpful as a means of learning about other projects and obtaining moral support in the face of difficulties.

Referees

• Most researchers considered that the referees’ comments had been helpful by providing intellectual support, forcing them to rethink their arguments, or pointing out gaps in the research.

• Referees were generally believed to have had the right skills, comprising a mix of specific professional and methodological expertise.

• Most respondents thought that projects needed two or three referees, although those with more complex projects tended to consider that more referees were needed.

Issues for the HTA Programme

• Some considered that methodological questions could be bolted on to other research, such as clinical trials.

• The view was expressed that methodological reviews should be updated as needed. Those who carried out the initial review should be asked first because they could do such work efficiently.

• Opinion was divided on whether the Programme should fund more large studies or small projects designed to scope a topic rather than deal with it comprehensively. There was also no marked preference for more primary or secondary studies, as this depends on the questions asked.

• A number of ideas for future research were proffered, but there was also interest in more attention being paid to getting the results of research into practice.

The Methodology Programme: views of those involved in its creation and development
This section summarises the views of key people who responded to a letter containing three principal questions.

Aims of the Programme and their achievement

• The aims of the Programme were seen to be identifying and answering important methodological questions relevant to health technology assessment and raising the profile of health services research methodology.

• Four respondents thought that the Programme had been highly successful, with impressive output in terms of both quantity and quality. This was ascribed to enthusiasm from researchers, assisted by the Methodology Projects Group, and good steering from the Methodology Panel itself.

• Doubts about the Programme included whether the right research had been funded and whether the research had been well managed by the commissioning body and research teams. Two respondents considered that a shortage of trained researchers had hampered the development of the Programme.

Noteworthy projects

• Two interviewees identified the characteristics of a successful project. Both focused on the qualities of the researchers: there is a need for the involvement of senior researchers who are familiar with the field and the Programme, and who are able to think deeply about the research and understand the target audience. A multi-disciplinary team was also seen as important.

• Various projects were noted as having been well executed, often following the particular interests of the respondent. Two noted that the projects concerned with randomised trials were particularly valuable.

Future directions for the Methodology Programme

• It was argued that the dissemination and use of methodological research needs to be addressed.

• Attention to research management was seen to be needed. Projects should be required to involve senior staff and the submission of early drafts of the final report for discussion.

• There was concern that the decision to widen the portfolio beyond health technology assessment would bring new problems of defining both the content of the work and who the customers are; in consequence, an evaluation of the Panel when its remit widens was proposed.

• Other suggestions included the need for systems for updating reviews and to address the problem of systematicity in the context of methodological work. It was proposed that methodological gaps in the Service Delivery and Organisation agenda should be identified and new primary research commissioned.

Interest in the projects

• On the basis of “hit rates” for the relevant website, there is considerable interest in these projects. The monthly hit rate for methodology projects was not only high in itself (median 264/month) but was also more than twice that of other HTA projects (median 102/month).

• More recently published projects were found to have higher hit rates.

Conclusions

Reflections on the findings

• These studies were commissioned at a time of high enthusiasm for systematic reviews and meta-analysis, resulting in both epistemological and practical problems for some of those undertaking them.

• The importance of systematicity rather than exhaustiveness needs to be recognised.

• The question of whether these reviews should be seen primarily as data collection or thinking exercises pervaded the interviews and has important practical implications.

• Contrary to our expectations, little was made (in this sample) of additional research methods, with few attempts at triangulation. Researchers were not opposed to changing methods in principle but did not do so in practice.

• Researchers seemed very conscious of the problem of bias and undertook an impressive range of steps to reduce its impact.

• The organisation and management demands on these projects were not notably different from those for other studies, including the need to plan carefully from the outset and for close research management by senior staff, but some issues may be heightened in this context.

• The Programme was largely seen as a success, covering a wide range of issues and helping to develop a pool of researchers familiar with the field.

• The website analysis provided a quick illustration of the considerable interest shown in methodological and other reports.

• The significance of the Methodology Programme should be seen to lie not simply in the reports produced, but in the diffusion of knowledge it facilitated.

Reflections on the methods

• The telephone interviews worked well, although they were time-consuming and tiring; they elicited very full responses and much valuable material.

• The letters to key people were clearly less successful, with a very poor response rate. Those who did reply generally provided very thoughtful responses.

• The website analysis was a useful addition to the research but, like citation analysis, cannot be used to infer the quality or impact of the research.

Recommendations

Recommendations for researchers

The conduct of research

• Researchers should consider methods beyond the review of ideas and even the review of data, for instance, networking and other means of primary data collection (e.g. methodological studies attached to primary health technology assessment).

• Because systematic reviews in this context are very different from traditional “Cochrane type” reviews, methodological researchers should not try to chase every reference, but ensure that they search widely (i.e. consider disparate databases and sources).

• Some overlap of the various stages of research – searching, analysis, synthesis and writing – should be encouraged because this can help to clarify the nature of the research.

• Researchers should publicise their studies early on, to help to short-circuit extensive search processes and stimulate ideas.

• All studies should include a short summary of key findings, which should include practical solutions to identified problems, to assist future researchers.

Reducing bias

• A variety of safeguards to reduce potential bias should be built in to research, including the establishment of a steering group, multidisciplinary teams, peer reviewing of applications and final reports, a report for the commissioning body, and a clear intention to publish in widely disseminated journals.

The efficient management of research

• Senior staff need to be closely involved throughout the research, both to assist with data analysis and to maintain good research management; this should be reflected in the costing of grants.

• A clear plan of action and research timetable should be developed, including a plan for the report writing, with a preliminary structure to influence both data collection and analysis.

• Multisite projects should be considered carefully before they are set up, with respect to both the logistics and the willingness of the parties to work together.

Recommendations for the Methodology Programme

Project management

• The Programme should continue the fairly light style of research management that it has used to date, but remain accessible to researchers who may need to discuss problems or changes to the initial plan. Programme representation on a project steering group is one means of achieving this.

• The Programme also needs to continue to advise researchers on overlapping projects; care needs to be given to avoiding duplication of effort through discussions with project advisory groups. A master steering group for all projects would be one way of achieving this and reassessing resource needs.

• Prior to commissioning projects, the Programme could introduce a more iterative approach, to ensure that it has correctly specified the research problem.

• Particular attention should be given to the dissemination and use of the findings of research already commissioned by the Programme and others; websites are clearly valuable for this purpose.

• The need for mutual support among researchers undertaking complex projects should not be overlooked.

• The new Programme could be evaluated to see if it continues to work as well as it did when its brief was more limited.

Future commissioning

• Methodological researchers should be encouraged to explore a wider range of methods, incorporating intellectual analysis and primary research, including methodological experiments.

• Some substantive researchers should be encouraged to add a methodological component to their studies, but not all subject areas can work within such a framework.

• The management of methodological research may itself be the subject of study.

• Arrangements could be set up for updating reviews as needed.

• Particular attention could be given to methodological gaps in the Service Delivery and Organisation agenda.

NHS R&D HTA Programme

The NHS R&D Health Technology Assessment (HTA) Programme was set up in 1993 to ensure that high-quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and provide care in the NHS.

Initially, six HTA panels (pharmaceuticals, acute sector, primary and community care, diagnostics and imaging, population screening, methodology) helped to set the research priorities for the HTA Programme. However, during the past few years there have been a number of changes in and around NHS R&D, such as the establishment of the National Institute for Clinical Excellence (NICE) and the creation of three new research programmes: Service Delivery and Organisation (SDO); New and Emerging Applications of Technology (NEAT); and the Methodology Programme.

Although the National Coordinating Centre for Health Technology Assessment (NCCHTA) commissions research on behalf of the Methodology Programme, it is the Methodology Group that now considers and advises the Methodology Programme Director on the best research projects to pursue.

The research reported in this monograph was funded as project number 96/71/01.

The views expressed in this publication are those of the authors and not necessarily those of the Methodology Programme, HTA Programme or the Department of Health. The editors wish to emphasize that funding and publication of this research by the NHS should not be taken as implicit support for any recommendations made by the authors.

Criteria for inclusion in the HTA monograph series
Reports are published in the HTA monograph series if (1) they have resulted from work commissioned for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

Methodology Programme Director: Professor Richard Lilford
HTA Programme Director: Professor Kent Woods
Series Editors: Professor Andrew Stevens, Dr Ken Stein, Professor John Gabbay and Dr Ruairidh Milne
Monograph Editorial Manager: Melanie Corris

The editors and publisher have tried to ensure the accuracy of this report but do not accept liability for damages or losses arising from material published in this report. They would like to thank the referees for their constructive comments on the draft document.

©2001 Crown Copyright

HTA Home Page | Details page for this publication | Publications listing | Publications search