Background A key objective of the Guidelines International Network (GIN) is to reduce duplication of effort. To address this objective, a working group was established to define a minimum dataset for inclusion in all evidence tables.
Methods A literature review was conducted to identify existing evidence tables, and GIN member organisations were asked to provide the tables they use. The results were used to develop a minimum dataset (template) for studies addressing intervention questions. The template was pilot-tested by a group of guideline developers and reviewed at GIN conferences.
Results The literature search yielded 65 articles. These dealt with reporting standards and trial quality (eg, CONSORT statement) rather than which data should be extracted from studies. However, the checklist items given were considered useful. Nineteen GIN members provided evidence tables; 17 tables were used for analysis. The number of items included in the tables ranged from 8 to 19, with several items common to all tables. Within individual items, the level of detail varied widely. The draught template included a majority of items relating to objective data. Pilot testing revealed that the median time to read a paper and complete the template was 2 h for a randomised controlled trial and 2½ h for a non-randomised, controlled intervention study. The median rating for both relevance and clarity of items was high.
Conclusion The template listing the items needed to summarise an interventional study is now available for large-scale testing by all organisations.
- Data extraction
- evidence tables
- intervention study
- practice guidelines
- clinical practice guidelines
- evidence-based medicine
Statistics from Altmetric.com
- Data extraction
- evidence tables
- intervention study
- practice guidelines
- clinical practice guidelines
- evidence-based medicine
Health costs have escalated in recent years and are still rising steeply, driven by ageing populations and the introduction of new technologies. Quality of healthcare is an important issue for health professionals, patients and carers, and many national governments have assumed that improving the quality of healthcare will help contain costs by reducing unnecessary treatments and procedures.
One of the promises of evidence-based medicine is that it will enhance the quality of healthcare. Various bodies produce health technology assessments (HTA) and clinical practice guidelines (CPG) on the basis of evidence reviews and syntheses. The production of these documents is time-consuming and costly, requires considerable resources and leads to substantial duplication.
Adapting guidelines from one country to another may save time1 but is not always feasible. Standardisation of evidence tables produced from systematic reviews could facilitate faster guideline production. Although most HTA and guideline agencies use such tables, the scope and quality of the information they contain can vary quite considerably. Standardising the data-extraction process and the format of tables should minimise duplication of effort. It may also reduce the overall cost of HTA or guideline production.
The Guidelines International Network (GIN) was founded in 2002 to ‘facilitate information sharing, education and knowledge transfer, and collaborative working between guideline programmes to promote best practice and avoid duplication of effort’ (http://www.GIN.net).2 A key concept underlying these objectives is that of globalising the evidence which should be the same wherever it is used.3 In 2005, a working group on evidence tables was set up to define a minimum data set that should be included in all evidence tables. The approach was similar to the work that has already been done on the reporting of clinical trials (CONSORT, etc)4–6 and the standardisation of guidelines.7
Providing summaries of evidence in a standard format is therefore a central issue in sharing work between guideline developers. It brings additional benefits for newcomers to the field in that it relieves them of the need to develop a core data set themselves. For established developers, standard formats are likely to enhance the usability, reliability and relevance of the evidence they review in support of their guidelines and, most importantly, avoid having to review studies that already been fully summarised by others.
This paper aims to describe the background to the design of the template used to extract data from studies addressing intervention questions and the results of a pilot test on template use.
In 2005, the GIN Board agreed to the establishment of an Evidence Tables Working Group on this topic. The remit of the Working Group was:
to carry out a review of the literature;
examine the evidence tables already in use by GIN members;
agree a definition of evidence tables;
propose a minimum dataset to be extracted from studies (template);
obtain feedback from GIN members;
perform a pilot study.
Copies of the evidence tables used by GIN members were requested by email. This was followed by a literature search (Medline, until 2005) using the following keywords: (‘Data collection’ (MeSH) OR ‘Data summaries’ (MeSH) ‘Evidence tables’) then (‘Quality Control’ (MeSH) AND (‘Publishing/standards’ (MeSH) OR ‘Research Design’ (MeSH:NoExp) AND reports(Title) OR report(Title) OR reporting(Title).
A first version of the minimum dataset was produced after a meeting attended by two experts with longstanding experience in guideline development (see Acknowledgements). The template was presented at the 2005 GIN conference. Feedback was collected from workshop attendees, and the dataset amended accordingly after discussion by the ETWG members.
The amended template was pilot-tested by 14 guideline developers with different professional backgrounds from the Haute Autorité de Santé, an organisational member of GIN. They were provided with a standard set of instructions and asked to complete the templates for three papers of varying quality which were under review at the time: two randomised controlled trials (RCTs)8 9 and one non-randomised, controlled intervention study.10 They rated the relevance of the items on a scale from 1 (totally irrelevant) to 9 (essential item) and clarity of the instructions on a scale from 1 (instructions unclear) to 9 (instructions clear and unambiguous) and estimated the time taken to read the studies and complete the template. The results of the test were presented at a regional meeting on guideline development for Eastern European countries (Vienna, October 2006) and discussed among the ETWG members present. After further amendment, the final template was presented at the 2007 GIN conference.
The literature retrieved 65 articles. All were concerned with reporting standards and trial quality rather than the separate issue of which data should be extracted from studies in order to inform content and quality. An updated search (2005–2008) provided no further information.
Nevertheless, the checklist items given in the articles were considered to be useful tools to help select the data to summarise studies. In addition, valuable advice was found from the Cochrane Collaboration:
…authors first should consider how much information they want to collect. Overly detailed collection can result in forms that are longer than original study reports, tedious and boring to complete, and wasteful of author time. On the other hand, if forms are not sufficiently detailed and omit key data, authors may have to reabstract studies using supplemental data collection forms. Having to review a study a second time can be frustrating and time-consuming.11
Survey of evidence tables in use
Responses on evidence tables relating to intervention studies were received from 19 GIN members, of which 17 were available for analysis. One respondent had no fixed model, and one did not use evidence tables. Four members used different tables for different types of intervention studies, whereas 13 used the same basic table.
The number of items included in the tables ranged from 8 to 19. Several items were common to all tables (intervention, patient characteristics and outcomes measured), or the majority of tables: comparator and effect size (n=16), duration of follow-up and number of patients (n=15), quality rating and comments (n=14) and study type (n=11). Other commonly included items were: the full reference (n=9), country/setting (n=8), funding source (n=8), partial reference (n=7), number of patients followed up (n=7), design details (n=6+2 ‘sometimes’), inclusion/exclusion details (n=6), harms (n=6) and the conclusions from the paper (n=5). Less common items were: objectives (n=2), recruitment period (n=2), healthcare context (n=1), number of patients entering the trial (n=2+1 ‘sometimes’), morbidity description (n=2), co-interventions (n=2) and applicability (n=1). Surprisingly, one table excluded the reference and another excluded effect size.
Within individual items, the level of detail varied widely. Details not given in the table were sometimes to be found in the accompanying technical report. The comments section was often used for items that other institutions detailed in their evidence tables. In addition, the survey identified an obvious need to clarify what should be included under each heading—for example, did ‘number of patients’ refer to the number recruited, completing the study or analysed?
Defining evidence tables and establishing the data to be extracted
After examining the results of the literature review and survey, the ETWG agreed on the following definition of evidence tables: ‘Methodological and outcome summaries that present data from a number of related studies. They answer a well-defined question in a consistent format, aim to demonstrate overall trends in the evidence, and enable the process of making recommendations.’
Its answer to the question on the type of data to be extracted from the selected studies was ‘the most objective and factual data that summarises the study. This should include key data for quality/validity assessment and the data that will avoid the need to review the study a second time. Each heading requires explanatory notes.’
Evidence tables provided by GIN members, the CONSORT statement and checklist for the reporting of intervention studies, the QUORUM and TREND statements and the ‘how to use series’ of papers were used as source material for the first draught of the minimum dataset to be extracted.12–14 The template had to be applicable to all types of studies addressing intervention questions and not only to RCTs. The draught was amended in the light of feedback obtained at the 2005 GIN conference. Most amendments concerned the wording of the headings and explanatory notes.
For the 14 reviewers, the median time to read the paper and complete the template was 2 h for an RCT and 2.5 h for a non-randomised study. The median rating was 8 for the relevance and 9 for the clarity of the items respectively. Most assessors (10 to 14 depending upon the item) found the instructions useful and fully explanatory. Twelve out of 14 reviewers were in favour of the inclusion of evidence levels. There was some debate about the amount of detail required and with regard to wording. Items that were considered insufficiently covered were the objective of the study, exclusion criteria, statistical methods, sources of bias and health economics data. Comments were also made on the quality of the reporting of the articles. The template was amended in the light of these comments and validated at the 2007 GIN conference (table 1).
Evidence tables play an important role in presenting research evidence to groups developing clinical guidelines and HTA reports. The work undertaken by the GIN ETWG has led to the creation of a template that lists the minimum dataset needed to summarise a study dealing with an intervention question. The template is designed to avoid a ‘second look.’
This template has several strong points:
It extracts objective factual data in a standard format. There is just one subjective section, and this concerns the critical appraisal of study quality.
It includes all recommended items for outcomes (benefits and harms).15
It provides information on the setting in which outcomes were obtained15
It includes a bibliographic citation, enabling users to identify the source of all the information provided.
Institutions will be able to use completed templates in their entirety (evidence tables) or select and extract the data that meet their specific needs. It is possible, for example, to identify articles suitable for inclusion in a meta-analysis or to obtain additional, more detailed information than that collected for GRADE evidence profiles16 or summaries of findings17 (eg, the precise selection criteria used in individual studies, sources of funding).
A number of outstanding questions were addressed at workshops:
Who should complete evidence tables?
Should a quality rating be used for studies?
Would guideline developers be willing to use evidence tables instead of original papers?
It was agreed that people with skills in critical appraisal are able to complete evidence tables but that problems are sometimes encountered by non-clinicians since clinicians may have a better grasp of the relevance of an intervention and the choice of study population. Comments on study validity were considered essential, as the lack of an internationally recognised rating system currently precludes the use of a quality score. The degree to which standardisation reduces duplication of effort remains to be tested in practice.
Our study had two main limitations. The working group was small, and all pilot test participants were from same institution in France. We now plan to open up participation to other guideline and HTA developers via the GIN website and are developing a glossary of terms. Validity and feasibility studies will be undertaken. We are also currently developing and pilot-testing a template for diagnostic studies and have scheduled work on templates for prognostic studies and health economics assessments.
In conclusion, GIN's achievement in standardising the template on studies addressing intervention questions is the first but crucial step towards the long-term aim of collating evidence tables and facilitating their sharing to reduce duplication of effort.
We would like to thank all pilot test participants: S Barré, health economist (Haute Autorité de Santé); A Bernard, methodologist (Dijon University Hospital); J Biga, public health physician/health technology assessor (Haute Autorité de Santé); C Carbonneil, immunologist/health technology assessor (Haute Autorité de Santé); M Dhénain, dermatologist/public health physician/guideline methodologist (Haute Autorité de Santé); S Laversin, general practitioner/guideline methodologist (Haute Autorité de Santé); I L'Hermite, paediatrician/health technology assessor (Haute Autorité de Santé); F Midy, health economist (Haute Autorité de Santé); C Moty-Monnereau, public health physician/health technology assessor (Haute Autorité de Santé); K Petitprez, immunologist/guideline methodologist (Haute Autorité de Santé); N Préaubert, health economist (Haute Autorité de Santé); F Quentin, biologist/health technology assessor (Haute Autorité de Santé); V Raimond, health economist (Haute Autorité de Santé); C Revel-Delhom, general practitioner/guideline methodologist (Haute Autorité de Santé). Guidelines experts: S Danet (Haute Autorité de Santé), S Laversin (Haute Autorité de Santé); T Ojasoo for her help in reviewing and providing important comments on this paper. The Guidelines International Network (http://www.GIN.net) is an international not-for-profit association of organisations and individuals involved in the development and use of clinical practice guidelines, recognised under Scottish Charity Number SC034047.
The GIN active Working Group members at the time of template design and testing (in alphabetical order): B Burnand, Unité d'évaluation des soins & Centre d'épidémiologie clinique, Institut Universitaire de Médecine sociale et Préventive, Switzerland; R Cook, Bazian, UK; H de Beer, Dutch Institute for Healthcare Improvement CBO, The Netherlands; R Harbour, Scottish Intercollegiate Guidelines Network, UK; T Kaiser, Institute for Quality and Efficiency in Health Care, Germany; E Ketola, Current Care Guidelines—Finnish Medical Society Duodecim, Finland; J Komulainen, Current Care Guidelines—Finnish Medical Society Duodecim, Finland; R Kunz, Basel Institute for Clinical Epidemiology, Switzerland; M Laurence, Haute Autorité de Santé, France; N Mlika Cabanne, Haute Autorité de Santé, France; S Sadasivan, Ministry of Health, Malaysia; R Shiffman, Yale Center for Medical Informatics, USA; J Slutsky, Center for Outcomes and Evidence, Agency for Healthcare Research and Quality, USA; S Twaddle Scottish Intercollegiate Guidelines Network, UK; C Whittington, National Collaborating Centre for Mental Health, UK.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.