Background Scales like AGREE provide a systematic means for appraising guideline quality, but they are lengthy, emphasise methodology over practicality, and are best applied by guideline experts.

Objectives Create a short instrument for guideline appraisal, based on widely accepted standards.

Methods The Institute of Medicine (IOM) identified eight principles that make a guideline ‘trustworthy’. We adapted each principle into an item graded ‘A’, ‘B’, ‘C’, or “NR” (not reported). Guideline assessments are presented as a grid rather than a single score, with each row representing an item, each column a guideline, and cells coloured green, yellow, red or white to reflect the above grades, respectively. Concordance tables mapping AGREE and G-I-N standards to IOM domains were also created.

Results Piloted use of the tool suggests it can distinguish guidelines developed using weak methods and those that are poorly documented. Grids highlight guideline strengths and weaknesses, as well as guidelines that are more trustworthy than their comparators. The concordance table found that AGREE lacks standards for guideline currency and updating, while IOM lacks standards for resource implications.

Discussion Our pilot use of this instrument suggests that while the overall trustworthiness of guidelines is important, using IOM domains to understand sources of guidelines’ weaknesses can help organisations select guidelines best suited for their needs. Further work will examine our instrument’s reliability across users with different levels of expertise.

Implications for Guideline Developers/Users Pilot use of this tool suggests it can be applied by clinicians and administrators who have limited training and time.