Background In 2011 the IOM recommended that every guideline recommendation should be accompanied by an indication of Evidence Quality and Recommendation Strength and called for strategies to encourage harmonisation of development processes.

Objectives To characterise the differences in systems for rating Evidence Quality and Recommendation Strength as a prelude to possible standardisation.

Methods We examined 17 international, English-language guideline development systems to identify rating parameters and applied descriptive statistics. We also searched for conceptual linkages in the rating system descriptions and identified systems where Strength of Recommendation was stated as an Evidence Quality parameter.

Results Rating systems were remarkably inconsistent in their application of category indicators—using letters, Arabic and Roman numerals and combinations. The modal and median number of Evidence Quality categories was 3 (range 0 to 10) and Recommendation Strength categories was 4 (range 0 to 6). 13/17 used randomised trials as indicators of highest quality evidence. 7 systems used “expert opinion,” 6 used “case reports” or “case series,” and 4 described “reasoning from first principles” to define lowest evidence quality. Definitions of intermediate levels varied considerably. 7 systems judged benefits and harms in deriving Recommendation Strength. In 7 rating systems, Strength of Recommendation was described entirely in terms of Evidence Quality.

Implication There is considerable disagreement about the requisite granularity and definition of categories of Evidence Quality and Recommendation Strength. Application of the concept of Recommendation Strength consonant with the IOM standard is limited. A straightforward mapping of rating systems to one another is elusive.