Background The terms applicability, generalizability, external validity, transferability generally describe one overarching theme: can available research evidence be utilised to answer the health care questions at hand, ideally supported by a judgement about the degree of confidence in this utilisation. This concept has been called directness.

Objectives To offer conceptual and practical guidance to those judging directness and using research evidence from non-randomised studies (NRS).

Methods We used a literature review and feedback from participants of a workshop funded by the Agency for Healthcare Quality and Research and the Cochrane Collaboration.

Results Guideline developers can use NRS as a source of complementary, sequential or replacement evidence for randomised controlled trials (RCTs) by focusing on judgements about the population, intervention, comparison and outcomes. They use NRS to complement judgements about inconsistency, the rationale and credibility of subgroup analysis, baseline risk estimates for the determination of absolute benefits and downsides, and the directness of surrogate outcomes. Authors use NRS as sequential evidence to provide evidence when the evidence from RCTs is insufficient (e.g. long-term harms). Use of evidence from NRS may also replace RCT evidence when RCTs provide indirect evidence but NRS provide overall higher quality, direct evidence. We developed a simple tool and algorithm to make judgements about indirectness more transparent.

Discussions These judgements need to be made in the context of other quality of evidence domains.

Implications for Guideline Developers/Users The transparency of the framework will support interaction with those making health care decision and policy.