Every profile includes an Evidence Rating, presented in the “Snapshot” and “Did It Work?” sections, which is an assessment of the quality and strength of the evidence that the results described in the profile are due to the innovation and not to other factors. The Evidence Rating is one type of information that a potential adopter of the innovation may consider when making a judgment about the likely benefits and risks of implementing the innovation in their own organization.
How to Use the Evidence Rating
The adoption decision is complex and must take into consideration other factors besides the innovator’s results, even when those are compelling. This is because local delivery systems, policies, resources, and culture vary, and may have a profound impact on the results of any change in service delivery practices (
- Information from the innovator in each innovation profile such as:
- Context of the Innovation– Describes the innovating organization.
- Adoption Considerations– Information on starting and sustaining the innovation.
- Planning and Development Process– Discusses the pre-innovation phase.
- Results and Evidence Rating– Describes the innovator’s evaluation methodology and findings.
- Contact the Innovator– All innovators have agreed to be contacted.
- References– Information on relevant research studies in other locations, and other material.
- In-depth information on Taking Innovations to Scale on this topical page featuring innovation spread stories, strategies, tools, and voices from the field (https://www.innovations.ahrq.gov/scale-up-and-spread/taking-innovations-scale).
- A Decisionmaker’s Guide to Adopting Innovations, a tool for assessing the “fit” of an innovation and understanding whether it will work for you (https://innovations.ahrq.gov/guide/guideTOC).
- A variety of articles on innovation basics and expert perspectives (https://www.innovations.ahrq.gov/articles-and-guides).
These resources can help place the Evidence Rating in context when assessing the ability of an innovation to address local problems.
Components of the Evidence Rating
The Evidence Rating is based on three types of information:
- Information about the design of the evaluation of the innovation
- The results of any quantitative evaluation of the innovation
- Non-experimental, qualitative, or anecdotal information about the effects of the innovation
The Evidence Rating draws on aspects of several established systems that implicitly or explicitly assess the strength of study design and comprehensiveness of reporting including the GRADE guidelines for rating the quality of evidence (
Many innovations have not been around long enough to have been rigorously evaluated. In fact, by the time many ideas are rigorously evaluated, they are no longer new or innovative. As a result, the Evidence Rating was designed to cover not only well-established, evidence-based innovations, but also cutting-edge ideas that are not as well studied.
Three Categories for Rating the Strength of Evidence
Drawing on elements of these established systems, the Innovations Exchange uses three categories to provide meaningful distinctions in assessing the strength of the link between the innovation and the observed results:
Strong: The evidence is based on one or more evaluations using experimental designs based on random allocation of individuals or groups of individuals (e.g. medical practices or hospital units) to comparison groups. The results of the evaluation(s) show consistent direct evidence of the effectiveness of the innovation in improving the targeted health care outcomes and/or processes, or structures in the case of health care policy innovations.
Moderate: While there are no randomized, controlled experiments, the evidence includes at least one systematic evaluation of the impact of the innovation using a quasi-experimental design, which could include the non-random assignment of individuals to comparison groups, before-and-after comparisons in one group, and/or comparisons with a historical baseline or control. The results of the evaluation(s) show consistent direct or indirect evidence of the effectiveness of the innovation in improving targeted health care outcomes and/or processes, or structures in the case of health care policy innovations. However, the strength of the evidence is limited by the size, quality, or generalizability of the evaluations, and thus alternative explanations cannot be ruled out.
Suggestive: While there are no systematic experimental or quasi-experimental evaluations, the evidence includes non-experimental or qualitative support for an association between the innovation and targeted health care outcomes or processes, or structures in the case of health care policy innovations. This evidence may include non-comparative case studies, correlation analysis, or anecdotal reports. As with the category above, alternative explanations for the results achieved cannot be ruled out.
If the available qualitative and quantitative information is insufficient to place the innovation in one of the three categories above, the activity fails to meet the minimum inclusion criterion for evidence, and therefore is not eligible for inclusion as an Innovation Profile in the AHRQ Health Care Innovations Exchange. It may, however, qualify for inclusion as an Innovation Attempt.
If you have questions about the Evidence Rating in general, or about the rating on a specific innovation, please contact us at firstname.lastname@example.org.