Evidence Resources /

Grading for Evidence-Based Guidelines: A Simple System for a Complex Task?

Editorial by Anne Lethaby, Senior Researcher, New Zealand Guidelines Group

One of the basic principles of the evidence-based medicine movement is to improve health outcomes by basing our decision making on the best available evidence. A systematic approach minimises the chance of making errors or making poor decisions.

Firstly, we need to ask ourselves what we want to know. Our query gets phrased into a question in a standard format to help us search for and identify relevant evidence that would answer the question. Once this evidence has been located, the next step is to make judgments about the quality of the evidence to help us to make the most appropriate decisions and to indicate how confident we feel about a proposed recommendation for doing, or perhaps not doing, something to improve a persons health or recovery from illness. Evidence-based guidelines provide guidance for health care decision making and a rigorous grading system enables us to be explicit and transparent about the guidance that is offered.

There are a plethora of grading systems used by guideline developers throughout the world to indicate both the quality and the strength of evidence used in health care decision making. The following systems are in regular use and have borrowed widely from each other: Guyatts Users Guide system, USPSTF system, Oxford Centre for Evidence Based Medicine system; ICSI system, SIGN system, NHMRC system; and the NZGG system.

There are common features in all these systems and the majority use a 3 step approach.

1. The first step is to critically appraise the study using the appropriate checklist for the design. There are optimum study designs for answering each type of question, depending on whether you are seeking to find out the most effective treatment for a condition, how to ascertain whether your patient has a condition, what the prognosis of your patient is, or whether he/she is more or less at risk than other people.
2. The second step is to assign a grade or level of evidence to the quality of an individual study that is relevant to answering a question. Typically, 3 levels are used to indicate excellent or good quality, fair quality and poor quality.
3. For the third step, the SIGN system has developed a unique process for considering the entirety of the evidence that has been gathered together to answer a question. This step requires judgments to be made about the body of evidence as a whole: whether there is enough evidence, whether it is consistent, whether it is applicable and/or generalisable to the setting or people that you are considering, whether it is of clinical importance and is economically feasible and any other relevant factors. Conclusions are then developed to summarise the main points and recommendations are then formed that suggest appropriate actions or inactions. These recommendations are assigned a grade according to the strength of the evidence. This last process requires subjective decision making and errors are best minimised by requiring the consensus of a large heterogeneous group of people representing different interests (in much the same way as a jury is formed to represent the breadth of opinion within a community).

The system proposed by NZGG uses aspects of many of the other grading systems in use. NZGG recommends that guideline groups use a Considered Judgment approach for step 3 to develop graded recommendations (details of this are available from www.sign.ac.uk).

The NZGG system has the following structure:

Study design	Level of evidence		Grading of recommendation
Systematic review Randomised controlled trial Cohort study Case control study Case series Cost study	+	No major flaws	A	The recommendation is supported by GOOD (strong) evidence
	~	Potential flaws, but these are unlikely to substantially alter the results	B	The recommendation is supported by FAIR (reasonable evidence, but there may be minimal inconsistency, or uncertainty)
	X	Major flaws	C	The recommendation is supported by EXPERT opinion (published) only
			I	There is INSUFFICIENT evidence to make a recommendation
			ü	GOOD PRACTICE POINT (in the opinion of the guideline development group)

This system is currently being piloted in New Zealand. No clear criteria are given for the grades of recommendations because rigid rules may prove inflexible in practice for different topics.

The proposed system attempts to strike a balance between the inevitable complexity of considering and synthesizing a number of different concepts in order to make good decisions and the requirements for simplicity and ease of use by developers and readers with different skills. It is yet another variant, but a Kiwi version, of a number of systems currently in use. There is a strong drive among international guideline developers to develop one standard system that can be used consistently in guideline development, but consensus on this goal is still a way off.