Methodological Guide

Scope and purpose

A priori development of a protocol serves as a reference for the guideline development (or adaption) group and it reduces bias in the development process. The protocol does not need to be excessively detailed, but it should provide fundamental methodological principles (e.g., key questions, evidence search and synthesis approaches, funding source, and panel members, among others).

Deviations from the protocol may be necessary, but these must be justified and reported in the executive summary. The topic of the guideline is usually prespecified, however the selection of specific question frameworks may be informed by the panel. The steering group may prepare the protocol in collaboration with the evidence synthesis group and the guideline panel. It may also be made available through international guidelines registries, the organizational website, the social media, or other means, for public input.

The guideline should have a specific and clear objective that drives its development (or adaption). Guidelines on surgical interventions may aim to improve the outcomes and experience of patients with a specific surgical disease or condition, to improve the diagnostic process, to enhance the prevention of disease and/or to inform policymakers and healthcare authorities.

Defining the clinical question(s) early in the process of guideline development is of paramount importance. The clinical question(s) should be structured within a question framework (e.g., PICO – patient, interventions/procedures, outcomes, for questions on interventions; patient, index test, reference test, outcomes for questions on diagnostic tests). Framing the question avoids making spurious assumptions and judgements when developing the recommendations.

Generic questions such as “Which is the best approach to treat patients with malignant biliary obstruction” do not define the competing interventions and will inevitably lead to confusion among the evidence research group, panel members, and other involved stakeholders.

A structured question, such as “Should percutaneous biliary drainage be preferred over endoscopic stent placement for the management of patients with malignant obstruction of the extrahepatic biliary system?” will facilitate individual processes of evidence outreach, summary, appraisal and development of recommendations.

Defining the outcomes is best made through panel consensus (including patient representatives or advocates), for example by grading their importance; therefore, outcome measures may not be available at the outset.

Stakeholders

Experts in the field facilitate the process of guideline development (or adaption). A guideline methodologist will define key methodological parameters, will instruct the panel how to appraise the evidence and will coordinate the process, will accommodate the discussion on how the evidence and other considerations will inform the recommendations (e.g., evidence to decision framework), and will safeguard the quality of the guideline development process. Evidence suggests that surgical guidelines produced by organizations with an ad hoc guidelines committee are of higher quality.

The guideline panel should include all stakeholders affected by the content of the guideline. These may be other than surgical and interventional specialists, such as primary care physicians, nurses, and physiotherapists.

For example, a guideline on robotic colorectal surgery might involve general and colorectal surgeons, oncologists, radiotherapists, pathologists, nurses, healthcare economists, surgical technology specialists, and patient representatives. Patient representatives or patient advocates should participate as ordinary panel members, with equal contribution and voting rights from the start of the guideline development process. The involvement of patient representatives might not always be feasible, for example in a guideline on orthopedic surgery in the octogenarians. On such occasions, patient advocacy groups or representatives of caregiver support groups might need to be involved. Guideline development (or adaption) groups should aim for at least two representatives from each stakeholders’ group. Particular care should be given to avoid the over- or underrepresentation of stakeholders, depending on the topic of the guideline.

Furthermore, the guideline development process should be supported from a diverse group of people, such as experts in evidence outreach (e.g., health information specialists), evidence synthesis (e.g., statisticians, methodologists), and health economists. Individual roles and tasks should be specific and defined at the outset.

The guideline should be developed (or adapted) to be used by specific stakeholders. These may be surgeons, physicians, nurses, allied healthcare professionals, policymakers, or the public. Different considerations apply depending on the perspective of the guideline.

For example, a guideline designed to inform policymakers has to consider cost and implementation issues. In contrast, a guideline developed to inform exclusively patient decision making might not need to focus on such issues.

Other considerations refer to the appraisal of the certainty (quality) of the evidence. Guidelines developed to inform population-based practices are best informed by intention to treat analyses (which are usually provided by randomized trials on surgical interventions). In contrast, from the patient perspective, per-protocol analyses are more relevant.

Evidence synthesis

Guidelines should be informed by all available evidence on a specific topic and avoid biased consideration of a fragment of evidence. This makes a systematic review of the literature imperative. The review is ideally performed de novo by an evidence review team. This is usually a labor-intensive process and requires time and financial resources, and methodological expertise. Furthermore, such process requires development of an appropriate search strategy, with careful selection of thesaurus and search terms, subject headings, truncated terms, search limits, and additional topic-specific search term combinations to capture the desired study design(s). Collaboration among the evidence search group, the evidence synthesis group, the guideline panel, and the steering group is of particular importance.

Alternatively, the guideline development group may use one or more existing systematic reviews to summarize the evidence that will inform the development of recommendations. However, these must be recent and of high quality (e.g., evaluated using the AMSTAR – A MeaSurement Tool to Assess systematic Reviews 2 tool), and they should provide evidence on the outcomes prioritized by the guideline panel. Another option is to update an existing systematic review, for example, through collaboration with the authors of the original review. This will allow updating of their work, tailoring it to the needs of the guideline project, and extracting information of interest that was not captured in the original review.

As in any systematic review, a guideline needs to specify criteria for selecting the evidence. These refer to the study design or type of evidence (e.g., randomized trials, observational studies, diagnostic test accuracy studies, surveys, focus groups) and the question framework (patient/population, intervention or index diagnostic test of interest, comparator or reference test).

These criteria are ideally specified at the protocol stage; however, the guideline development group might need to deviate from prespecified criteria and consider alternative sources depending on the available evidence.

For instance, a guideline focusing on the management of abdominal aortic aneurysms in patients over the age of 80 might need to be indirectly informed from evidence on patients over the age of 65, when appropriate evidence for the former group is not available. Such considerations should be transparently documented and be made available to guideline users.

The guideline should consider and document the strengths and limitations of the evidence that informed the recommendation(s). Limitations may be related to the risk of bias of individual studies, indirectness of the evidence (available evidence not addressing the guideline-specific population, intervention or diagnostic test, comparator, reference test, and/or outcome of interest), the certainty of the comparative effect estimate (imprecision), and other issues, such as heterogeneity and publication bias. Such considerations are summarized in the GRADE assessment of the certainty (quality) of the evidence [24] and can be presented in evidence tables.

Guideline developers (or adaptors) should explain how these limitations affected the recommendation(s). For instance, GRADE suggests that the overall certainty of evidence on a question framework be defined by the lowest certainty of evidence on critical outcomes. A strong recommendation can rarely be provided when the overall certainty is below moderate.

Development of recommendations

A guideline development group is expected to involve patients and/or the public in the guideline development or adaption process. Patient representatives or patient advocates should be involved from the outset, be regular members of the guideline panel, and have equal opportunities to express their opinion and the same voting rights as the other panel members.

Alternatively, patient/public views and preferences can be informed by available research, ad hoc surveys, or interviews with focus groups. Patients’ and public input is particularly important for selecting patient-centered critical and important outcomes, and for defining clinically meaningful differences. Guideline developers may find the AGREE-REX (AGREE Recommendation EXcellence) useful when formulating their recommendations under consideration of patients’/public’s views.

Guideline developers (or adaptors) should predefine the methodology that will be used to formulate the recommendation(s). A specific, predefined, transparent methodology to develop (or adapt) the recommendation(s) will avoid the common pitfall of arbitrary decisions on the strength and the direction of the recommendation(s).

For instance, GRADE suggests using an evidence to decision framework, which consists of the certainty of the evidence, the balance between benefits and harms, acceptability, feasibility, equity, economic/resources considerations, and patients’ views and preferences. Agreement among most of these parameters in favor of an intervention, in the presence of at least moderate certainty evidence on critical outcomes, suggests that a strong recommendation may be provided. Methods to reach consensus among panel members are ideally defined a priori and can involve iterative discussions, Delphi process, or voting.

In the context of evidence appraisal, harms of an intervention are outcomes in which the comparator is better than the intervention. It is of specific importance to consider both benefits and harms when developing recommendations. Failure to consider harms may result in the false confidence that an intervention is superior or inferior to the comparator. The balance between benefits and harms may need to be informed by the magnitude of the effect of the benefits and the respective magnitude of harms, along with the importance of these outcomes.

Guidelines are informed by research and empirical evidence, and through the input of stakeholders and experts in the field. Guideline users should be provided with sufficient information to review what type of evidence informed the recommendation(s) and how. Such information can be summarized in evidence summaries, systematic observation forms to retrieve expert-based evidence, and summaries of evidence on cost, patients’ values and preferences, acceptability and feasibility. For example, the link between the evidence and recommendations is summarized in evidence tables and the evidence to decision framework tables within the context of GRADE.

The recommendation(s) should be unambiguous, concise, actionable and clearly formulated. The wording of the recommendation(s) should be precise and reflect its/their strength (e.g., “We recommend…” for strong recommendations and “We suggest…” for weak/conditional recommendations). It should also indicate the direction of the recommendation (e.g., “laparoscopic hysterectomy over open hysterectomy”). Lengthy recommendations with complex language may result in misconceptions about the recommended course of action and the strength of the recommendation. Such wording as “Perioperative thromboprophylaxis may be recommended…”, “…could be suggested”, or “should be considered” produce confusion among target users as to whether an intervention is strongly recommended or is suggested as the best alternative in most situations. Furthermore, recommendations may need to specify the required setting, resources, or surgical/interventional expertise for their implementation; e.g., “Robotic prostatectomy is recommended over laparoscopic or open prostatectomy when a robotic platform is available, and surgical expertise is in place”.

Different interventional and non-interventional options and alternatives for treatment, management, diagnosis, prevention or screening might be appropriate for different patient populations or people at risk, and in different settings, regarding the availability of financial resources and infrastructures, surgical or interventional experience and expertise. Such options should be taken into consideration when formulating the recommendations.

Recommendations should be highlighted in the guideline publication, executive report, organizational website, etc. They must be clearly distinguished from supporting information. The authors may provide a list or table with the recommendations, and they may also be highlighted in the text, to facilitate identification. Algorithms or flow charts may be helpful in transferring a recommendation or a set of recommendations to clinical practice and decision making. Notably, the guideline manuscript should not contain in the supporting text statements that may be considered recommendations.

Resource considerations, including the cost of interventions, instruments, hospitalization, reinterventions, infrastructures (e.g., the availability of a hybrid angiosuite for combined surgical and endovascular interventions), and specific features of organizations or health services should be considered when developing the recommendation(s). This is particularly important for guideline developers, who wish to adapt existing guidelines to be used in specific settings with defined resources. Every effort should be taken to identify relevant published evidence (e.g., economic evaluations, such as cost-effectiveness analyses), or to perform such analyses during the guideline development or adaption process. Suppose no such information is available and there are no sufficient resources to perform de novo analyses. In that case, the guideline development group may seek input from an expert in economic analyses, or summarize the panel’s collective experience regarding the use of resources. High demands on resources (e.g., high cost, limited availability of an interventional technology, or lack of wide-scale expertise) may prompt guideline developers to provide a weak recommendation or even recommend against an intervention, even when it is more effective than the comparator.

Surgical and interventional experience and expertise may affect the outcome of an intervention. Studies comprising the background evidence of a recommendation may report on the experience and expertise of those performing the interventions. The guideline should specify the experience and expertise required to perform the recommended interventions (e.g., previous courses, hands-on training, previous experience with a specific number of procedures/interventions, training of operation room staff). If expertise in an intervention is not widely available in a given context or geographical region, the guideline panel may abstain from providing a strong recommendation. Alternatively, they may issue a strong recommendation in a given context, such as in the presence of interventional or surgical expertise, or after appropriate training. Such considerations will also inform the decision of guideline adaption groups to issue a strong or weak recommendation in a specific setting.

Editorial independence

The guideline should be independent of any type of financial, intellectual, or professional influence. The funding body (which includes for-profit or non-profit organizations) should not influence the content of the guideline, and its role should be transparent, specific and defined in advance.

All individuals who contribute to or influence the content of the guideline should disclose any direct (financial) or indirect (intellectual) conflict of interest. The Guidelines International Network recommends that the guideline chair and the guideline panel be free from either direct or indirect conflicts, and that professionals with expertise but with conflict of interest related to the interventions, diagnostic tests etc. discussed in the guideline may act as consultants, but do not participate in the decision process of the direction and the strength of the recommendation(s). A strong opinion favoring an intervention, course of action, diagnostic test etc., may also constitute an indirect conflict of interest. The guideline steering committee, the guideline chair(s) and/or the methodologist may need to reallocate functions and responsibilities, or exclude (from parts or the whole guideline) contributors with relevant conflicts of interest.

Conflicts of interest should be documented at least at the outset and upon completion of the guideline development process. Standardized declaration forms of peer-reviewed journals usually do not require declaration of indirect conflicts; therefore, a detailed statement may need to be made available by different means; e.g., in a supplementary file or file repository.

Implementation and update

The guideline development (or adaption) group should identify any potential barriers to the implementation of the recommendation(s), such as lack of wide-scale experience and expertise, limited resources (e.g., financial, technologies or instruments), resistance to change, organizational culture, limited awareness of the evidence, etc. Furthermore, the guideline development (or adaption) group may identify potential facilitators, such as cost-effectiveness, reduced requirement of resources, or ease of implementation. Both facilitators and barriers should be considered when developing or adapting the guideline. They may also inform the decision on the strength of the recommendation(s), e.g., through an evidence to decision framework [27]. Such information can be collected directly from stakeholders (e.g., the guideline panel) or through pilot testing of the guideline before widespread implementation.

The guideline should specify its validity period, which may inform the timing of its updating. This may be related to expected advances in the field, identification of ongoing trials, publication of new research, or accumulation of experience with new technologies. The guideline development group may need to perform a scoping search of clinical trial registries to identify ongoing trials, or they may map the trend of new publications over a specific time period, which may predict the publication of new evidence after a specific period of time.

The guideline development group may consider providing tools and resources to facilitate application, such as guideline summary documents, patient or public versions or lay summaries, treatment/management algorithms, how-to manuals, solutions linked to barrier analysis, tools to capitalize on guideline facilitators, the outcome of the pilot test and lessons learned, decision aids, or smartphone applications.

Assessment of the applicability, uptake and impact of the guideline should be among the objectives of guideline developers. This can be achieved through a survey of target users a period of time after publication, in order to appraise applicability, and to develop and measure indicators (e.g., reduction in the incidence of surgical site infection), related to key recommendations. This can be provided in a supplementary file or other resources.