How AGREE-S was developed

This was a tripartite project (Guideline Assessment Project: Filling the GAP in Surgical Guidelines), which was designed in line with guidance for developers of health research reporting guidelines1. A protocol was developed by the steering group and was published in advance².

Through a systematic search of medical databases and the grey literature, we identified a sample of 67 surgical guidelines produced by surgical organizations with an international scope.

Assessment of these guidelines using the generic AGREE II appraisal instrument, suggested that 40% were of below-average quality and may therefore not be recommended for use. In exploratory analyses, we found that guidelines produced by organizations with high guideline output (at least 1 guideline per year) and organizations with a guidelines committee were more likely to be recommended for use. Furthermore, using the GRADE methodology increased the odds of a guideline to be recommended for use by 8 times (odds ratio 8.2, 95% confidence interval 2.5 to 26.3). We coded these findings into items, which were nominated for inclusion in the extension document.

In exploratory analyses, we have found that guidelines produced by surgical organizations with a high output and those produced by surgical organizations with a guideline committee have higher odds of reaching sufficient quality and being recommended for use.

More details on GAP I can be found here.

The second part of this project focused on statistically calibrating the AGREE II instrument in the context of surgical guidelines. Specifically, we used the assessments of the same set of 67 guidelines and we applied reliability analyses, correlation analyses, factor analyses and the item response theory, hereby tailoring the AGREE II instrument in the context of surgical guidelines based on statistical modelling.

Of note, the model suggested that items from the domain Clarity of presentation be moved to Rigor of development, and rearranging items among domains to improve reliability.

More details on GAP II can be found here.

The third part of the project consisted of a Delphi process of stakeholders to inform the development of the extension checklist. We invited professionals from a variety of surgical specialties, representing General Surgery, Thoracic Surgery, Vascular Surgery, Pediatric Surgery, Head & Neck Surgery, and Urology; representatives from America, Asia, and low/middle income countries; representatives from the GRADE Working Group, representatives from the Enhancing the Quality and Transparency of Health Research (EQUATOR) Network and from the Guidelines International Network (GIN); a surgical journal editor, a guideline implementer, a national authority representative, and a patient representative. Stakeholders were informed about the findings of the previous parts of the project (GAP I and GAP II).

In a round of open-ended questions, participants were surveyed to nominate items that they consider important to be included in the extension checklist. Their responses were grouped, and they were introduced in a Delphi process, together with the findings of GAP I and GAP II. Items that were approved by at least 80% of participants were candidate for inclusion. After two rounds of adaption and modification, 5 new items were nominated for inclusion, 1 AGREE II item was nominated for inclusion after modification, and 4 items were nominated for exclusion from the extension checklist.

In a face-to-face consensus meeting of the steering group, development of the extension checklist was informed by the findings across the three parts of the project.

More details on GAP III can be found in the below publications:

Pending

Links

The following organizations participate, support and/or fund the GAP III project.