Using threshold analysis to assess the robustness of public health intervention recommendations from network meta-analyses: application to accident prevention in households with children under five | BMC Public Health

Cindy F. Cape

In both network meta-analyses, the most intensive intervention was identified as the most effective. The usual care intervention was identified as the least effective intervention in both of the networks, along with free/low-cost equipment interventions which was also identified as being least effective in the poison prevention network. For the poison prevention network, the results indicate that no distinct intervention could be recommended as the most optimal intervention, this is also illustrated in the study and contrast level threshold analysis. In the NMA, this is inferred by the credible intervals for the effect estimates and the overlapping intervention rankings. In the threshold analysis, this was reflected in the small thresholds identified in the analyses, which meant that a small change in the evidence would result in an alternative intervention being most effective. Furthermore, the intervention recommendation from the poison prevention NMA was not robust, as the effect estimate was sensitive to the level of imprecision in the evidence and potential bias. On the contrary, for the falls prevention network, there was a distinct intervention that could be recommended from the NMA, and the threshold analysis identified this recommendation as robust.

As recommended by Phillippo et al. 2018 [4], any studies with reasonably small thresholds need to be assessed for risk of bias by using the tools discussed previously. From the threshold analysis applied to the poison prevention network, there were 3 studies with thresholds less than 0.5; these were studies 6, 7 and 15. By referring to the study quality assessment in Table 2, these studies did not appear to be particularly at risk of bias and did not have any major issues with their quality.

A limitation of this work is that the published NMA examples only had a small number of studies contributing to each of the networks. There was little evidence for many of the intervention contrasts. As well as this, there was no distinct or clear intervention recommendation from the poison prevention NMA as all effect estimates contained 1, and the rankings overlapped. However, this example still illustrates the use of NMAs and threshold analysis in the context of public health and highlights that any recommendations from this example are not robust.

Threshold analysis allows researchers to identify and quantify the robustness of intervention recommendations from NMAs to any potential bias in the evidence. The use of this method provides researchers and policy makers with the confidence that their results from NMAs are robust to changes in the evidence that might be due to potential risk of bias or imprecision. It is important to note that threshold analysis does not investigate the presence or absence of any particular bias and does not make any assumptions on the type and source of the bias. Threshold analysis is more concerned with the implications, if there is any bias present, that such bias would have on the intervention recommendations and resulting decisions [4, 12].

There are several other tools available to assessing the quality of network meta-analyses and their results. The Grading of Recommendations Assessment, Development and Evaluation (GRADE), also formerly known as GRADE NMA, has been developed to assess the quality of evidence contributing to the intervention contrasts for every pair of interventions. The quality of evidence for each contrast in the network is rated as high, moderate, low, or very low across five areas: inconsistency, study limitations, indirectness, imprecision, publication bias. However, as networks become larger, loops of evidence become more complex leading to GRADE NMA becoming insufficient [6].

Another example of a tool to assess the quality of NMAs, is the recently developed CINeMA (Confidence in Network Meta-analysis) which is accompanied by user-friendly software. CINeMA, unlike GRADE NMA, can be used for any type of network [13]. Both GRADE NMA and CINeMA consider the plausibility of assumptions but do not give numerical indication of the certainty of recommendations from NMAs, which could be more useful for decision-makers and guideline developers [6]. However, we are not stating that one method here is better than the other, each method/tool has its own advantages and disadvantages and have different aims, which should be considered at the users own discretion.

Threshold analysis could be extended to incorporate GRADE judgements in the analyses, as seen in the paper by Holper 2019 [14]. The use of GRADE judgements alongside threshold analysis offers a qualitative judgement as well as quantitative. Threshold analysis could also be incorporated into a cost-effectiveness analysis to consider the robustness of decisions on the cost-effectiveness of interventions.

A further application of threshold analysis could be to components network meta-analysis. Component network meta-analysis expands on the NMA framework and splits the interventions into components to consider which combination of components is most effective. The interventions in the NMA assessed in this example consist of several components, for example, education, fitting, and home safety inspection, so it could be more appropriate to explore which combinations of these components, not just the ones observed, are most effective. As well as this, in recent literature, threshold analysis has been applied to continuous and binary outcomes. These methods could be extended to look at other possible outcomes.

There still should be some careful consideration when applying complex evidence synthesis methods to highly heterogeneous data, as threshold analysis is not a way to fix the issues that arise. The primary consideration with heterogeneity is that we should account for it appropriately rather than avoid complex analyses due to the arising issues. Heterogeneity is inevitable, especially in public health intervention appraisals. The use of advanced methods for evidence synthesis, including the appropriate account of the heterogeneity, can lead to more detailed and robust conclusions, which will improve research and aid the decision-making process [3].

https://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889-022-13377-5

Next Post

Samsung Galaxy Z Fold 4 leak points to a serious camera upgrade

We’re rapidly approaching the expected launch window of the Samsung Galaxy Z Fold 4 – some point in August, as far as we know – and a new leak sheds some light on the rear camera setup that we can look forward to from this flagship foldable. Seasoned tipster Ice […]