The Case of the Blind Allocator

In the modern world of evidence based medicine we exist in a perpetual state of doubt, continually attempting to perceive truths through the veil of science. Far too often our sample cohort deviates from the population it intends to represent. Hypothesis testing and frequentist statistics are tools intended to quantify the extent to which the observed results are due to random errors in sampling. And yet, there is an entirely different type of error that our statistical instruments are far less adept at appraising. This non-random form of error comes in the form of bias. This post will explore a number of common forms of bias and their extensive effects on data.

Despite the long standing belief that central venous catheters (CVC) placed in the femoral vein are at increased risk for catheter-related blood stream infections (CRBI), recent evidence has suggested that in the modern era of sterile insertion practices, the rate of line infections due to femoral catheter placement is no greater than cannulation of either the internal jugular (IJ) or subclavian (SC) veins.

In 2012, Marik et al published a paper in Critical Care Medicine with the intention of demonstrating this very assertion (1), conducting both a systematic review and meta-analysis of the existing data comparing the rates of CRBIs associated with each respective insertion site. The authors examined data of 17,376 central catheter insertions from 10 publications and concluded there was no difference in the rate of CRBI between the femoral inserted lines and their cephalad comparators. The relative risk cited was 1.02 (95% CI 0.64–1.65, p = .92) for femoral compared to SC, and 1.35 (95% CI 0.84–2.19, p = .2) when compared to IJ. Over 17,000 observed catheters insertions demonstrated no statistically significant difference in the rate of CRBIs between the various catheter insertion sites. And yet, despite the robust nature of its sample size, the validity of this meta-analysis has been questioned mostly due to the quality of the underlying data. Only 1,006, a small fraction of the total catheters placed in this analysis, were from RCT data. Of which, the majority of these originated from a single trial examining emergent dialysis catheters placed in either the femoral or internal jugular vein, which found no significant difference in the rate of central line infection between these two sites (10). But it is unclear if dialysis catheters, which are kept impeccably clean and accessed only for dialysis, translate to the heavily exploited standard CVCs used in the critically ill.

Observational cohorts, totaling 16,370 catheters, accounted for the remainder of the data in the Marik et al meta-analysis. When comparing outcomes between groups, observational data presents a number of methodological problems. In this instance, the location of catheter placement was not randomly assigned. Leading to an immense potential for selection bias, as the factors that determined site of cannulation may directly influence the likelihood that the catheter becomes infected. For example, in patients with severe respiratory distress, the cannulation of the SC vein may be avoided due to a fear of causing a pneumothorax. This leads to the placement of IJ and femoral catheters in a sicker subset of patients who are, in turn, at a greater risk of infection. Additionally, due to the pre-existing bias of many clinicians, femoral lines may have been removed earlier than either IJ or SC lines. The risk of central line infection is directly related to its time in situ, and thus their abbreviated use may underestimate the true risk of infection associated with femoral venous cannulation.

To further complicate matters, Marik et al eliminated two large trials from their analysis claiming they were statistical outliers (1). Although such a deletion may be statistically appropriate, the redacted trials demonstrated a far higher rate of line infections when the femoral site was utilized (2,3). When these trials are included in the analysis, the difference in the rate of CRBIs between the femoral and IJ insertion sites becomes statistically significant.

Essentially, there are too many confounding variables to be able to clearly interpret the data utilized in the Marik et al meta-analysis. Mathematical manipulations of this data, in the form of regression analyses, do not clarify the matter. This type of error is difficult to correct through statistical modeling and can only truly be controlled using randomization. Randomization accounts for confounding variables by randomly distributing them amongst the study arms. When implemented correctly, one may assume the observed differences are caused by the treatment effect in question.

A recent trial published in the NEJM sought to do just that. Parienti et al examined 3,471 catheter insertions in 3,027 patients in ten ICUs throughout France (4). Lines were inserted by “experienced” house staff, each required to have at least 50 previous line insertions. All lines were inserted using strict sterile precautions and Seldinger technique, though the use of ultrasound guidance was left to the inclination of the clinician performing the procedure. Patients were enrolled if the treating physician determined that at least two of the three sites (IJ, SC, or femoral) were appropriate for cannulation. At which point the patient was randomized to site.

The authors found a significant difference in their primary outcome, the rate of catheter-related infections and symptomatic deep-vein thrombosis, between the patients randomized to undergo SC line placement when compared to both IJ or femoral placement. Overall there were 8, 20 and 22 events in the SC, IJ and femoral sites respectively, which translates to 1.5, 3.6 and 4.6 events per 1000 catheter-days respectively. This was offset by an almost identical increase in the rate of mechanical complications (arterial injury, hematoma, pneumothorax or other), observed in patients randomized to the SC insertion site when compared to both the femoral or IJ groups (2.1%, 1.4% and 0.7% respectively) (4). This difference was made up entirely of an increase in the rate of pneumothoraxes observed in the SC group. And yet despite the randomized nature of this trial, the methodology utilized by Parienti et al makes interpretation less than straightforward.

As discussed, the major flaw in the Marik et al meta-analysis was the fact that the majority of the data was obtained from non-randomized cohort data, making it extremely difficult to account for the confounding variables that might have influenced site selection. Ideally randomization should eliminate these biases. Unfortunately because of a number methodological concerns, the Parienti et al trial failed to control for bias as well as we would have hoped.

For randomization to be valid, it is vital the participating clinicians are not aware of patient group assignment prior to randomization. This is what is called allocation concealment. Prior knowledge of such events will lead to a selection bias, as there is a tendency for clinicians to exclude certain patients based on their own beliefs regarding the validity of the treatments being examined (5,6). For example, a patient with severe respiratory distress may not be enrolled in the trial if the physician had prior knowledge that the patient would be randomized to SC site insertion, primarily due to potential for pneumothorax. Improper allocation concealment will exclude a certain subset of patients and produce results that systematically deviate from reality (6). Although Parienti et al did attempt to conceal allocation prior to randomization by the utilization of a permuted-block randomization with varying block sizes, they allowed the treating physicians to exclude one site prior to randomization, if it was deemed not suitable for clinical use. This allowance was probably unavoidable, as it is not uncommon for one or more vessels to be inaccessible in clinical practice, but this concession allows for the introduction of the very selection bias we were hoping to avoid through randomization (6).

Of the 3,471 catheters placed, 2,532 (72.9%) were placed in patients in whom all three sites were deemed accessible. This leaves 940 catheters (a little more than 25%) that were placed in patients in which the treating clinician had eliminated one site prior to randomization. The majority of these exclusions (570) were of the SC site, because the treating physician felt the risk of pneumothorax or bleeding was unacceptably high. Another 277 of the exclusions were of the femoral site, 45% because of “site contamination”. These exclusions potentially prevented the highest risk patients from being randomized into the SC and femoral insertion sites, leading to the very type of bias found in the observational data in the Marik et al meta-analysis we hoped to eliminate.

A further source of bias in the Parienti trial, can be traced to its inability to blind practitioners to the treatment group after allocation. For obvious reasons such blinding would have been unfeasible in a trial such as this, but it does allow for the introduction of yet another source of bias. When RCTs lack adequate blinding, the risk of ascertainment bias is prominent. Ascertainment bias is the systematic, non-random distortion of the measurement of the true frequency of an event because of the investigator’s knowledge and assumptions of the group allocation (7). In this case, patients randomized to the femoral site had their CVC in place significantly shorter than patients randomized to either the SC or IJ sites (mean catheter days approximately 5.9 +/- 4.8 for femoral and 6.5 +/- 5.4 for IJ and SC). Since risk of infection is directly related to length of catheter duration this difference could potentially skew the results in favor of the femoral site.

The authors attempt to control for these confounders through the use of regression analysis and analyzing catheter events per catheter day, rather than per insertion. Just as we discussed regarding the Marik et al meta-analysis, these types of statistical compensations cannot support such methodological frailties.

Despite its flaws, Parienti et al have gathered the largest, most complete data set in existence addressing the complication rate of CVC insertion. I suspect their results are as close a proximity to the truth as we currently have. As such, if we are willing to accept the slight increase in the rate of pneumothorax, the SC vein may be the preferred initial option for central venous cannulation, with the caveat that the true pneumothorax rate might be higher than observed due to the large number of exclusions prior to randomization (4).

So often in the interpretation and translation of medical literature we find ourselves lost in the statistical minutiae, citing p-values and confidence intervals as if they hold intrinsic value. And yet these statistical manipulations are for the most part concerned with quantifying the extent the results observed are due to random chance. Their mathematical constructs cannot account for the non-random error caused by methodologic missteps. Collecting data in the face of these flaws and attaching a statistical judgment to the results does nothing to legitimize its validity.