Evidence based assessment/Step 4: Revise probabilities based on intake assessments

Medical disclaimer: This page is for educational and informational purposes only and may not be construed as medical advice. The information is not intended to replace medical advice offered by physicians. Please refer to the full text of the Wikiversity medical disclaimer.

With regard to the concern that the DLRs are influenced by the reliability, convergent, and discriminant validity, and validity of the collateral informants, this is all true. It also is already “baked in” the DLR. The DLR is an alternate form of effect size; in fact, it is possible to convert r or Cohen’s d into an area under the curve (which is a plot of paired coordinates of sensitivity and specificity, meaning that the DLR is tied at a deep level to the effect size -- Swets, Dawes & Monahan[1] talked about the slope of the line tangent to the AUC at a given threshold as a way of expressing this relationship). Why bother with a different effect size? Because DLRs make it possible to translate the assessment result into an updated probability for the individual client. Cohen’s d or Pearson’s r do not by themselves, and they would require even more arcane formula to get there (see Wiggins. 1973, Chapter 4, for example). The strength of the EBM approach to DLRs is that it establishes a common framework for translating all assessment results into a common metric that updates the probability for an individual client. If the reliability of the measure attenuates the observed effect size, then the DLR would be smaller (and the amount that it would change the posterior probability would shrink). If the convergent validity is higher, then the effect size would be larger (and the DLR would be more informative again). More subtly, if the discriminant validity is lower, then the scores would often be elevated for reasons besides the presence of the intended target. This would degrade the specificity, again reducing the AUC (or the Cohen’s d for the separation between the distribution of scores for the target condition versus the distribution for the comparison condition). Again, the net result would be a lower effect size, and necessarily a more modest DLR. At a deep level, the psychometric concerns are already summarized in the DLR -- with the same caveat that the DLR is a property of test scores or assessment results in the context of a particular sample and for a particular purpose… the same caveat that applies to all assessment per current standards (***).