False Discovery Benjamini-Hochberg

False discovery using standard statistical methods is a perennial headache.

Indeed false discovery has been blamed for the non-repliccability of many studies.

The Benjamini-Hochberg, BH, procedure is widely recommended to help solve this problem

This blog provides an EXCEL spreadsheet to estimate the number of ‘true’, i.e. after correction for alse discovery using BH procedure. It also rpesents a behavioural scenario example of the dilemmas posed by false discovery.

Design

Data analysis

Binary logistic regressions are conducted separately for all 3200 units.

Response is valence, V. V = 1 if response is agree or strongly agree, V = 0 for all other responses. Predictor is Year, confidence level p is .05, 2-tailed.

Results

No control for false discovery

352 (10.9%) units had change significant at 95% confidence level

215 units improved, 137 units declined

Benjamini-Hochberg Control for false discovery

The Benjamini-Hochberg procedure with false discovery rate q =.05 gave p = .0008 as the corrected decision criterion. This led to inference that

7 (.2%)units had changed 4 units improved, 3 units declined

Conclusions?

At the organisation level, the change was clearly, negligible after correcting for false discovery.

BUT

What about the 347 individual units with significant change at the p =.05 level, but no change at the p = .0008 ‘protected’ level? At the single individual unit level, should assessment be affected by what did, or did not, happen in the other 3218 units?

Comments & advice welcome

Advertisements

Share this:

Like this:

2 Responses to “False Discovery Benjamini-Hochberg”

Thank you for this excel calculator, but shouldn’t the instructions be:
1. Clear p-values from column A
2. Insert your p-values in column A (not C)
3. Sort in ascending order column A
etc. and the B-H adjusted p- value will appear in cell G3