Methodological flaws

The 11 March 2015 NHMRC media release misinformed the public that [1]:

“there is no good quality evidence to support the claim that homeopathy is effective in treating health conditions”.

The NHMRC Information Paper refers to the homeopathic studies considered being 'poor quality' 40 times in 40 pages - strongly (over) emphasising this message; unusual in science reporting.

44 of 176 trials evaluated by NHMRC were categorised as ‘quality unreported’, meaning that it is unknow whether quality has been correctly assessed (as being poor) for a quarterof the evidence in the Homeopathy Review [2].

Instead of checking any of the original studies to fill in this large gap in the data (a relatively simple task), the HWC instead created an arbitrary ‘rule’ to "assume" these studies were 'poor quality’ [3].

NHMRC’s published finding of 'no good quality evidence' therefore represents mere opinion; it is not evidence-based and the public have been misled.

Making such large-scale 'assumptions' about data then making definitive conclusions on the basis of what amounts to guesswork, does not equate to 'rigorous' scientific analysis as claimed. This is a bizarre practice employed by an organisation that describes itself as (response to stakeholders, 22 Dec 2015):

“an established leader in the development of high quality, evidence-based health advice”

Use of an inappropriate method:

To assess the evidence, NHMRC decided to conduct an 'Overview', referenced against Chapter 22 of the Cochrane Handbook [4]. The Cochrane 'Overview' method is intended to summarise the findings of systematic reviews (SRs), which themselves summarise the findings of original studies. Thus, the Overview approach is a 'summary of summaries'. For this reason, the Overview approach is designed to evaluate high quality Cochrane Intervention reviews, or SRs of equivalent high quality.

NHMRC/ HWC identified 57 SRs for the Overview, summarising the findings of 176 studies. NHMRC did not report that it only rated 7 out of the 57 (12%) of the SRs it identified as being sufficiently robust to be suited to the Cochrane ‘Overview’ method referenced. An inappropriate method was therefore used for the Overview.

Freedom of Information (FOI) returns reveal that the first (terminated) review it conducted between April & August 2012 (see 'The First Review') did screen the data to excluded poor quality SRs, as per the intent of the Cochrane Overview method (emphasis added) [5]:

"The aim of this overview is to identify all well-conducted systematic reviews of homeopathy as an intervention (treatment) for any condition. …Exclusion criteria: Systematic reviews with major flaws (AMSTAR rating of 3 or less).”

Irrespective, for the second (Optum) review NHMRC proceeded to make definitive conclusions on the basis of secondary sources it predominantly rated as not fit for purpose. Because the SRs only summarised original studies, they were significantly limited by incomplete and/or missing data. While NHMRC acknowledged this 'limitation' (IP, p. 25), it did not quantify or report the major impact it had on the analysis (as detailed in this and the 'Procedural irregularities' section).

NHMRC did not apply the same caution when drafting definitive conclusions on the basis of these predominantly ‘substandard’, inconclusive sources. It is unclear how this constituted a 'rigorous' approach using 'internationally accepted methods for assessing health evidence' claimed.

Limitations of approach - incomplete & missing data:

Arguably the most serious ‘limitation’ NHMRC encountered in trying to apply the Overview method to SRs of insufficient quality was that essential data about the original studies described in the 57 SRs was either inconsistent across different SRs or entirely missing. This is because it is not the purpose of SRs to include the type of information required to assess primary studies (noted in undisclosed expert peer review feedback to NHMRC on 9 July 2013).

NHMRC acknowledged this as a 'limitation' of the Review methodology (IP, p.25) - but did not quantify in its report the major impact this ‘limitation’ had on the analysis, lacking transparency. For example, the SRs made no mention of the quality of 44 out of the 176 included studies (as above), which were then simply "assumed" to be not god quality..

By not publishing such quantitative analysis, NHMRC concealed the significant impact from public and expert scrutiny, which if disclosed would have raised serious doubts about the Review's credibility.

Expert peer reviewer criticism of NHMRC's approach withheld:

The 'Overview' approach was an unprecedented method in NHMRC evidence reviews, representing a 'short-cut' method - as it meant NHMRC did not need to retrieve or assess any original studies​. The NHMRC Information Paper (p.25) informed: “The overview was based on finding systematic reviews of homeopathy, rather than searching for all individual published studies of homeopathy."

This was criticised by two out of three expert reviewers in 2014. One reviewer strongly criticised the approach as a breach of established NHMRC best-practice - feedback that was not disclosed by NHMRC, despite publishing a dedicated 'Expert review comments' document (revealed through FOI returns) [6]:

'Overview' method intended to summarise and report the findings of SRs - yet were dismissed:

NHMRC ignored perhaps the most useful information provided by the SRs: their conclusions. Yet the main purpose of an 'Overview' is to summarise the findings of high quality SRs, to provide researchers with an 'overview' of research findings in a particular area. The NHMRC Information Paper (p.25) even informed:

"The advantage of this [Overview] strategy was to make use of the large amount of work that had already been done by researchers around the world in finding and assessing studies and to provide an overarching picture of the whole body of evidence.” (emphasis added)

Yet incongruously, the HWC dismissed the findings of all 57 SRs, despite 'the work having already been done' by SR authors who did assess the original studies and the purpose of an 'Overview' being to report their findings. Information Paper (p.11):

“The professional research group did not accept the conclusions or interpretations of the systematic reviews, but instead considered the included studies.”

​Thus NHMRC ignored the work done by the SR authors, yet re-assessed the original studies without retrieving or assessing any of them, instead relying on incomplete/ missing information about these studies as provided in the SRs. This is a bizarre way of assessing evidence, which does not align with any accepted research method, not even the Cochrane guideline referenced (Chapter 22 of the Cochrane Handbook stipulating:

"Cochrane Overviews [...] summarize existing Intervention reviews rather than find and summarize or synthesize original studies."

A number of the 57 SRs identified reported positive findings of the evidence on homeopathy in their assessment of the original studies, as one of NHMRC 2014 expert reviewers informed NHMRC in (undisclosed) feedback critiquing the approach (NHMRC FOI 2014/15 004-Section 62):​

​"Some systematic reviews conclude homeopathy is more than placebo (Cucherat et al 2000; Linde & Melchart 1998; Linde et al 1997; Kleijnen 1991; and many of the reviews in the Swiss report found a trend in favour of homeopathy)."

​Accepted NHMRC research methods abandoned midstream:

The NHMRC Information Paper announcing the findings of the Overview assured the public that in the Review NHMRC employed:

“standardised, accepted methods for assessing the quality and reliability of evidence” (IP, p.5); and

"When evaluating health evidence and drafting health advice, NHMRC uses a rigorous approach that has been developed by Australian experts in research methods." (IP, p.9)

​But this was not the case.

The Information Paper did not inform the community that midway through the Overview, NHMRC abandoned its established, accepted method of reviewing health evidence as intended in the original research protocol, once it became apparent that the secondary data it exclusively relied upon (SRs) was not sufficiently robust, due to ‘limitations’ associated with missing and incomplete data.

This fact was not mentioned anywhere in the Information Paper - only on p.20 of the Optum Overview Report - thus remaining unnoticed by the media.

The formal investigation into the NHRMC Homeopathy Review has also uncovered that almost none of the criteria NHMR applied to assessing the evidence were 'standardised, accepted methods' for reviewing evidence (see below and 'Procedural irregularities' section).

Adoption of arbitrary criteria never used before:

One of the most serious methodological anomalies associated with the Homeopathy Review was the creation of a unique concept of 'reliable evidence', which was underpinned by arbitrary criteria never used before or since by any other research group (including NHMRC).

​For a trial to be considered 'reliable', it had to have BOTH:

More than 150 participants - that dismissed 146/176 studies (83%) from the findings

Have an unusually high 100% quality rating (5/5 on the Jadad or equivalent using another scale) - or it was considered to be 'poor quality' - dismissing 25/30 of the remaining studies from the findings.

​NHMRC regularly funds and collaborates in trials with far fewer than 150 participants and no published Cochrane Overview (whose guideline NHMRC referenced) has used 'quality rating' scales as a threshold for trial 'reliability'; also accepting a quality rating of 3/5 on the Jadad scale as 'good quality'.

How the N=150 sample size exclusion threshold impacted the findings:

The following illustrates how the N=150 sample size exclusion threshold was used to dismiss the findings of positive, good quality trials in several medical conditions, to conclude 'no reliable evidence':

Figure 1: The full evidence base for individualised homeopathic treatment of children with diarrhoea

The way in which NHMRC assessed the evidence for isopathic treatments for allergic rhinitis (hay fever) (Figure 2) illustrates how NHMRC used the N=150 sample size threshold, alongside other unusual criteria, to dismiss good quality positive trials from being considered 'reliable' and hence being excluded from the findings:

A ‘good’ quality trial showing efficacy of Pollen C30(Reilly 1986) [7]Specifically, the homeopathy group in this trial had a significant objective improvement in overall symptom severity compared with the placebo group (p=0.02). Dismissing this trial as ‘unreliable’ only on the basis that ‘the findings of this study were not confirmed by other good quality,sufficiently sized studies’ is scientifically unjustifiable. Although replication of this study with another large trial would be advantageous, the fact that this has not yet happened does not make the Reilly et al. 1986 study in and of itself ‘unreliable’. NHMRC also fail to mention that this study followed on from, and confirmed the findings of, an earlier pilot study testing the same isopathic medicine (Reilly & Taylor 1985), which adds weight to the findings.

​

A ‘good’ quality trial showing efficacy of ‘individualised immunotherapy’(Taylor 2000) [8]Individualised immunotherapy involves giving each patient a homeopathic remedy made from the specific allergen to which they were sensitive e.g. house dust mite or cat hair. This RCT, which was published in the BMJ and has a quality score of Jadad 5/5, found that the homeopathy group had a significant objective improvement in nasal airflow compared with the placebo group (p=0.0001 - a highly significantly positive result).

The combined evidence from four RCTs of Galphimia glaucaConsidering that three SR authors who drew specific conclusions about Galphimia glauca [9,10,11,12] all concluded that there was evidence in support of the effectiveness of this remedy in the treatment of allergic rhinitis, this is a serious omission.

A ‘good’ quality study which found the homeopathic nasal spray Luffacompositum to be as efficacious as conventional cromolyn sodium nasal spray (Weiser 1999) [11].As this trial has a quality score of Jadad 5/5, one must assume that it was dismissed for having 146 instead of 150 participants. The lack of scientific justification for this threshold in general has already been established, but in the case of this particular study it is irrational.If NHMRC had read the original study they would have seen that the researchers carried out a ‘power calculation’ which showed that a sample size of N=144 would be needed for the trial to be sufficiently ‘powered’ to produce ‘meaningful results’; as the trial was N=146 it was of ‘sufficient’ size.

NHMRC’s conclusion that there is ‘no reliable evidence’ for the effectiveness of homeopathy for people with allergic rhinitis is inaccurate, denying the existence of good quality studies that suggest several potentially effective treatments for hay fever sufferers i.e. Pollen C30, individualised immunotherapy, Galphimia glauca and Luffa compositum nasal spray.

Therefore it is patent that NHMRC did not provide an accurate or fair assessment of the evidence on homeopathy for allergic rhinitis and its conclusion is highly misleading.

The same applies to several other medical conditions assessed (such as those indicated in Figures 1-4 above). Similarly misleading and inaccurate analyses of these and other conditions made by NHMRC/ HWC is detailed in the complaint to the Ombudsman.

Exclusion criteria dismiss the results of 97% of the data from any consideration in the findings:

​Neither the 'sample size' nor the 'quality rating' exclusion criteria align with any known 'accepted research methods' (either individually or in combination) - as NHMRC assured the public it used.

​Most damning, together these criteria dismissed the results of 171 out of the 176 (97%) Overview studies from being considered as part of the Review's findings (at all) - a fact not reported by NHMRC in over 944 pages of report documentation. The graphics at right show the impact that the N=150 exclusion threshold had on the evidence base in several medical conditions (see also 'Procedural' section).

Worse still, FOI documents reveal these criteria were developed and applied entirely post-hoc:

Research protocol retrospectively modified:

Another serious methodological/procedural issue identified during the investigation into NHMRC's conduct in the Homeopathy Review, was extensive changes made to the research protocol by the Office of NHMRC (ONHMRC)/ Homeopathy Working Committee (HWC) throughout the Optum review - well after the protocol had been "finlaised" in December 2012 and Optum commenced their review on 3 January 2013.

The Optum Overview Report (p.12) notes the protocol was further “developed through active consultation between the members of the HWC and the evidence review team” - with none of the specific changes to the protocol disclosed. The final version of the protocol was only officially confirmed in July 2013 and the method of assessing evidence continued to be modified up until August 2013 – well after Optum’s evidence assessment had already been completed.

Thus the changes were made retrospectively: after the original protocol had been finalised and after the contractor (Optum) had already completed its evidence assessment - removing probably the most important safeguard reducing/preventing reporting bias in scientific studies.

Making post-hoc changes to the research protocol is a serious ethical issue in scientific inquiry, as it exposes such processes to risk of bias (the intentional or unintentional tailoring of the outcome according to pre-determined expectations).

More details of these post hoc changes to the protocol are provided in the 'Procedural irregularities' section - summarised below (revealed through Freedom of Information returns referenced in the 'Procedural' section):

Dec 2012 - The original research protocol was agreed and finalised between the Office of NHMRC, the Homeopathy Working Committee HWC) and the contractor Optum.

​

Mar 2013 - Optum completed its evidence assessment.

​

Apr 2013 - A ONHMRC/HWC 'Sub-Group' was established to further refine the research protocol.

​

End Apr 2013 - The 'null hypothesis' approach was first introduced, where homeopathy was 'assumed to be ineffective unless 'reliable evidence' shows otherwise'. At this stage, the definition of 'reliable evidence' or the criteria that comprised it did not yet exist.

​

May 2013 - ONHMRC formulates an 'adapted GRADE' tool to assess level of confidence in the trial results (Element 2 of the Review's evidence statement criteria). This criterion did not pass either round of expert peer review feedback (Australasian Cochrane Centre); ignored and not disclosed.

​

May 2013 - ONHMRC/ HWC decided to add 'conclusive statements' (Element 3 of the Review's evidence statement criteria), instead of 'informing the community of the evidence' as intended in the original research protocol.

​

Late May/ Jun 2013 - The decision to set a minimum sample size threshold for trial 'reliability' (Element 1 of the Review's evidence statement criteria) was developed; initially set at 200 trial participants for a trial to be sufficiently 'powered'. At this stage, no concept of trial 'reliability' existed.

The research protocol is then further modified; the 'N=200/ adequately powered' criterion is abandoned; the 'N=150 sample size threshold for trial reliability' is introduced for the first time.​

​

11 Jul 2013 - HWC formally approves N=150 participants as a minimum sample size threshold for a trial to be considered 'reliable'; linked to a BMJ study - which actually warns against its findings being extrapolated to the types of studies assessed in the Review (suggesting the BMJ paper was not properly read).

​ ​Notably, all of these changes:

Were formulated by the ONHMRC/ HWC, not the contractor Optum (who implemented them)

Represented the criteria that underpinned the Review's published findings, none of which were part of the originally agreed research protocol

Are not accepted research methods - why they have never been used before or since by NHMRC or any other research group.

Changes to the research protocol not reported:

A pre-requisite of ethical scientific inquiry is that all changes to the research protocol must be fully disclosed and justified.

None of the key changes to the research protocol outlined above were reported or justified. Agreeing a protocol before a review process starts is a recognised safeguard against scientific bias, so making such significant post-hoc changes fundamentally undermines the credibility of the Review results. ​

Extensive study exclusion criteria applied to the Homeopathy Review:

The 11 March 2015 NHMRC media release announcing the findings of the Homeopathy review misinformed the public that the Review's findings and conclusions were:

"... based on the findings of a rigorous assessment of more than 1800 papers"[1]

whereas only 176 papers were included in scope, with the rest ignored altogether (see 'Inaccurate Reporting').

While only focussing on assessment of efficacy data (i.e. randomised controlled trials) is not unusual or unreasonable when reviewing health evidence, the following aspects were unusual:

NHMRC greatly exaggerated how much evidence was actually assessed in its reporting to the public, as if to create the impression that the Review was far more comprehensive than it actually was

NHMRC described the Review as a 'Health Technology Assessment', which by definition assesses parameters that were entirely excluded from scope (e.g. real-world clinical studies, safety, quality and cost-effectiveness)(see 'Procedural irregularities' section).

Applying further exclusion criteria to the 176 included studies; criteria that are not scientifically justifiable and have never been used before or since by any research group; also developed and applied to the data well after the research protocol had been finalised and the contractor Optum had already completed its evidence assessment.

NHMRC misled the public that its Review was 'thorough' and 'comprehensive' [1], when it wasn't.

Poor quality reporting - Lack of consistencies & errors:

Some minor inaccuracies due to human error are to be expected in a large report, but the pervasive lack of clarity, combined with the quantity of errors and inconsistencies throughout the Homeopathy Review call into question its fundamental credibility.

NHMRC’s failure to provide their quality assessments for each trial, nor to provide any details of their ‘level of confidence’ calculations using their ‘adapted GRADE’ method, have both been mentioned earlier; these reporting flaws need to be reiterated here as they are unusual and serious omissions.The inappropriate use of the Cochrane Overview method for the specific data set involved in the Review has been outlined above, but the extent of the errors created by this flaw becomes apparent when NHMRC’s summary of the evidence base for a specific clinical condition is examined closely:

Example: Sinusitis chapter in Overview Report:

For example, in the Sinusitis chapter (Optum Overview Report, p.203), which only describes three studies, we find that:

Two further trials covering sinusitis are missing, even though they are included in other chapters of the Overview report (Ammerschlager et al. 2005 Level III-2 study in ‘Rhinitis’74 and Steinsbekk et al. 2005 Level II study in ‘Upper respiratory tract infection’ [17])

The paper referred to as Weiser & Clasen 1994 [14] is from a non-peer reviewed journal and as such should not have been eligible for the review. The correct peer-reviewed version of this article was submitted to NHMRC during public consultation but was incorrectly excluded as ‘Study included within a systematic review in the Overview Report’ [18].

As the above errors all relate to a section of the Overview Report that is only one page in length, analysing the full extent of this problem across a Review comprising over 600 pages is beyond the scope of the complaint to the Ombudsman, however multiple examples identified throughout the Review show that this problem is not confined to the Sinusitis chapter, raising serious questions about the accuracy of NHMRC’s findings in general.

If the HWC had included a topic expert, as is usual in such Reviews, their specialist knowledge of the homeopathy research field would have significantly reduced such errors (see 'Procedural irregularities').

For an organisation that describes itself as “an established leader in the development of high quality, evidence-based health advice” [19], the lack of clarity, multiple inconsistencies and simple errors found in this report raise the question of whether such poor quality work was intentional or unintentional.​Certainly, the result is the production of a document with such obfuscation of the data that we contend it would not have passed peer review for external publication: the Homeopathy Review should therefore be set aside in its entirety, as academically it is not fit for purpose.

The inaccuracy of the NHMRCs conclusions are also reflected in another fundamental flaw at the heart of the report – i.e. considering the results of all trials for one condition together as a whole, even though the individual trials were assessing very different types of homeopathic treatment.

To illustrate this flaw, the NHMRC reviewers asked, “Is homeopathy effective for condition A?’, working from the premise that a positive trial showing that one homeopathic treatment is effective is somehow negated by a negative trial which shows that a completely different homeopathic treatment for that same condition is ineffective. This is a bizarre and unprecedented way of assessing scientific evidence (referred to as ‘heterogeneity bias’, as per Cochrane guidelines that NHMRC referenced).

In conventional research the question asked would be, “Is treatment X effective for condition A?”, not “Is conventional medicine effective for condition A?” based on combining the results of all drug trials together. Some treatments work, some don’t.

The whole point of medical research is to establish which treatments are useful and which are of no value. This is no different in homeopathy. Ignoring such a basic premise of scientific assessment is not a hallmark of rigour, but a limitation of the short-cut method adopted by NHMRC (due to no original studies being retrieved or assessed during the Review, with the secondary sources relied upon being limited by incomplete and missing information) [20].

​“If the intent is to provide general statements about the effectiveness of homeopathy, then ‘no reliable evidence’ may not adequately reflect the research. For example, when a substantial proportion of small (but good quality) studies show significant differences, [...] 'no reliable evidence' does not seem an accurate reflection of the body of evidence."- Australasian Cochrane Centre expert methodological feedback to NHMRC, 30 Aug 2013 (Freedom of Information return)

​“The dismissal of positive systematic reviews compounded with the lack of an independent systematic review of high quality randomised controlled trials leaves me uncertain of the definitive nature of the Report’s conclusions.”​- Expert reviewer feedback to NHMRC, May 2014 (Freedom of Information return)

"High quality RCTs with narrow confidence intervals (Level 1 evidence) should have been searched for and included in this review."- Expert reviewer feedback to NHMRC, May 2014 (Freedom of Information return)

“…it is important that the methods to be used should be established and documented in advance.… It is important, however, that changes in the protocol should not be made on the basis of how they affect the outcome of the research study. Post hoc decisions made when the impact on the results of the research is known, such as excluding selected studies from a systematic review, are highly susceptible to bias and should be avoided.”- Cochrane Handbook Chapter 2.1(‘Rationale for protocols’)