Background: Diabetes remains a major health problem worldwide. Low-risk low-cost alternatives to pharmaceutical interventions are needed where lifestyle modifications have failed. We conducted a double-blind randomised placebo controlled trial to investigate the efficacy of a Chinese herbal formula, Jiangtang Xiaozhi, in treating impaired glucose control and insulin resistance in persons with prediabetes and controlled diabetes. Methods: Seventy-one patients with prediabetes or 'controlled' diabetes were randomised to receive 3 capsules of Jiangtang Xiaozhi (n = 39) or placebo (n = 32) three times daily for 16 weeks with a follow up eight weeks later (week 24). The primary outcome was change in glycaemic control as evidenced by fasting blood glucose (FBG), post-prandial plasma glucose and glycosylated haemoglobin (HbA1c). Other measures included change in fasting insulin, insulin resistance and sensitivity, lipids, C-reactive protein (CRP), body mass index (BMI), waist girth, blood pressure (BP), health related quality of life (HRQoL) and safety. Analysis of covariance (ANCOVA) was used to model outcomes at 16 weeks, by treatment group corrected for baseline level of the outcome variable. Results: In patients receiving Jiangtang Xiaozhi, FBG was not significantly different (p = 0.73) compared to placebo after 16 weeks of treatment (6.3 ± 1.1 mmol/L vs 6.7 ± 1.3 mmol/L). There was a significant difference (p = 0.04) in the mean levels of fasting insulin between the treatment group (11.6 ± 5.5 mmol/L) and the placebo group (22.1 ± 25.9 mmol/L). Insulin resistance slightly decreased in the treatment group (1.58 ± 0.74) compared to that of the placebo group (2.43 ± 1.59) but this change did not reach statistical significance (p = 0.06). Patients taking Jiangtang Xiaozhi had a significant improvement in high-density lipoprotein (HDL) level compared to the placebo group at week 16 (p = 0.03). Mean levels of cholesterol, triglycerides, BMI, waist-girth, HRQoL, BP, CRP and insulin sensitivity were not significantly different between the two groups. The herbal medicine was well tolerated. Conclusions: In the current study, the 16 week Jiangtang Xiaozhi treatment did not lower fasting blood glucose, but it improved serum insulin and HDL cholesterol in a Western population with prediabetes or controlled diabetes. Our trial may have been underpowered. Dosage needs to be considered before commencing a longer adequately powered trial. Trial registration: Australian New Zealand Clinical Trials Registry ACTRN12612000128897; https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=362005.