We study quantum phase transitions in the Bose-Fermi Kondo problem, where a local spin is coupled to independent bosonic and fermionic degrees of freedom. Applying a second-order expansion in the anomalous dimension of the Bose field, we analyze the various nontrivial fixed points of this model. We show that anisotropy in the couplings is relevant at the SU(2)-invariant non-Fermi-liquid fixed points studied earlier, and thus the quantum phase transition is usually governed by XY or Ising-type fixed points. We furthermore derive an exact result that relates the anomalous exponent of the Bose field to that of the susceptibility at any finite coupling fixed point. Implications for the dynamical mean-field approach to locally quantum critical phase transitions are also discussed.