The word “herbal”

While working on a project that involved translation (but was not itself a translation project), I found out that the formulas prescribed by traditional Chinese medicine practitioners are called “herbal remedies” in English.
The other day, I happened to want to use a bag of off-the-shelf medicine that I had bought earlier, so I opened the bag, and—surprise—the first thing that popped up in my mind was “But this is not herbal!”
While I knew there to be non-herbal ingredients in some formulas (say gypsum), I had always thought that all the plant-based ingredients used are herbal. (Let’s say I knew some of these “herbal” ingredients are fruits and somehow I never had a problem with that.) Yet what I saw were pieces of wood—definitely plant-based but not herbal.
Yes, in Chinese these formulas can indeed be referred to as “herbal medicine”. But is it not true that there is a certain element of inappropriateness when we continue to call them “herbal” in English?