I think of Cale as an androgynous name. I knew a girl Cale, and yeah at first her name made me think of a vegetable. I don't think I ever lost the vegetable association in my mind, but it wasn't a bad association. If anything, it seems to increase the "snappiness" and freshness of the name. She was the only Cale I knew, so it also seemed unique.

What if you used it as short for Caleb? You could still use Cale, and no one would say you "named your kid after a vegetable," if that possibility bothers you.