Habanero peppers

I am having some trouble getting peppers on my habanero plants, which I
started from seed in late March. The plants are in full sun here in Palo
Alto, CA (which is in the San Francisco Bay Area). I have three plants
together, all are about 14 inches tall and appear healthy. They have an
abundance of flowers, enough that there is a sweet smell and bees are
attracted to them.
The problem is that the flowers simply dry up and fall off, never producing
fruit. If anyone out there has any idea why this is happening, and what to
do to get some peppers, I would appreciate the help. I am a mechanical
engineer, not a botanist.
Thanks,
Tom