Does this setup cool the motor enough?In that setup the air will not flow completely around the motor. Most of the cooling will be at the top of the motor and the air then gets diverted away from the motor so that there is little cooling on the bottom of the motor.An enclosure that directs the air flow completely around the motor would be better.

That enclosure will help cool the motor better than no enclosure since all of the air flow is forced to stay close to the motor. In you first drawing most of the air will be deflected by the motor so that the lower sides and the bottom of the motor does not get air flow.

I assume this is a brushless DC motor as commonly found in electric RC applications?

Remember that they do run a lot hotter at low speed /high current operating points than at higher speeds where the current is reduced. They will run all day at full power if you let them spin quickly, but pull down the speed and they can overheat or overload the motor controller.

Forgive me if I am pointing out the obvious, but I have seen students burn out motors / controllers because they didn't consider this.

If you are still worried about overheating maybe you can use special rounded heatsinks designed for motors . You can find them in hobby shops .Also you can try using a peltier module to cool it , if it's an inrunner that is

You could conceivably machine or cast appropriate-size round heat sinks yourself, using thermal compound to fill in the slight gaps, but that seems like a lot of work :-) If you can enclose the motor and move a bunch of air over the extent of it, that will probably work well (as long as you don't overtorque the motor way too much!)