Miss a lens at the long end.

I shoot a D7000 and I use a 16-85mm lens as a walk-around lens, a 50mm 1.4 D for low light conditions and a 105mm macro for my passion with flowers. I like portraits but I shoot mainly groups of persons and flowers with geared head and a rail when stacking. I am missing a lens at the long end. I am considering of course the 70-300 VR (500$), the 300mm F4 (1300$) or the 70-200 VRII (2400$) (in case I am swept into the D600 or even D800 crowd one day). Just writing those prices down makes me favour the 300mm F4 lens at the moment. Is this the sweet spot decision?