I have the Manfrotto 322rc2. I am not sure where that sits in relation to the model you have, more upmarket or less. For me this works fine with D70 or D300 + 300mm/F4. I also use it with my 80mm Celestron spotting telescope (=long and heavy).

I can leave this kit set up for days and when I next look it's all still pointing where I put it. Is there perhaps a friction/tensioning screw on the model you have?

Although I like this ball head, actually I wish I'd bought a traditional pan and tilt head which would be far more suited to making panoramas - something I like doing.