Savage Build Problem

So I have built a 260 Rem with a Savage 110 action. I used a Pac-Nor barrel. It has a Bell&Carlson Duramaxx stock. I have got the barrel broken in and now, shooting groups, it will shoot bullets where the holes are touching but every few shots, it send one about 4 or 5 inches high. I couldn't figure out what causes it. The barrel is completely floated. Could the recoil lug be the cause of this? It is the factory lug and it doesn't look like it makes good flush contact near the top of the action.

I am shooting 130 Grain Berger VLDs with 39 grains of IMR4350. The shooting I'm doing is at 100 yds. I have checked the scope mounts. They were tight. I have had good luck with vortex scopes in the past but I will give that a try too. Any more ideas?

Wow. I was gonna say it could be variation in factory load but that just got blown to Hades!

How's the stock fit around the bolt handle? I had a stick one time that was doing the same sort of thing. I found out that the handle was touching the stock a little if I didn't have the bolt fully down. I took a little sandpaper to it and it's shot straight ever sense.