That "wax stuff" is the seal. It sounds like your replacement seal is the newer waxless ring. The wax ring is still the seal of choice by many. I believe your problem is just what you suspected, the flange is not anchored and the toilet is moving. The reason the flange screws are not holding is the floor is rotten. The toilet must be anchored to the floor, so what you need to do is cut the flooring back to solid wood, inspect the subfloor, and replace anything that is rotten. You'll have to fit the flooring around and under the flange in pieces like a jigsaw puzzle. Put a new layer of whatever you have for a floor covering under the flange and over the new wood. It doesn't have to match or be a professional fit on the seams, just close. Fill any spaces in the fit with rock hard putty. Screw the flange to the floor with screws long enough to go into the subfloor. (I like stainless steel, but brass is OK) The outlet on the toilet is made a bit smaller than the wax ring. When you set the toilet on the ring, rock it gently and push down to make sure it seats into the ring. Tighten the nuts on the flange bolts snug, but don't crank on them. The toilet edges should be resting on the floor so the bolts just hold it from moving.