The difference is, whether you take the total (5.138 x 15) and assign a percentage to it, or if you multiply the first half of the equation, 5.138, by 15%, which gives you the lower number.

Q4: What is this spawning equation applied to? Is it the possibility of finding one or more creature groups per entire terrain square, or of finding a group per individual section of a 20 x 20 terrain area you search in?

Q1: Yes
Q2: One would have to assume
Q3: It would be multiplied in that example by .15, but that final number is a percentage in decimal form, so it's still 77%
Q4: I think it's for the creature spawning somewhere in the terrain square, seems to make the most amount of sense

I can see the need to remove creatures once you leave a 22x22 square, for the very reasons you state. And since it appears that your NPCs stop following you when you're more than a square away, I could see the creatures being "retired" when you leave the square.

I'll post back here if anything unusual happens with creatures in my wanderings.