The problem that you are having is that the ground is not flat and your graphic is. What you need is an accurate 3D model of the ground that you can apply a projection mapped texture to. This is extremely difficult to do in AE. Let me explain with something as simple as a cube. Let's say you have a camera move around a cube and you want to add a graphic that wraps around the cube. The only way that you can accurately do this is to project or line up your graphic with each face of the cube. Inside AE we have a pretty good camera tracker. When you track your shot and then move over the tracking points you'll see targets at all kinds of angles. The tracking points on the side of a hill will give you a target that is laying on the side of the hill so it looks distorted or at an angle. Where the ground is flat you get a target that is at a different angle. What you need to do to get your graphic to stick to each point is to generate an accurate 3D model of the terrain then use projection mapping to lay the map over the ground.

There are two ways to generate the map. The first is to use a isometric view of the property (no perspective) like a satellite image from Google Maps. The second way is to draw your accurate from a still frame of the property from your footage and use that view as the starting point. The first way is better.

The second step is to generate an accurate model of the terrain using a program like Photo Scan which I have used for similar projects. Using about 100 frames of an aerial video you can generate a 3D model then apply a texture map to the model, then export only the map as a 3D model and then combine the footage and the 3D model of the map in AE using camera tracker.

You can also use a program called Syntheyes to basically do the same thing.

The long answer to your question is that it's going to take some more software and a bunch of research to do this project correctly. The short answer is that you cannot do this accurately in AE with the tools provided from Adobe.

Another Option would be to take a look at Blender or any of the paid 3D apps like C4D and do your camera tracking and 3D model and projection mapping inside that app. I have done some fairly complicated Camera Tracking and modeling in Blender.

I thought of another option. You could dig up a topographic map of the area and use that to create a displacement map for FreeForm Pro then use projection mapping to lay your map to the freeform layer and track that to the camera. Here's a Freeform Pro Projection Mapping tutorial... How To Dent a Garage Door VFX | Mettle