Of course, you could and should improve the fertility of at least some of that soil. First, have your soil tested. If the test reveals major deficiencies, correct them with organic fertilizers such as fish meal (for nitrogen), bonemeal (for phosphorus), and greensand (for potassium). From then on, use compost and cover crops to maintain your soil's fertility.