Crazy to buy modular home just to renovate?

My husband and I have discovered we cannot finance the custom built house we've been dreaming of. We have this idea of buying a standard factory-built home (not a mobile, just a prefab house) and then poking away at it over the years to make it something we can be proud of, all the while having a place to live on our rural property. Are we crazy? Will a structure of this quality stand up to time and investment over the years to improve finishes, fixtures, insulation, windows, etc.? We are NOT planning to add on in ways that would challenge the engineering, just gut as far down as we want to create a really nice living space. Has anyone done extensive reno on a factory-built, not mobile home (saw the mobile home thread).