I'm working in a newly installed version of Captivate 4. I am going into edit preferences for the recording mode. I am changing the default from demonstration, to custom, or training simulation. (I've tried both.) Even if I resave my project immediately after, if I try to record, it records in demonstration mode instead of what I had selected. So I went back in and changed the project back to training simulation, and closed the edit preferences box. When I immediately reopened the edit preferences box and looked, it had defaulted back to demonstration mode. I know that I am not accidentally hitting the restore defaults button. How can I get this to record in anything other than demonstration mode?

Unfortunately it would appear that you are misunderstanding how these actually work.

When you click Edit > Preferences > Recording > Modes you see a list of different modes that you may record in. Think of each of these modes as being similar to a recipe. You configure the ingredients used in the recipe here. So later, if someone wants to use the Demonstration recipe, the ingredients are what you configured in this area.

Now, when you prepare to record your project, you choose the recipe you wish to use by clicking the icon in the gray strip. This should produce the flyout menu where you may choose the recipe you modified earlier.

"......but still didn't understand the "recipe" concept in the Edit/preferences section."

You might have noticed that various modes of Automatic Recording i.e. Demonstration, Assessment, Training and Custom differ from each other on the basis of objects(for eg - captions, mouse, click boxes etc).

In the Edit-> Preferences, Recording:Mode(s), you can configure what objects you want your recording session to generate for a selected mode.