I do most of my development on a HP slimline, with a max display resolution of 1024 x 768. The college that I attend have displays that are 1280 x 1024. And then I know there are other graphic modes that display different resolutions.

I know that during design time, it is possible to set the current game mode using the GUI option (F10) but I would like for the 'torquedemo.exe' program to detect what video card the gamer has on his system, determine what graphics mode (800 x 600, 1024 x 768, 1280 x 1024) the card supports, then set the game to use the best graphic mode!

You can see there, that it checks for shader version, if the display driver is intel and the amount of video memory.
Maybe you can build something based on that, if not, you have to look for more variables the engine can look up, or build some on your own.
To detect the resolution before launch, you have to run that function on launch obviously, which can make problems, thats probably why Torque launches by default in 1024x768 and then lets you increase it manually.

For now I would just suggest setting it to the lowest resolution the machines can have you want to run it on and then let the users adjust it later if they want bigger resolution.