* deny switching to fullscreen mode if the resolution is too high
and print a message to the console

Very good idea.

* use windowed mode as fallback option if we are already in
fullscreen mode and the new resolution is too high and print a
message to the console

Do you end up with a window bigger than the screen, then? Is there a
chance the user can escape from this situation, i.e. reach all parts
of the virtual screen to find the switch for setting the resolution?

Another option would be to simply display an "Out of range error"
across the screen, like a real monitor would do. Usually, operations
systems feature a protection against setting a resolution higher than
supported by hardware already (set back to a lower reolution after
some delay).