Command Tilde in Finder, prevent focus on Desktop at end of cycle

I begin focussed on A, then ⌘ + ~ to focus on B, finally ⌘ + ~ once more, and it still leaves B in front, but B is now slightly grayed out (same style as finder window A when it's behind other finder windows).

This 3rd location is focussed on the Desktop, how do I prevent the Desktop from receiving focus on ⌘ + ~? I only want actual Finder windows to receive focus?