That's why it is done.
I practiced this method for some time with a zen teacher.

Quote:

... and it drives me crazy! I eventually just get disheartened and stop.

Continue. - If you like.
Talk to you teacher.
Maybe change your "focus" of awareness from the outside to you yourself. This method is not about connecting or atuning to the surrounding. It is indeed about getting able to lock it out.
At least this was the point of my teacher then.