Doubting the effects of Gödel's incompleteness theorems on Physics

It is sometimes said that Gödel's incompleteness theorems imply that the existence of a TOE is impossible.But I can't accept this.

Gödel's incompleteness theorems don't seem to be as broadly applicable as it is being applied in such discussions!
I also have read a book on mathematical logic and although I can't claim that i understood it in detail,I know enough to tell that Gödel's incompleteness theorems are talking about certain axiomatic systems which are somehow related to arithmetic.I don't know how to explain it,but it just seems they're pushing it so far!!!and I don't understand such ideas!

For any formal effectively generated theory T including basic arithmetical truths and also certain truths about formal provability, if T includes a statement of its own consistency then T is inconsistent.

It is said that the theory should include basic arithmetical truths!But why a TOE should contain such things???I see now reason!
And also,a TOE doesn't have to include a statement about its own consistency.There is no reason to include such a thing in a TOE!We will just have in mind,that its OK.

Even if I forget about the above arguments,I can tell that Gödel's incompleteness theorems are just restricting the idea of TOE,mathematically.I mean a physical theory is a bunch of thoughts which initiate some calculations.It doesn't have to be such rigorous and hard,it doesn't have to be that much formal.We can have our ideas in mind and do calculations and be happy for having a TOE,but still when we hand it to mathematicians,they just turn around saying "Mathematically,this formal system is not capable of explaining everything!"blah blah blah!!!...But who cares?!