The New York Fed is refusing to tell investigators how many billions of dollars it shipped to Iraq during the early days of the US invasion there, the special inspector general for Iraq reconstruction told CNBC Tuesday.

The Fed's lack of disclosure is making it difficult for the inspector general to follow the paper trail of billions of dollars that went missing in the chaotic rush to finance the Iraq occupation, and to determine how much of that money was stolen.

The New York Fed will not reveal details, the inspector general said, because the money initially came from an account at the Fed that was held on behalf of the people of Iraq and financed by cash from the Oil-for-Food program. Without authorization from the account holder, the Iraqi government itself, the inspector general's office was told it can't receive information about the account.

The problem is that critics of the Iraqi government believe highly placed officials there are among the people who may have made off with the money in the first place.

And some think that will make it highly unlikely the Iraqis will sign off on revealing the total dollar amount.

“My frustration is not with the New York Fed, it is with the Iraqis,” said Stuart Bowen Jr., the Special Inspector General for Iraq reconstruction (SIGIR). “They haven’t been sufficiently responsive.”