Judge Jed S. Rakoff on Monday rejected a settlement negotiated between the SEC and Citigroup over a $1 billion mortgage fund and said that he could not determine whether the $285 million settlement was “fair, reasonable, adequate and in the public interest.”