WASHINGTON—The awarding of a Department of Homeland Security contract to a company accused of defrauding the government is facing growing criticism from the losing contractor and U.S. lawmakers.

FCi Federal, the company that currently handles some of the agency's administrative work, filed a protest Thursday with the Government Accountability Office against the decision to hand the work over to the professional-services arm of U.S. Investigations Services Inc., the government contractor the Justice Department has accused...