(CNN) -- A Massachusetts man taken into custody in a law enforcement sweep after the Times Square bombing attempt has been ordered back to Pakistan by a federal immigration judge, a court spokeswoman said Friday.

Aftab Ali, also known as Aftab Khan, was ordered back to Pakistan by Judge Robin Feder, who issued a written opinion Thursday, said Kathryn Mattingly, spokeswoman for the federal Immigration Courts.

Khan was one of the men picked up in a terror sweep earlier this month after the botched May 1 car bomb attempt. He had a hearing last week in Boston Immigration Court, Mattingly said.

Mary Attia, an attorney with the firm representing the man, told CNN no decision has been made on whether to appeal.

Khan, his uncle Pir Khan, and a third man named Mohammad Rahman were picked up when search warrants were executed during a sweep in the Northeast. They are accused of having been involved in an informal money-transfer network that provided cash to Faisal Shahzad, the suspect in the car bomb attempt.

However, they may not have been aware of any terrorist plotting, authorities have said.

Pir Khan and Aftab Khan were picked up in Massachusetts; Rahman was detained in Maine. The three were being held on immigration charges.

Saher Macarius, another attorney for the Khans, said earlier this month they had no connection with Shahzad and told CNN his clients didn't know the suspect and did not give him money.

Macarius, whose practice is based in Framingham, Massachusetts, has said the three men had been booked to fly from New York to Pakistan on June 6, using the same route Shahzad had booked before he was arrested trying to leave the United States.

The lawyer said Pir Khan had booked his flight April 27 and had even informed immigration officials of his travel plans.

"If he was involved in anything shady, why would he wait until this explodes?" Macarius asked. "He would have left earlier."