Wal-Mart Stores Inc., Gap Inc. and other large U.S. retailers are nearing an agreement to establish a $50 million, five-year fund to improve safety conditions in Bangladesh garment factories, a person familiar with the proposal said.

The deal could be announced as soon as mid-July, but the details are still being worked out, said the person. The $50 million fund, a critical component of the deal, is contingent on the Bangladesh government meeting certain criteria that would ensure accountability for safety improvements,...