WASHINGTON—U.S. regulators have spent the past six years forcing banks out of businesses seen as risky. Now, they are beginning to worry that they have pushed some financial activity into the shadows and outside their legal reach.

Regulators on the Financial Stability Oversight Council on Wednesday said they are monitoring new practices by nonbank financial firms—including mortgage-servicing companies, insurers and asset managers—over...