WASHINGTON—Malicious cyberattacks increasingly are aimed at core infrastructure of the securities markets and could present risks to the entire financial system, according to a research report released Tuesday by a group of international securities regulators.

The report, by staff of the International Organization of Securities Commissions, warns that cybercrime has become significantly more sophisticated, making it more challenging to defend against. Hackers are increasingly focusing on destabilizing attacks, particularly...