BEIJING—China is trying to bolster the legal basis for its maritime-security forces operating in contested areas of the South China Sea, threatening to complicate fraught relations with southern neighbors and drawing quick criticism from the U.S.

A new regulation enacted by the southern island province of Hainan requires non-Chinese fishing vessels wanting to operate in the South China Sea to first obtain permission from China's...