U.S. intelligence agencies and the Federal Bureau of Investigation have spent months trying to substantiate explosive claims, compiled by a former Western intelligence official, that Russian government operatives engaged in an extensive conspiracy with advisers to Donald Trump’s presidential campaign and employees of his company, people familiar with the matter said.

The unverified allegations—including a claim Russia has material that could be used to blackmail Mr. Trump—were deemed sufficiently significant by senior intelligence...