WASHINGTON (Reuters) - North Korea withdrew its invitation to a U.S. envoy who was headed to Pyongyang to request the release of imprisoned, ailing American missionary Kenneth Bae, the U.S. State Department said on Friday.

North Korea cancelled talks with Robert King, the U.S. special envoy for North Korean human rights issues, who was expected to visit Pyongyang on Friday and Saturday.

"We are surprised and disappointed by North Korea's decision," State Department spokeswoman Marie Harf said.

"We remain gravely concerned about Mr. Bae's health and we continue to urge the DPRK authorities to grant Mr. Bae special amnesty and immediate release on humanitarian grounds," the spokeswoman said in a written statement, referring to the North by its formal name, the Democratic People's Republic of Korea.

There was no immediate explanation for the decision from North Korea, which does not have diplomatic relations with Washington.

King's trip, announced this week as he was visiting U.S. allies Japan and South Korea, was seen as a potential signal of the start of a gradual thaw in relations between Washington and Pyongyang.

The State Department had termed the trip a "humanitarian mission" and played down any connection between Bae's release and diplomacy over the North's sanctioned nuclear weapons program.

Bae's sister in Washington state issued a statement saying their family was disappointed and worried about Kenneth Bae's health, but are "not giving up hope for a peaceful and timely resolution."

"We hold on to faith that DPRK and U.S. diplomats will resume talks soon, ultimately leading to my brother being released," said Terri Chung, Bae's sister.

Representative Rick Larsen, who represents the district where Bae's family lives, urged the North Koreans to free Bae.

"The North Koreans gain nothing from this course reversal. It is time to let Kenneth come home to his family and get the medical attention he needs," Larsen, a Democrat, said in a statement.

King secured the release of another Korean-American missionary, Jung Young Su, in 2011 as part of a trip to assess North Korean pleas for food aid.

FROSTY RELATIONS

Relations between Washington and Pyongyang have been frosty since the collapse of a deal in early 2012, when North Korea broke its promise to end its long-range rocket launches and prevented nuclear inspectors from examining its nuclear stockpile and production.

Nuclear talks involving the United States, China, Japan and the two Koreas have been deadlocked for five years, although Pyongyang in 2005 had signed on to a deal in which it would have frozen its nuclear program in exchange for economic and energy aid. North Korea has conducted three nuclear tests since 2006.

North Korea's human rights record has recently come under international scrutiny. A U.N. Public Commission of Inquiry in Seoul on August 20 was told by witnesses that public executions and torture are daily occurrences in the North's prisons.

Bae, 45, was sentenced to 15 years hard labor for attempting to overthrow the North Korean state by spreading anti-government propaganda, according to North Korean media. He has diabetes and his health has deteriorated since he was jailed.

North Korean state media said Bae started his plot to "topple" the country's government in 2006, a date that coincides with his own testimony about his arrival in China.

It accused him of spreading "false propaganda" and of bribing North Korean citizens in a bid to bring down the government.

Bae lived in a Chinese town that borders North Korea and worked for a tour company while undertaking missionary work inside North Korea.

North Korea says it permits religious freedom, but religious expression is tightly controlled in a state that acknowledges total loyalty to the Kim dynasty that has ruled for three generations. North Korea lands at the bottom of most independent surveys of freedom.

In online postings of one of his speeches on his missionary work, Bae described himself and a party he took to North Korea as "warriors for Christ" and told of holding a prayer meeting on a beach.

The postings have since been removed, as have all traces of Bae's involvement with a tour company operating out of China.

In a videotaped sermon, also removed from the Internet, Bae discussed bringing 300 people to a coastal town in North Korea to emulate the biblical destruction of the walls of Jericho.

Bae's family has acknowledged his deeply held religious beliefs but have suggested that his sympathy for North Korean orphans may have been behind his arrest.

(Additional reporting by Arshad Mohammed, Eric M. Johnson in SEATTLE, and Elaine Lies in TOKYO; Writing by Paul Eckert and David Chance in SEOUL; Editing by Paul Tait, Vicki Allen and Stacey Joyce)