RAFAH, Gaza Strip — Three brothers from this dusty Gaza border town paid smugglers to spirit them through underground tunnels across the border into Egypt. There they joined the Sinai affiliate of the Islamic State, which is battling the Egyptian Army in the Sinai Desert.

One brother was killed about 18 months ago, at the age of 20. Last week the eldest, Hamza al-Zamli, 25, showed up in a shocking video, railing against Hamas, the Islamic group that dominates Gaza, and describing its fighters as “apostates.”

In the finale of the 22-minute production, Mr. Zamli, a firebrand with long hair flowing from a black turban, instructs another fighter clad in camouflage to shoot to death a kneeling captive accused of smuggling weapons to Hamas.

The video exposed new levels of enmity between Hamas and the Sinai branch of the Islamic State, injecting another layer of instability into an already volatile region. And it has roiled Gaza, prompting two families whose sons are shown in the video to disown them.

The video accuses Hamas of betraying Palestinians by imprisoning extremists in Gaza, failing to prevent the American recognition of Jerusalem as Israel’s capital, and being supported by Iran.

It urges attacks on Hamas’s members, courts and security positions, as well as on Shiites and Christians in Gaza, according to the Washington-based SITE Intelligence Group, which monitors Islamic State propaganda.

Tensions have long simmered between the two groups, even as common interests have led to cross-border cooperation in the past, particularly in arms smuggling, according to officials and experts in the region. But in declaring war against Hamas, the Sinai group has surrounded itself with enemies — Egypt, Israel and now Hamas — and given Hamas a common cause with Israel.

One of Hamas’s main crimes, the Islamic State argues, is its participation in Palestinian elections, which the Islamic State views as putting man-made law above God’s law. Another factor in the dispute is Hamas’s efforts to improve relations with Egypt as that country tries to broker a reconciliation deal between Hamas and the Palestinian Authority, its rival based in the West Bank.

As part of that effort, Hamas has been tightening control along its border with the Egyptian Sinai in recent months, constructing a buffer and installing cameras and barbed wire.

The group has since become one of the Islamic State’s most effective local affiliates. It downed aRussian jetliner in 2015, killing 224 people, and appears to have been responsible for an attack on a Sufi mosque in north Sinai in November, killing 311 people in Egypt’s worst terrorist attack.

Hamas is essentially a Palestinian national movement whose main effort is directed against Israel. It has periodically cracked down on more extreme jihadists in Gaza — who are ideologically closer to the Islamic State and Al Qaeda — including in a recent wave of arrests as extremists fired rockets into Israel to protest President Trump’s Jerusalem move.

Islamic State sympathizers argue that those arrests served only Israel.

Salah Bardawil, a senior Hamas official, described the video as a “Zionist” production. Another senior Hamas official, Mahmoud al-Zahar, said the Islamic State’s Sinai branch “does not want there to be weapons in Hamas’s hands to resist the Israeli occupation.”

Generally, though, Hamas has remained tight-lipped about the video, not wanting to draw more attention to it. Families whose sons have joined the Islamic State are reluctant to talk about them for fear of repercussions from Hamas, which dominates the Palestinian coastal territory.

“There is an undeclared war between Hamas and Daesh,” said Mkhaimar Abusada, a political scientist at Al Azhar University in Gaza, using the Arabic acronym for the Islamic State. “These are guys who disassociated themselves from Hamas and joined Daesh because they disagreed with Hamas’s participation in the 2006 elections. They don’t like Hamas’s behavior as it doesn’t enforce Shariah” — Islamic law — “and there are aspects of corruption regarding its rule in Gaza.”

Ehud Yaari, an Israel-based fellow of the Washington Institute for Near Eastern Policy, said that in the past Hamas had provided the Islamic State’s Sinai branch with training and advanced weapons, and had allowed wounded fighters to come to Gaza for treatment. The recent shift, Mr. Yaari said, was “a typical story of Middle Eastern changing alliances.”

In Gaza, the video’s horror is intimate.

Mr. Zamli, the video’s main figure, grew up in the Shabora refugee camp in the Palestinian part of Rafah, which straddles the border with Egypt. The man who fired the pistol, Muhammad al-Dajani, from Gaza City, was a Hamas fighter who defected to the Islamic State from Hamas’s military wing. A Hamas official in Gaza identified the accused smuggler as Mousa Abuzmat and said he came from the Egyptian side of Rafah, just across the border.

“It pains me that bloodshed was committed by my own son, Hamza,” said Mr. Zamli’s father, Adel al-Zamli, 51, a teacher and an imam in a local mosque. Speaking in a room lined with religious books, he said Hamza had taken his younger brothers “to perdition.” Hamza’s mother, Mariam, said that after seeing the video, she was sorry she had given birth to him.

Soon after the video emerged, the Dajani family issued a statement saying they were proud of belonging to Hamas and its military wing and describing their son’s act as “criminal” and “contrary to our religion and our people’s values.”

The Zamli brothers’ journey to the Islamic State began about three years ago when the two older ones left for Sinai. They had been jailed in Gaza for stealing a gun, and had embraced an extremist ideology, their relatives said. Hamza left behind a wife, whom he divorced, and a daughter. After Hamza’s brother Muhammad was killed, another one, Ibrahim, 19, left Gaza and joined him to fight in Sinai about nine months ago.

Muhammad al-Dajani, 24, the man seen pulling the trigger in the video, had been a member of Hamas’s military wing. He left a wife and two daughters in Gaza and made his way to Sinai about four months ago.

“He is no longer my relative,” said his uncle, Hosam al-Dajani. “He is a stranger to all the Dajani family. We hope to kill him with our own hands.”