An Azov battalion soldier stands next to an armoured personnel carrier at a checkpoint in Mariupol on 4 September. Photograph: Vasily Fedosenko/REUTERS

"I have nothing against Russian nationalists, or a great Russia," said Dmitry, as we sped through the dark Mariupol night in a pickup truck, a machine gunner positioned in the back. "But Putin's not even a Russian. Putin's a Jew."

Dmitry – which he said is not his real name – is a native of east Ukraine and a member of the Azov battalion, a volunteer grouping that has been doing much of the frontline fighting in Ukraine's war with pro-Russia separatists. The Azov, one of many volunteer brigades to fight alongside the Ukrainian army in the east of the country, has developed a reputation for fearlessness in battle.

But there is an increasing worry that while the Azov and other volunteer battalions might be Ukraine's most potent and reliable force on the battlefield against the separatists, they also pose the most serious threat to the Ukrainian government, and perhaps even the state, when the conflict in the east is over. The Azov causes particular concern due to the far right, even neo-Nazi, leanings of many of its members.

Dmitry claimed not to be a Nazi, but waxed lyrical about Adolf Hitler as a military leader, and believes the Holocaust never happened. Not everyone in the Azov battalion thinks like Dmitry, but after speaking with dozens of its fighters and embedding on several missions during the past week in and around the strategic port city of Mariupol, the Guardian found many of them to have disturbing political views, and almost all to be intent on "bringing the fight to Kiev" when the war in the east is over.

The battalion's symbol is reminiscent of the Nazi Wolfsangel, though the battalion claims it is in fact meant to be the letters N and I crossed over each other, standing for "national idea". Many of its members have links with neo-Nazi groups, and even those who laughed off the idea that they are neo-Nazis did not give the most convincing denials.

"Of course not, it's all made up, there are just a lot of people who are interested in Nordic mythology," said one fighter when asked if there were neo-Nazis in the battalion. When asked what his own political views were, however, he said "national socialist". As for the swastika tattoos on at least one man seen at the Azov base, "the swastika has nothing to do with the Nazis, it was an ancient sun symbol," he claimed.

The battalion has drawn far-right volunteers from abroad, such as Mikael Skillt, a 37-year-old Swede, trained as a sniper in the Swedish army, who described himself as an "ethnic nationalist" and fights on the front line with the battalion.

Despite the presence of these elements, Russian propaganda that claims Kiev's "fascist junta" wants to cleanse east Ukraine of Russian speakers is overblown. The Azov are a minority among the Ukrainian forces, and even they, however unpleasant their views may be, are not anti-Russian; in fact the lingua franca of the battalion is Russian, and most have Russian as their first language.

Indeed, much of what Azov members say about race and nationalism is strikingly similar to the views of the more radical Russian nationalists fighting with the separatist side. The battalion even has a Russian volunteer, a 30-year-old from St Petersburg who refused to give his name. He said he views many of the Russian rebel commanders positively, especially Igor Strelkov, a former FSB officer who has a passion for military re-enactments and appears to see himself as a tsarist officer. He "wants to resurrect a great Russia, said the volunteer; but Strelkov is "only a pawn in Putin's game," he said, and he hoped that Russia would some time have a "nationalist, violent Maidan" of its own.

On one afternoon earlier this week the Guardian travelled with a group of Azov fighters to hand over several boxes of bullets to Ukrainian border guards. During an artillery attack outside Mariupol in the days before, the border guards had come to the rescue of a group of Azov fighters, and the bullets were their way of saying thank you. "Everything in this war is based on personal links; Kiev does nothing," explained the Azov's Russian volunteer, as we sped towards a checkpoint in a civilian Chevrolet; the boot full with the boxes of bullets and rocket-propelled grenade launchers; one of the windows shot out by gunfire during a recent battle.

"This is how it works. You go to some hot spot, they see you're really brave, you exchange phone numbers, and next time you can call in a favour. If you need an artillery strike you can call a general and it will take three hours and you'll be dead. Or you can call the captain or major commanding the artillery battalion and they will help you out straight away. We are Azov and they know that if they ever needed it, we would be there for them."

For the commanders and the generals in Kiev, who many in Azov and other volunteer battalions see as responsible for the awful losses the Ukrainian army has suffered in recent weeks, especially in the ill-fated retreat from Ilovaysk, there was only contempt. "Generals like those in charge of Ilovaysk should be imprisoned for treason," said Skillt. "Heads are going to roll for sure, I think there will be a battle for power."

The Ukrainian armed forces are "an army of lions led by a sheep", said Dmitry, and there is only so long that dynamic can continue. With so many armed, battle-hardened and angry young men coming back from the front, there is a danger that the rolling of heads could be more than a metaphor. Dmitry said he believes that Ukraine needs "a strong dictator to come to power who could shed plenty of blood but unite the nation in the process".

Many in the Azov battalion with whom the Guardian spoke shared this view, which is a long way from the drive for European ideals and democracy that drove the protests in Kiev at the beginning. The Russian volunteer fighting with the Azov said he believes Ukraine needs "a junta that will restrict civil rights for a while but help bring order and unite the country". This disciplinarian streak was visible in the battalion. Drinking is strictly forbidden. "One time there was a guy who got drunk, but the commander beat him in his face and legs until he could not move; then he was kicked out," recalled one fighter proudly.

"The failure to stop abuses and possible war crimes by volunteer battalions risks significantly aggravating tensions in the east of the country and undermining the proclaimed intentions of the new Ukrainian authorities to strengthen and uphold the rule of law more broadly," said Salil Shetty, Amnesty International secretary general, in Kiev.

Fighters from the battalion told the Guardian last month they expected a "new revolution" in Ukraine that would bring a more decisive military leader to power, in sentiments similar to those of many Azov fighters.

Despite the desire of many in the Azov to bring violence to Kiev when the war in the east is over, the battalion receives funding and assistance from the governor of Donetsk region, the oligarch Serhiy Taruta. An aide to Taruta, Alex Kovzhun, said the political views of individual members of Azov were not an issue, and denied that the battalion's symbol had Nazi undertones.

"The views of some of them is their own affair as long as they do not break the law," said Kovzhun in written answers to questions. "And the symbol is not Nazi. Trust me – some of my family died in concentration camps, so I have a well-developed nose for Nazi shit."

As well as their frontline duties, the Azov battalion also functions as "a kind of police unit", said a platoon commander who goes by the nom de guerre Kirt. A medieval history buff who takes part in Viking battle reenactments and once ran a tour firm in Thailand, Kirt returned to east Ukraine to join the Azov. He took the Guardian on an overnight patrol through the outskirts of Mariupol and the villages around the front line.

Part separatist hunters, part city cops with no rules to restrain them, they travelled in a convoy of three vehicles, all heavily armed. As midnight approached we set off across the bumpy tarmac roads to the outskirts of Mariupol, and soon came across a parked car by the side of the road that the men found suspicious.

Fighters dashed from the front two cars and rushed at the vehicle pointing their guns at it. A startled man got out of the passenger seat, then a sheepish looking woman in a cocktail dress and holding a half-smoked cigarette emerged, smoothing her hair. The Azov fighters apologised, but only after demanding documents and thoroughly searching the car.

As we edged closer to the front line, Kirt and the others scanned the skyline with binoculars, on the lookout for snipers and separatists. Later, fighters sprinted towards a suspicious jeep parked on the beach while the sea was scanned for hostile support vessels, but it turned out that again the men had stumbled upon people just trying to have a good time: a group of women drinking sparkling wine out of plastic cups on the beachfront.

The Azov have been partially brought into the military and officially function as a special police unit. There are discussions that Azov and other battalions could be integrated into the army or special forces when the conflict is over.

Some of them, however, are hoping Ukraine will look very different in the not-so-distant future. And while they may be a tiny minority when it comes to Ukraine as a whole, they have a lot of weapons.

President Petro Poroshenko will be killed in a matter of months, Dmitry said, and a dictator will come to power.

"What are the police going to do? They could not do anything against the peaceful protesters on Maidan; they are hardly going to withstand armed fighting units."