The Italian castle where the poet Ezra Pound retreated in the 1950s to work on his epic poem The Cantos could scarcely be more remote, perched on a sheer rock outcrop against snowy peaks on the Austrian border.

More than half a century later, Pound's 86-year-old daughter still savours the isolation of Brunnenburg castle's lofty wood-panelled and book-lined rooms, where she reads the poem to visitors and mulls over her extraordinary upbringing by the renegade US poet.

But Mary De Rachewiltz's peace has been shattered by the murky world of Italian neofascist politics and the fallout from a bloody shooting spree through Florence's ancient piazzas that left two immigrants dead in December.

Angered by the decision of a rising fascist group called CasaPound to name itself after her father in honour of his support for dictator Benito Mussolini, De Rachewiltz is seeking through the courts to force the group to drop the name, and has accelerated her efforts after a CasaPound sympathiser shot dead two Senegalese street traders and wounded three others before turning the gun on himself.

"I was ashamed and hurt by the news," she told the Observer. "The fascists want to claim Pound, but they have nothing to do with Pound. They are a nuisance and there has to be something I can do to stop them."

Striding up the stone spiral staircase of the castle, De Rachewiltz heads for a corner study that commands breathtaking views up to rock faces and a piercing blue sky and down past the family vineyard to the valley far below.

An accomplished poet herself, De Rachewiltz has surrounded herself at Brunnenburg by Pound's possessions, including sculptures and old school books in which he drafted, in different coloured inks, sections of The Cantos, the historical work seen as one of the 20th century's major poems. She calls it "her bible".

De Rachewiltz's devotion to her charismatic father has never flagged, despite being given up for fostering at birth in 1925 to a peasant couple in South Tyrol by Pound and her mother, the US violinist Olga Rudge, who was the poet's mistress. Before her birth, Pound had settled in Italy with his wife after blazing a trail across the London poetry scene, en route to becoming a fixture on Paris's Left Bank, where he helped TS Eliot edit The Waste Land and mentored Ernest Hemingway, who tried to teach him to box.

In total contrast, De Rachewiltz's early upbringing revolved around bee-keeping and sheep herding, albeit broken up by trips to see her father and his mistress at the home they kept in Venice.

"The Tyrolean peasant is still ingrained in me," said De Rachewiltz, who recalls smashing a violin her elegant mother gave her. A tense relationship between Rudge and her dialect-speaking daughter ensued. "I think I was influenced by my foster parents, who probably envied 'the lady in the silk stockings' who would pick me up," she said.

With Pound she forged a strong bond as they roamed Venice together, visiting bookshops and taking the vaporetto waterbus out to swim at the Lido. Later, Pound would ask his daughter to translate the poem he had started, The Cantos, into Italian. "In 1943, at the height of the war, when I was back in South Tyrol for safekeeping, Pound made his way from Rome to make sure I was safe," De Rachewiltz recalls. During the visit he finally confessed to her that he had a wife in Rapallo on Italy's Ligurian coast. By that time Pound was a wanted man in the US after endorsing Mussolini's brand of fascism and making a string of radio broadcasts – which were riddled with antisemitic statements – to persuade America to stay out of the war.

Arrested in 1945 by American troops and kept in a cage for three weeks, where he showed signs of a mental breakdown, Pound was charged with treason and spent the next 12 years in a mental institute in the US.

De Rachewiltz has since fought a lifelong battle to separate Pound the poet from Pound the fascist and antisemite, which is why the emergence of CasaPound – now boasting 5,000 members – is so painful.

She rebuffs the suggestion that CasaPound's lionisation of him is no more than he deserves. "Pound just quoted what Mussolini said," she said. "This organisation is hiding behind Pound's name for intellectual cover," she added. "He made mistakes and we have to take the good part of him, just as he did with others. He fell into certain antisemitic clichés that were rampant in Europe and the US at the time."

Pound later told the American poet Allen Ginsberg that his worst mistake in life was his "stupid suburban antisemitic prejudice".

Released by the US government in 1958, Pound returned to Italy, where his daughter had married an Italian Egyptologist of Russian descent, Boris de Rachewiltz. They had purchased Brunnenburg, then in ruins. "It cost more to put in windows than to buy the castle," she recalls.

Pound moved to the castle to continue work on The Cantos, joined by his wife Dorothy, "who stayed in her room reading Henry James", recalls De Rachewiltz. But after years in an overheated mental institute, Pound was not fit for drafty castle life, and returned to Rapallo, where Olga cared for him until his death in 1972.

De Rachewiltz now sees that rivalry with her mother for Pound's affections clouded their relationship. "Anyone who met Pound loved him," she said. "When he wrote 'Pride, jealousy, and possessiveness/ 3 pains of hell', he was writing about all of us, including his wife. We all wanted him for ourselves. I re-read my mother's letters now and see how much she loved me."

Decades on, surrounded by her family, De Rachewiltz reads and writes, stopping for lunches of bread and cheese with a glass of red wine from her vines, followed by a cup of Earl Grey at four. TV soap operas are an occasional pleasure.

The CasaPound legal case, she insists, will not break up her routine. "I do not want to meet them," she said. "If he was here, Pound would say they need to go back to kindergarten."