Zsa Zsa Gabor’s husband Frédéric Prinz von Anhalt is speaking out against claims he is holding his wife’s ashes against the wishes of her estate.

“I had a service at a church, and then I took the urn home, which is a normal thing to do,” von Anhalt tells PEOPLE. “It’s standing on the fire place, and that’s it. It’s my urn, it’s my wife’s urn and I’ll do what I want with it.”

But Edward Lozzi, Zsa Zsa Gabor’s one-time publicist who also represented her late daughter, Francesca Gabor Hilton, tells PEOPLE that Gabor’s “ashes were suppose to be interred at West Wood Memorial [Park Cemetery] on the Dec. 30, right after the funeral at the Good Shepherd.”

Prinz Frédéric von Anhault places his wife's ashes in her favorite dog carrier case at the funeral service in Beverly Hills.

Willy Sanjuan/Invision/AP

However, as of Wednesday, Lozzi says the ashes have not arrived. “Not only was it her wish to be interred at the cemetery, the plot is already bought and both [her sister] Eva Gabor and Francesca are already buried there.” Zsa Zsa’s sister Eva died in 1995 and her daughter with Conrad Hilton, Francesca, died in 2015. Gabor’s other sister, Magda, died in 1997.

Asked about the accusations, von Anhalt denied Lozzi was ever his wife’s publicist, telling PEOPLE, “This guy was never the press person, he must be a lunatic or something, I don’t know who he is.”

Lozzi says his work for Zsa Zsa and her daughter is “well documented.”

Lozzi believes that von Anhalt is holding the ashes to get revenge on him and Gabor’s celebrity friends for boycotting the funeral von Anhalt had planned. While many of the celebrities von Anhalt had on his guest list did not attend the funeral, he attributed their absence to bad timing.

RELATED: Zsa Zsa Gabor’s Life in the Spotlight: Her Biggest Scandals and Sad Last Years

“The church is not full today because the timing was bad,” von Anhalt said at the funeral, which was live streamed online. He added that many would-be guests were out of town for the winter holidays or had passed away.

At the end of the funeral service, von Anhalt placed his wife’s urn in what he said was her favorite dog carrier case and exited the church holding the case.

At the time, Page Six reported that von Anhalt said his wife’s ashes would be interred with her father in Budapest.