The family of a prisoner who was executed in Ohio on Thursday using an untested combination of medical drugs that appeared to cause him prolonged distress are planning to sue the state for inflicting cruel and unusual punishment, in violation of the US constitution.

Dennis McGuire, 53, was put to death using an untested two-drug protocol involving the sedative midazolam and painkiller hydromorphone. Before the execution, which took an abnormally long 25 minutes, Ohio courts were warned by an anaesthesiologist who served as an expert witness for McGuire that the procedure and the doses of drugs to be used would inflict untold suffering upon the inmate. The state decided to go ahead.

McGuire was executed for the 1989 rape and murder of Joy Stewart, who was 22 years old and about 30 weeks pregnant. Her unborn child also died. Before the execution, members of Stewart's family said in a statement that the manner of McGuire’s death would be more humane than the brutal way he had killed her.

McGuire's daughter Amber, son Dennis and daughter-in-law Missie were present in the death chamber at the Southern Ohio Correctional facility in Lucasville and watched as he struggled to breathe and tried to sit up over a period as long as 15 minutes.

According to a timeline put out by the Ohio department of corrections, the syringe of midazolam was inserted at 10.27am and 49 seconds, and the hydromorphone was injected at 10.28am and nine seconds. At 10.42am the medical team recommended waiting a further five minutes to ensure that McGuire was dead, and it was not until 10.52am that it was recorded that there were no further heart or lung sounds from the inmate.

In a press conference on Friday, Dennis McGuire, the inmate's son, described what he had witnessed.

“I watched his stomach heave, I watched him trying to sit up against the straps on the gurney, I watched him repeatedly clench his fist … [it] appeared to me he was fighting for his life while suffocating,” he said. “The agony and terror of watching my dad suffocate to death lasted more than 19 minutes. I can't think of any other way to describe it other than torture.

“Until yesterday I did not understand what cruel and unusual punishment was. Now I do, I witnessed it. No one should have to die the way my dad did.”

Last week an expert witness for the defence, David Waisel, a professor of anaesthesia at Harvard medical school, told an Ohio court that the use of midazolam was inappropriate in an execution and that, in employing it, the state ran the risk of the inmate being conscious for up to five minutes while suffering through the sensation that he was suffocating. Eyewitness accounts of McGuire’s death appear to correlate with Waisel's prediction.

On Friday there were expressions of astonishment and outrage from numerous groups about Ohio's use of the new drug protocol. Maya Foa, of the human rights group Reprieve, which has been central in inspiring a European-wide boycott of medical drugs for use in US executions, said it was shocking that the state had gone ahead despite the expert warnings.

“This is what happens when you have human experimentation,” she said. “It goes beyond anything that can be described as constitutional, and I only hope that it will give other states thinking of copying Ohio's use of midazolam pause.”

Deborah Denno, a law professor at Fordham University in New York who is a specialist in execution methods, said: “The cat is now out of the bag in terms of what these states are doing. They are out of control, taking ever greater risks with increasingly inappropriate drugs.”

Mike Brickner of the American Civil Liberties Union in Ohio said all eyes were now on 19 March, when the state has scheduled the execution of Gregory Lott for the 1986 murder of a Cleveland man. “Unless our governor or the courts do something immediately, there is a danger they may go ahead with this same botched procedure in March,” he said.

Dennis McGuire was executed in Ohio on Thursday. Photograph: AP

The McGuire family has instructed a group of Dayton-based lawyers, led by Jon Paul Rion, to prepare a federal lawsuit that is expected to be filed next week. The suit will ask the federal courts to halt executions in the state on the grounds that the department of corrections is violating the eighth amendment protection against cruel and unusual punishment.

Dennis McGuire said that the aim of the lawsuit was to stop all executions in Ohio. “I don't think any family should have to deal with what we dealt with yesterday,” he said. “Nobody deserves to go through that.”

Speaking at the press conference, Rion said: “If you were to read a medieval journal about something that happened hundreds of years ago when they strapped an individual to a board, deprived him of oxygen for over 15 minutes, and have the world sit there and watch him die, that would be interpreted as cruel. A culture that would inflict that kind of injury on a person would be questioned for its civility.”

Richard Schulte, co-counsel on the lawsuit, said the legal team would be focused on trying to find out where the drugs used in McGuire’s execution had come from and whether the manufacturers had been aware of what their product was being used for.

• This article was amended on 17 January 2014 to correct the description of Missie McGuire, who is the daughter-in-law of the executed inmate, not his wife.