It wasn't the farewell Billy Dee Williams originally anticipated, but when the Star Wars icon bid adieu to Dancing with the Stars on Monday night because of injury, he held his head high.

"I had a great time," the star, 77, told PEOPLE just after announcing his decision to depart. "I had a blast. I wish I could continue."

Despite coming in with the lowest score just before the first elimination, the fans voted to bring Williams back for another week of competition.

"I'm sorry that I'm not going to be around, but my back just can't handle it," he said. "My health is more important."

Williams's partner Emma Slater is in full support of the actor's early exit from ABC's hit show.

"I knew the back was a problem for Billy," she told PEOPLE. "We were already kind of taking it slowly, but for me, surprisingly, this decision wasn't as difficult as it could have been because it really was the right thing to do. We made peace with that. We more directed it as a celebration of the time we've had together."

Williams, who attributes his back problems with years of film stunts, doesn't have regrets.