It is hard to believe that Serena Williams is out of the Australian Open. She was so dominant all the way up to her fourth-round match against Ana Ivanovic that it seemed she had only to turn up to get through the final stretch and secure her 18th grand slam title.

Well, Ivanovic was having none of it. Nor did either of them buy into the theory that Williams was suffering from the after-effects of a long season. And there was no doubting the diagnosis that her back needed extended rest.

Her grand slam ambitions, which seemed realistic only a couple of days ago – nailed on, in some estimations, although not her own – lay in tatters on Sunday at the feet of Ivanovic, who had never won more than four games in a single set in their four previous meetings. It was the biggest upset of the tournament by a distance.

Williams, the world No1, had to be bullied, almost, into confirming later that she had considered pulling out of the match beforehand with a back injury that has been with her for at least a couple of days. She tried as valiantly to deflect questions about her fitness as she did to heap praise on Ivanovic, the 14th seed who did, indeed, play as well as she has done in a little while.

Ivanovic recovered from a tentative but energetic start then flourished, irresistible at the end against her stricken foe, to win 4-6, 6-3, 6-3 and go through to the quarter-finals.

Under persistent questioning Williams conceded: "Yeah, I almost pulled out. I probably should have but I'm such a competitor. I guess the secret is out but I obviously wasn't hitting the way I normally would hit and wasn't moving the way I normally would move. It wasn't the best."

What was "it", exactly? "To be honest, I don't know. I have no idea."

How long had she had it? "A few days. It's not anything that's life-threatening or anything. I feel like it will get better. I just need a couple days off."

If Williams had defended that well on court, she might have won.

"It was all Ana today," she said. "I thought she played really well. I made a tremendous amount of errors (31 unforced, against 27), shots I missed I normally don't miss, that I haven't missed since the 80s."

The Serb's victory was greeted with resignation on the other side of the net, and jubilation around the Rod Laver Arena. Perhaps it was because she did to Williams what their own hero, Sam Stosur, had done in the final of the US Open in 2011 but had been lamentably unable to repeat ever since, leaving this tournament meekly.

"The crowd often goes for the underdog," Williams said. "And she was hitting a lot of winners. I don't think the crowd was against me. I think they wanted Ana to play well."

Ivanovic said: "I just stayed in the moment physically. I didn't think much about the occasion and who I was playing because it can get overwhelming. I could have just made few more errors. But I really just believed in my game and stepped up when I needed to. I had to break a spell, fourth round, and what's the better place to do it than here against such a champion?"

Ivanovic said she was unaware Williams had a back problem but added, "I think she didn't serve as fast as in her previous matches. I really hope she can recover fast."

Yet Williams hit a tournament high for her of 194kph (120mph) and averaged 171kph.

Rumours swept the tournament that Williams was having off-court problems, unspecified but troubling enough to affect her mood. When she walked on to the court at the start of play, her coach and partner, Patrick Mouratoglou, was carrying her bag, trailing in her wake. It was a strange sight.

She had pulled out of the doubles with her sister, Venus, and did not practise the day before; her movement was sluggish, her face sullen. What one could reasonably say is Serena, on court, was far from serene.

She hit five winners off the ground against 23 from Ivanovic, a statistic that would have made more sense, normally, if reversed.

But Ivanovic deserved the garland. She out-hit her opponent in rally after rally. She won the 2008 French Open then lost all form but looked reborn here. If it had been a night match her victory smile would have illuminated all of Melbourne.

So Williams's slam aggregate stalls at 17, alongside Roger Federer on the men's side and one behind both Martina Navratilova and Chris Evert, five behind Steffi Graf. For now her claims to be regarded as good as or better than those wonderful players idles until further proof is produced.

"I'm going to go home and see my son," she said. "His name is Chip. We're going to spend some time together. I miss him."