Michu puts Swansea 1-0 up against Arsenal to spark a frenetic second half at the Liberty Stadium. Photograph: Adrian Dennis/AFP/Getty Images

The defensive frailty that threatens to cost Arsenal a place in next season's Champions League was evident again at the Liberty Stadium, when Arsène Wenger's hit-and-miss Gunners went from a goal down to 2-1 up, only to concede an 87th-minute equaliser which forces them to replay on 16 January for the right to meet Brighton in the fourth round of the FA Cup.

The second attempt between these two will be at the Emirates, which makes Arsenal the favourites to progress, despite the fact that the Swans won 2-0 there in the Premier League. Nevertheless, Wenger was left to rue the failure to hold on to a winning position and settle it in 90 minutes. "We should have won the game", he said. "I'm frustrated that we didn't." For the French eminence grise, it has been a familiar experience this season.

In reality, the result was the most equitable outcome to a real slow burner that only caught fire late in the second half. The first goal was delayed until the 58th minute, the last three came in a frenetic final 10.

The first thought of the afternoon was that in Wales the magic of the Cup is about as successful as Tommy Cooper's. On Saturday Cardiff fielded a team of 11 reserves and were eliminated, with no regrets, by non-League Macclesfield, opting instead to concentrate on their promotion bid. Here, Swansea rested Ashley Williams, their captain, and right-back Angel Rangel with Wednesday's first leg of their Capital One Cup semi-final, away to Chelsea, in mind. They also left their leading scorer, Michu, on the bench until the second half, when he plundered his 15th goal in 24 appearances just 73 seconds after getting on.

Arsenal recalled Olivier Giroud to lead their attack, with Theo Walcott supporting from the right. So much for his run as the main through-the-middle striker.

In an aesthetically pleasing but goalless first half, Danny Graham fastened on to a long, lofted pass from Dwight Tiendalli and held off Per Mertesacker on the 18-yards line before letting fly with a pulverising drive which demanded a spectacular save from Wojciech Szczesny.

Arsenal took 35 minutes to muster a decent goal attempt, Santi Cazorla blazing over. Swansea were the better side at this stage and their quick response saw Kyle Bartley heading firmly against the crossbar from Jonathan de Guzman's long free-kick from the right.

Into the second half and Walcott set up Giroud for a crisp drive, well saved. Aaron Ramsey, already booked, could have been sent off for diving after 52 minutes, when he clipped his own heel and went down in the penalty area without being touched. Wenger, myopically, claimed Arsenal should have had a penalty, for a trip by Tiendalli.

The biggest cheer of the afternoon greeted Michu's introduction from the bench, and in little more than a minute he cranked up the decibel count still higher by putting Swansea ahead. Michel Vorm's long clearance was transferred by Nathan Dyer and Graham to the Spaniard, who nodded the ball past Laurent Koscielny before shooting low, beyond Szczesny's right hand, from the edge of the area.

With a lead to defend, the Swans sent on Kemy Agustien, their midfield enforcer, in place of the diminutive Leon Britton. They needed all the defensive manpower they could muster as Arsenal pressed with insistent urgency in pursuit of equality. In belated response to the gathering crisis, Wenger sent on Lukas Podolski for the last 20 minutes but two gilt-edged chances came to nought.

Walcott ought to have equalised but shot wastefully wide and Podolski was guilty of an even more culpable miss from close range before his blushes were spared by an offside flag. The German atoned in full in the 81st minute by hooking home the equaliser from near the penalty spot, the Swans' defence having failed to clear the ball after a corner.

Arsenal's transformation from deficit to profit was complete after 83 minutes with a thumping left-footed volley from Kieran Gibbs, but Swansea were not finished. They hit back hard, and after Ki Sung-Yueng had forced a notable save, Graham made it 2-2, driving in high and handsome from a testing angle on the left after a corner taken by Ki had been inadvertently headed to him by Podolski.

In a pinball finale Vorm denied Koscielny in the 90th minute, then Tiendalli drove into the side-netting. The scorers, inevitably, take most of the plaudits, but in an all-hands-to-the-pump rearguard action when the force was with Arsenal Swansea were particularly well served by two young defenders, Bartley and Ben Davies, who boast just 25 appearances between them. Bartley joined from Arsenal, after playing for them only once in a Champions League tie away to Olympiakos.