Social media was awash on Monday night with examples of customers complaining of being left high and dry by NatWest and RBS. Photograph: Jonathan Mitchell /Demotix

A technological banking glitch on one of the busiest online shopping days of the year left millions of shoppers unable to pay for transactions using their credit or debit cards.

Customers of the Royal Bank of Scotland (RBS) group, which includes NatWest and the Ulster Bank, found that their payments were declined, while others could not access the banks' websites or smartphone apps.

Last night the banks were grappling with a new and potentially shattering blow to their brand reputation and growing customer anger after similar glitches 18 months ago when IT problems locked many customers out of their accounts for several days.

The cause of the chaos is unclear, with speculation ranging from suggestions of a cyber attack to a repeat of the previous problem.

Initial reports yesterday that transactions were being declined en masse began to filter through at around 6.30pm, one of the busiest times of the day for shoppers stocking up on groceries or filling their cars at pumps. Social media was awash with customers complaining of being left high and dry for a number of hours, and warning that the problems were the final straw.

Another tweeted that the problems had ruined her plans to take advantage of reductions available at many online retailers on one of the busiest online shopping days of the year: "And there goes my plan to get some Xmas shopping bargains online tonight. No #CyberMonday for me. #RBS is ruining Christmas."

NatWest's press office appeared to be as much in the dark about the cause of the problem as any of their customers, referring early evening queries to the bank's Twitter account and releasing a two-sentence statement pointing out that it was aware of "technical issues" and apologising to customers.

Shortly before midnight, it released a new statement pledging to compensate customers who had been "left out of pocket" as a result of the problems.

It said: "We are very sorry for the system issues that affected our customers this evening. Our customers are reporting that services are coming back on line. We will confirm when all systems have returned to normal service.

"If customers have been left out of pocket as a result of these system problems, we will put this right. If any customer is unable to resolve an issue caused by the disruption, they should get in touch with our call centres or come into a branch in the morning where our staff will be ready to help."

Last year, when systems affected more than 16 million account holders at RBS and NatWest, Ulster Bank customers were left without access to their bank accounts for a week.

Ulster Bank directors did not take their bonuses for last year as a result, although there was no such gesture in the case of their colleagues at RBS and NatWest. Reports have suggested that bankers at the group are set to claim £500m in bonuses this year, although pressure for them not to do so is expect to mount in the wake of yesterday's chaos.

It took place towards the end of a day when some hundreds of millions of pounds were expected to have been spent online as the Christmas shopping season took off.

Retailers hope that that the Mega Monday online shopping spree will mark the start of a Christmas rush after industry figures showed disappointing sales in November.

Predictions that as much as £450m would have gushed through online tills – making it Britain's busiest day on record for cyber-shopping – have brought comfort to retailers after last month saw sales growth drop to its slowest pace for seven months.

Other reports suggested the make-or-break month for retailers had got off to a strong start.

Visa Europe estimated that £450m would have been spent online in 7.7m transactions by the close of business on Monday night, which would make this Mega Monday the busiest online shopping day recorded.

Mega Monday is one of many days retailers seek to highlight in the runup to Christmas. It marks the day when shoppers have just been paid for November and, having supposedly spent the weekend browsing, they then splash out online – allowing plenty of time for deliveries in time for Christmas. Retailers mark the day with offers and discounts, following hard on the heels of American import Black Friday – the day US retailers offer post- Thanksgiving discounts.

In a sign cash-strapped shoppers were putting off festive food buying and holding out for pre-Christmas discounts on gifts, the British Retail Consortium said November's sales were up just 0.6% on a year earlier once the effect of new store space was stripped out of the figures. It was the weakest performance since takings slumped amid a cold snap in April.

Demand for winter and party wear helped sell clothes and a more buoyant housing market saw homewares fare better, but food sales fell for the fourth month running according to the BRC's sales monitor, compiled with accountants KPMG.

There was "good demand" for festive desserts, party food and Christmas hampers and the "baking frenzy" continued, the report said. But consumer confidence weakened and, coupled with falling food prices, knocked sales for the category overall.

"It seems that once again, shoppers are leaving their festive food buying until later. Six in 10 tell us they will spend more time than last year hunting for the best deals, and four in 10 seeking the best quality," said Joanne Denney-Finch, chief executive of grocery researcher IGD, which provides the report's food data.

Overall, sales growth was ahead of where it was in the runup to Christmas 2012, when November sales rose just 0.4% on the year. The BRC said this November saw a slow start and then growth "picked up in a big way" in the final week, when falling temperatures helped boost clothing sales.

Helen Dickinson, BRC director general, said: "Christmas spending looks on course to be an improvement on last year, but December will decide it.".