Responding to the rejection, ACI said it is "prepared to do what is necessary to make this (deal) happen."

"ACI remains ready and willing to complete this transaction."

S1 chairman John Spiegel said the board was of the view that ACI's proposal was not in the best interests of S1 and its stockholders.

"I think it's S1's way of putting back the ball in ACI's court and it looks like ACI will come back with a modified offer, which might be slightly higher than their original offer," analyst John Kraft of D A Davidson told Reuters.

On a post-earnings conference call, S1 CEO Johann Dreyer did not answer questions on whether it was in talks with ACI.

ACI, which makes software for financial companies, had offered to buy smaller rival S1 Corp for $540 million in July, which had raised questions whether S1 would go ahead with their proposed merger with Fundtech.

In June, S1 Corp had announced that it agreed to buy Fundtech Ltd in a stock deal valued at about $700 million, just weeks after the Tel Aviv-based Fundtech confirmed it was in talks with potential buyers.

A merger with S1 would help ACI boost its sales to mid-sized financial companies and help it move into emerging markets.

The S1-Fundtech deal will see the combined company have a wider international reach.

"While the Fundtech deal offers benefits in the longer term, the ACI deal would lead to immediate payment, which is why it might find wider acceptance, because it cuts out the risk associated with the Fundtech merger," analyst Kraft said.

Shares of S1 Corp, which touched a year high after ACI's takeover offer in late July, were trading up about 1 percent at $9.28 on Tuesday morning on Nasdaq.

ACI shares fell 2 percent to $35.50 in afternoon trade on Nasdaq, while those of Fundtech were flat after trading up 4 percent before the bell.