If you had a computer connected to the modem prior to trying to connect the PS3, unplug the modem for about 30 seconds and after plugging it back in connect the PS3 and it should work. This is to reset any IP address your modem may have stored.

You could give it a IP manually. Check your comps IP by going into START>RUN and type in CMD. In there type IPCONFIG /ALL and it should show your IP in K CONNECTION. Then set up your PS3 connection and use the comps ip but change the last tenth digit one point higher.