Linksys wrt54gs high ping time

I have a linksys wrt54gs wireless router and It's got this weird problem where after starting it (reset) everything is ok for about 3 minutes, I get a normal ping time 10ms - 30ms and after a few minutes the ping goes up to 1800ms - 4000ms with no apparent reason. This happens both on wireless and cable.