Downtime seems to have improved lately until we started having issues with one of our uplink providers.

3 out of 4 segments were down. We've opened a support ticket with them and working in conjunction to get this issue fixed.Check back here for updates. This is an ongoing issue at this time, so if things don't seem to respond, packets don't seem to find their way to our network, this is why.Servers affected: pepsi, moonshine, tequila, mail01 and mail02.We'll keep you posted.

Details on what happened to the core router. To be honest, I don't exactly know. It just won't initialize all network interfaces and that caused iptables to to be all screwed up.Out of 5 currently used segments only 3 would come up in random order; very weird indeed.

After several power cycles, the problem persisted. We first thought it was the drive. After replacing it with a backup, the problem persisted.

Finally, after re-sitting the RAM it all went back to normal.

We'll keep a close eye on this bad boy and see what happens in the next few days.

Like always check back here for updates. :P

----

Update on this issue: 12/16/07The problem was obvious yet frustrating. The router would not detect eth0 at boot time and shift all interfaces up one step. That caused all sorts of packet storms going down the other border routers and messing up the traffic flow.We finally realized this after several hours and immediately fixed the issue. Just FYI we've got a second core router setup and ready to take the load if any other issues come up along the way.Things seem to work fine now.