2003 Reports

24 November 2003

It appears that comcast.net (including attbi.com addresses) has gone haywire this morning and is bouncing emails that are sent to their addresses. We are in touch with Comcast and will post an update to this page when we have additional information or when they have resolved the problem.

24 November 2003

Comcast.net actually contacted us (very impressive!). We have talked with them and they have told us that the problem will be fixed by the morning of Tuesday 25 November 2003 at the latest (it should be fixed within the hour). If we forward email to a comcast.net or attbi.com address for you, please contact us at 919-852-5262 if you would like for us to set up temporary email forwarding to a different address.

11 November 2003

The mail server got very slow this morning so I had to take it off line for a short period of time to do some work on it. The whole process should be over by 1500 EST if not before. No mail will be lost because it just stacks up for delivery when the server comes back on line.

7 October 2003

ALL the telco is back online (Yeah!). Apparently some cretin at BellSouth decided to leave all but one of our lines unplugged. We have sent Guido and Vinnie to his house :)

7 October 2003

Our telco is provided through GTE/Genuity who was recently bought by Level 3 Communications. During the overnight, Level 3 migrated our telco from the Genuity system to the Level 3 system.

They did a great job! We were down for only 5 minutes as promised and everything was looking great, until this morning...

As some of you have noticed, the connection to PagePlop has been slow today. Connections were getting through, but definitely more sluggish than normal. We have been working with Level 3 all day in order to track down the problem. At approximately 2 PM, we were told that the problem was coming from within our own system.

We have had isolated problems before where a machine will become infected with a virus, etc. and will flood our T1 lines. So, I started systematically unplugging every computer in the facility to find the culprit. No luck :(

I called Level 3 to give them an update and to dig further. After speaking with the representative, he informed me that the LINE (singular) was still at capacity. That is when I asked, "You did move ALL the T1s, didn't you?". "Uh, let me check - hm, you are right, there are multiple T1s on the router, but I only have 1 here."

So folks, Level 3 is now in the process of completing the move of ALL of our T1 lines so that we have our full capacity. I do not have a current ETA, but expect it will be complete sometime this afternoon or early evening and will update when I know more.

28 February 2003

Beginning at 0045 EST, we are taking down the primary email server to completely rebuild it. We are installing new, bigger drives, a faster processor, and optimizing the file structure. The entire process will take about three hours. In the meantime, no mail will be lost; it will all be properly sent as soon as the server comes back on line.

20 February 2003

What a day. It seems that business in the northeast has gotten back into full swing as folks dig out from the snow and make it back into work. At the same time, every kid is still out of school and sitting at home hammering the internet. Put them together and the major metropolitian peering points have been a mess today.

There have been routers flapping all day long and the delays in data transmission have been extraordinary. The biggest manifistation of the mess involves delays in the delivery of emails. Things are moving and getting delivered, but the whole east coast has been backed up.

Today is simply a day to be patient.

15 January 2003

During our nightly maintenance, we began moving approximately 25 clients to a new machine. Unfortunately, that move did not go as smoothly as planned because we began having problems with the machine we were moving the sites from.

No data or sites were lost, but the move has gone significantly beyond or maintenance window. We expect all affected sites to be online no later than 12 noon.