1999 Reports

2 December 1999

On Wednesday morning, AT&T Worldnet did something to their DNS servers
that resulted in a complete failure of their entire network. All 1.9 million
dial-up subscribers are off-line, unable to reach any part of the internet
-- and that includes checking email on our system. As of this morning, service
has not been restored and AT&T has no time estimate of when that service
will be restored. Please note that PagePlop does not use AT&T for its
backbone connectivity and is not affected by the AT&T outage.

Any of our clients who are affected and are unable to retrieve email,
send email, or make alterations to their web sites will either have to wait
it out or use an alternate dial-up provider to do any work necessary. In
the meantime, I will be more than glad to rely email messages, open client
email and read it to them over the phone, or make changes to their web sites
as needed until this outage is over. Unfortunately, there is little I will
probably need to actually do since no one who is using AT&T and affected
by the outage can read this page to note my offer.

But it's a nice gesture. :)

27 November 1999

I just wanted to put up a note on what I think is going to happen as
the result of Y2K with respect to your web sites serving properly.

There are absolutely no issues involving Y2K problems with respect to
the PagePlop system, it's hubs, switches, or routers, or its connection
to the backbone of the Internet. BellSouth, Sprint, and Carolina Power and
Light have assured us that their entire systems are fully ready to go and
have been repeatedly tested. That does not mean that there will not be issues
to deal with, however.

I anticipate that most of the world will be picking up their home phones
shortly after midnight to check if they still have dial tone. If too many
of them do so, they will indeed not have a dial tone. They will find the
same conditions as they do on Mother's Day or Christmas morning when the
phone systems are overwhelmed. That means that if you need to call us, you
probably will not be able to do so until things settle down. The voice phone
system is not the same as the Internet-connectivity system, though, so our
digital lines are not affected at all.

But everyone will also try to hit their web site at the same time to
see if we and other providers are still serving. That will probably result
in a massive overload of the Internet backbones, though PagePlop is fully
capable of handling just about any traffic you can throw at us. Even if
every one of our clients hit their home pages at exactly the same time,
we estimate it would take less than 20 seconds to clear out the requests.

So the question becomes, if I can't get to my website because the Internet
is jammed and I can't call because my phone is jammed, then how do I know
what is happening? Well, you don't. All you can do is sit tight and wait
for folks to eventually realize that there is no problem. They will stop
picking up their phones and they will stop trying to hit web sites and things
will calm down.

Now, the more serious issues...

Since there are no major points of failure anywhere in the telco or power
systems, I would anticipate that more than a handful of ignorant cretins
will attempt to cause havoc by introducing viruses or denial of service
attacks against the more high-profile web sites and the backbone providers.
They are all aware of the potential of malicious intent and will be at the
ready to intervene. That does not necessarily mean that there will be no
significant problems, though. It simply means that any problem that arises
will not be the result of a Y2K problem.

Then there is the water supply. I anticipate a very large, but temporary
problem with the water supply especially in major cities. What I think will
happen is that folks will be curious as to whether the water is still on
and flush their toilets or turn on faucets pretty much at the same time.
The resulting drop in water pressure will be similar to what occurs at half-time
of the Super Bowl when the entire nation flushes simultaneously.

I also anticipate runs on cash, food, and medicine which will deplete
banks, grocery stores, and pharmacies nationwide. Unfortunately, I fear
that a synergy will set in. In other words, the shelves will be sparse so
folks will think the problem is with the food distribution system and horde
even more. What we will end up with are a whole lot of households which
contain three months worth of toilet paper and green beans for absolutely
no reason. That will also result in a situation where folks for weeks or
months will spend less than normal throwing off store buying patterns. We
may even end up in a small recession as consumer spending drops and interest
rates rise because of increased demand by businesses for short-term, cash-flow
loans in the first quarter of 2000, loans they would not need if people
just maintained their normal buying patterns.

I can also envision a dramatic spike in crime, particularly burglaries.
Think about it. People start taking money out of their accounts and the
only place they have to keep it is at home or work. Then they will go out
to party leaving their homes and offices without protection. That is an
open invitation for the scum of the earth to break in and take goodies.
What concerns me most about that scenario is when burglar alarms start sounding
at police stations and they either assume that it is a Y2K glitch or they
are so overwhelmed with alarms that they can in no way respond to them all
in a timely manner. As soon as (or if) criminals figure out that the police
are overwhelmed, that pretty much gives them an open season on committing
further crime like looting. It may not be very pretty until the sun comes
up on January 1st.

But it won't have a thing to do with any direct computer problems, just
human stupidity.

13 October 1999

FTP access is currently turned off for approximately 100 clients because
we are having some problem with one particular server. We are afraid that
the latest version of MGI is somehow conflicting with the FTP server and
are trying to identify the exact cause. I anticipate that FTP access will
be turned back on about 2 PM EDT.

21 September 1999

Things are completely back to normal at least in Raleigh. If you go not
20 miles east of us, you find a 120 stretch miles of the worst flooding
in North Carolina history. Whole towns are under water (like 30 feet of
water,) Interstates are closed, and there is a tremendous worry about disease
since some 500,000 hogs, a million turkeys, and thousands of people are
floating dead in the miles and miles of deep water. By the way, the people
are not as a result of the death toll from the storm itself -- that numbered
about 40 -- they are from grave sites where the caskets popped up to the
surface. Really nasty. Unfortunately for many folks living in eastern North
Carolina, the rivers have not even crested yet and the remnants of Tropical
Storm Harvey are due to dump another three to five inches of rain there
today and tomorrow. Fortunately for us, it is all happening to our east.

In the meantime, I am backed up about 150 emails that really needed to
be answered over the past several days, but we have been busy with clean-up.
I am going to get to them all today and tonight, catching up completely,
then things will get back to normal. So bear with me today and all will
be well.

16 September 1999, 11:49 am EDT

The power just went back on and the sun is starting to peak through the
breaks in the clouds. I must say that the sky is never so blue as the time
just after a hurricane. Personally, I'll take the pollution and haze any
day and leave the cleansing by hurricane to someone else.

I am going to leave the generator up for about another three or four
hours until I am sure that the power grid in our neighborhood is stable,
then I am going to sleep. In other words, don't expect me to answer any
emails until significantly later this evening.

16 September 1999, 9:00 am EDT

The real nasty stuff is pretty much over now for us. The rain is down
to a medium rainfall (perhaps one-quarter inch per hour) with an occasional
burst to heavy (one inch per hour rate,) but not sustained for more than
a couple of minutes at a time, and the winds have dropped off to about 20
MPH with an occasional gust to perhaps 35 MPH. The North Carolina Office
of Emergency management is already stating that this is the worst flooding
ever recorded in North Carolina and the eastern part of the state still
has a long way to go. Some areas of the state have received over 25 inches
of rain and the heavy rains continue in the northeast part of the state
(about 100 miles from us.) We have had about 9 inches of rain where we are
with maybe two more inches to come. Floyd is supposed to pull completely
out of North Carolina by noon or so, but the rivers will continue to rise
for several days. Parts of the Tar River are already 15 feet above flood
stage.

Hurricane Floyd is still a strong category one storm moving to the northeast
at 25 MPH. CP&L now estimates that over 500,000 homes in eastern North
Carolina are without power with 120,000 in Wake County alone. CP&L has
over 1,400 workers available to repair power lines and will begin as soon
as the wind gusts die down enough so folks working on poles and around trees
will be safe.

16 September 1999, 6:10 EDT

That was stupid.

I went out to check the generator fuel gauge (it's still above full)
and the sump pump (it's still pumping fine.) The winds were not too bad.
So I decided to go to the car and get the big flashlight I left in there
in order to see the trees and stuff in the neighborhood.

Our front door faces south. The generator is set on the southwest corner
of the building and the door to the crawl space is on the west side of the
building. The car, however, is in the parking lot on the east side of the
building.

It's amazing how much a building can block the wind. I am not amused.
If anyone finds sheets of skin to our southwest, please return them to me.
And the flashlight wouldn't even work since it runs off the only type of
battery that I don't have -- those big, blocky things.

16 September 1999, 5:34 am EDT

It looks like we got fairly lucky. Floyd has moved in more of a NNE direction
after coming in over the Wilmington area. Since Raleigh is due north of
Wilmington it means that the center of circulation is moving away from us.
We are currently in the northwest sector of the wind circulation with sustained
winds of 40 MPH and the highest recorded gust so far at 52 MPH. That is
at the airport about 8 miles to our west and away from the storm center.

The storm is now moving NNE with a forward motion of 23 MPH (up from
14 MPH,) and the top sustained winds at the center are down to 105 (from
140.)

We have been listening to the police command center on the scanner and
trees started coming down big time about two hours ago. Most of the major
roads in Raleigh are impassable and it is unsure how the secondary and residential
roads are faring since it is still dark and no one can get to them at this
point. All of the streams and creeks are above their banks now and Crabtree
Valley Mall (the largest mall between Washington, DC and Atlanta) is about
to flood out; the parking lots are already full of water and they have hours
of heavy rain to go.

Heavy rain is anticipated to continue through at least 10 am with winds
maintaining their current speeds until that time. At this point, the huge
problem we have are that the tree roots are loosening up in the wet ground
and winds of even less sustained speed than we are currently experiencing
are capable of doing severe damage as those trees come down. Remember, oak,
elms, and pines don't bend for the most part like palms.

All telco is still up and working fine. We have full dial tone on all
the POTS lines and the data lines are running clean with no CRC errors or
drops. BellSouth has generators at every POP in the area and the Central
Office for the Raleigh area is on high ground.

Kenly seems to be the hardest hit inland town at this point with flooding.
The entire town is under water; Kenly is about 30 miles to our east. CP&L
is estimating that 270,000 homes are currently without power in eastern
North Carolina with much higher totals in South Carolina. It appears that
this is going to get much worse before it gets any better since the rivers
and creeks will continue to rise for hours or days even after the rain stops.
Virginia is next.

I-95 is completely closed between exit 119 and 121 (due east of us by
about 50 miles) as the Tar river is completely over its banks.

The generator is still humming and the sump pump is still pumping. We
are stilling in very good shape, though it may be days until grid power
is restored.

16 September 1999, 2:51 am EDT

Hurricane Floyd is close to making landfall on the lower North Carolina
coast. Raleigh is still in the west path of the storm with gusts up to 80
MPH expected through early afternoon on Thursday.

The power went out for good at 2:30 am, the generator went on at 2:31
am. With the combination of backup UPS units and the generator, we have
not experienced any outages. We will be pulling an all-nighter monitoring
the situation (while we watch TV, eat and work on the computer - so, we
have a really BIG generator). We will post further updates here as they
occur.

15 September 1999, 6:30 pm EDT

We are finally done with all the set up. The generator is set up and
grounded. The windows are completely boarded up. And I am kinda ticked off
at Travis, one of our programmers. Some 150 of my muscles are laying around
in the front yard and he refuses to go out in the rain and pick them up
for me. I guess I'll just have to flop around for the time being.

Right now, we have heavy rain -- after a day of heavy rain -- and the
trees are ripe for the winds. It now looks like the eye of the storm is
actually going to pass about 40 miles to our east, but that will still leave
us with several hours of sustained hurricane-force winds with gusts to perhaps
90 MPH. Not many of the big trees are going to survive.

But we are ready. CP&L is out in full force. There have been trucks
rolling all day long. BellSouth is on emergency status and are ready to
tackle any problem, even though they don't anticipate any problems in the
Raleigh area since everything is underground. We have food which is perhaps
the most important aspect of this entire situation.

The bulk of the storm is supposed to start coming in about midnight and
last a good 8 to 10 hours.

I'll keep you posted.

15 September 1999

Here we go again...

Not two weeks after a near miss by that pesky Hurricane Dennis (which
got us really wet, but without any other consequences,) we are about to
enter the Big Time.

Hurricane Floyd is bearing down on North Carolina. The center of the
storm is currently located off the Florida coast moving directly toward
us at 14 MPH. Current storm projections predict that Floyd will strike land
somewhere in northern coastal South Carolina on Wednesday night then move
inland Thursday morning. Though the eye of the storm is projected to move
to the east of Raleigh, the storm is packing winds of 140 MPH and has a
hurricane-force wind shield over 300 miles wide. Tropical storm force winds
extend almost 300 miles from the center in all directions.

Current predictions for the Raleigh area include sustained winds in excess
of hurricane force with the possibility of winds in the 95 to 115 MPH range
if the storm tracks slightly to the west from its projected path. Rainfall
is expected to be in excess of 10 inches. Even if the storm moves farther
to the east, the impact on Raleigh will be severe and sustained throughout
the day Thursday.

The generator is due in this morning. All the windows are boarded up.
And all the hard drives are in the process of being backed up as I type.

I apologize in advance if we do not answer the phone today or respond
to email in a timely manner, but we're kinda busy. Things will get back
to normal by Thursday night -- particularly if I can obtain the services
of an orthopedic surgeon to repair the damage I did to myself yesterday
preparing for this monster.

And if anyone knows of a good substitute for human skin, please let me
know. In that context, I am pleased to announce that the small, razor-sharp,
holly bushes buried amongst the lovely bushes outside my windows hve been
utterly destroyed.

22 July 1999

BellSouth is working on the F2 circuits today from the crossbox to the
doubler. At any give time, we may have only a single T1 line functioning,
but we will not lose complete connectivity. The work should be completed
by late today. Unless you are doing a massive download via a T1 line or
better from your side, you should not see any problems at all. If you are
on a T1 and downloading large things, you may see some speed reductions
until all lines are back up.

15 July 1999

BellSouth was testing one of the T1 lines last night for about three
hours. There was a slight problem which was corrected. No loss of connectivity
resulted, however some folks may have experienced stalls of about ten seconds
as the line came down then back up and as the load balance readjusted itself.

3 July 1999

IT IS ALIVE!!!!

The offending server is back up and functioning flawlessly. We are happy,
though tired. It turned out that there were three problems that happened
at the same time. One, a SCSI cable went south. Two, the FTP software corrupted.
And three, we somehow got corruption of the primary drive directory structure
such that some folders were actually on there twice, but only appeared once
in the file list. Needless to say, any one of those would have given us
fits, but all three together resulted in thoughts of homicide dancing in
our heads.

The up side is that the only time the machine stopped serving was during
reboots and only FTP was directly affected during the move. Also some folks
had problems seeing their sites not because of problems on this side, but
because their ISPs are too ignorant or cheap to abide by my DNS time-to-live.
You see, I have all DNS entries set to expire in one hour so that if I move
a domain from one machine to another, any information reflecting the old
IP number is supposed to die at ISPs in one hour then be refreshed the next
time someone on that ISP requests the site. Most ISPs abide by that rule,
but some are just plain ignorant and cache the dead information for one
to three days. They do not play by the rules and their clients suffer as
a result. The fix is two-fold. First, call up your ISP and tell them to
flush their DNS cache; that solves the immediate problem. Then chew them
out for not following the standards of recognizing TTL parameters.

2 July 1999 - Late

It's now about midnight and the server is completely reconfigured with
all new software. I am going to take a break and grab dinner, then start
reconnecting all the FTP accounts. Each one needs to be manually configured
and there are about 400 of them. Each one takes a bit less than a minute,
so you can pretty well figure out what I am doing till the sun comes up.
All should be completely back to normal by morning.

2 July 1999

That did not go very well at all. We have suffered failure of the FTP
software on one server that handles some 400 domains. It is serving web
pages just fine, but fixing the ftp server will require some downtime of
the server. Therefore we will be monitoring the machine all day and will
address the software failure later this evening when the machine traffic
slows down. We're aiming for no more than 15 minutes of server downtime
in the middle of the night (tonight). Thank you for your patience while
we solve this issue.

1 July 1999

The servers are fine, but ftp is temporarily turned off. We are upgrading
some of the hard drives on our servers and doing so seamlessly so that your
web site keeps serving. Since we are moving files, we chose to shut down
ftp so that we do not overwrite any new files that you might put on the
server during our upgrade. We estimate that the ftp for all accounts will
be restarted by 6 AM 2 July 1999, although your ftp may be available earlier
as we restart each one by one.

26 April 1999

Had a strange outage that only affected four clients, but it was weird
nonetheless. We had a 12-volt power block plugged into the UPS unit that
served four colocation machines. That power block was barely resting on
the top part of the master on/off switch. Over the past several months,
the vibrations of the machines sitting on the same shelf as the UPS unit
caused that power block to settle into the switch and turn it off. Needless
to say, when you shut off power to the computers, then tend to turn themselves
off.

16-20 April 1999

We will be testing our new automated rebooting system this weekend. At
times, we will be intentionally crashing the servers to check recovery and
remote rebooting times. In early trials, we are able to automatically reboot
any crashed server in less than two minutes. We are happy. And since we
can now fully reboot any hung machine from anywhere, we can actually do
things like normal people do such as eat dinner out. We are still happy.

8 April 1999

We took a five minute telco bounce for no apparent reason right at midnight.
Given that the time was so precisely at the top of the hour, I thought it
was BellSouth doing some routine maintenence and forgetting to tell us.
Rather, someone with a large butt apparently swung too wide at the NOC and
knocked something loose. The appropriate people have been yelled at.

16 February 1999

For the past week and for the next week or so, the various machines will
be bouning up and down for very brief periods of time every couple of days.
The typical outage will last about three minutes. We are in the process
of loading the new version of MGI on all the servers and though we will
try to confine our configuration to the middle of the night, sometimes things
need to be done in the day time. We will keep any disruption to a minimum.

4 February 1999

The phone's been ringing off the hook today with folks complaining that
they can not reach their web sites, FTP to their home space, or check email.
There is nothing wrong with our system nor is there anything wrong with
BellSouth. There ARE major problems affecting AOL, Mindspring, SWBell, and
a handful of other providers. Therre may also be problems at the major peering
point, but I can not effectively verify that. Today has apparently been
a rough day on the Internet and the only thing you can do is ride it out.
Perhaps it will be better by late tonight.

22 January 1999

Around 5:45 this morning, some genius in BellSouth Provisioning decided
to install a new circuit without telling anyone (there or here), taking
down most of Raleigh in the process. After about a one hour outage, we are
taking stable traffic again. The responsible party shall be duly flogged
with a wet noodle!

14 January 1999

Whoops...

As you may have noticed, FTP services are down. I did a major reconfiguration
of the web server this morning and every alias for every client broke. fortunately,
nothing happened to the web serving capabilities of any web site, but FTP
is down hard. I have to reset each alias by hand -- one by one -- to bring
them back up. That should take at least ten hours (like I didn't have anything
else to do today.)

If you are trying to FTP in and need to change files before I get to
your specific domain, call me at 919-852-5262 and I will skip to yours and
hook it back right then. And don't be shy to call. That's what I'm here
for.

10 January 1999

Now THAT'S the way things are supposed to work. Some idiot BellSouth
contractor, while picking his nose by the neighborhood crossbox, decided
to mess with one of our T1 lines. Of course, it went down for five minutes.
But the load sharing software worked perfectly and the other lines picked
up the slack without even batting an eye, something which can not be said
about the BellSouth contractor who is presently trying to pull a 2X4 out
of his pupil. How the 2X4 got there is something I probably do not want
to get into at this moment since there is still 6 years, 364 days, 23 hours,
and 52 minutes before the statute of limitations runs out. I will, however,
entertain a medical discussion on the merits of various methods used to
dislodge a cinder block suppository, such a discussion being appropriate
to the above-mentioned BellSouth contractor as well.

9 January 1999

The web server crashed and was automatically rebooted today. Total downtime
was a bit under three minutes. Since I was in Florida basking in 80 degree
weather, I am not exactly sure why it crashed, but it didn't happen again,
there was no need to get Chad or Travis on the problem, and it has been
stable since. It was probably hungry.

3 January 1999

We had a glitch of the stupid graphic counter software today that interrupted
service to all graphic counters for who knows how long. I really hate that
thing. Fortunately, we are within two weeks of replacing it entirely with
something we wrote and know works properly. When that is done. I am going
to take the existing graphic counter program, put in on a floppy disk, and
toss it into the fireplace.

2 January 1999

We did some work on the secure server last night and had the thing bounding
up and down for about six hours during the early morning hours. It was never
down for more than about ten minutes at a time. I need to finish up with
the software install at some point, but I will wait until the weekend to
do so.