Yeah, the damn thing screwed itself good last night.... Second time since
we started. Blue screen by this morning and a lot of people said they couldn't
get in. And the DNS server (a third party) had a problem yesterday. Sorry
about the hassle. Its up again now.

The good news is PacBell finally got our 384kbps DSL connection up at a location
where the server can be monitored 24/7. The current server only has human
companionship <GRIN> 06:30 to 15:30 daily. We are testing the connection
for reliability and setting up a new server machine. Then we will register
a nicer domain name for the techref and move the techref and piclist sites.
The old server will continue for other things and refer techref and piclist
people to the new address.

I'm also experimenting with Linux (Red Hat 6.1) as a router and backup web
server. I've seen some interesting things done where an NT box and a Linux
box ping each other and if one dies, the other takes over. The idea is that
even if something external (hacker, virus, anything that triggers an OS flaw,
etc...) kills one, its very unlikely to kill the other. Since I use a lot
of ASP and 32-bit machine language MASM code, in my case the Linux box will
only have a "We are experiencing technical difficulties" web page and it
will spend its time screaming (pager, audio alarm, phone calls, etc...) for
help and maybe rebooting the NT server. So far the Red Hat installer has
some kind of security problem with the partition table, but I've not had
time to really trouble shoot it.

All last week I've been having horrible problems with the server getting
overloaded. % Processor Usage goes to 100% and stays. Web site response is
more and more sluggish and the number of users connected climbs to insane
levels. The TaskList shows that inetinfo or mtx is takeing 99% of the available
cycles. Useing Performance Monitor to watch the threads of those processes
shows that different threads are maxing out at different times. Running
windbg, attaching the process and viewing
a stack dump for the maxed out threads always shows that MSVCRT.DLL is running.
No info in MSKB related to that DLL other than that it is the Visual C runtime
library. I don't run any Visual C code. Tried shutting off all CGI-BIN exe's.
Upgradeing that DLL, etc... has no effect. Stopping the site or the entire
web service has no effect (!). And then... We notice that stopping the
content index service frees up the processor and disconnects most of the
connected users. No content index corrupt messages... server ran fine
for an entire day with the search functions disabled. Rebuilding the content
index had no effect. I've disabled the search functions (but kept the index
server running) and the processor never makes it above 5%! And I've served
more pages than ever! Doing one or two searches at a time (locally)
sends the processor up to about 80 or 90% for the duration of the search.
Doing 10 searches at once pegs the meter and it never recovers.

So... the secret is: Don't run a busy NT web server with more than a Gig
of text in the search engine unless you throttle its use. I'm frantically
searching for a way to do that. For now, it just records the starting time
of each search and will not allow another one for a number of seconds after
that.

A new, faster, server would allow more so I'm adding requests for support
on all the search pages and during the time the search is running. If that
doesn't generate some funds for a faster server, I'll look for sponsership
or advertizing for the search pages ONLY! The main site will
always be free and accessable.

ISP service outage. Road construction nearby cut the cable. Second time
(for the same reason) in years.

20020110

Moved from Rancho Bernardo to Temecula. DSL is the only option. Verizon has
us on a 384k up and 384k down modem.

20020112

Second day and the DSL connection slowed to a crawl. Support at Verizon will
not talk to us unless we diconnect our router and connect only one PC. After
verifying that the connection is at about 64k, a real tech was called and
found that our account was configured incorrectly in one of the three systems
that control connections.

20020125

A lineman decided that our wires were not punched down neatly enough at the
main terminal block for the building, called up the main number, got a girl
in the office and told her (not asked) that the phones would be down for
2 minutes. He then ripped us out and re-wired us... DSL, fax lines, and all.
Took about 10 minutes.

20020226

Down for a while this am at the office: First it looked like the router might
have locked up as I couldn't reach it on the internal network... but the
DSL modem was also flashing its Data light which according to the manual
is abnormal. I power cycled both and all was well... for a while. Then
the data light went out and the modem light on the DSL modem started blinking
and that lasted for about 10 min and 2 power cycles. Sat on hold for 7 min
for Tech support and they couldn't get the modem back up so they put in a
trouble ticket, and about a minute later, it came back up. Tech got here
just before closeing and it had worked all day...

20020321

Up and down this afternoon from 2 to 3 at the office. DSL modem would flash
its data or modem lights on and off. Support doesn't want to send a tech
because it started working while I was on the phone with them.

20020322

Up and down this afternoon from 12 to 1 at the office. DSL modem would flash
its data or modem lights on and off. DSL sucks. Support will send a tech.
The tech found a bad connection on the breakout panel in the back of our
suite. Seems ok for now.

200301061324-200301061509

After a long time of not haveing any problems, I apparently set the firewall
box too close to a server monitor and it overheated... I know that internet
access stopped suddenly at 1:24pm, and I could not access the firewall config
page. I found the top of the unit was very hot and the vents were being blocked
by the side of the monitor. After verifying that the problem was not with
the DSL modem, I reset the firewall to factory defaults and started setting
it up again (note to self, print out and keep handy all the settings... Doh!).
Finally back up at 3:09pm

The SQL server took a dump. BSOD. Restarted and all is well. Only affect
at this point is the email archive, but as I add more ecom stuff, that could
cause real problems. Anytime you add more machines to a solution, the risk
of failure increases. Sigh.

I applied some updates from $MS last night, and restarted, and then about
11pm there was a big windmill, then a spike in the index service cpu usage
(not unusual) and the inetinfo service started sucking all the available
cycles. Commited bytes slowly climbed to about double the norm and avaiable
bytes oscillated wildly. Network utilization was also spiking with the windmills.
This AM the server was reporting 500 Internal Server error on piclist, sxlist
and massmind and running the others really slow. By the time I got to the
office (9am) the system was still sluggish, but task manager did not show
any unusual activity. The web sites were now reporting HTTP 1.1 Application
Restarting. I restarted the machine and all seems to be well.

Well, the NTFS can only do so much I guess. <grin> The hard drive fried
last night. I took it as a sign to move to the new server a bit early. I've
been getting it ready but there are some rough edges... let me know if you
find one.

I had good backups from the old server, but it took a while to get them all
together. I made the mistake of using the new server as a development platform
to try to work out some new features and solve old issue, but that ment that
it was not in sync with the old server. And once i got that sorted out, there
were the standard wierd problems to work though:

SEARCH DISABLED: I get runtime error '800a01ad' and can't create object
"ixsso.Query"
Stopping and reStarting the index service makes no difference
ixsso.dll is present v 5.00.1696.1
dependancy walker says it is having a problem with query.dll over the URLencode
export.
Query.dll is present and other exports from it seem ok, its version is
5.00.1782.4
Trying to re-install IIS4 from option pack 4.0. The original query.dll was
5.00.1696.1 which looks like a better match...
Arrgh... I can't copy it in since the original query.dll is in use. How do
I do that?
ahhh! from winnetmag.com article 14585:

Start the registry editor (regedt32.exe not regedit.exe)
Move to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager
Double click on PendingFileRenameOperations (or create of type mutli_str if it does not exist)
On the first line is the name of the file that will be replacing the current file with \??\ in front, e.g.
\??\d:\time\ntfs.sys
On the second line is the file to replaced with !\??\ in front, e.g.
!\??\d:\winnt\system32\drivers\ntfs.sys
Click OK

"HTTP 1.1 Application Restarting" started about 23:00 and continued even
after the regular restart a few hours later. I finally drove in to the office
at around 14:00 on the 13th. I started and stoped the site, tried to stop
the web service (not responding) and restarted the PC. Still the same. And
then I realized that I had been editing an include file used by default.asp
from another PC via the network and had left that file open. I close it and
suddenly all was well. Apparently, IIS was trying to rebuild the application
and couldn't get access to that file because it was locked by the other PC?
Just my guess. Note to self: Don't leave important files open!
+

Another of the lovely power fail / shutdown started / power back on before
shut down complete / system doesn't restart hang-ups. APC with NT is just
lovely. Happens every year when the heat waves come and the power grid can't
keep up with the Air Conditioners.

Started getting this "Couldn't load application" error on the web site. Event
log is full of

DCOM got error "Logon failure: unknown user name or bad password. " and was
unable to logon MASSMIND\IWAM_XX in order to run the server:
{3DFEFADE-B61A-4096-90D8-F3C0F137D9BB}

and

The server failed to load application '/LM/W3SVC/2/Root/xxxx'. The error
was '80080005'.

where xxxx was each of the applications in turn. E.g. techref, dict, and
so on.

I deleted and re-created each of the applications for the virtual directories
that have them. That seemed to solve the problem. No freaking idea what caused
it except that I had been playing around with the old web server and I think
it syncronized the SAM database and changed the password on that account.

None of the web services are running after the normal restart this AM. And
they won't start from the Internet Service Control Panel. The WWW service
is running, so I stoped it and started it and then the web sites, etc...
could be started in the ISCP. Event log shows: Event ID: 4098 Source: Transaction
Server

The run-time environment has detected the absence of a critical resource
and has caused the process that hosted it to terminate. HRESULT: 80070006
(Microsoft Transaction Server Internals Information: File:
x:\viper\src\runtime\mtxex\vipthrd.cpp, Line: 862)

I can't seem to find anything about that on the net... Nothing wierd in the
log files, just seem that something didnt start right this AM. I hate crap
like this. Yesterday and the day before I was seeing really high counts on
"Current Anonymous User" without a corresponding increase in traffic.

Happened to notice that the site was running sort of slow and the processor
seemed to be working harder than usual. After looking around, I found that
none of the applications where running in "separate memory space (isolated
process)". Oh... no problem, just some leftover from the issues of a day
or so ago, so I'll just turn that on... And then the web site was down.
"Application failed to load"

Event log is filling up with 80004005 "failed to load"

Oh boy, oh boy... turn that back off... Ok for now. But why can't I run in
seperate memory space? Anyway, long story short: http://support.microsoft.com/kb/297989/
Was the deal. After I got that worked out, all was well. Not only the IUSR
but also the IWAM accounts had gotten thier passwords changed by the backup
domain controller and they needed to be reset. I also ended up having to
delete two of the packages out of the Transaction Server \ Computers \ My
Computer \ Packages Installed list and re-create them via the Internet Service
Manager. Not sure what was wrong with them, but they would not accecpt the
seperate memory space option: it would just delete the application when I
turned that on.

Perminant "Application Restarting" again. As far as I can tell, the backup
(which runs from another server) froze with the application .asa file open
(open for read not write, but IIS seems to have issues with anyone else even
looking at that tile) and restarting both machines did the trick.

Power failure. UPS held the server up for about 20 minutes and then it died.
The power came back on about an hour later... for about 2 minutes... Then
went down for another hour and a half or so. Finally came back on around
5 and stayed up. August sucks. Of of these years, I have to move to a hosted
server.

Hard drive lost some chunks and for some reason the server didn't restart
on it's own, so it didn't start working on the standard restart chkdsk until
I restarted it in the morning around 9am. And the chkdsk ran REALLY slow
for some reason (probably because it was finding and correcting the problems)
so it didn't get done until around 3pm. Yeap: Fried the entire day. And me
biting my nails to see if the drive was toast. Once it finally finished,
everything reports good and the server is back up.

Power failure. The triplight rack mount 750 kept the server up for about 10 minutes, but the monitering software on the servers was setup to shut down after a few minutes and totally failed to do so. I happened to be in the office and so I shut down manually after discovering that the monitoring agent service was not running and could not be started. It had apparently failed after an IMF update had been installed earlier that day. So why did it fail? And why didn't the monitoring agent (which was running) log or broadcast any error message when it's service wasn't running? Details will be posted after TrippLight support had a chance to respond.

just a thaught for you, if the search engine is the hard part the simplest
answer is to let sombody else do it ;-> say perhaps google? they have
a search within a site type thing (eg the microchip site) also what search
algorithm is used here? i have some skill with ASP i'll help if I can. (thaught
that springs to mind is dump the MS indexing stuff and put the search stuff
into a MySQL database, that should be *plenty* fast.

James
Newton replies: Thanks for your suggestions: Yes, you can use google
to search the parts of the site that they have bothered to index. Unfortunately,
that doesn't seem to cover much.

The algorithm is proprietary to Microsoft and inside the search engine so
I have no idea what it is.

The one huge advantage of the MS Index engine is that it gets a notification
of any changes from the OS itself... e.g. it does not have to actually index
the site to know what has changed. This alone is a massive advantage. Since
the site is now approaching 3 GB of content and is constantly growing (about
200 new documents or changes per day) and the server is a very slow old Celeron,
my efforts now are, I think correctly, directed to hardware upgrades... The
new server has arrived, and I just need to get the OS installed on it and
move the site over.

James answers: Thanks for the input Nomel, but I have the latest Norton on
the machine. And I found the problem was in the Content Index as seen above.
I was supprized that others haven't run into this... And I do appreciate
your concern.

Questions:

asks:

I'm sure you have a lot of time invested in the current search engine but
have you given any thought to maybe porting it to a *NIX type system? In
my experience *NIX systems scale much better then NT not to mention handel
load better. Using a very simple shared SCSI system with OpenGFS (Global
File System) you could add more servers with ease. I want to make it perfectly
clear I'm not bashing the current system it is just that even (especially)
you seem understand that something must be done. Would you be open to the
idea of other list members giving their constructive input on how the system
could be upgraded? --adam

James Newton answers:

I did try with Linux (Red Hat) but I was spending all my free time
(of which there isn't much) either A) learning how to patch / hack / compile
/ etc.. the OS or B) keeping the script kiddies out. I asked for volunteers
to help, and I even offered to pay or turn the use of the box over to another
in return for hosting my site... I had no takers until just last week and
then only to help a friend solve an email problem. I want to spend my time
concentrating on the content and MS NT has been very, very good to me on
that point. My uptime on this little box has been quite good.

As far as I can see; With *nix, you pay for the people or time and with NT
you pay for the OS.

The server is an NT box which is a bit starved for ram at the moment: 96MB
only. But I'll be upgrading it again soon.

The only problem with mirroring is that the site accepts updates instantly
via the form you used to post your offer. Keeping two sites in sync is something
I haven't worked out, but I'll take any help I can get on that issue.

Ivan "
James
Newton replies: Thank you kindly. Frankly, I have neither the time
nor the desire to learn Linux. With the searches limited to one every few
seconds, the server does just fine.

If you were interested in mirroring the site or providing a search engine
from your own server, I would be willing to try to set up such a thing.

At one time I was looking in to using RSYNC (I think?) on the NT box to send
out notice of changes and keep a remote mirror up to date. The idea was that
the remote mirror could provide the search function (as well as the content
if desired) and so remove the burden from the local box.

Again, I'm NOT interested in hosting, learning, or in other ways touching
*nix. Nothing personal, just my preference based on my lone experience.

http://www.devx.com/asp/Article/21782/0/page/3
Shows how to decouple long running jobs from the ASP pages that are requesting
them. The initial request is assigned an ID, then the job is started in a
seperate process on the server via MSMQ. During execution, an ASP page asks
the client to repeatedly check the status of the job. When the job finishes,
the results are written to an SQL table and the next status check then retrieves
the job. Using this for the new dictionary
site

Every now and again, I find "RUNDLL32 SETUPAPI,InstallHinfSection DefaultInstall
1 {out}.inf" running in an unattached process. It turns out that is part
of a batch file that I wrote years ago to restart the server every night.
The batch file gets called by task manager and trys to run dll this inf file
that causes the server to restart. If it doesn't restart, for what ever reason,
the process sits there doing nothing forever.

After you find an appropriate page, you are invited to
your
to this massmind site! (posts will be visible only to you before review)
Just type in the box and press the Post button.
(HTML welcomed, but not the <A tag:
Instead, use the link box to link to another page.
A tutorial is availableMembers can
login
to post directly, become page editors, and be credited for their posts.

Link? Put it here:
if you want a response,
please enter your email address:
Attn spammers: All posts are reviewed before being made visible to anyone other than the poster.