Posted
by
kdawsonon Tuesday May 26, 2009 @01:15AM
from the what-matters dept.

Philip writes "Three years ago I was appointed as a network manager to a barely functioning MS-based network. Since then I've managed to get it up and running — even thriving — but have been guilty of being too busy with the doing of it to document the changes and systems that were put in place. Now as I look back, I'm worried that I am the only one who will ever know how this network works. If I get hit by a bus or throw in the towel for any reason, I'd be leaving behind a network that requires some significant expertise to run. Ultimately, this won't be a good reference for me if they are trying to work out technical details for years to come. It looks like I'm going to have to document the network with all sorts of details that outside consultants could understand too (no, I don't want to be the outside consultant), especially since it's likely that my replacement will have less technical expertise (read 'cheaper'). Are there any good templates out there for documenting networks? Is anyone who has done it before willing to share some experiences? What did you wish your predecessor had written down about a network that you inherited?"

Something I found to be VERY good policy that was implemented at my former position (as Network Admin) was to hand to the boss (CEO, CIO or whatever) a sealed envelope with EVERY relevant password (most importantly the admin password:) ), to be held in the company's vaults (if your company has such a thing of course, or similar).

Whenever an important password was changed, I would hand over the new envelope:)

I was taught that the proper way to store admin passwords for an emergency situation (such as admin gets hit by bus) was to1. Write each password on an individual piece of paper2. Seal each piece of paper in an envelope3. Store the envelopes in a Safe/Deposit box that a limited number of people have access to, (company owner, CTO, CEO, IT Manager)4. Put policy in place that requires passwords to be changed and re-recorded any time the envelope it is stored in is opened.

You can encrypt the file in such a way that any of a set of private keys can decrypt it. You can also encrypt it in such a way as that a combination of keys are needed (i.e. 5 keys used in encryption, and you must have 2 in order to decrypt.)

You can also send that machine's logs to all 5 owners, so that an intrusion by any one will be more likely to be noticed (and it will be harder to tamper with the logs.)

Public-key cryptography to the rescue! You can encrypt the file in such a way that any of a set of private keys can decrypt it. You can also encrypt it in such a way as that a combination of keys are needed (i.e. 5 keys used in encryption, and you must have 2 in order to decrypt.)

I don't think this solution is nearly as good as the envelope system. In particular, I don't expect the company owner or the CEO of the average small to medium business to have the expertise keep their private key files and password accessible to them and no one else. That means that either you're actually relying on two out of the other three keys being available (if those two fail to keep their private key files accessible to themselves) or you have poor security (if those two fail to keep their private key files accessible to no one else), whichever is worse.

In contrast, all of the trusted people probably know how to keep a physical key reasonably secure and accessible, and if you use a deposit box at a bank there may be identity checks as well.

So what happens if said predecessor gets hit by a bus, has a heart attack or a stroke and can no longer tell you the passwords? Or, worst case scenario, the whole IT team gets taken out in a road accident on the way to a team building session for example? I've read a few "deserves to be fired" comments on/. and usually tend to agree (or occasionally get embarrassed because I think hmm, that's me!), but in this case you are being a fool.

Of course, if you are dead then you won't care if they have the passwords, but some of us actually like our places of work and even our colleagues, and want our place of employment to be able to chug along even in our absence.

Reboot the DC into "Directory Services Restore Mode" (this is on Server 200 and above) and the local Security Accounts Manager is used again, not AD.

This is actually a way of resetting the admin password on a domain, if you ever need to. Boot from the Linux password reset CD (http://home.eunet.no/pnordahl/ntpasswd/bootdisk.html), blank the Administrator password (which resets the _local_ admin password), then reboot into DS Restore Mode. Log in and then copy cmd.exe over on top of logon.scr and reboot into "normal mode" (with AD functional).

Wait until the "screensaver" pops open and you have a command prompt that just opened that's running with SYSTEM privledges. From there you can run mmc.exe (or whatever else you like).

Oh and of course, this is exactly why you don't allow physical access to servers....

In reference to the PDC statement, I'm not aware of any way to use local accounts on NT 4 or below, but I think he meant just DC instead of "PDC."

If you are primarily a Linux shop you might use Coda or NFSv4, but if you still have Windows clients you might also run Samba. Actually, Samba can serve CIFS and not just SMB; CIFS has extensions for Unix features. So it's not a bad choice for serving files to Unix, but it's no NFSv4. (In some ways, that is a good thing.)

If the predecessor does write the passwords down, he deserves to be fired.

That's knee-jerk stupidity, and you should be ashamed of your non-thinking fundamentalism.

Passwords do need to be written down, and stored in "escrow". I put the list of passwords in an envelope, lick seal it, sign and date the seam, and then seal it again with clear packing tape. Give it to the boss to put in his safe.

Try spraying the envelope with refrigerant. The paper becomes translucent when wetted and you can sometimes read what's inside, and then it dries without a trace (unlike wetting it with water, which swells up the paper fibers leaving the telltale signs of tampering.)

Learned this one from a history of the U.S. Black Chamber [wikipedia.org].

Try spraying the envelope with refrigerant. The paper becomes translucent when wetted and you can sometimes read what's inside, and then it dries without a trace (unlike wetting it with water, which swells up the paper fibers leaving the telltale signs of tampering.)

Learned this one from a history of the U.S. Black Chamber [wikipedia.org].

Does anyone not use "security" envelopes anymore? (The ones that are printed with a dense line pattern on the inside?) I didn't know they sold plain envelopes anymore, except for greeting cards.

If the predecessor does write the passwords down, he deserves to be fired.

Either he writes the passwords down, or he uses weak passwords that a human mind can remember.

Besides, a password is a security token. A piece of paper or a little plastic card with the password printed on it or a USB stick with SSH key or whatever saved on it are also security tokens. They aren't inherently less secure than memorized passwords; you simply have to secure the physical object by, for example, locking it in a safe.

"Passwords should never be written down" is an idiotic rule, right there with "never use goto".

If the predecessor does write the passwords down, he deserves to be fired.

You can't always take it for granted. I've been on the cleanup-end more than once. Sometimes it's "engineered job security"/"they don't dare fire me", sometimes it's "I don't have time for that", sometimes it's just plain forgetting about a piece of hardware you set up the first week you started work there, and sometimes it's a legacy password issue. ("nobody's been able to login to that box since Phil left in '05") The rare treat is finding a mystery box that nobody knows what it does nor has any idea how to login to it. Too many managers don't understand the danger and consider it a waste of time or a bad risk to try to fix problems like that.

Sometimes it's an uphill battle when taking over, too. You want to document the settings on all the routers, but two of them are "legacy" password issues, and the router only supports hard reset (clears password, AND all settings) so you can't get the settings from it once you reset it, and you need the settings from it in case it gets reset. That's never fun, but you will find yourself in that catch-22 occasionally, and it's hard to blame someone for not fixing it because it's utterly unpleasant to deal with. (hint: get another router and program it how you think the mystery box is set up. swap. test. immediately swap the mystery back in. adjust the settings on the new one and test. Swap back out. repeat until you get it right, and don't reset the mystery immediately, keep it onhand for at least a month in case some uncommon thing requires a setting you haven't yet discovered)

Occasionally you can get lucky - contact the hardware vendor and see if the box has an undocumented soft reset. ("open it up and short together the two pads left of D-15, and you can login for 5 minutes with no password")

Or how about a new CTO pissing both admins off and one walking out as the other was fired for no good reason.

I've had to walk in behind something like that several times and reset the passwords or load the password hashes into some cracker in order to find the passwords. A lot of times, you can pull them from workstations the old admins used and they are easier to crack then the newer MS servers. The funniest thing is that the CTO usually wants me to stay on full time and gives me a dirty look when I won't do it because he made life so miserable that two other people walk off the job under his supervision.

This may sound funny, but I recently had the same experience. I took over the position of CTO of an electronic payment company, and after one week, I figured a lot of critical systems are missing root password, including Linux, AIX, HP/UX and SCO Unixware. No one knows the password, it's been changing hands so many times, and the people who were responsible for those machines have left, without leaving the passwords behind.

Those are critical systems that must run 24x7. We had to rebuild the system on new machines, re-route transactions to the new machines, and shutdown the old ones to recover (single user mode).

And that's a platform handling over 400 billion in transaction per year. Scary. But that's the easiest problem I have inherited, mind you.

I note that GP was very careful to mention "systems" as a plural a lot. There is a cluster near me at the moment which nobody knows the root password to (until a recent local kernel root exploit at least) because of a very similar situation. Thankfully we got them back with a lot less trouble than this one, but there it is.

The nice thing about post-its is that they can be updated very easily when you're fiddling with the device.
That doesn't work if you're configuring it remotely, though.:|

The next step up from a Post-It, though, is a snazzy label-maker. My portion of the company uses these extensively to document our development lab (we do some NMSy stuff). Of course, it's not a production network, and standards are a little different.

Your successor will never find any documentation that you leave behind (or if you show it to them they won't bother with reading it) and by the time they notice it they'll have already screwed things up to the point where the documentation will be obsolete. This means you can save yourself the trouble of doing the documentation unless that documentation is going to make you more effective while you're there.

Nor again is there anyone who loves or pursues or desires to obtain pain of itself, because it is pain, but because occasionally circumstances occur in which toil and pain can procure him some great pleasure. To take a trivial example, which of us ever undertakes laborious physical exercise, except to obtain some advantage from it? But who has any right to find fault with a man who chooses to enjoy a pleasure that has no annoying consequences, or one who avoids a pain that produces no resultant pleasure?
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain.

When I was part of an IBM sales team in The Old Days we used to scare prospects by asking to see their network diagram. Almost never appeared.Then we'd ask who owned the network. That was good for laughs.Finally we'd ask why the most important part of the network (the end user) didn't appear on the diagram (assuming they could produce one).By the looks of it nothing's changed.

Yeah, I think that's more or less true. At one of my previous jobs, I had a guy try to imply that I didn't deserve my pay because I "wasn't doing much". When I asked him what I should be doing, he said, "It's just that you have a really easy job. The IT guy at my last job had it much harder. He was always running around, fixing things. You just sit at your desk because nothing ever breaks."

I can't remember now, but I think I might have done a literal facepalm right then. I said something like, "Has it occurred to you that, if you think none of our IT stuff ever breaks, I must be doing a good job? If the IT stuff at your last job kept breaking all the time, he was doing a worse job than me?"

Sounds like a very easy way to over work and over stress your self, get some help one way or another. Summer is coming and I'm sure there are plenty of Comp Sci/Network Engineer/IT students that could of help. It may not be a bad idea if you make a plan of some kind before you go head in.

1. Simply begin documenting, its a work in progress..never finished.
2. Select a worthy staff member from your org, with the brain cells (and desire)to start learning networking, and begin to train him/her on what you are documenting.
2.a refrain from selecting the network thinks-he-knows-it-all type, instead pick anyone else with the skills listed in 2.

A good tool like dia which can allow you to create a network diagram. When it comes to documenting a network, a picture can be worth a thousand words. Or you could also use MS Visio as it is, perish the thought, a good tool. A good, detailed diagram can come in very handy as a reference tool for your own use in case of a failure.

Etherape is gorgeous. I wish I had time to understand what it all means, but still I love firing it up and staring at the network traffic.

Another tool that I've enjoyed much more personal success with is cheops-ng. For documenting a networking, cheops-ng and a decent icon collection provides a pretty snazzy view of what NMAP can see. (NMAP being another tool I haven't had time to fully grok, yet)

A summary diagram (dia/visio) is still pretty key, but nmap (or specifically the latest versions of zenmap (the gui frontent to nmap) provides a good detailed view of network topology and if you include the capture file, you can annotate specific computer details in the notes section (ofc you have to assume your replacement will be familiar with zenmap for that to be useful though)

i just got a work-study job with the campus adming at the community college i attend. hes been there almost a decade and has no network monitoring system, so he has no idea when something goes down until he gets a complaint or cant get something to work himself. i thought an interesting project during my time with him might be to see if hed let me help implement a network monitoring system, and ive seen a couple of people use The Dude before a

I've found that starting out with the very basic physical layout and working your way up in complexity is greatly beneficial.

i.e. start out documenting network cable runs including cable type. follow it by switch layout. follow that by routers and vlan setups. follow that by the servers that provide basic network functionality(e.g. DHCP, etc...). If this is a windows network, that would likely mean detailing the domain controller setups. From their systematically document the systems in order of importance to the business, etc...

Short answer: don't worry about it too much. Put together enough that it looks like you've done something then go have a beer.

You could have the most amazing docs the world has ever known - with passwords and clear instructions - ad the odds are about 20% that the next guy will even read them.

The next guy will figure that he/she knows much more than you as evidenced by the fact that they are there and you are not. And, the cheaper they are (read: inexperienced) the more likely this is to be the case. When things go wrong, they will blame you anyway.

So document away, but for YOUR sake so that if/when you are called in after the new guy horkens everything, you can have an easy time putting it all back together. But don't wait for the call... people will put up with almost anything when pride is on the line.

No wonder our field and many of our professions have such a bad reputation.

I have read only a few posts and two (moded up 5) say pretty much to ignore the issue.

In several networks I have worked with fundamental information was non existent. This translated in lost time, down time and actually losing money (if you lost your job in one of those companies recently, the indolent SAs or Network administrators may be partly to blame).

You never know who the next guy will be, if he is less experienced or capable then the documentation will be very valuable, if he is more experienced or capable then you would have saved their time to do some real work, after all they (and you) have not being hired to do forensics.

How a professional can hide behind the "let's be real" nonsense is beyond the pale.

For me, visio's are great and everything, passwords too, but really the most valuable thing you can do is document single points of failure, outdated software/hardware, etc., license keys/expiration dates, cert expiration dates, personal support contacts you have and all vendor relationship details as well are essential. Do you use change control? If you do, go back and comment your changes, if not, do the best you can at explaining why things are the way they are. Get some open source software that is good at indexing data and create a searchable knowledge base from the information above. Don't concentrate on docs that can be found on the web at first because any admin worth their salt will know where to look for how to's, etc. Focus on the why's, the where's and disaster recovery.

1. Viseo overview of the network drawing with complex areas drawn out specific detailed viseo's (even a scanned sketch or paint drawing is better than nothing)2. A spreadsheet with circuit ID's mapped to router and interfaces.3. Document the trunk interfaces as well as the LAG's (Link Agrregation Groups, port channels, whatever you want to call it)4. TACACS passwords / domain logins in a secure location (or radius or diameter or whatever you use)5. Data center capacity as a function of 1. Rack Space, Cooling Capacity, Electrical Load.6. Write brief knowledge articles describing any problem areas and explaining a history of anything you think would be hard to figure out easily. No need to go hog wild, just re-brand the RCA documentation you have. You do have Root Cause Analysis right?7. Network protocol hierarchy map. Where are your major redistribution points, what is your routing strategy etc.8. If you have a voice network document all your DID's, PRI trunks, Gatekeepers, Dial Plan, and any translations you use on h.323 gateways or how MGCP or SIP is configured. If you have a complex call center you should probably pay to have it professionally documented in down to the minute detail.9. SSID's and BSSID's for any wireless you may have as well as passwords, 802.1x authentication methods, along with linking documentation.10. Make the documentation part of your CMR process (Change Management Review) and incorporate it into the time allotted for a change.

I know these are just rough ideas and you should get many more ideas from all the smarter people on here than myself, but whatever advice you get I would say you would need to have the documentation update able via subversion, or some document control system and have some kind of review process for it, even if it means getting together over pizza with some of the other groups and asking them about their environment and getting pointers and possibly help on documenting it all. Documentation is a full time effort and IMHO there is no such thing as too much documentation. You would be surprised how good documentation can aid you in problem resolution down the road or aid vendor support in helping you resolve a major outage. The three basic principles of network care are document document document.:)

Network diagrams should be at a network, physical and datalink layers. Only the simplest networks can have all this information on a single diagram and have it be useful. Seperate the network drawing from the datalink and physical drawings as requred but be sure to leave enough detail to connect the drawings (Visio has a nice linking feature for this). Also keep a spreadsheet or database of assigned networks, IP ranges, and assigned static

Actually, it's a simple test designed to separate the people who should be working with the corporate wide area network from the people who should be forcefully prevented from coming anywhere within fifteen metres of the comms cabinet, using a taser if necessary.

If you have to ask which group your answer has placed you in then I'm not going to tell you.

On the side, I manage a small network, and I've also wondered the same sort of thing: if someone else needed to find their way around, where would they start.

A Wiki makes for a really nice way to document things, not least because you can include all sorts of cross references. For example, a list of servers, with links to the services they provide - and a list of services, with links to the servers. But Wiki's normally run on servers, which leaves your successor with a chicken-and-egg problem.

A bit of random surfing turned up TiddlyWiki [tiddlywiki.com], which is a Wiki in a single HTML file. A really elegant bit of engineering, and very handy for self-contained documentation. Since the entire Wiki is just a single file, it's easy to protect. I wound up with two: one with "public" information describing the general architecture and one with private information (including passwords). The private one you can put on a USB-stick in a safe, hand to your boss, or whatever seems appropriate...

At my last few companies and my current one that I work out, one of the first things I do is setup an internal only Wiki server.

Not only does this let me document everything I can about the network but I also try an train my co-workers in using it to document information they feel is important for their job too.

The effectiveness though seems to be related to the level of computer literacy of the user, i.e. my last company the software developers went crazy with it and documented everything they could think of. But other than them or us in the I.T. dept, no one use would hardly touch it at all.

Either way it's still a simple method for your to store notes, diagrams or any information about your network in an easy to find single location that you feel would be important to the company should you leave for any reason.

Unlike software documentation, in a network there is more than one approach, pretty much a function of the amount of people that have to fiddle with the network at any given time. Usually there is physical laying of cables (where are they), box location and naming (labeling and Visio-sheets), peripherals (printers, faxes, phone system), box setup (OS and processes), server process configuration specifics and client requirements. And then there's that (what I think) very important document that describes w

Am I the only one who first thought, "If you have done the network correctly, it should explain itself"?
Overly-complex networks take overly-complex documentation and overly-complex people to run and maintain. Mind you, there's a difference between correctly-complex and incorrectly-complex, but at the same time, every level of difficulty you go upwards in the configuration scale you exponentially increase the need for carefully calculated and layed-out systems.
Ok, ok...now that I'm thrust back into rea

The phone numbers/emails for points of contact in other departments/companies.You likely don't run *everything* and the new person needs to know who to contact when the interaction between inside and outside fails.

If the network itself (switches and routers) is built on Cisco, there are commercial (SolarWinds) and Freeware (NeDi [www.nedi.ch]) tools to document the interconnections, VLANs, and configs.
Assuming you've left CDP enabled and standardized your SNMP community strings, NeDi does a fine job of documenting a network with minimal effort. I've found switches the network team didn't even realize existed just from using the NeDi discovery mode and adding 'public' to the list of SNMP strings to try.

Present client I am at I inherited a network of about 15,000 clients that was previously managed my a very incompetent IT department. Started by looking at the existing flowcharts and discovered that almost everything that was documented was wrong... Long story short I have been spending a fair bit of time reverse engineering their production environment so that we could accurately document it. Unfortunately we had come to the conclusion that we can use/nothing/ that the previous administrators left behind for documentation. You don't want someone like me coming in and looking at your documentation and declaring you incompetent, it can cost you your job.

You haven't detailed the size of your organization to know if you will need sign off from other departments or not. If possible try to get sign off so that they have a reference and you can create a standard that can be used to fix things and to ensure your designs don't get trampled by a new admin in another department. You really need to provide more detail on your environment for people to answer you.

I do most work in Visio, starting at 50,000 feet and working my way down. At this level I need to document network topology, server distribution and database server distribution. I work my way down from there using a zoom in style that has served me well for 30 some clients. Depending on the size, complexity and your area of responsibility you may need to flowchart anywhere from a 2-3 levels to potentially dozens of disparate processes. You haven't mentioned much about process development, I assume you want people to know how to do at least critical portions. Never write a process without flowcharting it, this will save you grief by getting people to focus on the process instead of a step by step set of directions. It takes someone fairly good to document the complex and make it look simple, that is your job at this point in time.

The bottom line is that your documentation should show dataflow for each critical system. As long as you can do this someone else can step in and work with what you have, even if they may not understand a given piece. One of the big advantages of flowcharting everything (especially processes!) is that this will readily show you weakness and holes that may have been previously overlooked. When flowcharting complex processes don't be afraid to have a single point represent an entire additional complex process that can be distictly referenced of it's own accord (as an car repair manual of mine once described the process to replace a crankshaft "Step 1. Remove Engine".) If you try to put to much detail in a given process you lose your audience and the value of the documentation.

Bottom line when I am done with a design document it covers server, network, database and client topology in varying levels of detail with dataflow. A typical design document I would turn over would be 150 pages with most of that broken down into different sections describing what was done, why it was done, the best practices followed for build, and best practices for lifecycle. The document typically does not get read by any one person, instead it would be a reference for a number of different departments that will each reference it according to their own needs.

Ok, I think that establishes you as a system auditor, rather than an administrator. Without having seen the documentation you produce, I'm not going to judge either way (I've seem some 150 page documents that are invaluable as a crib sheet for some systems, and I've seen way too many 150 page documents that aren't worth the paper they were printed on).
From the sound of the original poster, he's in a tough spot. Way too much to do, and not enough time to do it (sole network admin).
I think this Dilbert ca [dilbert.com]

You are both on and off the mark. First, I'm an enterprise architecture consultant for a living, I've done lifecycle administration in the past and do some now. I've certainly done audits, even to the point of being brought overseas, but that was only about 20% of my work. Once the audit is done my job was typically to follow up with how to bring things up to par. Staffing, architecture, servers, licensing and bandwidth considerations all come into play and receive my recommendations. I am far more likely t

If you are in the server room, and you have:A: a spreadsheet that your predecessor made.B: a post-it note on the switch saying it what it does.Which one do you trust?

For the physical/low-level network the documentation should be in the network. Just like source code should contain comments about this particular piece of code, a similar approach works reasonably for the physical network. I see no point in a having an outdated spreadsheet. It is more useful that the cables and ports are labelled and numbered, that there is a post-it note on a switch say where the links go, etc.The grand overview should be in electronic form, though. A scanned hand drawing is fine. A photo of a whiteboard drawing is fine too.

The MACK Truck Rule (MTR for short) is a measuring stick which we use do determine if a solution is good for us. Basically, it's an objective measurement of the level of expertiese required to do something. Basically, the MTR has you ask yourself (Or your team) the following question:

If the person(s) responsible for a task was suddenly hit by a MACK(TM) truck, How much time would it take for somebody else, untrained, to complete that task if needed?

If that amount of time is unreasonable*, It doesn't follow the MTR. Notice the caveat for unreasonable; this is the subjective part. What' unreasonable for one may be reasonable for another. This needs to be decided for yourselves.

Documentation always helps difficult tasks pass the MTR. So can good support. I try to leave a readme in the place where the installer is for a difficult program. I'm now begining to use FreeMind to map out networks and servers. I have a good ticket system for all our repairs. Hopefully these things will make things easier the day I want to take a vacation.

We used to have a similar 'BUS' rule (ie, what if so-and-so got hit by a bus) until someone we all knew got hit by a bus. That sucked, he was a good guy and we had just referenced that joke a week earlier.Now we have the 'LOTTO' rule (ie, what if so-and-so hit the lottery and left the company to be independently wealthy.)We all miss Dave. And most of us secretly wish he had documented his fucking code before getting hit by that bus.

I have a wiki set up for the company I admin. Each server on the network has its own page, with a standard set of categories...

Purpose

Access (IP addresses, names, special ports)

Services provided (and descriptions of the services)

Maintenance (to do at intervals)

Quirks (stuff that tends to go wrong and what I've done about it)

Installation Notes (anything special I did when installing the server or any software on it)

Captain's Log (an entry for each incident involving this server, what its symptoms were, what I did to fix it, etc.)

I have a nicely formatted template page with all those categories set out. I also maintain a page of IP address assignments and an inventory of harware specs of all the machines in the office (which is helpful in the cases of "We need to reproduce a bug that only happens on ____ processor with ____ video card" and of "We're getting new machines. Who is in most dire need of an upgrade?").

I write down everything in these, and find myself referring to them very often. My predecessor gave me a Word document with all his notes in it, which has been very useful, and I used that as a starting point for my pages. The wiki has saved me a ton of time, kept me organized, and serves as a great reference for me and for the inevitable next admin.

The only caveat is if the wiki (or the server it's on) goes down. This has happened once, and my instructions for fixing the wiki were... on the wiki, so extra troubleshooting for me. Thus, I find it good practice to maintain a hard copy of the wiki pages, especially the page that tells how to fix the wiki.

I'm running this on Redmine, which has proven to be bleedingly simple to use and administer, and much easier than trac, which we used before. It's especially nice having it on the intranet, as I'll just have a browser open to the wiki as I work on systems and refer to and update it as appropriate. It's very handy to document exactly how I performed a strange or experimental installation of some software that I'll want to replicate later without making myself crazy, and I'll take the extra few seconds to retype the commands I just used into the wiki from anywhere in the building, though I probably wouldn't do the same into a Word doc.

It's not so much the mundane day-to-day that I find that important to document. It's the weird fixes, the trouble spots, the command line parameters, the installation procedures, the changes that shouldn't have fixed it but did, and the horrific chain reaction situations that make one piece of software crash because a seemingly unrelated piece of software has the wrong version of the 64-bit library. Things that take 4 hours to figure out and 3 seconds to implement... those are the ones to document, and those are the ones that I'd be kicking myself 20 months later for neglecting to write down. In an afternoon, any schmuck could walk into the building and figure out which network cable goes where. Documenting the strange bits (and the frustrations), though, can get a malfunctioning mail server back up and running in 3 minutes instead of 3 hours (which, of course, is secondary to good administration keeping the server from going down in the first place).

There are a few things that are often overlooked and outright forgotten when documenting networks. I had to take over a few networks, let's see what I usually miss:

Every admin remembers to hand over passwords. Except for the routers.Routers and other "managed black boxes" are notoriously being left out from the list of passwords. Fortunately, more often than not it's the standard password because "nobody has to touch them but me anyway" (ignoring that, if people only touched what they should, passwords would be moot...)

Every admin remembers to draw you a network layout. They don't tell you WHERE those switches physically are, though.In large companies (read: Lots of room to cover, independent of the number of people working there), this can indeed be a problem. Especially when there's not one single server room where everything is collected, when you have switches and routers hidden in cupboards and other "innocent looking" furniture, cables that appear out of nowhere and disappear into walls, without an indicator where they surface again. Or what purpose they serve, first of all.

What HAS to be documented is the reuse of resourcesThat's the worst of the "undocumented changes". When you find a switch that shouldn't be there, you know you have to investigate, you know something wasn't documented. When you find a certain box sitting where it is supposed to be, you don't investigate. You expect it to do what it allegedly does. If it does not and has been "recycled" to fulfill another role, the whole documentation goes out the window. Because now you start questioning EVERY piece of hardware.

My favourite technique for making sure documentation is done and updated, get the new guy to do it. Then he/she has to go all around the campus, locating servers and getting serial numbers form all sorts of odd equipment and making sure all of the support aggreements are current and the contact details for the vendors are accurate.The other favourite is if I find new equpment that has been installed and is not labelled or documented, I get the installer responsible to audit all similar equipment to make sure there are no other ones missed out. After haivng to crawl around dozens of risers and labelling or confirming all switches etc are correct and documented, they don't often make the same mistake twice.We also have a password management system which also allows details like how to install the management console or the URL to access a system for management to be stored.My answer to any question about "What is the password for X", or "what the hell is the name of the server for X applicaiton" is "Its in the store" Then if it isn't, we add it 8) Only takes a few times for the newbies to start looking up the information themselves first.

The other key file is a massive Visio document with a summary page with a managment style overview, and then a document with everyhting in it in layers like an electircal diagram or building plan.Lay in the workstaitons VLAN, the switch management VLAN, the Servers VLAN, link to things that are self contained like all of the Firewalls and DMZ configurations.etc.

First: You must make everything as self-documenting as possible. Label every server, every cable, every power lead to within an inch of its life. And establish processes which say "when a cable is moved or added, labelling is updated accordingly". If you don't have a labelling machine, buy one.

That deals with basic "what's plugged in where" and is far more likely to stay up to date than a spreadsheet or wiki page.

Second: Whatever you choose, it must be something which can scale to your needs and which you can live with.

It will need regular updating - and quite frankly, very few people are able or willing to regularly update a single 200 page Word document complete with embedded spreadsheets, diagrams and photographs. A wiki - or even Sharepoint, if that's your thing - may be better. But if you do take the Wiki route, make sure you keep hard copies of the documentation which says "If the sh1t hits the fan, this is what you need to do to recover".

Others have said "don't bother, your successor won't read it" - I say balls. Documenting is more than just helping your successor - it also helps you remember what is set up, clarify how things work and as part of the process you start to look at things and think "hang on a minute.... this document I've written describes something quite absurd. Are we really doing that?"

If you're a masochist, you can always try and follow the DoD Architecture Framework [wikipedia.org] which defines multiple views of architectures (including networks). Once finished, there shouldn't be any question of what your network is, what it does, and how it does it, but you'd probably need an army of peons to put it together.

At home I have an inbox which gets ~200 spams a day at the moment. 1000 at the peak. Every day I browse the Sender field just in case it contains a legit message, then I delete the lot. Last night I noticed an email from an old friend who I hadn't heard from for three of four years. I opened the message and sure enough it was spam. I am pretty sure it came from her PC. My old address must still be in her address book.

Attempting (even facetiously) to blame SPAM on Windows is wrong. If every copy
of Windows on the Internet somehow magically disappeared, the SPAM problem would
not abate. Bot herders and spammers would simply shift their efforts to other
platforms.

OK fanboi, did you even read the link I referred to? Here's an excerpt:

Safari on the Mac is easier to exploit. The things that Windows do to
make it harder (for an exploit to work), Macs don't do. Hacking
into Macs is so much easier. You don't have to jump through hoops and deal with
all the anti-exploit mitigations you'd find in Windows.

It's more about the operating system than the (target) program. Firefox on
Mac is pretty easy too. The underlying OS doesn't have anti-exploit
stuff built into it.

"The network started horrible, here is how I cleaned it up" is a GOOD reference. I have killer references from two jobs I automated myself out of this way. Each time I got a more interesting more challenging better paying job by doing so.

Those who don't document don't have job security. They are an insecure leach sucking up a paycheck fearing -- and rightfully so -- the day they are going to be replace.

Those who DO document show their value to the organization, and should have no fear of being replaced. Their position is secure -- and should they go elsewhere -- they have something to show of and for their work.

I disagree with the parent vehemently and will say so based on years of experience as a techie, a techie manager, a manager of techies, an executive, and (thankfully) a techie again. You can never document too much, but those who don't cost the organization more in the long run each and every time.

I'm familiar with the ideas behind electronic discovery, and I think ultimately it is just going to go away. It is the brainchild of people who want to treat the entire electronic world the same way they treated the paper processing world, and they have no idea how much data they're actually trying to wrangle (much less the costs involved in wrangling it). Entire industries have cropped up to feed off of the eDiscovery nonsense, and you can directly measure how much productivity is being sapped from indus

I also try to stay away from documenting things in static formats. If you ask me, incorrect documentation is even worse than no documentation at all, and the fastest way to get incorrect documentation is to create a process that relies on a person doing all of the updating manually.

I know a lot of people love their spreadsheets and their diagrams, and maybe they update them religiously. Nonetheless, that process is *always* prone to error. And if a technician goes to a document for information and finds out that the document is wrong, the document loses its credibility. If that happens a few times, the technician will simply stop trusting the document, and it will just fade into obscurity.

If you want to document a system, look for ways to make the system document itself. Switches keep real-time lists of the MACs that are connected to them. Routers keep real-time lists of which MACs map to which IP addresses. Routers and switches will always tell you their current configuration if you ask, and you can automate the process of asking and storing and checking for changes. Most servers will tell you their serial numbers automatically if you ask them, so you should automate the process of asking them and storing that information. The same goes for what kind of hardware is in the server, where the server is attached to the network, etc.

So much information can be collected automatically rather than recorded by hand, and when you collect the information automatically, it will always be up to date. It will not matter if a tech decided to re-rack a server in the wrong place -- even if they didn't write it down, your network knows that it moved, and it will tell you if you ask it. So the next time you sit down to write a 200 page document describing the network, you should ask yourself a) how much will it cost in time and effort to keep this document relevant, b) how likely is it that the document will become out of date either through accident or negligence, c) how quickly will people abandon this document if it does become out of date, and d) aren't there huge parts of this document that could be totally dynamic instead of written in static text on a page?