Don't Do This: a thread for enterprise IT anti-patterns.

You should not have your receptionist using the secondary DC as her personal PC. really, it isn't a good idea.

Wait, what? Someone made the secretary a domain admin?

Apparently so. The story was related to me by a co-worker. He was investigating logon issues and came across this.

Small company I consulted for had the CEO's secretary using the tower terminal server as her day to day desktop. About 30 other people remote logged in to access the inventory app. We knew all about this because the company thought it was "silly" to have an unused tower "just sitting there" and wouldn't buy her a desktop. She liked IE toolbars and bonzai buddy style apps...

It's great, I have several hundred servers arranged by tags, and can access many simultaneously in a single window, with tabs for each environment. (note tags/tabs difference!)

Open-source.

This. I tried most of the open source options, Terminals was the only one that stuck. Tagging was quite helpful for organization, and you can import/export to easily keep in sync between management computers.

We moved from RoyalTS when it started to only come in MSI (Windows Installer) format, because we can GET Zip files through the firewall, but not .exe or .msi, AND we don't have administrative access 'on the box' (on our workstations) to install foreign MSI's.

You should not have your receptionist using the secondary DC as her personal PC. really, it isn't a good idea.

Wait, what? Someone made the secretary a domain admin?

Apparently so. The story was related to me by a co-worker. He was investigating logon issues and came across this.

I had a client's office manager (and wanna be SysAdmin) who dragged a server out to his desk and set it up on a KVM a couple years ago. He did this so he could have 2 workstations since "the server isn't really doing anything anyway". *sigh* About $1200 worth of my time later, he had been told in no uncertain terms he needed to do his job and only his job. A couple of months later, the client had a new office manager who's much easier to work with. Not only does she knows what not to do with the computers but she's smokin' hot and likes to talk tech. If only she weren't married ....

I had a client's office manager (and wanna be SysAdmin) who dragged a server out to his desk and set it up on a KVM a couple years ago. He did this so he could have 2 workstations since "the server isn't really doing anything anyway". *sigh* About $1200 worth of my time later, he had been told in no uncertain terms he needed to do his job and only his job.

I was doing work for a company that had a branch office in Las Vegas. The owner wanted all of his branch offices to be able to communicate with the company's headquarters, so one of my colleagues flew out there, put in a FortiGate-50B, and set up an IPSec tunnel back to HQ (along with anti-virus, web content filtering, etc.). This setup worked great for about a year.

One day, I get a call from the company's operations manager, who tells me that the folks in Vegas weren't able to access the file server at HQ anymore. I checked it out, and I come to find out that the office manager there had, without informing HQ, a) switched to another ISP, and b) decided to replace the FortiGate with a Linksys wireless router. During this process, the FortiGate somehow got "lost."

I had a client's office manager (and wanna be SysAdmin) who dragged a server out to his desk and set it up on a KVM a couple years ago. He did this so he could have 2 workstations since "the server isn't really doing anything anyway". *sigh* About $1200 worth of my time later, he had been told in no uncertain terms he needed to do his job and only his job.

I was doing work for a company that had a branch office in Las Vegas. The owner wanted all of his branch offices to be able to communicate with the company's headquarters, so one of my colleagues flew out there, put in a FortiGate-50B, and set up an IPSec tunnel back to HQ (along with anti-virus, web content filtering, etc.). This setup worked great for about a year.

One day, I get a call from the company's operations manager, who tells me that the folks in Vegas weren't able to access the file server at HQ anymore. I checked it out, and I come to find out that the office manager there had, without informing HQ, a) switched to another ISP, and b) decided to replace the FortiGate with a Linksys wireless router. During this process, the FortiGate somehow got "lost."

Running Exchange 2003 in 2010 is maybe a bit behind times. But running a release candidate in production up to 2010 is just nuts. Since I have seen that, I consider Exchange more solid than before.

Having a big network printer in a server room with network equipment consolidates space and the AC makes sure that even during large print jobs it's not getting too hot and the ozone emitted is immediately disposed of. That is a valid point for employee health. After the second time you maybe learn to not use the same circuit for the printer and the main Cisco.After the first time the thing catches fire and burns your network down, even non-IT people start to get what potential problems such an idea may have, despite it's benefits. Installing AC and ventilation in another room for a printer would have been cheaper than some hundred people not working for more than a day.

And I somewhere have a pic of a server room that is too cliché. A wagon from the cleaning personnel, complete with a 5l of water and detergent on top, blocks the doors of the server racks. And I've been told that's where it always. With the water bucket on top.

A few jobs ago, after I've just joined this company. I'm asked to look into why a few simulation servers in HQ are running so slowly. After looking around I find that the are not ,as the folks using them thought, real servers, but instead Xen VM's.

I get them to hook up the LOM ports on the physical servers and get into a stripped down build of centos with a custom xen build. I also find the reason for them running so slowly.

The servers had been bought with 64GB of ram, and then that was swapped out for 6. The xen host was setup with 96Gb swap and the custom xen build was hacked up to minimize the lightly-hood of anyone detecting that they were running in a VM. I encounter more crazy like this and we find that the sys admin there has been stealing equipment and running his business out of hq and off the HQ netapp. No wonder he was so twitchy and reluctant in the extreme to let anyone have access. Douche ++

And I somewhere have a pic of a server room that is too cliché. A wagon from the cleaning personnel, complete with a 5l of water and detergent on top, blocks the doors of the server racks. And I've been told that's where it always. With the water bucket on top.

Anti-pattern: using a boiler room as a networking room in a campus building. When the boiler explodes some time later, the open telco rack will be doused with rust-filled water. The equipment remains in service for years afterward, still covered in adhered iron oxide where the boiler water evaporated.

Anti-pattern: making a mistake in infrastructure layout in a new building and needing to use a planned Human Resources secure storage area for an IDF. Relocating the storage area would unduly burden the department. The secure storage area will remain under the control of HR doyens and thus not accessible to technical personnel, especially during off-hours emergencies. It will also be a citable example of a common fire hazard.

Anti-pattern: Having an IT department that's isolated from the rest of the business for idiotic cultural reasons, which in turn forces the business to make extreme changes to the department simply at larger intervals.

Anti-pattern: RAID 5, and why are we still using it? SCSI drives no longer cost $600 for 9gigs, so stop aready with the delicate pampering of the parity stripe. How many times do data managers have to smash themselves in the hand with a hammer before they realize it hurts and keep losing terrabytes of data due to RAID 5 corruption? No, two drives didn't just fail in sequence. Your stupid EMC software is just reporting it as such because it's too stupid to see what's actually going wrong.

Anti-pattern: Letting layer 3 geeks over-see layer 7 priorities. A chair is for sitting, a glass is for drinking, and a switch is for switching. Why are we spending $150,000 for a switch upgrade and how will it improve our company's ability to do business? It wont....it just gives you cooler tools to play with. BTW - that chair and glass doesn't need a management console, but Cisco corp probably says it does so we have to hire somebody to manage it. As far as I'm concerned, the lower the number your OSI expertise is the less you matter.

Anti-pattern: Campus / central model networking. We have a shiny new data center, so lets consolidate all our remote node servers in one spot so we can justify our fiber channel switches while end users suffer with VDI and authentication performance that makes lantastic / Windows 3.11 / Pier to Pier look 'speedy'. You don't care - that's what you're paid to do (wrong).

Anti-pattern: shoving everything in a fancy, glass caged data center with superfluos blade density generating thermal at about the same core density as a Russian nuclear reactor. We all know of course that thinner blades are better blades. Oh yeah, don't bother to check that the 1,000,000 watt AC unit in the data center required to power all the virtualized desktops the end users hate isn't on the same circuit as the back-up generator. Make sure all those servers are running RAID 5. Thunderstorm over weekend - power goes out - blame it on maintenance.

Anti-pattern: Having an IT department that's isolated from the rest of the business for idiotic cultural reasons, which in turn forces the business to make extreme changes to the department simply at larger intervals.

Anti-Pattern: Silo'ed departments that interact on a very regular basis. Ex: Systems administrators, network administrators, SAN administrators, and all other IT administrators are in entirely different management chains (3 bonus points for each additional SVP involved, instead of at least rolling up to the same SVP - or whatever is one level below your board members). And let's not forget the union or external org that handles physical issues, like running ethernet or power. There's no way that kind of setup could turn the process of adding an additional SAN allocation to a server into an 18 month nightmare for every group, right?

Anti-Pattern: Allowing one-offs to happen more than once. AKA, the "just this once" principle. Some "emergency" is declared and you're going to do something out of process "just this once." Repeat every 3 months for the rest of your tenure at the company. It won't even matter if technology changes and obsoletes this one-off process; it's now a standardized one-off that is continually justified as an emergency, so it's going to happen and the only question is if you want to put your job on the line for it.

Running Exchange 2003 in 2010 is maybe a bit behind times. But running a release candidate in production up to 2010 is just nuts. Since I have seen that, I consider Exchange more solid than before.

Having a big network printer in a server room with network equipment consolidates space and the AC makes sure that even during large print jobs it's not getting too hot and the ozone emitted is immediately disposed of. That is a valid point for employee health. After the second time you maybe learn to not use the same circuit for the printer and the main Cisco.After the first time the thing catches fire and burns your network down, even non-IT people start to get what potential problems such an idea may have, despite it's benefits. Installing AC and ventilation in another room for a printer would have been cheaper than some hundred people not working for more than a day.

And I somewhere have a pic of a server room that is too cliché. A wagon from the cleaning personnel, complete with a 5l of water and detergent on top, blocks the doors of the server racks. And I've been told that's where it always. With the water bucket on top.

Wait. Am I understanding you right - this guy was running a release candidate of 2003? Recently? Wow.

Running Exchange 2003 in 2010 is maybe a bit behind times. But running a release candidate in production up to 2010 is just nuts. Since I have seen that, I consider Exchange more solid than before.

Wait. Am I understanding you right - this guy was running a release candidate of 2003? Recently? Wow.

Yes. But it gets better.The guy that told me this was hired as the new head of their IT. He pushed to update to Exchange 2010, the real version this time, among other things. He came to make the mess they called IT into something at least close to IT. I talked to him the other day again...

He is currently arguing heatedly to get more budget. After updating Exchange to 2010 and renewing some hardware, the upper management now thinks that with the SAN and its fault tolerance there is no need for backups or replication.A several hundred employee company, critically dependent on e-mail, is running Exchange 2010 without backups or replication. The same company that has seen only a brief time ago what happens if just part of their network equipment gets grilled.

He is currently arguing heatedly to get more budget. After updating Exchange to 2010 and renewing some hardware, the upper management now thinks that with the SAN and its fault tolerance there is no need for backups or replication.A several hundred employee company, critically dependent on e-mail, is running Exchange 2010 without backups or replication.

There's a difference between not wanting to spend hard dollars for backups or replication and not wanting to have backups or replication.

Sometimes the organization is pushing to have the task performed without cutting a purchase order. Sometimes the SA is gold-plating a solution because the tight fiscal environment makes it seem smart to overbuy because the opportunity won't come again, or is adding unnecessary complications or extras to the solution.

And sometimes people get confused about the difference between fault-tolerant storage and backups, and don't want to consider a disaster scenario where offsite replication would have been required for survivability. Considering the multi-gigabyte blobs of useless bits that pass for email today, it could be that the critical function that's performed in email should be moved to a more appropriate (e.g., more structure) system.

Don't look a gift horse in the mouth. No mail backups means no mail restores. At long last, there's a potential path out of the nightmare of unmanaged, constantly-growing mail spools.

Believe me, mail is critical for them and not easily replaceable. Of course lots of the mails are, for sure, nothing more than horse manure, but they have a legal obligation to archive every single customer facing mail for 10 years.And from what I know from him, he's not doing politics or asking for anything that is unreasonable.

If the snapshot size or IOPs on your Copy-On-Write block system start climbing, look for this anti-pattern. Especially fun when enabled... by automation... simultaneously... on entire servers farms... with thin-provisioned LUNs... which collectively oversubscribe the available raw disk.

I don't use Macs nor iOS at the moment, but deliberately designing a system not to support them is the height of ignorance. I use various flavors of Unix but it's harder to dismiss the point when I'm talking about the CMO's iPad, the CFO's iPhone, the CIO's Android and the CEO's Macbook Air.

Most of the heterogeneous systems hate is circular. One would have a hard time supporting anything but Windows when all of the systems run Windows and the systems architecture came out of a bin at a big-box store. I understand.

The pattern of consistency paving the way for IT at scale gives way to the anti-pattern of false consistency failing badly and creating far more complexity without any corresponding value added.

Anti-pattern: Having an IT department that's isolated from the rest of the business for idiotic cultural reasons, which in turn forces the business to make extreme changes to the department simply at larger intervals.

Anti-Pattern: Silo'ed departments that interact on a very regular basis. Ex: Systems administrators, network administrators, SAN administrators, and all other IT administrators are in entirely different management chains (3 bonus points for each additional SVP involved, instead of at least rolling up to the same SVP - or whatever is one level below your board members). And let's not forget the union or external org that handles physical issues, like running ethernet or power. There's no way that kind of setup could turn the process of adding an additional SAN allocation to a server into an 18 month nightmare for every group, right?

Bonus Anti-Pattern: An insane networking regime who applies principles in an insane matter. Dumb principles become aplocolyptic. Good ones become awful.

Example: Servers whose network interfaces span security zones must be isolated by a firewall and protected by an IPS. Enter the DR proxy server. Now by its nature, a proxy spans the client network and the Internet. So the server was delivered in 2003, and after the purchase and refresh of 3 firewalls, 2 ips devices and one fibre channel switch, four data center recabling jobs, two instances of union electricians walking off the job, one attempt to move the physical server's boot drives to SAN, ther server was turned over to the proxy team in late 2008, after they had forgotten about it!

Example: Servers whose network interfaces span security zones must be isolated by a firewall and protected by an IPS. Enter the DR proxy server. Now by its nature, a proxy spans the client network and the Internet. So the server was delivered in 2003, and after the purchase and refresh of 3 firewalls, 2 ips devices and one fibre channel switch, four data center recabling jobs, two instances of union electricians walking off the job, one attempt to move the physical server's boot drives to SAN, ther server was turned over to the proxy team in late 2008, after they had forgotten about it!

We just got one of those last week, courtesy of the ex-director who left two years ago.

Email 1: Your 'new' servers are delivered.

Email 2: You totally don't need to do anything with them, because no-one cares.

anti-pattern: Using zip ties in your cable plant or to mount ANYTHING in a rack.

I've spent hours of a tight window cutting them out from an installation where a lunatic had used bags of the things.

Nail clippers. There's a reason I have nail clippers in my desk at work, and it's not for personal hygiene. Aside from cabling, I also ship and receive on a daily basis crates that are both padlocked and zip tied. Nail clippers.

Quote:

Pattern: use only velco or waxed twine to secure cable, and only mount into a rack with the proper screws or shelves.

Velcro only. Once you've used it, there's no going back to anything else.

Nail clippers. There's a reason I have nail clippers in my desk at work, and it's not for personal hygiene. Aside from cabling, I also ship and receive on a daily basis crates that are both padlocked and zip tied. Nail clippers.

Electrician's scissors seem a bit more versatile. Less likely to join you on an airplane, and more expensive to replace, though.

anti-pattern (I'm looking at you, developers): Hard coding IP addresses into your applications. Ever. Even just a little bit. Just the tIP, I promise.

anti-pattern: Using zip ties in your cable plant or to mount ANYTHING in a rack.

Zip ties are fantastic for rack power whips though... but never on data/LV cabling.

I f'n hate zip ties - and my prev sysadmin wanted to show his enthusiasm back when he was just hired, retreated to the server room to organize things for a couple of days then proudly called me to take a look...

...I was close to get a heart attack right there, the entire server room gigabit/10Gb/etc cabling was zip-tied to the racks, shelves etc.

anti-pattern: Using zip ties in your cable plant or to mount ANYTHING in a rack.

I've spent hours of a tight window cutting them out from an installation where a lunatic had used bags of the things.

Nail clippers. There's a reason I have nail clippers in my desk at work, and it's not for personal hygiene. Aside from cabling, I also ship and receive on a daily basis crates that are both padlocked and zip tied. Nail clippers.

You don't have dikes? I spent hours doing the job with the proper tool. There were that many. A knife with serrations near the tip in a pinch, but it's relatively hard to find knives serrated all the way out.

I would agree, but why give the bad guys an easy way to scan your network? And what about smurf attacks?

What's funny is that I'm constantly getting tripped up by Windows Firewall. I just can't remember it's there and I'll waste anywhere from 50 minutes to an hour and a half trying to figure out why the Windows box isn't seeing the incoming traffic it's supposed to.

I would agree, but why give the bad guys an easy way to scan your network? And what about smurf attacks?

Why not give the good guys a way to use PMTU and everything else that is reasonable in a modern network? It's also a really bad mindset to have, as IPv6 be far more broken if you apply the same policy there.

Besides, you can scan the entire IPv4 address space in a few minutes with a decent botnet. You're not scaring anyone away from your network.