Posted
by
samzenpus
on Wednesday January 05, 2011 @04:25PM
from the we'll-do-it-later dept.

snydeq writes "The underfunding of routine hardware replacement purchases and the degradation of aging enterprise apps pose systemic risk for many IT organizations, thanks to a ballooning 'deferred IT maintenance debt' in the decade since Y2K fears pushed enterprises to invest heavily in essential system upgrades, InfoWorld's Bill Snyder reports. And with sysadmins 'scrambling to keep systems up and running with budgets that barely cover the basics,' this 'IT debt' promises only to increase in the coming years, especially as IT continues to defer routine maintenance in favor of new 'cost-saving' initiatives, particularly around the cloud."

I fail to see why this is newsworthy? Is it just because IT people whine louder?If you are in the US---just look around. Infrastructure systems are crumbling away because of "deferred maintenance". It's not just IT. It's roads, bridges, state governments, municipalities, houses, businesses---it'severything!

Well, it is sort of a "duh" story the way it is written, but OTOH the subject is not without merit.

I have been involved with infrastructure assessment of companies prior to acquisition and some stuff is just shocking. Publicly owned companies are driven by return to the shareholders; one way to keep the dividends flowing when the economy is in a downturn or when the business plan isn't working is to reduce operational expense.

Releasing employees is very effective to reduce the spend side but usually that means there is less available effort to work on maintenance. It looks good to have all employee time capitalized on projects but who is keeping stuff working? Also, each person out the door takes expertise with them that is lost to the company. After a while, the company may not even have enough knowledge internally to understand that their boat has holes in it and that patching isn't happening.

This isn't smoke; I've seen it. Data centers with overheating problems and with inadequate standby generators. Power is distributed unwittingly to cause a cascading failure if one breaker trips. Leaking roofs over financial servers (plastic tarp and bucket gave that away). Licensing that has not been kept up to date because no one has a good inventory and no one wants to look-see. So... Oracle enterprise instances running in non-secure network zones and without proper licensing ( potentially million$ in back costs). A database server being used as a network monitoring node and firewall because funds were not available to separate the functions.

Deferred infrastructure investment and maintenance investment happens and it is a ghastly mess to clean up. I am not surprised that more of this is happening.

Some admins are just scared of things going wrong during maintenance do don't do it. My boss at the last place I worked was like that. He never did updates on servers if he could possibly avoid it. The fear that the server might not reboot and he would have to drive all the way out and spend part of his weekend fixing it stopped a lot of preventative maintenance getting done. It was easier to blame users when the RDP server was hit with a 2 year old virus using a vulnerability that was long since patched. D

I think it's a setup for the "IT Industry Invaded by Incompetent Idiots" and "CIOs Found Replacing Working Systems with Crap Made By Their Hunting Buddies" articles.

A large portion of/. readers are in IT and already knew this. However, seeing it in "print" in a newsrag you might find in a CIOs office is a little noteworthy. It means it's only a matter of time before someone comes rushing to your desk to say "Our CIO just read an article about infrastructure and we need an ans..."

Sometimes cheap routers are the way to go -- I replaced 2 Cisco routers at a remote site in a seaside warehouse (one due to a power surge when a generator failed, one due to water from a leaky roof) before switching to cheap Netgear routers at about 1/10 the cost. Redundancy? We had a spare configured and ready to go the foreman's truck toolbox and another at his house.

One of the Netgears even survived a similar water deluge to the one that took out the Cisco (but then the Netgear didn't have a fan to suck

Sure, the rule may have been bogus, but we had to obey nonetheless. That's one of the joys of dealing with a government bureaucracy where the guy who manages the facility gets to interpret the rules no matter how capricious and arbitrary they may appear. Things were a lot smoother at that facility after we found out that the harbor master liked jelly donuts.

I can't say I particularly care for DLink, but for the cost of a (used) Cisco, I can pick up two Procurves with better specs and a management interface which anyone with basic networking comprehension can master in a couple hours. I'm not hearing anyone say the Procurves are shit (because they aren't). With d-link, I'm sure the same is true - and I can have a stack of hot spares for the same cost, too - Just In Case.

I suspect two reasons: 1)(and most important): This is being published by Infoworld, ergo it focuses on IT stuff. 2) Much of the worst rot in IT is largely invisible to the layman.

Slow computers with styles that were pretty neato back in 2000 are obvious to the poor office drones who have to endure them; but anything that new can, largely, be forklift upgraded for the cost of the new systems and some grunt labor. Turning a 3 year desktop refresh cycle into a 5 year(or 7 year, *cough* *cough*) desktop refresh cycle doesn't make anybody happy(particularly once warranties run out, the scavenging and improvising begins); but is architecturally a small problem. You don't really accrue much "debt" over time. The cost will be "1 forklift upgrade to present day PCs" whether that upgrade takes you one generation ahead or three.

It's the complex software, the highly specialized proprietary industrial controller cards, and suchlike widgetry where there is real hell to pay, and most of that is invisible...

Scavenging for desktop parts is the "little devil". Scavenging for people who know how the bloody things work more than 3 years after for IT systems is the real nightmare.

If a system has been in the field for 3+ year nobody knows what are its real dependencies and what does it really take to augment, add capacity or do any changes. The people who knew have left, gone to pastures new or have forgotten what the problems used to be and no documentation can help you here (even if there is any suriviving docs on the design of the system in question). This is valid for almost all classes of IT and telecoms systems and is the real cost factor in IT "maintenance debt". If we use a real-life analogy IT maintenance debt is like a discounted mortgage. You pay virtually nothing for 2-3 years and after that the lender skins your hide.

I agree wholeheartedly with you.
I think another large part of the equation which our fellow IT workers fail to admit is that our ilk are incredibly stubborn about replacing and fixing things. IT workers are notorious for telling management that they can make things work with a hodge podge of coathangers and toenails, not because it is the best solution, but because they can.
The problem lies on both sides of the coin. Management not wanting to spend money and IT workers not setting a realistic expectat

Hey now, don't know Y2K vintage hardware. Place I worked spent a lot on some beige and white Dell workstations that year. They were spiffy at the time. The boss/owner was ultra proud of his Dells and made sure they sat on TOP of the desks so clients and whatnot would see them and be impressed. Woooh he spent money on Dell.

Boring beige Dell was actually an improvement. Prior to this, the place was run off an old infected Vision PC they got from a radio ad. I upgraded them with almost no budget to DIY s

It's not just bespoke unfortunately. If, for example,the vendor decided some time after 2003 to lay off their programmers, support staff etc and get a huge suite of software rewritten in India in java but have not finished yet then you just have to have the in-house expertise that the vendor can no longer provide. It sucks having to edit install scripts of expensive commercial software just to get it onto the machines and then alter undocumented scripts to get it to do plot outputs or do remote tape acces

Oh Please! That is SUCH an easy one to fix! You either run XP Mode in Pro or just load up XP VMs. No you want to talk about "IT debt" try some of the places I walk into, where there is ALWAYS a "mission critical app" that is this horribly mangled piece of badly coded VB+Access mess of no comments anywhere junk, and then they expect YOU to deal with it! Hell one place I walked into in mid 09 had a NT 4 box running a VB3 "app" because each guy they brought in took one look at that beast and said "fuck that!".

Man I can hear the real programmers right now screaming out in pain just at the thought! You want to watch a "real" programmer wet his pants in fear you hand him a huge 14 page VB mess written by a half a dozen guys over the years, NONE of whom ever heard of a comment, with shit all over the place and nothing indented or even calling in a logical order, unless "insane band aid" is considered logic.

You want to know why there is an ever increasing IT debt I'd say that is a BIG part of it. All across the country you have this huge mess of apps written by some Joe Schmo that was bought ages ago and nobody knows how to live without and it DON'T run on anything but what it was written for and even then it is fussy as hell. And that of course don't even take into account the lovely crap like that ISA C&C controller written for DOS 3 that runs a $75,000 piece of machinery made by a company that has been DOA for a decade plus! I have stared into the abyss pal, and not only did it stare back it gave me the finger to boot!

Silly young grasshoppah. There is no "just do..." in IT. The mythical solution you're referring to is a cruel joke told by vendors.

huge 14 page

So, a little script, then?

Something on my table right now: 15k (in-file - and probably significantly more on the printed page) lines of PHP3 with nasty embedded SQL up the wazoo. It ties into half a dozen (literally, 6) other 'mission critical' applications and is customer facing as well as providing internal network management functionality. And this is small fry compared to some of the crap out there.

You want to know why there is an ever increasing IT debt I'd say that is a BIG part of it. All across the country you have this huge mess of apps written by some Joe Schmo that was bought ages ago and nobody knows how to live without and it DON'T run on anything but what it was written for and even then it is fussy as hell

I couldn't agree more. We've had entirely too many Boy Geniuses in decision making places in IT who think they've got something special and unique which will have Totally Awesome Results. They don't bother to think through their decisions.

The proper approach to something like this isn't to fix it. It's to replace it outright with something that does 90% of the task, better, with 50% fewer inter-dependencies by modularizing things as much as possible. Re-implementing, feature for feature, is quite often quicker. Just make sure you don't make the mistake of so many before you and re-implement it - poorly. If you can't do it, find someone who can.

IMO, the key to a successfully maintainable software infrastructure is to KISS and leave things as White Box as possible. When you can't keep things generic, you keep things isolated and modular. When you need something custom purpose that your users rely upon, you make damn sure it's standards based and that there are alternatives available.

(I don't even want to THINK about where we will be with things like Sharepoint in 3-5 years. Likely, another lengthy, drawn out, and costly migration project. This time, maybe back to something like, oh, NFSv5.)

Ohhhh...sounds like the abyss has given you the finger as well. And to be fair we are talking about 14 pages of badly written VB4 with about half of the pages nothing but fricking GOTOs bouncing all over the place.Now I'll admit when I've had to do something quick and dirty I've thrown in the occasional GOTO but the GOTOs all went to the same place whereas this massive pile of shite had more twists and turns than a bad detective novel!

But since you have seen the "true horror" of IT I'm sure you can see why I sometimes want to bash my head on the desk when dealing with FOSSies. They always seem to think "All you have to do is replace Windows and Office, and all will be hearts and flowers!" when it is NEVER Windows and Office that is the problem. Hell most companies can't even upgrade to the latest windows version for fear of that giant reeking mess of garbage code they've come to depend on will come falling down like a house of cards in a hurricane.

And while I agree 110% that the goal should ALWAYS be KISS, the problem is the PHBs at these places will never ever in a million years shell out what it costs to actually get all the data out and build a REAL solution, not until they have the crap they are depending on fall apart like some giant train wreck from hell. Like that VB4 app I mentioned early on, I ended up just jamming the thing in a Win2K VM and letting it rot for the next guy. Not because that is what I wanted to do, but because you couldn't pay a college kid what they wanted to pay to have it rewritten, much less get a REAL programmer of any skill.

And that to me is the problem in a nutshell and why the shit isn't gonna get any better for the foreseeable future. It is because businesses in the USA can't pull their heads out of the stock page to realize that working long term solutions cost money no different that decent roads or schools, and look how well THOSE are faring here in the US. IT has it even worse because as long as the thing works that day they don't care if it is a single power surge away from taking down the whole company. To them ANY expense related to IT is just a "waste" unless it is something like an iShiny for themselves or the CEO. That is why I ended up getting out of dealing with corporate, because I frankly got tired of people with impossible problems that wanted to pay pennies to fix years worth of neglect. I mean with the corporate attitude in the USA, is it any wonder nobody young is going into IT anymore? You'd have to be nuts!

Ya, well I'm the Joe Schmo that gets tasked to fix this sort of stuff.

The conversation usually goes like this:

Me: "To fix this properly you will need to do this, buy this software, redesign this system, it will take X amount of dollars and Y amount of time."Boss: "We don't have the money. You need to fix it as best as possible using the materials at hand with no budget, and a limited time."

Thus, slaps together whatever I have to get it working, cheaply and likely crudely. Forget documentation, you don't hav

Because much of this IT is stuff that affects individuals who have no influence over it.

When a company puts off investing in security, for example, and when they also collect and store my credit card info / medical info / personal demographics / shopping history / etc., they are putting me at risk.

I have to trust that their IT department is on the ball. Something I am beginning to think is never a good idea. But it's impossible to not give companies some info on me and still be a normal modern human, and thus I am forced to trust them all the time.

So if they're further neglecting their IT, it means my data is more vulnerable. Not that's there's a damn thing I can do about it.

Just one minor gripe with the parent - a lot of times, what should be weeded out isn't the "one-offs" (which are often times built way under budget with way more capacity and way less maintenance cost), but the actual official enterprise standard that got put in because some CIO was buddies with some sales rep. "One-offs" are a signal that the current standards (either of technology, or product development), are having problems. While not all "one-offs" may be worthy of keeping, when going through the wee

I think the one-off's he's talking about are things like the purchase tracking system that Finance installed on a workstation under someone's desk. IT was not aware of it being purchased and of course it has no way to export data into any of our other systems so we can integrate it with the rest of our financial systems.

Eventually we sucked it into our VMWare infrastructure to get it off of the desktop machine (no RAID disks, and the backup consisted of someone in accounting copying the database to a flash

You think that's bad. A former organisation I worked at had a similar thing happen where the Finance group bought a software package which was sold on "no IT department required" however the way we found out was even before they deployed it. It was of course a computer program so instead of the Finance department using their own budget allocation, the bought it out of the IT departments budget. When the CIO and the CFO had a meeting the shouting could be heard from the other end of the building.

I live and work in Latin America. My residence is in Colombia, but for work I am in most LATAM countries, trust me, the US has wonderful infrastructure compared to us. And any latin country i have been in.

Probably the estimated amount of money it would be needed to catch-up with the backlog (the so called "IT debt") and a few words on how the estimation has been made? Also, where these money would be needed? TFA:

$500 billion -- it's a number so big you'd assume it's a component of the national debt. It isn't. Instead, it's what Gartner analyst Andy Kyte calls the IT debt. "

The "debt" really has two major components: One is underfunding and even neglect of routine but important hardware replacement purchases and software upgrades. The other is the slow degradation of enterprise applications.

...

Is that $500 billion number too high? Kyte says he derived it by analyzing several large Gartner clients that generally do a good job of keeping applications up to date. That led him to estimate that a typical Fortune 2000 company would require upgrades costing more than $200 million each.

I admit, these as scraps of "meat", but they are nevertheless still meat.I imagine that this may be the first step (as an argument) in a push the big IT houses (hardware/software/services) will make to get so more revenue... Something on the line of "well... Apple and Android tablets and phones (that

I'm guessing it has something to do with the fact that IT managers read InfoWorld. Some read slashdot, too. Many of them are dropping the ball on this, big time. "We don't have the money this quarter" comes up time and time again, and they don't realize that the decreased maintenance is going

The only people who really realize this kind of thing are sysadmins. Unless the sysadmins have a huge amount of power within the organization (including their own ability to prioritize the budget and 'vision' of the org

Now how is pushing services to a cloud exsaserbating the problem? Sure going to the cloud has it pros and conns. But if you are retiring your old servers to a cloud in terms you are Upgrading your infrastructure and removing your old stuff.

And with sysadmins 'scrambling to keep systems up and running with budgets that barely cover the basics,' this 'IT debt' promises only to increase in the coming years, especially as IT continues to defer routine maintenance in favor of new 'cost-saving' initiatives, particularly around the cloud.

The point of using "the cloud" (a hollow buzzword, I admit) is that you can offload the servers, software, and maintenance to a firm that specializes in such things. Theoretically, taking advantage of the cloud where it fits your organization will offset the "maintenance debt" problem. YMMV, of course.

Theoretically, taking advantage of the cloud where it fits your organization will offset the "maintenance debt" problem.

"Cloud" (as in, dynamic server provisioning) has very little to do with it.

Outsourcing IT functions to a firm that is contracted to actually perform the maintenance that was being deferred on the in-house systems (whether hardware, infrastructure software, application software, etc.) obviously can address problems related to deferred maintenance, not because of the outsourcing itself, or because the vendor to whom the operations are outsourced happens to use "cloud" technology to power its offerings, but be

"The point of using "the cloud" (a hollow buzzword, I admit) is that you can offload the servers, software, and maintenance to a firm that specializes in such things."

Yes, because it's a demonstrated hard fact that those companies providing infrastructure for the cloud can't lower their operational costs by neglecting maintenance; of course they wouldn't do that anyway since it's those infrastructure companies' very valuable data what is at risk if maintenance is neglected instead of their customers'.

Too many CIOs of too many western corporations report to the CFO, not the CEO. There are WAY too many CIOs who come into organizations with an eye, or a reputation, for cost cutting instead of tech innovation. Pick up any copy of CIO magazine and look at the toadies who make the top CIOs in the nation, and ask yourself - what innovation did they bring to make that list? What business process did they improve with tech? Only a handful make the cut. Most are there because they are good at pinching out costs, kicking out the older IT workers and either outsourcing or bringing in college grads.

I routinely see job ads for experienced Java developers, people with hard core experience in integration, esp. with telephony or security technologies, need 5-10 good years, offering $70k tops. Good luck with that, but again it is the CIOs who get the jobs telling people they can staff cheaper, run leaner, cut the corners - that get the job because it is the CFO who is doing the hiring and the performance reviews.

The big corporation IT C-level execs are a fear driven lot, there are no Gates or Zuckerburgs in their midsts. The action is being with the cloud providers, or the web service providers themselves. Enterprise IT is really a shit place to be outside China. It's a world full of EDS consultants and chickenshit CIOs who won't think how a business could use IT to expand. And the social media space is going to tear a bunch of them new assholes, because none of them know how to leverage it. The startups do.

Nail, head hit. I have worked for people who had bosses who had zero interest in anything other than cost. The same people that I lambast for putting basic security precautions as an extreme low priority, due to their attitude of "security has no ROI."

What is ironic is that even though this makes the quarterly figures look good, it is killing competitiveness long term. Another example is R&D. Not product research, but true R&D where people discover something, shrug, put it on a shelf for 20 years, then it becomes immensely marketable (think Corning and Gorilla Glass as the prime example of this.) Instead any groundbreaking research ends up being done overseas or by small firms which are bought out, or have their IP infringed upon. If research is done, it is for a product to cough up this quarter or perhaps this FI, and usually it is how to add a gewgaw to something existing and palm that off on the market.

You mention China. Chinese companies know how to lock their doors down. They know what happens if you run your company with your fly open, and most of the companies over there wouldn't even be in business had it not been for "borrowed" IP from the West. Take Foxconn as an example. For a company that size making so many Apple products (including ramping up production on unannounced items), they are quite airtight about what their factories produce even with the hordes of workers they have. Had this been a company run by the typical PHB here in the US (with their usual lack of interest in security), everyone and their brother would know what the iPhone 5 looks like, and perhaps even have the source code for that rendition of iOS.

I wish the US would start borrowing from China in this regard. Even with security aside, just because a company's IT infrastructure works today doesn't mean it will work 5 years, or perhaps even six months from now without major issues. Taking a charge off quarterly earnings to fix problems now means a lot of less wasted money when the upgrades have to be done post-haste.

"I wish the US would start borrowing from China in this regard." I wish you were right. However, if an American firm did this, they'd be sued by Chinese companies in American courts using American lawyers...and their Imaginary Property would be protected and the American firm set up for serious damages. And the Chinese know this.

The same people that I lambast for putting basic security precautions as an extreme low priority, due to their attitude of "security has no ROI."

To be fair, on the normal USA business timescale (that is, the next two or three quarters), it really doesn't. Security only matters if you're still heading the company when something bad happens, which to most of these people is pretty much unthinkable - because they know they'll be gone in a few years.

Had this been a company run by the typical PHB here in the US (with their usual lack of interest in security), everyone and their brother would know what the iPhone 5 looks like, and perhaps even have the source code for that rendition of iOS.

I wish the US would start borrowing from China in this regard.

Give them a few years, they will. It'll start to happen when the first PHB demands evidence that a chinese factory has had no more than three information leaks in the previous 2 years, and they'll get a very bemused email back asking them what sort of information leaks would they like engineered because that's the only way anything's getting out.

Not only is the nail on the head, it is worse. The CIO reports to the CFO but doesn't actually know any of the terms that the CFO uses, so all they have are "saved X by not doing Y". A REAL CIO would have the CFO terms down and be able to say "by doing T we can do S things more productively".

I remember when this hit me about a dozen or so years ago, when I was the business manager for a small regional ISP (when they still existed), and had to process some logs for billing purposes. I had one computer that I

My point got lost here. The point is that Chinese companies actually seem to devote resources to computer security and IT infrastructure because they are seeing the mistakes companies are making on the other side of the Pacific. Even the Chinese government is laying fiber like there is no tomorrow.

Here in the US, we don't need corporate dorms -- we escaped that stuff in the Gilded age (but with the political climate, we may be heading back towards that direction.) Instead, more than just a token effort i

That's not entirely the CFO's fault, a lot of that has to do with the US tax code. If the capital gains rate for investors didn't start to kick in until 2 years down the road and took another year to fully kick in, you'd see longer holding periods. Also, if the short term tax rate was higher for individuals holding for less than a month.

As it is there's very little reason for corporate executives to think down the road by more than a couple quarters, as chances are the people who own shares now won't still own any much beyond that.

Too many CIOs of too many western corporations report to the CFO, not the CEO. There are WAY too many CIOs who come into organizations with an eye, or a reputation, for cost cutting instead of tech innovation.

Assuming the corporation is not in the IT business but only uses IT, I thing that's only fair to use risk management as the second term of the contrast (instead of tech innovation).

It's amazing how right this is. The problem also stems from companies that could or should be using IT as a way to improve their core competencies and improve their competitive position aren't because of the recent economic issues. Many people are getting power and influence by riding the penny pinching wave instead of making good long term decisions. We're going to be facing the aftermath of having these people over promoted for a long time to come.

Companies which have been improving their services and products through better IT practices are actually not doing too poorly. And no, I'm not wholely talking about IT industry companies, either: I know of manufacturing, architectural, and a handful of general 'service' companies (including banks) which have significantly improved their IT utility, both internally and as they face the customers.

No, XP isn't off the desktops at many of these places. (They are at the architectural place.) But many of the mach

Low cost needs to be balanced against getting the job done, and reliably done.

Because if your IT starts to have frequent outages or lose valuable data, it can be more expensive than investing in decent equipment and competent employees.Since/. likes car analogies:In the 90s Opel, a German branch of General motors, was a bit too aggressive in cutting manufacturing costs. The resulting quality problems were quite damaging to the brand and customers started to look elsewhere for their next car.

One way or another, it's going to be paid. Either buy and deploy internally or pay an outsourcer to buy and deploy. Meanwhile, you can't just export re-writing your old creeky VB6 apps to the cloud. Even if you find a cloud vendor offering a suitable replacement app, you'll have to pay for that and pay to have your vital business data pried from the cold dead fingers of your old app and imported to the new app.

It may come out a bit cheaper having the cloud provider do it, but be sure of the business arrange

The price of replacing things may be getting cheaper faster than the implied cost of the risk of not replacing it is going up. It may be cheaper to wait until it breaks than to buy something with a rapidly depreciating value. The extra cost of dealing with an emergency may be paid for by the lower cost of the gear and labor. And for those risk events that never happen, the ratio of preventative replacement cost to emergency replacement cost will be infinite.

"The underfunding of routine exercise programs and the degradation of aging overweight sysadmins poses systemic risk for many IT organizations, thanks to a ballooning 'deferred weight loss program' in the decade since Y2K fears pushed enterprises to invest heavily in dudes who live in their parents' basements", InfoWorld's Bill Snyder reports. And with sysadmins 'scrambling to keep their bodies up and running with foods that barely cover the nutiritional basics',' this 'IT chub' promises only to increase in

With hardware prices dropping every year, the longer you can defer hardware upgrades, the less money it will cost you. Given that basic piece of information, it's hardly a surprise that companies don't upgrade until they absolutely have to (anyway: why would they until there's a need?). If they can give their kit a mid-life kicker with some more memory or swapping in some faster CPUs, isn't that better than spending 10s of thousands or more on a new box. Better, that is for everyone except the hardware manu

...worked anywhere where someone actually swapped CPUs in a server from a real vendor (ie, not some BS whitebox)?

I *added* a CPU to an HP server once (single to dual CPUs) and it was super expensive and not all that easy to get the part. From HP it was OMFG-are-we-really-spending-this-kind-of-money expensive, a "re-certified" part from a third party was still way more than the $300 you'd spend for your home system as the part and its corresponding VRM were proprietary.

...worked anywhere where someone actually swapped CPUs in a server from a real vendor (ie, not some BS whitebox)?

I *added* a CPU to an HP server once (single to dual CPUs) and it was super expensive and not all that easy to get the part..

I did and while the parts from HP were outrageously expensive we ordered it from a third party. HP does not have an exclusive market on Xeon chips or the plug-in voltage regulators they used to use (maybe still do?). The tricky part was the heatsink which needed to be a funky size to squeeze in the space allotted but that too was solved from a third party vendor. Total cost was about a third what HP wanted. We did it as a test case to make a single CPU machine with 6 total slots into a dual. It was even

Yes, I've personally seen it done on two seperate occasions. In both the situation was the same. A software vendor that licenced on a per-processor basis (Oracle). One box was a 12 processor macine, the other was a 16 processor job. In both cases it was far, far cheaper to swap the CPUs for faster ones than to add more of the same speed. In both cases that was my recommendation - based purely on cost savings and in both cases it worked very well.

If they can give their kit a mid-life kicker with some more memory or swapping in some faster CPUs,

That's just stupid. Simply "dropping in" a faster CPU doesn't do all that much, because you keep the same bus speeds on the motherboard that limit bandwidth. As for memory, haven't you noticed that buying older, slower memory tends to be more expensive than buying the new faster memory, because the old stuff is no longer in production?

Aside from those factors, there's the added labor in installing those components and testing them. On top of all that, most mass-market desktops that are used in this space ar

Those who have no experience and say it's "stupid" should look and learn from those who are actually doing it and getting their employers good value for money in proven, real-life situations. It *does* work. It's a quick, reliable and scalable solution in many, many cases. Obviously you need a background in performance analysis so that you've correctly identified and quantified the bottleneck - and no, merely playing with a "point and shoot" GUI "tune your system" toy program that you downloaded for free o

I am having trouble getting basic hardware replaced - I can't get a 500-750 dollars to replace some network switches let alone enough scratch to update my primary DC.
Our Budget Analyst does not see the need to plan for future needs, or periodic replacement of vital equipment as warranty cycles expire.
I have documented our needs, but my boss the CIO is afraid to push the issue.

I see replacing equipment as less of issue than purchasing new equipment for additional capacity. Assuming you are doing backups, you can always move hard drives from old-system to a new-system. Yeah that is some work to get it back in running, but it is still a possibility.

However, if you have servers running at capacity, and they refuse to purchase new equipment, you are asking for problems.

And that's exactly the problem, my friend. Too many people round here seem to imply that "deferred it maintenance" means not replacing servers when the guarantee period ends up. But maintenance means having two sysadmins when you formerly had three or maintaining the three sysadmins when capacity has grown 50%.Lowering maintenance costs means that it has been a year that nobody has the time for a test restore so nobody has noticed that the backups are failing since six mont

I am having trouble getting basic hardware replaced - I can't get a 500-750 dollars to replace some network switches let alone enough scratch to update my primary DC.
Our Budget Analyst does not see the need to plan for future needs, or periodic replacement of vital equipment as warranty cycles expire.
I have documented our needs, but my boss the CIO is afraid to push the issue.

My advice: add some risk analysis argumentation. You know? Something on the line of:
1. probability of equipment failure [omdec.com] over time - use the "cumulative hazard function" not the "probability distribution function".
2. impact the server crash will have on the business (make sure you slip-in some "lost face" apropos - after all it would be the manager's face to be lost). If you can express the impact in $$$ and plot the "risk x impact", chances are the budget analyst will "get the picture" easier.

Forget the risk analysis arguments. If people refuse to see simple logic, hitting them with more of it will do no good. These people will dismiss your reasoning with superstition, thinking along the lines of, "My VCR is 10 years old. It works fine. I see no reason to upgrade our network equipment until my vcr dies."

For things like that, the only reasonable solution is a loose mains wire and some bridging rod to the offending equipment in question. Then, when you get your new equipment overnighted at 20% mar

I understand that eventually apps will fail to be supported by the developers, won't potentially work on more modern operating systems, and in some cases require updating in order to be able to work correctly with the rest of the world.

But it's a bit disingenuous to call this "degradation". The app continues to do what it always did. You're just wanting more out of it than you did before. The app didn't change, you did.

1) The languages, special hardware, libraries and controls become unsupported on new hardware.2) The languages, special hardware, libraries and controls become unsupported on new versions of the operating system.3) The operating system becomes unsupported.4) The hardware becomes unsupported.

2010 Hardware and operating systems to support VB6 start becoming unavailable. All are unsupported by vendors.

Cost to redevelop VB6 program-- about 1.6 million dollars.

At some point- basically find a new packaged product (cost $100k + $500k user licensing & support + loss of ability to differentiate business) which provides 80% of functionality of the VB6 program and toss it. Can't be changed to match your business - you must change business to match it.

At some point- basically find a new packaged product (cost $100k + $500k user licensing & support + loss of ability to differentiate business) which provides 80% of functionality of the VB6 program and toss it. Can't be changed to match your business - you must change business to match it.

Larry Ellison? Is that you? That's been standard Oracle-speak for nearly a decade, now.

I think you may have meant it sarcastically... but I think it's a good idea.

Sometimes the differentiating business practices is your competitive edge. Infact, it many times is just that.

I should know - we do loose some business because we cannot adapt the software to suit our needs (closed source portions/original design approach), and are forced to have our business to fit within the software specs. That sucks.

For startups the standard software package might allow cheaper start up, but eventually there will be things which does not suit the business.

There is a solution for 1 through 4, and it's not one you'd like (unless you have a shotgun):

Virtualization. Virtualization will give those ancient systems another stay on life, and you will be able to "phase them out" (not really) over the following decade, as other products start to get used.

Virtualization turns your old crap systems into ever-living zombies. "This is old crap, we'll just virtualize it on new hardware". Doing this should last the system itself 1-2 hardware refreshes. By the time it's time

Maxo gave a great answer, but there are matters of internal compatibility also. Generally an IT app relies on lots of other IT apps to function, all of them are changing, and if YOUR app does not change you are retarding what can be done in other applications, or forcing maintenance of a backwards compatibility layer.

There are some systems that can really just sit there and don't interface with much, really core systems... but then you run into that systems support issue and there is nothing corporate IT p

But it's a bit disingenuous to call this "degradation". The app continues to do what it always did. You're just wanting more out of it than you did before. The app didn't change, you did.

Hmm... While theoretically you are right (the app doesn't change, the environ/data changed), it doesn't make the application run properly over time, the effort/cost to correct the application is still needed.

Examples:
a. application/system that don't function as expected anymore due to a security patch applied on the OS (damn;d if you don't apply the patch, damn'd if you do).
b. applications/system that don't scale anymore with the number of concurent users
c. applications/system with performance strongly a

It's not so much that the app rots, but it's support structure and fitness do. Requirements slowly grow until they exceed the app. The OS the app depends on gets a stream of updates until one of them introduces an incompatibility. People who actually know that app move on, demand for knowledge of that app declines and so less bother to learn it. Vendor EOLs the app or one of it's dependencies and support for it. Ancient install media gets damaged or lost.

Everyone I've been talking to my field is telling me that corporations are spending like crazy on IT in the last two quarters, and are going to continue spending large amounts for at least the next year. There has been some slow down after the economy tanked, but from everything I've seen, the cash flow spigots are opening up.

In my own experience I just got a new job six months ago and it has been non-stop, balls to the walls busy since I walked through the door. We're hiring new people and spending milli

I have been a UNIX sysadmin for years, and have seen this at many companies. For a long time. Back in 1997 I worked at a company that had a lot of critical applications on an old, old Sun Netra. Lots of organizations have a few of these machines around - they are out of maintenance, the people who built it have left the company, it is out of warranty, and people generally don't touch them and hope they keep running. After years of meetings, presentations, budgets etc. we finally got a new machine shipped - and then a more politically connected department heard the machine had arrived, and had it stolen for their department on the day it arrived. I guess everyone thought it would be cheaper to page me in the middle of the night if the old machine failed and blame me the next day. The replacement machine being pulled is one of the main reasons I quit the company.

I worked for another company that had a lot of money, but one thing we had to deal with was printing. Print jobs would come into our machines from strange places (IBM mainframe machines, from programs that were written 40 years ago) and go out to strange places (old dot matrix printers in a field office out in some obscure city in India). Thus I was sometimes left to puzzle why some program written in PL/I, coming from a mainframe which I don't have access to, is not printing to some ancient printer in Bangalore which is hooked to some ancient PC's parallel port.

My former company from 2009 had some machines like this. Two very old Ultras running StoryServer and who knows what else. The StoryServer license had long fallen out of use, the machine firmware and Solaris OS had not been upgraded or patched for years. It sent e-mail through, for some reason, four Macintoshes. The Macs did not even run MacOS X, they were previous MacOS versions. E-mail starting with the letters A-F went to Mac1, G-M went through Mac2 etc., if a Mac crashed, mail to those letters would stop going through. The developers did not want to spend the time migrating to a new system, and I don't blame them, the oldest long-time developer there who dealt with such arcana was laid off, while the people building the latest new and shiny that the business wanted had the most secure jobs. Aside from this, we did not ever patch or upgrade our Red Hat Dell servers or firmware, we had no scheduled system downtimes etc. Our major Java application server had had its license run out. As I was leaving, the operations boss (soon to be fired) was considering not re-upping our Red Hat licenses.

If a sysadmin goes on a job interview, and is not desperate, these are the types of questions they should ask, at least on the second round of interviews. Are all of the machines, OSs and applications I'll be responsible for under license? Are they all fully patched and upgraded for firmware, OS and application on a regular basis? What is the oldest machine still under responsibility - is it older than three years? Because all servers should be phased out every three years - at the very least. Try getting Dell/HP to support a 7 year old server decently. Also, do you have scheduled downtime once a week? Meaning do you have the option of rebooting and patching your main database machine, even if it is early Sunday morning? If they want 100% uptime it would necessitate paying for the infrastructure for high availability.

Why should they spend the money when they can just call you in the middle of the night, to continue keeping it running with duct tape? Then they can blame you the next day after it broke. And you get no credit for it continually running either - the time you spend keeping it running is not counted, only time you devote to the latest shiny they want to implement. In fact, too much time devoted to keeping the machines they decided not to spend money on keeping up can cost you your job - if there's a choice between laying off the guy maintaining legacy stuff, and the guy who makes the new shiny for the business group and management and who deals with the

I think anyone who works in IT long enough comes to think of printing as the biggest waste of money in corporate America. How many forests have ended up as paper jams in a printer because a manager wanted to print his email.

So true. It gets even more fun when you're in an environment that is unfriendly the many small moving parts of a printer. Such as areas with large amounts of dust or sand in the air. We used to have printer parties . We'd take a bunch of old (mostly broken) printers out back and find various creative and entertaining ways to disassemle them so they would fit in the disposal barrels.

I work in enterprise tech support, and finding a setup with unsupported hardware or code is the highlight of my day. No problem is easier for me to close than one where the customer has let equipment drop out of warranty and/or software support. Just did one today... box has been in continuous service (without poweroff) for six years; the model was discontinued in '05 and dropped out of support on 12/31/09. They have a difficult problem that likely would have taken me a day or two to solve... instead I m

In the age of BS corporate leadership, who *doesn't* want to be the guy who cuts costs by 25%, gets promoted up into the suites, then lets one of his successors take the fall when the shit hits the fan? I'm more concerned with our public infrastructure BTW.

Our service management team haven't applied a single patch to antivirus in over two years. Firmware on storage silos is pre 2007 level. OS levels and middleware are all out of support, and the application software will probably fall out of support in March unless someone has the balls to sign an extension before the end of the financial year. I won't bore you with the details on ancient client computers, which can't be upgraded to supported levels of software. When I point out the obvious, rising risks and