A third of London councils and more than a quarter of England's metropolitan authorities have admitted to using unsupported server software – and three are still running Windows Server 2000.
The figures were revealed under a Freedom of Information request that asked the councils which of the following they ran anywhere in …

I can remember in around 2002, we were looking at upgrading our then aging UNIX platforms. We were using AIX back then, with most systems running 4.1.* or 4.2.* so were all years out of date (some quite a lot of years!).

The plan was to shift everything to 5.* which also including new hardware in many places. (We did no Open Source at the time, so Linux wasn't even discussed)

One major bit of critical licensed software we were using (around £200k per year for a single server licence), turned out, that even the latest releases from that year (2002), were only certified for AIX 4.2.1 & 4.3.1 (1997 and 1998 respectively). it hadn't even been certified for the newer point releases for 4.3.x, let alone 5.n!

In the end we just took the risk and tested the software ourselves. We told the vendor that if they refused to still provide support, we'd go elsewhere for a new product.

About 2 years later, they still hadn't certified for AIX 5, and as we were planning on a major update to the platform, we just went elsewhere (it was a data integration and transformation engine). We also told the vendor specifically why we were not renewing with them.

"You'll probably find that a lot of these outdated versions of Windows are being used because they're running really old software that can't be upgraded and isn't compatible with more recent versions."

I once did a fix on a PC running Windows NT4 at a very specific patch level for Rolls Royce. It was one of a pair running a pair of specialist machines for finishing jet engine parts. The machine control s/w would ONLY run at that patch level, nothing higher, nothing lower. Some software is just very poorly written or takes specific advantage of buggy or transient "features" so it's no surprise that large organisations will run ancient kit to support some apps or hardware that needs to be retained for whatever reason. In the case of councils, as others have mentioned, there will be legal requirements to retain access to some systems or data that can't be economically moved to newer systems, especially as central government contuies to squeeze there budgets.

Considering the recent headlines over Northhamptons budgets, I'm surprised they didn't get an honourable mention in the list.

Indeed, I work for local Govt these days, and the sheer number of applications we support is bewildering. When I worked for an ISP, we had 'CableMaster' the app that managed customer accounts, and sent signals out to (then analog) set top boxes to provision their packages, some telephony management intermediate software, file and print, email, and the usual HR / Payroll. That was it. In local govt, we do a lot of things. We have a database covering every tree along every road and pathway for instance, which we inspect. A database for road maintenance, we run leisure centres, recyling centres, we register births, deaths, and marriages, licensing for boozers, housing, benefits, planning applications, museums and galleries, social care, education, libraries, ... so some of these apps just do what is needed, and sometimes the software vendor shows very little interest in upgrading the app for a handful of clients,.... they'll charge for changes, but that's about it. Therefore moving platforms just isn't possible often.

I'm not sure if suggesting OpenBSD is sarcasm m0rt, but if it isn't you clearly haven't used it much.

I am an OpenBSD fan and use it frequently on the desktop as well as infrastructure, but the support policy is *one year* - current and previous release, new version approximately every six months. There are no LTS versions.

It's true that OpenBSD is secure, and that there have only been two remote holes in the default install, but the default install is mostly limited to OpenBSD specific infrastructure software (firewall, email, very basic web server, routing, dns). If there is no local execution of programs or third party software then yes, upgrading is less essential, but many people need additional functionality.

If there is a need to go beyond the default install ports/packages are not audited to the same extent as the base install, there is no binary compatibility, and the ports tree is a moving target, so a couple of releases on it's possible it will not build against an unsupported release.

Furthermore, OpenBSD's policy is pretty much 'security before all else' - firewire, bluetooth, and the Linux compatibility layer were dropped completely because they weren't being adequately maintained and security couldn't be guaranteed. Currently hyperthreading is disabled by default under OpenBSD due to the speculative execution information leakage issues, which is certainly an effective mitigation but (on Linux) leads to around a 30% performance degradation in some scenarios.

If a council can't upgrade beyond a Windows Server version released in 2000, I'm not sure BSD is really the best idea..

Using Linux or BSD doesn't change anything. It is the attitude behind such decisions that needs to change.

I know one company that was still pushing out servers with SUSE Linux 7 on new machines to their customers in 2015! Version 7 was released in 2000! They only switched to a newer OS, because the new RAID controllers couldn't be used.

The attitude was, "its Linux, it doesn't need patching".

That said, I've also seen systems on Windows 2000 hanging around for compliance reasons. So, with a 10 year storage requirement for historical data, I can imagine that 2020 is about right for replacing Windows 2000, if support ended in 2010. If it is an old system that is no longer actively used, you have to either keep it around, spend big bucks on a newer version, just for it to sit in a corner and gather dust, "just in case" there is a problem with an old customer or spend big bucks to have the historical data transferred to a new system, where it just clogs it up.

A lot of people will settle for the "risk" of having that old kit around, "just in case", rather than investing heavily for no gain. If those systems are then air-gapped from the rest of the network, many can "live with it".*

Re: Universities

@Sal II

My experience too,... I've worked for two different Universities (well, one was a Poly when I first worked there) and we were early adopters, at one point we had the largest deployment of Microsoft Exchange in Europe (so the guy from MS told me at the SIG meetings we used to attend), that was Exchange 4.0. (there was a DOS client back then,......)

At another Uni (a real one this time) we were quick to adopt Linux, had a web server when the web was really just academics, and made some of our data publicly available via the web (I won't bore you with the detail, but the guy that came up with the idea had a light bulb moment down the pub, while looking at their CD Jukebox)

Re: Universities

Most of them very probably do have at least some legacy kit for specific reasons. I deal with a number universities and as others have just said, universities are likely to be at the leading edge if not actually bleeding edge for most of their kit. (one Uni I did some work at had just completed a complete server room refresh and I notices a beige box in the corner, still powered up. I asked what it was. It was a Windows NT Server box with a multiplex modem in it for those professors who still insisted they needed dial-up access when off on expeditions, sabbaticals etc. (this was about 5 years ago, so hopefully that's now gone to the bit bucket in the sky!)

Lies, damn lies and FOI

We ran a server 2003 instance until very recently, and I constantly got criticised for the "gross security risk" that represented.

This is WRONG for *some* use cases.

On a well designed infrastructure, it is more than possible to design the network operations in such a way that an older, but still critical, application can run on unsupported Hardware/OS/Application framework and etc. safely - if it is only used internally, and cannot reach/see the internet.

It takes effort and planning to ensure that it cannot be reached except as required to provide the "service" it exists to provide, and is only accessible by the clients and methods essential to that service... but that's why internal DNS, subnetting, VLANS, Reverse Proxies and Firewalls exist: to mitigate, control and contain risk.

So MUCH of my staff's time is wasted responding to FOI requests that are just used to sell my details to marketing droids... that I don't want to hear from (and no I don't want your white paper, didn't give you permission to store my details, so GDPR them off your contacts system, please, thank you and goodbye).

Re: Lies, damn lies and FOI

In my career, spanning Windows NT4 through 2016, I can remember all three times that I have actually approached Microsoft for support and on all occasions it was in the first year of release of the OS. There are so few issues that actually require escalation to Microsoft that cannot be dealt with by a competent support team.

If your database fluffs up, Microsoft couldn't care less unless it's a bug in their software. You probably would have found that diring the first 5 years of running the database on the platform.

Running on out of vendor support OS is completely safe if the network is correctly configured to isolate the OS and only permit access to the application. No internet access from/to the server blah blah. In fact, that 15 year old environment not get patched, poked and prodded all the time by external influences is likely to be rock solid.

Re: Can we trust the answers?

Also I expect the survey didn't ask what proportion of their business systems are running legacy OS's and the importance of such systems to the business.

I have a WfWG laptop that gets powered up occasionally - when I have need to go that far back; I also have a laptop running XP with a suite of useful tools; however, all my everyday business systems are running OS's that are in support and thus receiving security updates.

It's all about the money...

We have all seen it and been there - a company/organisation that you work for is running outdated and obsolete kit, desperately in need of an upgrade. However the management won't listen to you, don't believe that IT systems are important, and use any company cash for a nice pay rise and a final salary pension.

One company (that I know of) in the Temple (in London, an area for Solicitors and Barristers) is still running Exchange 2003!

Re: It's all about the money...

Re: It's all about the money...

Could be worse, when I moved to one company thinking I could move beyond creaky 2003, I found out they were still using ES5.5. Fortunately now on hosted Exchange, and it works fine.

Not that I can speak, one customer is still running ES2K3 because the gateway (written by me) isn't being updated. Blame Microsoft for that, the interface changes after ES2K3, the customer is unlikely to pay for a gateway upgrade, and there are no economies of scale to fund ongoing development (it's a very specialised gateway used by a tiny number of customers).

Potentially not THAT bad

If they're running in isolation, locked down, maybe just for compatibility with an old application then that seems pretty reasonable.

I had to repair a system running Windows 98 a few months ago. An embedded system that's been running every day for 20 years (rebooted daily, mind- it's still Win98) and shows no sign of failing, except for the date/time backup battery being too old to work (it was a soldered-on model, it's been out of service for about 10 years but they decided to renew it). I've no significant worries about that system outlasting the rest of the system it's a part of.

Somerset?

The FoI – a glorified sales pitch

I have come across quite a few of these. You get a request that is a thinly disguised pitch for information so that someone can try and sell you stuff that you clearly aren't looking for.

This gets the whole idea of the FOI Act a bad name. That beloved public figure Tony Blair allegedly said that passing this was the worst thing he did whilst in power. I am not convinced of that but it is certainly used for purposes very different from what were originally thought of for it.

Re: The FoI – a glorified sales pitch

I remember working in the public sector and being cold-emailed by an events company. Ever dealt with events companies? They're about one step up from recruiters most of the time. Anyway they wanted a copy of our org chart, probably so they could add a few senior names to their cold-calling rota.

They also made threatening noises about invoking an FOI request if they didn't get the info. I assume this was because they thought public sector bods would fall over themselves to be helpful at the merest hint of being slapped with an FOI request.

Thing is, this info wasn't that easy to get hold of, I had better things to do like read El Reg, and I'd taken a pretty instant dislike to this company. So I just replied with the contact details of our FOI team and, after another couple of huffy emails, the events company buggered off. As far as I know they never submitted a formal request.

Actually, an admission; If I was answering a survey similar to this honestly then i'd have to reply that i'm still running either NT4 or Win2k. (not actually sure which) Additionally, i'd have to admit that it's never had a security patch since being installed and that I have no plans to touch it.

This is because it runs the firms voicemail system, and came with the telephone system a very long time ago and has quietly kept ticking on since. It's connection to the outside world is via a bank of 56k modems, which receive telephone calls and also do the usual voicemail playback stuff for internal staff. It doesn't even have a network card, being of the vintage where motherboards left network and USB connectivity to be provided by PCI cards, rather than being baked into the motherboard.

The only way of getting information out of it would be direct physical access to the console (bringing your own PS2 mouse & keyboard + DSub monitor) and then writing something to transfer the data via the serial port. It's sort of more "no risk" than low risk when you consider remote data compromise. It's (still!) got an external support contract for BCM, which ends any concern about it still being kicking around after what now must be about 20 years.

I have yet to speak to somebody else at an industry event who won't admit to having something really old like this sitting around somewhere.

"The only way of getting information out of it would be direct physical access to the console (bringing your own PS2 mouse & keyboard + DSub monitor) and then writing something to transfer the data via the serial port. "

Heh

I once did some contracting work for a 'Major Financial Institution'. They were running NT4 servers and a mix of Win 98 and Win 98 SE desktops. This was in 2007-8. They were thinking about moving to Win2k servers and either WinME or Win2k desktops 'within 18-24 months'. Maybe. Win2k3 and XP were not even considered. Macs were 'too expensive'. Linux did not exist, so far as they were concerned.

Thy did hot-desking. The hard drives in their desktops were, umm, inadequate. This meant that a common problem was that users could not log in, because there wasn't enough space available to store their profiles. You read that correctly. The admins would log in via an local admin on a machine, clear the accumulated profiles, and move on. They spent a significant portion of each day doing this.

I recommended Win Server 2003 or 2008, and Win XP or Vista clients. This would have meant a complete replacement of their kit, as the Win 98 desktops could not run XP or Vista, and the servers were barely up to NT4. Some still had SP 3 installed because they 'could not run' SP 4. None had SP 5 or later.

After management recovered from the heart attacks produced by reading how much this would cost, they thanked me politely, paid my final fee, and updated to Server 2k and Win ME before the end of 2008. Exactly where they got the licenses and how they managed to get the things to install is unknown to me. They were still using 2k and ME at least until 2014.

How can they still be PSN certified?

I'm an ex local gov employee (outside of London, but not too far) and worked as an ICT manager and know that most LAs have to comply with the PSN Code of Connection (https://www.gov.uk/government/publications/psn-code-of-connection-coco). Running these unsupported servers is a direct violation of this agreement...so the question is, how can they still be allowed to operate on the PSN network (which handles sensitive DWP data amongst other things)?

I know where I worked, we had 2x 2003 servers still running and we were allowed to keep operating with W2003 as long as we could show a clear path to migrating off of the unsupported software. However, this is farcical as this remained the case for at least 3 years in a row (and was still the case in April this year) and basically showed that the PSN CoCo is toothless and LAs are getting away with risking ALL of our data!

This just goes to show how lax central gov is when it comes to this compliance and also how underfunded LAs are. In our situation, it wasn't because we wanted to remain on legacy software, we just couldn't afford to replace the systems running it, but where the MoD have the luxury of paying for extended support, LAs can't afford to do this.

Come on Central Gov...wake up and smell the sh** that LAs are coping with!

So did you say this..

Science Museum

Once upon a time I went to London and visited the Science Museum. They had a great many legacy technology on display, including Sptifire and Hurricane fighters, which makes me wonder if all those exhibits still enjoy regular support and maintenance from their original factories.

Re: Science Museum

Once upon a time I went to London and visited the Science Museum. They had a great many legacy technology on display, including Sptifire and Hurricane fighters, which makes me wonder if all those exhibits still enjoy regular support and maintenance from their original factories.

. . . Actually, yes. They do.

In 1963, a Mr Bill Lear Jr was living in Geneva, Switzerland and flying a surplus P-51. After numerous problems with the starter clutch on his Packard-built Merlin, he contacted Rolls-Royce. They instructed Lear to send them the clutch, which was quickly repaired and returned. Lear adds:

“I called my benefactor to thank him and to ask him when to expect an invoice. His reply was: ‘My dear Mr. Lear, Rolls-Royce-designed products do not fail. They may require occasional adjustment, but this is covered by our unlimited warranty. So there is no charge, sir.’

I was blown away. The engine and clutch had been manufactured under license in the U.S.A. by Packard in 1944, yet Rolls still stood behind them in 1963!”

Apparently this has been found quite handy by the RAF with the Battle of Britain Memorial Flight, but I suspect that the science museum doesn't really require that much in the way of support.

Xp is still alive

Re: Xp is still alive

I have an embroidery machine that needs XP. The manufacturer's suggested upgrade is to buy a new £5,000 machine. I met their UK MD recently, and explained their competitors support Linux, and IF I replace the machine, I will NEVER buy into Windows dependence again. I have learned my lesson.