Posted
by
timothy
on Wednesday November 24, 2010 @01:08PM
from the nothing-can-go-nothing-can-go-nothing dept.

sustik writes "For a month now the New Orleans real estate market has been crippled by a computer crash that caused the loss of online data from the late 1980s that should be researched prior to the closing of any real estate transactions. 'The clerk of Orleans Parish Civil District Court said Tuesday that her office continues to make progress in resolving the computer problems that have been holding up real estate transactions in New Orleans for the past month, but there still was no indication of how soon the crisis might end.'"

Pencils were banned. Someone tried to sneak one in so that they could work on fixing the problem, but the newly installed full body scanner at the office entrance detected it and the weapon was confiscated.

I'm still trying to figure out from the article...exactly why a computer problem in Jefferson Parish, is affecting the Orleans (New Orleans) Parish housing market so much? They're two separate entities...

It is difficult to catch a break when most of your problems are self inflicted. Even with Katrina, the problem was the hurricane as much as the levys. Oh, and the whole "Yes, the city is sinking, and yes this land is already below sea level, but it is still a great place to build houses!". Then, I don't feel all that sorry for people who bought houses with and 3 years later the APR jumped up to 15%, while the values dropped. At some point, people have to take a *little* personal responsibility for their actions. And stupidity.

They probably requested a backup and some vendor simply partitioned the hard disk into two and made the backup save to the second partition, meanwhile telling them that the machine has two drive letters.

Meanwhile, they likely paid the vendor millions to maintain the system...

Yeah, I've been dealing with this issue for the last 30 days as I sell properties across the county and a few in New Orleans. It's been a nightmare. But apparently, the back up procedure wouldn't work for some reason. They've got it mostly up now though and or only missing the last year's transactions. Still a pain in the ass but real estate is slowly moving again.

Might just be calling it a "crash" so that almost anyone can understand that the computer is borked and you can't get the data out. It's a lot like how most people will say they have a computer "virus" when it could be a trojan or a worm. It's a catch-all term that everyone, even non-technical people, can understand fairly easily. Even if you've never used a computer, you can at least associate it with a car crash and come away with the conclusion that something bad has happened.

Hate to reply to myself, but here's an earlier article from when the incident occurred. [nola.com] The article states: "The problem, which has been traced to a failure in the hard drive—" and "'The original real estate records HAVE NOT BEEN LOST,' Atkins said Thursday in a written statement." which suggests that it probably was a disk failure that wiped out at least the directory structure for the files or the index of the database in which the records were stored. So a "read-head crash" could be the actual culp

When I see the phrase "The original real estate records HAVE NOT BEEN LOST," I interpret that to mean that they still have the deeds, surveys, sale contracts, liens, covenants and easements on file, on paper, in a cabinet.

Which is good. It just means they'll have about 30 years of data entry to do...

When I see the phrase "The original real estate records HAVE NOT BEEN LOST," I interpret that to mean that they still have the deeds, surveys, sale contracts, liens, covenants and easements on file, on paper, in a cabinet.

Which is good. It just means they'll have about 30 years of data entry to do...

No, it means that the records haven't been LOST.

They've just been, er, 'misplaced'. Somebody put them in the round bins near their desk one day and they were taken away for storage. We'll find them soon. Not Lost, not at all.

Most likely it all got deleted, but was originally copied over from paperwork (I doubt they were storing records in a computer back then) So they're all probably furiously re-entering the data by hand, not realizing a scanner and a little help from google could get it done in a few hours.

You do realize that the term "crash" isn't a particularly technical term, right? A computer crash could be anything from a hard disk head crash to massive data corruption leading to the box not booting to a literal "the computer just fell out a window and crashed on the sidewalk scenario." As for that last one as dumb as it is, I bet somebody here knows of a place where that happened.

It's been the term used for a generic catastrophic computer failure for as long as I can recall, and that goes back decade

That's bad, but what's worse is how all the geeks seem to think that a CPU is made by either Intel or AMD. Those are processors that they're primarily known for and the CPU is the box that holds the processor, RAM and the rest of the goobins in.

Now for all those that will correct me on that, give off my gall darned lawn.

Their problem is that they've lost indexing data, not the underlying documents. So just make the documents, which are public records, visible to Google. Google will index them and anyone can then search.

Doesn't work so well if the underlying documents are scans of paper documents - I don't know if Google's PDF searching deals with OCR'ing text. And even if it does that assumes the scans are actually stored in a reasonably well-known format like PDF.

So, when did your data become important to you? Before or After you lost it...

You wouldn't believe how many people don't properly backup critical data. If it's important, really important, here are a few tips:- Have current backups- Test the backups to ensure they work- Keep multiple backups- Keep the backups in separate locations, preferably separate sites if possible, and if really critical, separate cities (Disasters happen)- Keep backups in a fireproof safe or equivalent, it should be waterproof as well, and probably airtight- Even though you need to keep the backups secure, you need multiple people that can access it when necessary (Accidents happen, people die, people lose keys and forget combos)

Those steps are simple, and a business can easily do all of them. Individuals may have less capability to implement everything. If you choose to do less, you are balancing the value of your data against the probability of losing it. I dealt with many many people who didn't follow those rules and lost their data. It happens, a lot. Business records, bank statements, novels, doctoral thesis, family photos, source code, chat logs, porn, contact lists, and more. Too may people blow off the importance of preserving their data until after it's gone, and when that happens, there are only two things that you can do. First, hope that a data recover place can recover some of it (all is a really long shot) but they'll charge you through the nose. Or two, deal with the lose and suffer the consequences. There are no miracles or magic pixie dust in data recovery.

Tip for data recovery. If something happens and you need the data back, I wish you luck, but here's some things to do that may improve your odds. TURN OFF THAT MACHINE AND REMOVE THE DRIVE THAT THE LOST DATA WAS ON! Your computer is doing things even when you don't tell it to. If it writes to the drive, it may very well write over where your precious data was. If that happens, it's gone, period, for-ever. No data recovery place on the planet gets back data that's been written over. They may be willing to try, and charge you an outrageous fee even if they fail, but the will fail. Usually only part of the data is written over, so something can be recovered, but it may be useless. After all, half an exe is pretty useless, but half that novel might help you out. Sorry about ranting, but seen way too many bad ones, and I know you don't want to go through that.

It constantly amazes me how hard it is to convince people to backup their data.

At my previous job I did outsourced IT support for local businesses. We'd have a hell of a time selling them any kind of a backup solution. They'd rather just trust that things were going to keep working than spend a couple thousand on software and a tape drive. Or they'd never, ever change the tapes. Or they'd keep using the same tapes for years. Or they'd store the tape right on top of the server, so that any disaster that

Uh, dude, if they had a backup, they wouldn't be coming to the computer shop for data recovery. But good on you for treating people like shit and assuming they know as much as you do about computers. Administering a backup system is a non-trivial task for novices. I'm sure people enjoyed hearing it was their own damn fault though. The little human touches are what makes being a computer technician all worthwhile.

Uh, dude, if they had a backup, they wouldn't be coming to the computer shop for data recovery. But good on you for treating people like shit and assuming they know as much as you do about computers. Administering a backup system is a non-trivial task for novices. I'm sure people enjoyed hearing it was their own damn fault though. The little human touches are what makes being a computer technician all worthwhile.

Perhaps I wasn't clear...

Basically any time we got a call for data recovery, from a home user or a business, the first question is "do you have a backup".

It actually does make sense. Why waste hours trying to recover data from a hosed HDD only to find out that there's a tape in the closet nobody mentioned? It has happened.

For business customers that we may never have worked with before, this is a very reasonable question. They may have a tape drive ticking away doing backups, but no idea how to actually

Thousands for software for backups?Stop scamming small businesses, amanda or bacula are free and a very good fit for such places.

Ummm... What I actually said was:

They'd rather just trust that things were going to keep working than spend a couple thousand on software and a tape drive.

Emphasis added for the comprehension-impaired.

I never handled the pricing or billing, I just fixed stuff... So I really have no idea how much we may have gouged our customers...

But we'd generally sell them some flavor of Symantec Backup Exec, or maybe Acronis True Image - both of which retail for a few hundred dollars. Throw in an LTO tape drive of some sort for another few hundred dollars. Maybe a SCSI/SAS/whatever controller for the server itself, another hundred or tw

The cost of hard drives is so low today, that tape backups rarely make sense. A hard drive in a USB enclosure that can easily be taken offsite makes much more sense. It is far easier to see if the backup is successful (files don't need to be compressed) and restoring is also much easier (since you don't have to deal with multiple tapes, compressed files, etc.)

Please show me the drive that can take 100MB/s 100% of the time and is ~1TB in size. It must also last years unpowered and be rated for such use. Max cost should be $30, the cost of an LTO4 tape.

Hard drives are not for backups, they are not designed for that use nor should anyone trust them for such use. Nor are they available as WORM devices. Hard drives have a use, storing your backup data is not it.

"So, when did your data become important to you? Before or After you lost it..."

Exactly! I've seen so many servers with hardware that dies. You manage to find a way to bring it back online and warn them that they need to replace the old equipment but they often don't. Generally, people think if it's running today it will be running tomorrow.

I am amazed that title insurance still exists. Well, not amazed - the answer is simple - lots of people make money selling it.

This is a problem that has a VERY simple solution. It was solved for cars ages ago.

The local government keeps a registry of all local properties (gee, they have to do that for taxes anyway). When property is bought or sold the registry is updated. When leins are placed on properties, or removed, the registry is updated.

It's called Torrens title [wikipedia.org].
Of course, backups are still important, it's just the amount of data is smaller and the format likely to be friendlier than digitised versions of some ancient papyrus.

Because of the havoc that the storm [Hurricane Katrina] caused, Atkins' office had hoped to prevent future snafus by hiring a company called i365 to back up the data regularly. But, Atkins said, all that information wasn't being backed up.

When the problem was first detected, "we were told it was a system failure, and they could get us up and running," Atkins said. "I don't think the court was made aware of the severity of the problem until late last week."

What would have been due diligence on the part of the court clerk to verify that i365 was doing their job? And why hasn't this problem been resolved three weeks later? I can see why realtors have asked the governor of Louisiana to get involved.

I dunno. The problem with this attitude is that it leads to one big race for the bottom.

Whose fault is it that wood glue ended up in infant's formula in China? I'm sure the guy who bought the formula signed a contract. Never mind that he bought it for 1/10th the going rate, I'm sure that on paper it was called formula and not glue.

Due diligence isn't about reading the paperwork - it is about doing your job.

Can't say that I fault a non-techie for hiring someone they think is reputable and from that assuming they are covered. That said, nothing wrong with carefully shutting down the system, hiding it, and calling said provider and saying "it got stolen how fast can you have me back up and running?" That would have uncovered any flaws in the backup process.

From my perspective, if it is located in the same building as the source data, it is a copy, not a backup. If it is offsite, but has never had a test re

...through the seas of hyperbole and ill informed journalists who in turn try to dumb down articles for the masses. For starters article in question is talking about them working through the backlog created by the original crash. The system went down, most of it is back, but the office is swamped trying to catch up.

"loss of online data from the late 1980s"

The data wasn't lost. From T original FA:

"The original real estate records HAVE NOT BEEN LOST," Atkins said Thursday in a written statement." (emphasis th

Perhaps there was no budget for such systems. Often the higher-ups have no understanding of why they should ever want to purchase extra equipment which isn't (visibly) doing anything to make things go.

You can lead a horse to water, but you can't make it drink, no matter how long you hold its head under water.

I find the use of phrases like "educate the higher-ups" charmingly naive. They're higher-ups. They don't need anything from you but compliance and endless status reports. And GOD FORBID if you somehow get the idea that you know something they don't.

In other words, your statistical sample of exactly one is not useful. The singular of "data" is not "anecdote". Dilbert is non-fiction.

Hmm, the only way to make sure everyone knows that a backup process is a) necessary and b) actually functional is to schedule a "business continuity" exercise and perform a simulated disk failure.

Get approval and buy-in from those higher-ups first, if you want to keep your job, though:-P It probably also helps if you keep a stack of dead drives on your desk with a skull and crossbones on top of them to drive home the fact that those things do indeed have an MTBF.

Doesn't work that way. I remember at a previous job needing extra batteries for the radios. The radios which were our only line of communication if we needed help. But they couldn't find the money for that even as they were pushing for more aggressive means of dealing with trespassers.

IT is a lot like that as well, just because there's a pressing need doesn't mean that there's somebody to convince that cares about anything more than the bottom line. A job like that is going to fill, somebody is always that desperate, but suggesting that the admin has any responsibility for that is pretty ignorant. You can't convince somebody that isn't listening.

It is part of the job of the admin to have them listen. If you are any good, in my experience, they will listen. The resoning why the extra expense is warranted and needed requires research, total knowledge of the factors and good communications. If have dealt with many admins who could/would not do that. A valid presentation as to why the expense is needed will, usually, get you the approval.

It's not necessarily that simple. Backup tends to get no respect or funding. A horrifying number of sites don't include backup solutions as a part of the cost of funding new machines. And if it was there that long it's entirely possible that whatever backup solutions were available and used then aren't going to be useful now.

Unfortunately just because the volume of data increases doesn't mean that the systems used to back it up are so easily scaled, the increased need doesn't guarantee extra funding either.

Or it could be an incompetent admin. Wouldn't be the first, however it's more likely that the problem is higher up in the chain.

Backups are one of the most misunderstood and neglected concepts of computing as we know it. Between laziness and vendors selling their appliances and gadgets, there are a lot of misconceptions about proper backups.

Horror story #1: The guy with the term paper on the laptop which gets backed up to a USB flash drive. Roommate gets kicked out of the university, and grabs laptop + drive as a consolation prize. Result: Retaking a course. Moral: Backups to another drive are good, but don't address the problem.

Horror story #2: Business had two machine which rsynced with each other for offsites. One of the sysadmins was disgruntled, rm -rf-ed the files on one end, rsynced that, then rsynced some large blobs so the deleted files would be overwritten.

Backups are easily forgotten about... until they are needed. I have seen a lot of deer-in-the-headlights looks from people who thought they had working backup systems, but in reality, they backed up the wrong data, overwrote the wrong items, had great encryption and no recovery keys, or the tapes were safe at Iron Mountain... but nobody had an account to access there.

Like security, PHBs consider backups pointless because they have no obvious ROI. Of course, this comes to kill businesses if something does fail. Here in Austin, there was a textbook seller for the University of Texas called Texas Textbooks. They were on top of the market. Then they had HDD problems and lost all their data with no ability to recover. End result, a few months later, their doors were shuttered.

Backups are not rocket science. You have a way to copy data to an onsite repository, then a second way to copy it offsite (be it to a cloud, to tapes or other media that you move offsite) This applies to everyone from a SOHO business to the big guys. You then validate that the data is readable, and every link in the chain is present, from having the license keys for the backup software, to having the software somewhere, to the right hardware for reading the media, etc.

Exactly. Copying to another server might help against hardware problems, but all it takes is an admin who wants to do some damage, root, and a dd command to ruin that completely.

The ideal system is D2D2T. Have lots of disk space onsite that backups to go. Then copy it every so often to a tape library, write-protect the tapes, and have those offsited. With WORM tapes (LTO and DLT have this capability), this also takes care of tamper-resistant archiving, as the data on the tapes can only be destroyed, not

It's not necessarily that simple. Backup tends to get no respect or funding. A horrifying number of sites don't include backup solutions as a part of the cost of funding new machines. And if it was there that long it's entirely possible that whatever backup solutions were available and used then aren't going to be useful now.

What happens most of the time is you've got a small group/department/business, they hire a consultant group to come in and set things up, and then call them when they have a problem. T

Backups are also often seen as an end. "We take backups every X time." Or "We are looking for backup soft/hardware.". I start with the REAL end goal: restoring.How easy is it to restore data and/or systems?

I am sure we all know or lived stories where backups did not work for various reasons.

How often are restores tested? Most often they are not. A backup is a waste of time if the restore does not work.

If it's the same IT as the one that runs the connecting City Hall, it depends on the day. They've had 2 administrators go to jail in the last 5 years.
Google "Nagin emails" and see what we're working against down here.
We have a new mayor and I think he'll turn the city around, but there is a lot of junk to clean up.

Anything dealing with realestate seems to have this lalala mentality to it. When I used to do work on various MLS systems for offices here in Ontario, I was working with netware 1.0 to netware 3. They had no backups, no tape drives, no remote site copies, no UPS system, etc, etc, etc. When I told them repeatedly that backups were required to ensure that they remained operational, I was dismissed and never got another contract with them.

I used to work at another company providing MLS services in Ontario. Our data was on a RAID, which saved the bacon a couple of times while I was there. We also shipped nightly backups to Iron Mountain (an offsite data storage facility). Every computer had a tape drive and every month the office computers would be backed up. Also, every month we would have a disaster recovery planning meeting and would plan out how to get the business running again if we found a smoldering crater where our office used to

MLSs only track currently offered properties, and the records themselves belong to the person making the listing (the agent doing the listing is responsible for making sure it's accurate, etc.) If an MLS died all of the sudden it would be a real inconvenience but the individual agents would just have to re-submit their listings, because it's their listings, not the MLSs. So the proprietary interest of the database is very different than in the case of a county recorder.

You still guys live in the wealthiest country on earth, enjoying the highest standard of living that any human being who has ever lived has had. The crime rate is the lowest it has ever been, and is getting lower every single year. The tax rate is lower now than it has been since the early 1900's and for the pittance you pay you get a social system that is actually run extremely well. Still, you imbecile objectivists ignore any evidence that disproves your dumbass ideology and trump up any news story that can be twisted to support it. You are no less than modern mirror-image Bolsheviks, looking to fundamentally destroy the system in order to build a bullshit utopian fantasy world that could never work.

Objectivists are nuts, but that Social Security ain't gonna look so pretty if we keep running a big fat deficit with a 350% and increasing debt-to-revenues ratio (higher than Greece).... to say nothing of the way that the public employee pension systems are built to assume 10% rates of return and contractually put the taxpayers on the hook for a shortfall.

"....but that Social Security ain't gonna look so pretty if we keep running a big fat deficit..."

Social security has a dedicated funding source. Which is why people are trying to cut it or privatize it. They want access to the money.

"...to say nothing of the way that the public employee pension systems are built to assume 10% rates of return and contractually put the taxpayers on the hook for a shortfall."

And this is different from any other pension system how? Public pension systems are extremely well designed and run. It is the legislatures that fail to fund them or treat them like piggy banks that are the problems. People who are anti-pension are anti-worker. Pure and simple.

You're living in the shadows of White Man's Burden with a hint of WMG. Self flagellation isn't necessary. Maybe if the locals would clean up their own nation's system of governance instead of the western world keeping the corruption on life support, perhaps then they too can live a 1st world life style. Culture also plays a large part in fostering societal development.

Some resources are finite, but often many more are either found or created as a replacement. The slice of each piece of pie does not grow sma

Great, then a meltdown of the financial system can never happen because banks and insurance companies are all private and extremely competitive. What about the oil industry? You can't go any more hardcore capitalist than that. That's why they never ever fuck up.

Banks and insurance companies are heavily regulated by multiple state and federal agencies and are pressured to "do things" that they would otherwise not be likely to do because of political decisions particularly in WDC.

The Community Reinvestment Act is inarguably at the start of the mortgage mess that started when banks were told to give loans in low income areas or risk loss of the right to stay in business in one form or another. That led to loans the banks didn't want and

Sure there is it's called "elections" or at least it would keep them on their toes if a substantial number of voters didn't reward bad behavior. Example, the Republican party ran up nearly $10tn in debt between Regan and the Bushes. And more recently gridlocked the legislative branch for the last couple of years and they get rewarded by being voted in as a vote for bipartisanship.

Democracy is a device that ensures we shall be governed no better than we deserve
George Bernard Shaw