Ars tracked down the original motion to drop the case against Armand Angulo, a doctor living in Iowa who had illegally sold millions of dollars worth of prescription medication online. The DEA started its investigation in 2003, and indicted Angulo and about two dozen other accomplices in 2007. The problem was that Angulo fled the country to his native Panama in 2004, and Panama has been uncooperative in extraditing Angulo to the US.

In the 5 years since Angulo's indictment, the DEA amassed "two terabytes of electronic data (which consume approximately 5 percent of DEA’s world-wide electronic storage capacity), several hundred boxes of paper documents, and dozens of computers, servers, and other bulky evidence. Continued storage of these materials is difficult and expensive," read a motion filed by US Attorney Stephanie Rose in July.

"Given the slim likelihood of Angulo’s extradition from Panama, and the economic and practical hardship related to continued storage of evidence in this matter, the United States moves to dismiss the Indictment, with prejudice, against Armando Angulo," the motion continued. Dismissal with prejudice means the case can not be reopened in court, but Angulo will also be prohibited from re-entering the US.

This particular case wasn't dismissed entirely because the defendant had too much evidence against him, but rather because all that evidence would likely have to be stored indefinitely. Still, 2 terabytes of electronic data consuming 5 percent of the Administration's electronic resources would mean the DEA operates with 40TB of data storage worldwide—a ridiculously small number considering the relative affordability of terabytes of data these days.

Speaking to the Associated Press, University of Iowa computer scientist Douglas Jones suggested that "the DEA's data server must be relatively small and may need replacement, a costly and risky proposition for an agency that must maintain the integrity of documents."

Ars contacted the DEA, but could not reach anyone who could comment on the matter.

Promoted Comments

I think I have nearly that much storage in loose drives in my desk drawers. These are all known good drives, too.

To be fair though, law enforcement requirements on data integrity and backups are pretty strict. They probably have at least 200 TB in storage capacity, with redundancy and backups. The last thing you want is a lawyer to nail you for not having chain of custody on a file...

I don't do server administration at my job, so I don't know what the actual numbers, but I wouldn't be surprised to find that we had double that amount of storage in this building. Not to mention a backup site and far-away disaster recovery location.

A few friends of mine have over 8TB total storage shared over networks. Pure storage that is, no OS.

Considering these days of large drives and serious storage systems like Backblaze pods (45x3.5" disks+single 3.5" boot disk in a single 4U case anyone?), it baffles me how you can run out of storage. Naturally, you double a few times for retention, but 40TB seems paltry to e when a single Backblaze pod will do 180TB with 4TB disks...

Hell, I'm planning on incrementally building one, just need to build the case at this point as I already have a spare C2D machine that currently serves as NAS/DNS/downloadbox (yay 20Mbit/s when you want Steam games fast).

Talking of whic, I still gotta rebuild that particular machine into a KVM virtalized box and have it take over router duties as wel and use my routers as pure APs instead...

I have double that amount of storage in 4U in our primary storage arrays. Probably 10x that in 4U in our test/dev storage.

But I can see how they could end up in that position, particularly if they're locked into a particular vendor (say, EMC). It can be EXTREMELY expensive to do a ground-up replacement of an enterprise storage system. Sure, storage is cheap, but storage with features (not limited to replication, snapshotting, and backup) isn't.

I have double that amount of storage in 4U in our primary storage arrays. Probably 10x that in 4U in our test/dev storage.

But I can see how they could end up in that position, particularly if they're locked into a particular vendor (say, EMC). It can be EXTREMELY expensive to do a ground-up replacement of an enterprise storage system. Sure, storage is cheap, but storage with features (not limited to replication, snapshotting, and backup) isn't.

Considering this is mostly archived data, I don't think snapshotting is much of a needed feature. As for backup and deduplication, its nothing a scheduled job and some secure VPNs can't handle.

I think I have nearly that much storage in loose drives in my desk drawers. These are all known good drives, too.

To be fair though, law enforcement requirements on data integrity and backups are pretty strict. They probably have at least 200 TB in storage capacity, with redundancy and backups. The last thing you want is a lawyer to nail you for not having chain of custody on a file...

The point is, it wouldn't take that much more funding to have an operational data server if they currently have 40TB of storage. They want to double their storage space? Maybe that costs them, what, $5,000?

I think I have nearly that much storage in loose drives in my desk drawers. These are all known good drives, too.

To be fair though, law enforcement requirements on data integrity and backups are pretty strict. They probably have at least 200 TB in storage capacity, with redundancy and backups. The last thing you want is a lawyer to nail you for not having chain of custody on a file...

To be fair, $2 billion budget and 200TB equates to $12,000 worth of harddrives (plus associated hardware, lets say another $50k). With their annual budget they could buy well over 30 thousand petabytes of storage, or using my above numbers over 6000 of hardware required.

It costs more to have someone manage the storage than it does to buy it. Odds are tripling what they have wouldn't add much strain to anything.

The point is, it wouldn't take that much more funding to have an operational data server if they currently have 40TB of storage. They want to double their storage space? Maybe that costs them, what, $5,000?

Well to be fair, enterprise storage costs a lot more than that. An EMC VNX with that much storage would be 250k+

The point is, it wouldn't take that much more funding to have an operational data server if they currently have 40TB of storage. They want to double their storage space? Maybe that costs them, what, $5,000?

Well to be fair, enterprise storage costs a lot more than that. An EMC VNX with that much storage would be 250k+

So they pop this story with all its' ambiguity.NOW they know who is the lawyer and how much storage each of you has.My advice is to read the silly story as if it were fact then, say "Hmmm.....iiieeeentersting." and then:SHUT UP!They are watching you........

The official reason stated: "several hundred boxes of paper documents, and dozens of computers, servers, and other bulky evidence"

While the data storage is doubtless absurd, it is probably this other stuff which really caused them to drop it.

They may have only 40 TB not because they are overflowing but because they are *not* overflowing and it was just incidental to their need to clean house here. After all, that space corresponds to 10's of millions of documents of the kind the DEA probably keeps, and doubtless their legal requirements to catalog it all, identify it, routinely verify it, etc., all the usual bureaucratic BS, is probably far, far more expensive than even the antiquated hardware it is on.

The space is doubtless replicated and maybe even backed up somewhere.

Humongous data, the kind you or I fill our drives with or that is kept in cloud data stores, reaches petabyte levels because most of it is junk. Systems like Hadoop are built to infer grams of gold from tons shit. But the DEA probably (and thankfully) probably has not reached that stage.

Where the hell is Big Brother when you actually need him? How about we disband useless organizations like the TSA and use their previous funding to be allocated across the rest of the 3 letter agencies for HDD space and maybe a server upgrade or two.

The point is, it wouldn't take that much more funding to have an operational data server if they currently have 40TB of storage. They want to double their storage space? Maybe that costs them, what, $5,000?

there are other nonsense regulation government agencies have to adhere to. Yes storage is cheap, but they can't just buy what they think is reasonably price is has to go out for bid to GSA approved places which means the price goes up. Then if it is a production unit you have to have SLA which is roughly 10% of the total cost of the unit per year. Don't forget that a lot of these server rooms being used by gov agencies weren't designed for much above what they had at that time and the bid went to the lowest bidder for building the requirement...therefore their AC may not be able to cool that room with additional hardware. There is also the oh so common x factor which I have had to deal with Ignorant SAN admins. Their solution is always keep the same archaic hardware and throw more drives at it. Instead of looking at a more logical solution like NetApp which may cost a little more but scales better uses less rack space and has better management for snapshots, redundancy, data deduplication etc. Thank GW for making Obama cut government spending too.

I can get the difficulties (and cost) of enterprise level storage which the DEA should be using. I'm baffled by their seemingly inability to move data off of their storage platform and into an off line archival system. While tedious and slow, moving old data to/from tape backup would make sense in this case. Some sort of backup system (either online or off) should be a requirement.

Christ, I have a 10TB media server that I assembled for under a grand. Enterprise grade? Nope. Triple the cost for enterprise grade and I could still quadruple their available storage for the cost of one mid-level paper pusher.

Christ, I have a 10TB media server that I assembled for under a grand. Enterprise grade? Nope. Triple the cost for enterprise grade and I could still quadruple their available storage for the cost of one mid-level paper pusher.

There you go wanting to cut jobs, and in this economy, have you no shame. /sarcasm

*real* storage is quite expensive! Go price an HP P4300 with 16tb storage: its about $25k for the entry level configuration, and you need another $10k in kit to properly connect it to 3 or 4 servers.

Despite all of that, they certainly have the money to spend a million bucks on storage.

Even EMC has plenty of solutions much cheaper per TB than what you are pulling out from your arse. From there, a router at each office that can handle the most basic of site-to-site VPN can do the rest.

It might not be fast with lots of users on it, but it would work. Adding in a few more SAN boxes at high traffic locations and doing a little replication is pretty simple to setup as well.

What usually costs gobs of money in these government contracts is their insistence on paying ridiculous prices for a proprietary one-off interface or database on the device instead of a standards based, scalable, fault tolerant infrastructure. Basically imagine someone buying $5k of server hardware and paying $80k to have someone throw a lightly custom Wordpress site up that has a total of 10 hours of work invested from the initial installation of Linux and Apache to "going live". Further, the government contractors turn around and hire other expensive contractors to fix trivial stuff that their vendor support contracts already cover, but they don't want to have to open tickets or get confidentiality agreements from the vendor support (I could give specifics, but I don't want to overstep NDAs).

If a government agency the size of the DEA has 40 TB of storage, it is because they had that much 10-15 years ago when they first bought the equipment and never setup a yearly accrual for future cap ex. This is exceptionally common in government spending as well and results in those sudden budget increases. Government spending pretty much follows a use everything you have this year, otherwise your budget will get cut next year, retarded form of thinking. Further, they rarely fully account for infrastructure costs as they scale up their workforces.

If a tactical military unit can afford a 60TB solution that costs less than $100K (Dell M3000 with slaved M1000, each with 15 2TB drives), this speaks to an outright lie or an absolutely impotent relationship between DEA leadership and IT support staff. Either are believeable, actually...

And that was cheap as anything - we're a freaking software startup, not law enforcement. This is just a typical case of management not understanding that what was a lot a few years ago is not going to cover things today, or in the future.