Introduction

Well here we are again with this Samsung 840 EVO slow down issue cropping up here, there, and everywhere. The story for this one is so long and convoluted that I’m just going to kick this piece off with a walk through of what was happening with this particular SSD, and what was attempted so far to fix it:

The Samsung 840 EVO is a consumer-focused TLC SSD. Normally TLC SSDs suffer from reduced write speeds when compared to their MLC counterparts, as writing operations take longer for TLC than for MLC (SLC is even faster). Samsung introduced a novel way of speeding things up with their TurboWrite caching method, which adds a fast SLC buffer alongside the slower flash. This buffer is several GB in size, and helps the 840 EVO maintain fast write speeds in most typical usage scenarios, but the issue with the 840 EVO is not its write speed – the problem is read speed. Initial reviews did not catch this issue as it only impacted data that had been stagnant for a period of roughly 6-8 weeks. As files aged their read speeds were reduced, starting from the speedy (and expected) 500 MB/sec and ultimately reaching a worst case speed of 50-100 MB/sec:

There were other variables that impacted the end result, which further complicated the flurry of reports coming in from seemingly everywhere. The slow speeds turned out to be the result of the SSD controller working extra hard to apply error correction to the data coming in from flash that was (reportedly) miscalibrated at the factory. This miscalibration caused the EVO to incorrectly adapt to cell voltage drifts over time (an effect that occurs in all flash-based storage – TLC being the most sensitive). Ambient temperature could even impact the slower read speeds as the controller was working outside of its expected load envelope and thermally throttled itself when faced with bulk amounts of error correction.

Once the community reached sufficient critical mass to get Samsung’s attention, they issued a few statements and ultimately pushed out a combination firmware and tool to fix EVO’s that were seeing this issue. The 840 EVO Performance Restoration Tool was released just under two months after the original thread on the Overclock.net forums was started. Despite a quick update a few weeks later, that was not a bad turnaround considering Intel took three months to correct a firmware issue of one of their own early SSDs. While the Intel patch restored full performance to their X25-M, the Samsung update does not appear to be faring so well now that users have logged a few additional months after applying their fix.

In the fall it was confirmed by Samsung that stale data on some 840 EVO drives would suffer performance degradation and released a tool to mitigate the issue which Al reviewed here. The Tech Report recently heard of some cases of drives slowing even with the new EXT0CB6Q firmware installed and decided to investigate. They took a 840 EVO 250GB SSD which had been filled with files to test the patch and was then left forgotten on a shelf for several months and tested the speeds. The benchmarks showed an average speed between 35-54MB/s far below what you would expect to see from an SSD but in line with what users have been reported. On the other hand another 840 EVO which has been in constant use since the firmware update shows no signs whatsoever of slowing down, though NTFS compression was recently used on the drive which could have refreshed the flash. Obviously more testing needs to be done, keep your eyes out for updates on this new development.

"In October, Samsung patched its 840 EVO SSD to address a problem that caused slow read speeds with old data. Recent reports suggest the issue isn't completely fixed, and the results of our own testing agree."

If you own a Samsung 840 SSD, it appears that after much repeated and vocal pressure, Samsung has acknowledged the slow down also affects your drive. We're not talking about the EVO or the Pro, this is the original pure TLC model that launched (the EVO is a TLC+SLC cache hybrid while the Pro is all MLC). Here's the quote from Samsung, via Computer Base:

What? You can't read German? Neither can we, but paraphrasing from the poor quality translation from several online tools, we deduce that Samsung has acknowledged the issue on the 840, and is working on a solution as quickly as possible. This is similar verbiage to the statement issued for the 840 EVO acknowledgement.

** Update **

Thanks to Zyhmet, who commented shortly after posting, here's a human translation:

Because of the feedback we got, we realized that, accessing specific data with units of SSD 840 could lead to lower reading performance.

For the moment our experts are systematically examining the SSD-units with different system environments and we are working on a solution as fast as possible.

Due to different technologies the PRO-series (840 PRO and 850 PRO) are not affected.

Samsung

** End update **

Side note - of those who have used the 840 EVO Performance Restoration Tool, a few have reported an issue cropping up. The error manifests as a SMART data misreporting error:

What's odd about this error is that it was present on some of our pre-production test samples (firmware EXT0AB0Q), and was corrected once we updated those samples to the first retail build (EXT0BB0Q). The image above was an actual screen shot taken during our temperature-dependency testing of the slow down issue. While none of our samples had the issue return when updating all the way to the performance restored firmware, one of those updates did corrupt the Master File Table, rendering the majority of the SSD inaccessible. While we have seen no other reports of corrupted partitions, severalusers noticed the SMART reporting issue after updating. It's odd to see this sort of a regression with firmware updates, in that a bug fixed in the initial shipping firmware has returned (for some) in a subsequent update. If you've updated your 840 EVO with their Performance Restoration Tool, it may be a good time to check your SMART attributes. If you see the error above, please leave us a note in the comments.

Circling back to the slow down issue - given that it is present in two TLC-based SSDs from Samsung, one has to wonder if this issue exists in other Samsung TLC SSDs as well. Here's the list of potentials (thanks to an anonymous comment on a prior story):

840 EVO - 19nm TLC

840 - 21nm TLC

PM841 - 21nm TLC

PM851 - 21nm TLC (some SKUs)

845DC EVO - 19nm TLC

PM843 - 21nm TLC

PM853T - 21nm TLC

We have several questions out to Samsung on these issues, but to date they have not been answered. More to follow as we wait for an official (English) response.

Over the weekend Samsung silently updated their 840 EVO Performance Restoration Tool. The incremental update improved support for some system configurations that were previously not recognizing an installed 840 EVO. Samsung also improved how the GUI progress bar responds during the update process, presumably to correct the near silent failure that occurred when the tool was unable to update the drive's firmware. Previously, the tool would halt at 15% without any clear indication that the firmware could not be updated (this would occur if the tool was unable to issue the necessary commands to the SSD, mainly due to the motherboard being in the wrong storage controller mode or using an incompatible storage driver).

Still no word on relief for those owners of the original 840 (non EVO or Pro). We've also heard from some users with Samsung OEM TLC-based SSDs that showed the same type of slow down (some variants of the PM851 apparently used TLC flash). More to follow there.

Introduction

** Edit **

The tool is now available for download from Samsung here. Another note is that they intend to release an ISO / DOS version of the tool at the end of the month (for Lunix and Mac users). We assume this would be a file system agnostic version of the tool, which would either update all flash or wipe the drive. We suspect it would be the former.

** End edit **

As some of you may have been tracking, there was an issue with Samsung 840 EVO SSDs where ‘stale’ data (data which had not been touched for some period of time after writing it) saw slower read speeds as time since written extended beyond a period of weeks or months. The rough effect was that the read speed of old data would begin to slow roughly one month after written, and after a few more months would eventually reach a speed of ~50-100 MB/sec, varying slightly with room temperature. Speeds would plateau at this low figure, and more importantly, even at this slow speed, no users reported lost data while this effect was taking place.

An example of file read speeds slowing relative to file age.

Since we first published on this, we have been coordinating with Samsung to learn the root causes of this issue, how they will be fixed, and we have most recently been testing a pre-release version of the fix for this issue. First let's look at the newest statement from Samsung:

Because of an error in the flash management software algorithm in the 840 EVO, a drop in performance occurs on data stored for a long period of time AND has been written only once. SSDs usually calibrate changes in the statuses of cells over time via the flash management software algorithm. Due to the error in the software algorithm, the 840 EVO performed read-retry processes aggressively, resulting in a drop in overall read performance. This only occurs if the data was kept in its initial cell without changing, and there are no symptoms of reduced read performance if the data was subsequently migrated from those cells or overwritten. In other words, as the SSD is used more and more over time, the performance decrease disappears naturally. For those who want to solve the issue quickly, this software restores the read performance by rewriting the old data. The time taken to complete the procedure depends on the amount of data stored.

This partially confirms my initial theory in that the slow down was related to cell voltage drift over time. Here's what that looks like:

As you can see above, cell voltages will shift to the left over time. The above example is for MLC. TLC in the EVO will have not 4 but 8 divisions, meaning even smaller voltage shifts might cause the apparent flipping of bits when a read is attempted. An important point here is that all flash does this - the key is to correct for it, and that correction is what was not happening with the EVO. The correction is quite simple really. If the controller sees errors during reading, it follows a procedure that in part adapts to and adjusts for cell drift by adjusting the voltage thresholds for how the bits are interpreted. With the thresholds adapted properly, the SSD can then read at full speed and without the need for error correction. This process was broken in the EVO, and that adaptation was not taking place, forcing the controller to perform error correction on *all* data once those voltages had drifted near their default thresholds. This slowed the read speed tremendously. Below is a worst case example:

We are happy to say that there is a fix, and while it won't be public until some time tomorrownow, we have been green lighted by Samsung to publish our findings.

A quick note to users of Intel SSDs - specifically for owners of the 335 Series. Intel has updated their SSD Toolbox app to v3.1.2. This app is used for various tasks on Intel SSDs, such as secure erasure, performance optimization under Windows, and TRIM through RAID-0 under Intel RST / ICH / PCH motherboard SATA controllers. This update is significant in that it can in turn update the firmware of the Intel 335 Series SSDs to correct a bug in how those drives report wear. This bug was initially discovered by Kristian Vättö, over at Anandtech.

If you have a newer version of the SSD Toolbox, F9 will be listed as "Total NAND Writes", and list that value in MB. The issue with the original 335 firmware was that it incorrectly calculated the wear (the Wearout Indicator - E9 above) such that it would list the drive as worn out after ~1,000 total flash cell erase cycles (i.e. 1,000 x the capacity of the SSD). The firmware update corrects this value to ~3,000 cycles, which is more appropriate for the rating of IMFT 20nm flash. Updating should be non-destructive, but you should backup just in case. The update is not urgent, in that it only corrects how the drive does the math to calculate E9. The Wearout Indicator will change to the correct value after the update, regardless of when it is applied. Additionally, if the MWI reaches 0 prematurely, it should have no impact on operation of the SSD.

This morning, OCZ pushed out a new firmware, dubbed 1.4RC. This is a release candidate of the upcoming performance-boosting firmware, and is meant for "enthusiasts who like to tinker with their hardware".

As a heads up for those who are feeling froggy this Monday morning and choose to update their Vertex 4 - this is a destructive update and will wipe the drive. The updater runs within a Windows session with the Vertex 4 connected as a secondary drive. While you're 'under the hood', I'd also recommend performing a secure erase with the OCZ Toolbox software after you have udpated and power cycled the SSD.

I have been able to partially confirm the performance increases, and will be reporting full results later this evening (for all three capacity points). Stay tuned!

*Note* OCZ NDA'd this update for this morning, but we have not seen where they have posted it for download from their site. We will post a link in the comments below once it has become available.

The Tech Report has been investigating the OCZ 2.15 firmware update for their Sandforce SF2281 based SSDs. The firmware was released to fix several specific issues that users were encountering which caused BSODs or stuttering during normal usage. The testing was a little odd for The Tech Report, they certainly didn't see any BSODs after flashing to the new firmware, however they never saw any BSODs on their drives previously. A little investigation showed them a significant decrease in the number of people complaining about BSODs on forums which leads them to believe the firmware update is effective at what it does. Even better, the firmware has no real negative effect on the drives performance.

"SandForce SSDs have been dogged by reports of BSODs and other issues, but new firmware promises relief. We take a quick look at OCZ's recent 2.15 firmware update to see how it affects SSD performance and the BSOD bug."

Over the past few months, we had noted a seemingly disproportionate surge of negative reports from users of SandForce-2200 based SSD's. These include OCZ's Vertex and Agility 3, Corsair's Force 3 and GT, Patriot's Pyro and Wildfire, along with many others. The complete list is available in our handy SSD Decoder.

The issue at hand was random BSOD's, with the possibility of an eventual complete failure of the SSD, rendering it unrecognizeable to the BIOS or Operating System. More details (and the fix) after the break:

I witnessed this personally, as the SF-2281 pictured above suffered the same fate when we attempted to use it a few weeks ago.

OCZ is pleased to announce that the cause of a BSOD issue experienced by some SF-2000-based drive owners has been identified by OCZ and SandForce. A new firmware update which directly addresses this BSOD occurrence related to SF-2000 based SSDs is available here. All newly manufactured OCZ SF-2000 based SSDs will feature the new 2.15 firmware revision (which is based on SandForce firmware version 3.3.2.) We highly recommend that any customers that have experienced the BSOD issue update their firmware to 2.15.

We sincerely appreciate the support from our customers, and if any customers have any questions or require additional support please do not hesitate to contact a customer service representative and we will be happy to address any questions or concerns.

If you own any of the affected SSD's, I highly recommend updating as soon as possible. Until then, I also recommend you back up any data present on these drives, as the above statements confirm the presence of an issue that can potentially brick your SandForce SSD at any moment.

Remember, patch only applies to the 2200 Series controller (i.e. SandForce SSD's capable of SATA 6Gb/sec).

We've seen some recentmumblings about a corner case where inadvertent or improper power loss to an Intel 320 Series SSD would result in the drive getting stuch in an inaccessible mode where it appears as an 8MB drive. From what I've gathered, the issue seems rare and may be tied to some specific hardware configurations.

The SSD 320 we tested back in March (we couldn't get it to 'stick' in 8MB mode).