[ar:Tom Kopchak]
[al:Def Con 24 Hacking Conference]
[ti:Sentient Storage - Do SSDs Have a Mind of Their Own?]
[au:Tom Kopchak]
[by: DEF CON Communications (https://www.defcon.org)]
[00:00:00.20]
>> So uh welcome, thank you very
much for coming. It's always
awesome to see a room full of
[00:00:04.17]
people, and it's humbling for me
as a speaker to really have such
a great audience showing up and
[00:00:09.67]
listening to me say things. I'm
always shocked. [clapping] >>
Give your all self a round of
[00:00:14.00]
applause. [ Applause ] >>Thank
you very much. So without
further ado, I'm gonna talk a
[00:00:20.50]
little about myself and we'll
get to the actual cool part of
this presentation. I'm Tom
[00:00:24.97]
Kopchak and I am the director of
whatever at Hurricane Labs.
Where I am pretty much
[00:00:30.90]
responsible for taking the
customer's problems and giving
them to the Engineers to solve
[00:00:36.97]
them. So I don't actually do any
work, but I'm a manager. But I'm
still an engineer, and uh
[00:00:43.07]
researcher at heart; and this
solid state drive forensics
research has absolutely nothing
[00:00:48.47]
to do with what I do
professionally. But it's still
interesting and I think this is
[00:00:52.23]
one of those topics that we
really are lacking a lot of
information on and I had some
[00:00:57.30]
fundamental questions when I was
looking at the research that was
done on this topic that I really
[00:01:01.70]
wanted to find out the answers
to. And over the course of the
next forty minutes or so, I'm
[00:01:07.20]
going to go through those
questions. We're going to talk
about how Solid State Drives
[00:01:11.30]
operate and the technologies
behind them. And we're going to
talk about what you need to
[00:01:16.07]
consider if you're trying to do
a forensics investigation on a
solid state drive and want to
[00:01:21.00]
see if the data is still around.
To determine if it's actually
worthwhile to get that data
[00:01:25.70]
back, or if there's gonna be
something that's gonna make it
virtually impossible. Also
[00:01:32.23]
during the course of this
presentation, you might notice
that I make some jokes that are
[00:01:35.87]
quite bad. That's what the
hashtag Tom Joke is for.
Internally in our uh chat system
[00:01:41.83]
at Hurricane, we have something
that you type in bang Tom Joke
and it returns 4 0 4 humor not
[00:01:47.87]
found. So I'm sure you guys all
have access to things that can
post things on to the internet.
[00:01:52.30]
Either in your pocket, or in
your lap, or something like
that. So if I say something like
[00:01:56.17]
that, go ahead and use the
hashtag Tom Joke, and uh we'll
make it a thing. It's the Dad
[00:02:01.30]
jokes of IT. So without further
ado. Why we are here. Uh,
current forensics technologies
[00:02:08.07]
and practices for working with
hard drives, they're
well-defined. When you get a
[00:02:13.30]
hard drive, we've got,
conveniently enough, one right
here. Normal laptop hard drive.
[00:02:19.47]
If you're a forensic
investigator and you've got one
of these, you would know what to
[00:02:22.80]
do with it. It's well-defined.
Now solid state drives, they do
behave differently in a lot of
[00:02:29.03]
cases and they present a whole
lot of new challenges. And
really this presentation is
[00:02:33.57]
going to look to explore those
differences in detail. What
challenges you need to be aware
[00:02:37.57]
of, and considerations when
working with an evidence
bˇrance... bearing solid state
[00:02:41.70]
drive. But first, I want to make
sure that we get everyone up to
speed on hard drive forensics.
[00:02:48.00]
So I am going to do a quick
overview of Traditional Hard
Drive Forensics and what that
[00:02:52.47]
means to you. Just to make sure
we're all on the same page. It
should be review for a lot of
[00:02:56.17]
you in the audience and
definitely something that we all
pretty much have standardized
[00:03:01.70]
and understood. But really what
we know is if you have a hard
drive, one of these, you delete
[00:03:08.57]
a file off of it, it is often,
if not never truly deleted. It's
very easy to recover that in a
[00:03:13.57]
[00:03:16.37]
lot of cases, and that data is
not truly gone. And this has
really become a cornerstone of a
[00:03:22.10]
lot of the basic forensics
practices. Someone has something
they shouldn't on a hard drive,
[00:03:27.10]
they delete it, the Feds
undelete it, through not all
that uncomplicated processes.
[00:03:32.53]
Fellas come back, throw you in
jail for forty-seven thousands
years, and then everything's
[00:03:36.63]
good. Well, it's not good but
you know how that works. Now if
you were to be like I wanna
[00:03:42.37]
delete this data and you quick
format that hard drive, the
data's still there. Uh, it's not
[00:03:46.80]
truly deleted, it's not
overwritten. And In order for
data to be deleted from a hard
[00:03:52.23]
drive, it has to be completely
overwritten at least once. Uh, I
have not been able to locate any
[00:03:58.80]
published documentation that
says on a modern hard drive, you
can recover data that has been
[00:04:03.77]
truly overwritten once. That
isn't saying that it is not
impossible. That's not saying
[00:04:09.30]
that there hasn't been some
branch of the government that
has one this sort of thing. It
[00:04:13.03]
is just not something that
people are talking about as
being practical. Also a
[00:04:18.23]
traditional hard drive,
fundamentally is a rather stupid
piece of equipment. In the fact
[00:04:23.63]
that it is not doing anything
intelligent under the hood, to
move data around, manipulate the
[00:04:28.33]
data, optimize it, it gets the
data, it writes it on to the
platter and that data is there
[00:04:33.67]
on the drive. The drive is not
gonna go out on its own and say
Hey, I wanna move this block
[00:04:37.80]
from this block because I feel
like it. Your normal hard drive
is not going to do that. And
[00:04:42.33]
that's something that forensic
investigators recognize. And
they aren't gonna move blocks
[00:04:47.33]
[00:04:49.80]
independently of the operating
system. So any sort of
optimization of the position of
[00:04:54.20]
data on the platter; because
this is a big spinning desk of
metal, with magnetic bits
[00:04:59.13]
scattered all around, sometimes
its slow and inefficient to try
and find that data. So on the
[00:05:04.00]
hard drive the operating system
a lot of times will try to
realign the data... the blocks
[00:05:08.23]
to defragment the drive. That's
something that the drive is not
going to go out and do because
[00:05:12.70]
it feels like it. Solid State
Drive on the other hand, does
things differently sometimes.
[00:05:18.43]
The other thing that we know is
that you know this hard drive,
just a Seagate One, if I had a
[00:05:23.03]
Western Digital or an Hitachi or
another brand, it would behave
the same way. So if we're a
[00:05:28.63]
Forensics Investigator, we get a
laptop with a hard drive in it
or a desktop with a hard drive
[00:05:32.60]
in it. We can pretty much use
the same universal truths with
dealing with this data. Surprise
[00:05:39.20]
surprise though. Solid state
drives change all of these
fundamental theories and beliefs
[00:05:44.60]
that we have. So in order to
talk uh and understand about
Solid State Drives, I wanna talk
[00:05:49.60]
[00:05:51.60]
about how these drives are
constructed and talk about Flash
Memory. So Flash Memory is
[00:05:56.60]
[00:05:59.20]
really the replacement and the
equivalent of the magnetic
platters in a hard drive. So I
[00:06:05.60]
have an example SSD that doesn't
have a cover here, and over the
sticker of course, there are a
[00:06:10.83]
bunch of chips that are flash
chips. That's where the data is
actually stored on a Solid State
[00:06:16.27]
Drive. And basically these chap,
chips have a number of
capacities. They have different
[00:06:22.43]
amounts of data that can be
stored on them and they're, the
drive is composed of different
[00:06:26.60]
amounts of these chips in order
to reach the capacity. So if you
have a 32 gig chip and you need
[00:06:31.67]
to have a 256 gig SSD, you'll
have eight of those chips that
are all together. And the way
[00:06:36.67]
[00:06:39.20]
that those chips work together
is an SSD is by using the
controller. And the controller
[00:06:44.63]
is basically the heart and the
brain of the SSD that makes the
flash memory actually receive
[00:06:50.10]
data from the operating system
and it makes the SSD look like a
normal drive to the operating
[00:06:55.07]
system because the operating
system doesn't really care what
it's writing to as long as it
[00:06:59.40]
oohs like a hard drive. Solid
State drive or otherwise. So
that gap between the controller
[00:07:04.90]
here on the back of the drive
and the flash memory, that's
often referred to as the Flash
[00:07:09.03]
Translation Layer or FTL.
Obviously physically, these
drives look very very different.
[00:07:15.70]
Uh your hard drive has the
splinning, spinning platters,
it's more like a record player
[00:07:19.87]
with uh metal and magnets
instead of needle and scratchy
noises. Uh, on the solid state
[00:07:26.80]
drive, you have the control on
flash chips no moving parts
completely solid state as the
[00:07:32.00]
name implies. In terms of Flash
Memory itself, the architecture
is divided into different types
[00:07:37.00]
[00:07:40.80]
of flash chips and there are
different types of technologies
for this. There's technologies
[00:07:46.20]
called F uh, Single Level Cell,
SLC and MLC, Multi Level Cell.
And they're actually even
[00:07:51.20]
[00:07:53.30]
expanding to make this more in
depth where you have a third
level. Um most of the ones that
[00:07:58.67]
you see on the market right now,
more expense... least expensive
ones of the MLC but as we go to
[00:08:03.47]
the triple level, the costs are
going down because the more you
can store on a flash chip. Now
[00:08:08.40]
on SLC, it's just a one or a
zero, in the MLC you have
varying levels of charge. So if
[00:08:14.40]
it's a, zero, it's a, zero zero
it's a certain level charge. If
it's a zero one, it's a
[00:08:20.07]
different level charge, if it's
a one zero, and a one one,
different levels of charge.
[00:08:24.27]
Obviously this is a little bit
slower because you have to have
a more precise level of charge
[00:08:28.67]
as opposed to turning it on and
off. So there's performing
tradeoffs with this but the cost
[00:08:33.00]
does go down because you're
basically doubling or if you're
tripling the amount of data that
[00:08:36.20]
you can store on the same amount
of flash. And one thing that
always gets me is that a blank
[00:08:41.07]
cell is represented is all by
ones for whatever reason I think
it would be zeros but in the
[00:08:45.50]
case of flash memory and the
engineers among here can tell me
why but it's all ones. Um. Just
[00:08:52.20]
incase any of you have SSD
trivia that comes up any day.
Uh, in terms of how a flash
[00:08:57.20]
[00:08:59.53]
structure is actually
architected, uh the page is the
smallest unit of addressable
[00:09:04.80]
flash in a flash memory cell.
And pages, they cannot be
overwritten on individual block
[00:09:10.80]
by block or byte by byte basis.
The entire page has to be
refreshed. Um, so you can write
[00:09:15.80]
[00:09:18.07]
data into a page that has free
space but if you need to return
that page to the pool of
[00:09:22.83]
available pages that has to be
done separately because theres
risks that by doing that, it's
[00:09:28.23]
electrically complicated and
time intensive in terms of
computing time. That you're
[00:09:32.77]
basically running into situation
where you might modify or lose
data and that's bad. Uh so only
[00:09:37.70]
entire blocks are erased at a
time. And then when the data is
erased or deleted, uh the blocks
[00:09:44.57]
containing these data is
actually just marked as invalid.
So that the operating is just
[00:09:49.83]
like hey, we don't need this
anymore. Or the drive is saying
these drives.. blocks are no
[00:09:54.60]
longer needed, I'm gonna mark
those as invalid. And they can
not be reused until they are
[00:10:00.03]
completely erased because of the
risk manipulating data and it
does take a lot of effort and
[00:10:05.93]
time computationally speaking.
Uh so this is all fast in terms
of human time, but the slowest
[00:10:11.70]
thing the an SD can do is erase
data at this point and that's
why this is done. So the speed
[00:10:17.53]
of an SSD is really determined
on having available blank memory
that can be written to. Uh and
[00:10:23.53]
in order to do that, the
controller has to ensure that
there's as much of a pool of
[00:10:28.30]
free flash space at any given
point in time. The other thing
that we need to be aware of is
[00:10:33.30]
[00:10:36.10]
Flash Wear. So unlike your
traditional magnetic drives,
where you're dealing with just
[00:10:41.10]
[00:10:44.13]
magnet on metal, flash cells
actually have a life cycle. And
there's a limited number of
[00:10:49.43]
times, that can be, they can be
written or overwritten. And the
wear can be uneven across the
[00:10:55.53]
drive depending on how that data
is used. So you think about how
an operating system works. There
[00:11:01.30]
are things that are almost never
gonna change. Like your static
operation system files for
[00:11:06.50]
example are pretty much gonna be
written when the OS is
installed. Maybe updated as part
[00:11:12.00]
of a patch or something like
that but very very rarely are
you gonna see those files
[00:11:15.33]
change. Whereas your log files
for example might be something
that are continuously being
[00:11:20.77]
written to, continuously
changing and continuously being
modified. And that's gonna be a
[00:11:24.70]
high level of wear on the disk.
And if the drive didn't do
anything to deal with that you
[00:11:30.13]
might run into a case where that
part of the flash that has your
log directory is gonna be
[00:11:34.60]
constantly and constantly
written and wear out sooner.
Whereas your OS section is gonna
[00:11:39.43]
just be chillin' there like
what's up I don't know what's
going on here. So the way that
[00:11:43.63]
the solid state drive controller
handles that a lot of times is
it's going to move data between
[00:11:48.40]
those blocks in order to make
sure that wears even. And that's
the concept of wear leveling. So
[00:11:54.80]
when we go back to the solid
state drive controller, that is
really the heart and the soul
[00:11:59.37]
and the brain and whatever other
human analogy you want to come
up with of an SSD. And from the
[00:12:05.50]
perspective of the operating
system, that's what makes the
SSD look just like a drive and
[00:12:11.07]
it doesn't have to worry about
how to speak flash memory or
anything like that. That's all
[00:12:15.00]
hidden to the user and in turn
to the forensics investigator.
Uh and all of the managing,
[00:12:21.33]
reading, writing, flash cells is
something that the OS doesn't
worry about. And also, all of
[00:12:26.50]
the more advanced wear leveling
features that are part of the
SSD are going to be handled by
[00:12:32.30]
that controller as well. So
controllers are really an
interesting aspect of the whole
[00:12:37.70]
solid state drive composition,
because there are so many
different types of manufacturers
[00:12:42.10]
that create controllers and
different controllers and even
individual firmware revisions
[00:12:46.63]
inside of an SSD can really
depend on how the drive behaves.
Uh there are cases where a solid
[00:12:53.00]
state drive behave very
differently or had a flaw that
was fixed because the vendor
[00:12:58.07]
produced a new firmware patch
and that modified how the drive
behaved without having to make
[00:13:02.60]
any changes. So it's almost like
the Tesla of storage where they
just push out a patch and all
[00:13:07.43]
the sudden your car just drives
backwards. [Chuckles] So... some
controllers they include
[00:13:12.27]
additional optimizations,
including um deduplication where
if you're writing the same thing
[00:13:19.00]
to the flash memory it's only
gonna actually write it to flash
once and reference it through
[00:13:24.00]
the controller as these two
files hit the same spot of
flash. Now this isn't like
[00:13:28.77]
traditional deduplication inside
an operating system where it
gives you more space. In terms
[00:13:34.00]
of what you can actually see as
the user, it's the same size
solid state drive, but in terms
[00:13:38.40]
of the flash that's actually
being consumed from the
perspective of the controller,
[00:13:42.10]
it's a smaller amount of data.
So this reduces flash wear more
than anything. Um, that's
[00:13:46.60]
something that's pretty much
transparent to the user in a lot
of your SSDs. And then drives
[00:13:52.37]
also do garbage collection
proactively uh to recycle flash
box and this is really more of a
[00:13:57.57]
performance thing than anything.
And really that comes down to
like the drive needs to know
[00:14:02.90]
when the data is invalid and
when it can recycle the flash
box. And so how does it do that?
[00:14:08.10]
Well think about computing time
again compared to human time.
Now in terms on when a drive is
[00:14:14.20]
actually doing something,
reading or writing, on a
computational scale, it's very
[00:14:18.60]
very infrequent. So it might
seem to us humans that can
computers always doing
[00:14:22.43]
something. But really it's like
okay the person hits a key, now
I'm gonna wait for a really
[00:14:26.90]
really long time, they hit the
next key. Okay, I'm bored. So
when the solid state drive is
[00:14:31.70]
bored it does other things. So
the idle time is really good for
performing garbage collection
[00:14:37.80]
type techniques. Optimize the
drive behavior, relocating flash
cells to other sides of the
[00:14:43.43]
drive, to wevel layer, oh
wevel... level wear uh or to do
garbage collection techniques to
[00:14:48.43]
[00:14:50.90]
free up flash so that you have
good performance. So as you can
see, these solid state drives
[00:14:56.53]
are sentient, they are going to
be taking over the world one
computer at a time. So uh, your
[00:15:02.47]
laptop you better watch out.
This SSD is plotting against
you. [Chuckle] So um where we
[00:15:07.47]
[00:15:09.90]
actually come down to how the
drive actually knows what to do
a lot of times though is the
[00:15:14.27]
trim command. Uh so this is a
ATA command that's implemented
through the sata bus and it
[00:15:19.60]
basically is the way for the
operating system to tell the
drive controller when data is
[00:15:23.67]
deleted. So basically instead of
the SSD trying to figure out
when the data can be deleted and
[00:15:29.00]
when that data's invalid. The OS
proactively notifies the
controller and the controller is
[00:15:33.40]
like okay, I can just wipe
these. Uh and that really
frequently accelerates the
[00:15:37.60]
garbage collection process and
that's definitely something that
I am going to demonstrate as
[00:15:40.83]
part of this forensics
demonstration. But really the
thing to keep in mind here is in
[00:15:45.07]
general your SSDs are gonna
behave more and more like your
Enterprise SAN controller or
[00:15:50.87]
RAID array than simply a simple
dumb hard drive. So this is one
of those things where an SSD is
[00:15:56.87]
fundamentally different from
your normal storage and that
definitely has forensics
[00:16:01.30]
implications and really the
questions that I have when I
started looking into SSD storage
[00:16:07.53]
years ago, was how do we
determine if solid state drives
impact the forensics process?
[00:16:13.20]
And really what are those actual
impacts? Uh I had a lot of
questions along the lines of if
[00:16:19.17]
we delete a file from an SSD uh
people are saying that it might
be different but what are those
[00:16:24.33]
cases that cause that uh and
when can I say yes that data's
gonna be there or no it's not.
[00:16:29.60]
Uh how do we standardize that,
how do we come up with the
framework for understanding
[00:16:32.73]
that? And really the question
that is facing a lot of forensic
investigators, including a
[00:16:37.87]
number of them that I've
interviewed is they're treating
SSDs like normal hard drives
[00:16:42.20]
when they're doing
investigations and really if
you're looking into to this data
[00:16:46.27]
as someone who's an investigator
and not understanding the
implications uh you have a high
[00:16:51.37]
chance of not getting what you
expect from one of these drives
without knowing what you're
[00:16:55.20]
looking for. And you could be
wasting a lot of time while
trying to do this. So there has
[00:17:01.77]
been some previous research that
has been done on Solid State
Drives and the forensic
[00:17:05.87]
implications. A lot of this
research has looked at the fact
of filling an entire SSD with
[00:17:11.00]
the same string of text and
trying to see when that part of
the flash is manipulated. And
[00:17:16.10]
fundamentally there have been a
lot of studies that have shown
yes, there are some changes that
[00:17:20.43]
have happened. Data is
destroyed. Uh but none of the
research that I could find was
[00:17:24.80]
really looking at this across a
wide pool of solid state drives
in a very pain staking, here's a
[00:17:30.57]
file, I delete it, what happens?
Which in my mind, that was the
simplest brain dead question
[00:17:36.53]
that I had for when I was trying
to look at this research. So if
i deleted a file, is it gone, or
[00:17:41.17]
no? Which really,I think is a
good question to really start
out with. So, my research
[00:17:46.17]
[00:17:49.60]
represents really from what I
could find one of the most
comprehensive studies of the
[00:17:54.00]
recoverability of deleted files
from solid state drives to date.
So I had a pool of SSDs that uh
[00:18:00.33]
we're gonna go through and
eleven different two-part tests
that really built on each other
[00:18:05.30]
to trigger subtle difference in
SSDs to really understand when
data is deleted, when it can be
[00:18:11.60]
recovered, and why. And really
it's primarily focusing on
deleted file recoverability
[00:18:17.93]
because that's what at the basic
a lot of the law enforcement
forensics at uh the lowest
[00:18:23.40]
possible level really ties into.
So without understanding the
basics, we can't really go
[00:18:27.93]
further. So the purpose of my
research was really to
comprehensibly study the impact
[00:18:34.43]
of solid state drives on a
forensics data acquisition
investigation processes. And
[00:18:39.23]
really look at it from and if
I'm doing current forensics
practices against these drives,
[00:18:44.53]
are they valid? And can they be
used all of the time? Can they
be used none of the time, or
[00:18:50.07]
some of the time? What cases are
they okay? And really to
determine if the traditional
[00:18:54.67]
forensics approaches are
sufficient. Um, all of my
experiments for this research
[00:19:01.47]
were designed to built on...
build on each other. So every
possible change that I could do
[00:19:07.10]
for this were intended to be the
smallest possible difference.
Really trying to minimize all my
[00:19:12.03]
variables as much as possible.
And really incrementally try to
trigger certain difference that
[00:19:17.93]
I have seen or read about as
options for causing an SSD to
behave in a certain way. And
[00:19:23.67]
really every test uh was two
parts. One was a deleted file
test where I would have some
[00:19:30.33]
type of file on an SSD that was
deleted and as we know on your
normal hard drive, deleting a
[00:19:36.63]
file doesn't make it go away. So
really a simple way for me to
tell if a SSD is doing something
[00:19:42.73]
is to put a file on a whole
bunch of SSDs, make sure it's
written on that file or on that
[00:19:47.50]
drive for sure, delete it. Image
the drive, try to recover it,
see what happens. And it's
[00:19:52.23]
really simple but it is a very
blatant way of demonstrating the
difference. The next part for
[00:19:57.23]
[00:19:59.77]
every test was a quick format
test. Where I would quick format
all the drives and try the same
[00:20:04.97]
recovery including file
recovering techniques because
that's typically when the file
[00:20:09.23]
systems gone. And uh
fundamentally that really
demonstrated even further
[00:20:13.77]
differences. Uh so, the other
thing that I learned as part of
this research is trying to image
[00:20:19.27]
a whole bunch of SSDs for a
bunch of tests is really really
boring and annoying. So when
[00:20:24.03]
you're dealing with hundreds of
gigabytes of drive imaging over
a usb two point oh write
[00:20:28.60]
blocker, it's really slow. So
um, so all these tests, they
were designed to isolate
[00:20:33.60]
[00:20:36.43]
different variables. Some of the
variable that I wanted to test
were trim state. So this could
[00:20:41.87]
be done in a number of different
ways. Uh since trim is something
that's an ATA command that is
[00:20:48.43]
only passed over the sata bus uh
it's something that you can
basically control the state of
[00:20:54.30]
by having a sata bus or not. So
if you connect to drive via USB
it's not gonna pass the trim
[00:20:59.33]
command because it's basically
going block that from happening.
Operating systems you can turn
[00:21:04.00]
trim on and off. Uh in Windows 7
for example, there's a command
line perimeter for doing that.
[00:21:09.90]
There's also operating systems
that natively don't support
trim. Uh Windows XP is great,
[00:21:14.33]
but it natively doesn't support
a whole lot of other things like
you know, security. So...
[00:21:19.17]
[Laughter] Obviously there are
hopefully not too many people
using that, but it's something
[00:21:23.23]
that you might see because there
are plenty of agencies that are
still using XP for different
[00:21:27.87]
sorts of things unfortunately
and makes all the security
people cry. Um, there are other
[00:21:33.67]
types of things that I was
looking to isolate for this. Uh
the number of files present. So
[00:21:38.53]
is a single text file for
example gonna behave differently
than an image file because it's
[00:21:42.40]
a larger type of file. So would
we see the text file still be
there and the image file gone or
[00:21:47.33]
vice versa? Would having more
than one image file on the drive
make a difference? Or if I
[00:21:52.97]
deleted a file every minute over
the course of an hour, would
some of the files be there and
[00:21:57.20]
some of them be gone or would
they all be gone or would they
all be there? Uh, all those
[00:22:00.87]
different types of variables for
what I was looking to try to
isolate. So let's get to the
[00:22:07.17]
drives that I was using for this
testing. I ended up using seven
different drives. Six SSDs and
[00:22:12.93]
one controlled drive. And all of
these drives are really picked
for specific reasons to try to
[00:22:17.77]
understanding different
controller types or different
manufacturers to understand
[00:22:22.60]
they'd behave. Uh so my control
drive was jus the Seagate normal
hard drive that you find in any
[00:22:29.07]
laptop. Since laptop drives, or
normal hard drives are gonna
behave pretty much the same way
[00:22:33.87]
either way, this any control is
pretty much as good as ay other
for that. Then in terms of the
[00:22:39.30]
other SSDs. Have these handful
ones. So I had a Crucial SSD, uh
this one has firmware thats
[00:22:44.30]
[00:22:47.33]
written by Crucial on a MArvel
chip set. Um so they under, they
handle all the software in house
[00:22:52.33]
[00:22:55.67]
for this drive and it's not
based on a third party
controller. Likewise the same
[00:23:02.30]
for Intel who also has their own
controller. Uh and both the
Crucial and Intel controllers
[00:23:07.30]
[00:23:09.77]
are different. So I was looking
to see if those would result in
any different behavior. Uh then
[00:23:15.37]
we had the OZC and the Patriot
which actually both have the
same controller and they're just
[00:23:21.30]
different manufacturers. So I
wanted to see if having the same
controller and the same firmware
[00:23:25.63]
revision just with different
manufacturers would result in
different behavior. And a lot of
[00:23:30.57]
times, it turned out to be very
much the same except the OCZ one
died partially through the
[00:23:35.20]
testing. SO after it stopped
working, it didn't behave in the
same way. [Laughter] Uh then
[00:23:40.20]
[00:23:42.97]
there was the Samsung. Which
Samsung has their own controller
which is actually a three core
[00:23:48.13]
uh controller and I could find
zero documentation of any of
those three cores would actually
[00:23:54.30]
do. But it is their own firmware
and their own implementation. So
it is different than Crucial and
[00:23:59.30]
[00:24:01.73]
Intel and all that. And it
doesn't use a generic
controller, the SandForce one
[00:24:05.63]
that's in the OCZ and the
Patriot. And then there was also
this super talent one which was
[00:24:11.73]
interesting because it was one
of the first SSDs that I was
able to locate. Uh and inside of
[00:24:16.83]
this, this stripped case that
shows nothing, is actually a one
point eight inch SSD with a
[00:24:23.73]
parallel to sata bridge chip. So
this is like ancient SSD
technology from 2009. So...
[00:24:28.73]
[00:24:31.70]
[Laughter] In terms of ancient,
it's not really ancient by human
standards but by technology
[00:24:37.43]
standards it should be just
probably really old. Um, but
this was interesting for me
[00:24:43.20]
because of having a sata to
parallel to bridge chip. That
shouldn't allow any sata
[00:24:48.73]
commands like trim or anything
to even get the SSD controller.
So I wanted to see how you know
[00:24:54.43]
a modern SSD versus a really old
SSD behaved and compared that to
a normal hard drive. So all of
[00:25:01.07]
these were selected for
different factors and I really
wanted to try to look at this
[00:25:05.10]
from a picture of the SSD
landscape. Obviously I couldn't
go out and buy everything SSD
[00:25:10.67]
that existed because uh I don't
have millions of dollars laying
around. But this should be a
[00:25:14.93]
good picture of what we should
expect to see and these tests
can be used across new SSDs and
[00:25:20.80]
new manufacturers to understand
so if you run into something
you've never seen before, I've
[00:25:24.70]
never seen before, you can do
similar tests and really get a
basic understanding of what to
[00:25:28.63]
expect. Now my Forensics Lab for
this consisted of a dedication
evidence creation and evidence
[00:25:35.53]
collection machine. Uh I never
ran an operating system or
anything like that on the test
[00:25:42.33]
SSDs because I wanted to
eliminate the variables as much
as possible. So the only data
[00:25:47.23]
that was written or read for
these SSDs were the files I was
working with. I.. it would
[00:25:52.00]
definitely be interesting to
look at this from an operating
system perspective but tracking
[00:25:55.63]
that in an efficient and sane
way just was not something that
I really felt for this type of
[00:26:00.50]
research. Someone wants to run
with that, by all means go ahead
and look at that. It's
[00:26:04.43]
definitely something that needs
to be explored. Uh and I also
look to use Open Source. So the
[00:26:09.30]
Caine Linux Forensics
distribution uh was my choice
distribution for collecting and
[00:26:14.63]
analyzing all the data. And all
the evidence was collected using
a forensics writeblocker. Now
[00:26:20.23]
there has been research hat has
shown that an SSD on a
writeblocker has been
[00:26:23.80]
demonstrated to modify data in
some cases. Uh I wanted to look
at this from the perspective
[00:26:28.83]
from someone who was analyzing
the drive without doing any chip
by chip analysis or bypassing
[00:26:34.67]
the controller. So I was looking
at this as a traditional
forensics investigator using the
[00:26:39.10]
tools that would be accepted in
court to in order to analyze
data. So for one of our sample
[00:26:44.10]
[00:26:47.40]
tests and we'll just start with
what really was one of the
fundamental and most basic
[00:26:51.70]
tests. Was the idea of writing a
single image to a file. And by
image I just mean picture. Uh so
[00:26:56.70]
[00:26:59.63]
I took a picture, I wrote it to
all of these disks. And I
mounted the disks, and that's
[00:27:03.83]
important because caching is
something that could happen on
an SSD where it would just be in
[00:27:08.97]
ready only memory. And what I
wanted to make sure the drive
had that data truly committed to
[00:27:13.10]
flash. So that I would look at
it in another machine and it
would still be there. Uh so that
[00:27:17.10]
I knew it was in the cache, in
the memory and written to flash.
Uh and then remount the drive, I
[00:27:23.40]
would delete it and then make
sure that I would image the
drive and see if it was
[00:27:28.07]
recoverable. Now that seems
simple, and if you're knowing
how forensics are working. Which
[00:27:34.20]
is everyone here. You delete a
file off the drive, you don't do
anything special. It should be
[00:27:38.97]
there, right? Yeah. I'm getting
some nods, that's good. Now
across the tests that I did on
[00:27:45.50]
SSDs, it was not always
recoverable. Now that begs the
question, why. So different
[00:27:50.50]
[00:27:53.77]
variables, and I'm gonna get
into this in the results, but
the drives did fundamentally
[00:27:58.87]
behave very differently. So some
of these drives would go ahead
and almost purge the data
[00:28:02.87]
instantly once it was deleted.
And a lot of times, that was the
component of the trim state. A
[00:28:07.87]
lot of times, there were
different factors but that was
probably the biggest one. Um
[00:28:13.33]
that would result in the data
being basically completely
recover... un or uh completely
[00:28:17.33]
unrecoverable in that period of
time. Um, the second test for
every test run was a quick
[00:28:22.33]
[00:28:24.83]
format. Uh so I would take the
same drive, same data I imaged.
I'd go ahead and quick format
[00:28:29.83]
[00:28:31.97]
the drive, take an image of the
drive after I unmounted it and
try to do file recovery. And
[00:28:37.33]
this was a lot of time done to
be done with file carving
because the op or the file
[00:28:40.67]
system was gone at that point.
Uh most of the time, on uh I
would say all of the time on
[00:28:46.17]
your normal hard drive, that
data is more or less
recoverable. Uh on your SSDs a
[00:28:51.83]
lot of times it was completely
gone obviously any SSD where it
was gone in the test beforehand,
[00:28:57.50]
the first test, it didn't come
back. Uh, which is probably a
good thing. That would be really
[00:29:02.03]
weird. But for a lot of the
drives where it was originally
still there, the quick format
[00:29:08.40]
would make that go away as well.
So our understanding of data
sanitization on an SSD level
[00:29:13.40]
[00:29:15.60]
fundamentally does change as a
result of just you working with
an SSD. Uh but it's not always
[00:29:22.03]
the case. So as I was doing
these tests, I started to
noticed some patterns. So all
[00:29:27.40]
the files in the controlled
drive were pretty much
recoverable all the time. Uh
[00:29:30.87]
there was a very significant
difference in SSD behavior and I
started to see them break into
[00:29:37.57]
two different groups. Where
there were some that were
resulting in a very low
[00:29:40.87]
recoverability in a lot of
cases. But there were also a
number of SSDs behaved very very
[00:29:46.80]
similar to the control hard
drive and the ones that have
very low recoverability were
[00:29:52.33]
pretty consistent across the
board in those tests and the
ones that had reasonably high
[00:29:56.47]
recoverability in a lot of tests
were pretty common as well. Uh
so without further ado, I wanna
[00:30:01.47]
[00:30:04.27]
dig into the results of the
research and scare you with you
some bar charts which uh aren't
[00:30:09.93]
gonna be too scary because I
don't wanna freak people out
with numbers and stuff like
[00:30:14.53]
that. I tire to break this down
in a bunch of different ways to
really illustrate the points. Um
[00:30:19.40]
if there are questions about
this, we can definitely talk
later after this presentation as
[00:30:22.90]
well. Uh but really this is just
an overall scale of everything
that I saw across the test.
[00:30:28.97]
You'll see all the way in the
left is the Seagate hard drive
recoverability. And I apparently
[00:30:35.67]
don't know how to use this but
right here, we see the Seagate
drive basically matches the
[00:30:40.67]
[00:30:43.77]
Patriot SSD's behavior and the
Super Talent SSD's exactly
across all of the tests. The
[00:30:50.17]
thing that you'll notice here,
is, let me make that go away.
The OCZ SSD it's very close here
[00:30:55.17]
[00:30:59.23]
to the Patriot one. Um this was
the point where the Patriot
failed. So it was behaving
[00:31:04.90]
exactly identical and based on
everything I had seen. The same
controller, I would expect, I
[00:31:10.93]
guess I could make that yellow,
but I would expect that bar to
be exactly the same had the
[00:31:15.30]
drive continued working
throughout the test. Just based
on extrapolating the result that
[00:31:20.07]
I was seeing. But we also look
at these, there are some drives
here at the bottom. The Crucial,
[00:31:25.07]
[00:31:28.23]
your Intel and your Samsung that
are a lot lower recoverable than
your normal SSD or your normal
[00:31:34.80]
hard drive. And that was
definitely interesting. But
really this graph by itself is
[00:31:40.50]
kind of misleading just like
every other graph. So I need to
go into some more detail to
[00:31:45.07]
really paint the picture of what
was happening and really make
this clear. So like I said, the
[00:31:50.53]
trim state was really the
biggest factor impacting
recoverability. So here on the
[00:31:55.63]
left is my result of recovering
data when trim was on. And then
on the right is when trim was
[00:32:00.63]
[00:32:03.20]
off. So you look at this from
just obviously you have trim,
it's going to make your chances
[00:32:09.17]
of recovering data a lot lower.
Now these ones right here at the
bottom, your Crucial and your
[00:32:14.40]
Intel were a single text file on
the drive saying, This is a text
file, because I'm really
[00:32:19.57]
creative. Uh that single text
file was not something that when
it was deleted was enough to
[00:32:25.63]
trigger garbage collection or
trim event where it actually
overwrote that part of flash.
[00:32:29.73]
Every other case for the Crucial
and the Intel it was an image
file that was large enough that
[00:32:35.73]
was something the drive
controller saw significant
enough to wipe that data
[00:32:39.37]
proactively. So in a, pretty
much any case with a large
enough file on those type of
[00:32:44.73]
SSDs with trim enabled, that
data's gone upon deletion.
Immediately. Um, the Samsung was
[00:32:49.73]
[00:32:52.13]
pretty darn close and then if we
just clear this out we look at
the other ones are all pretty
[00:32:57.13]
[00:33:00.23]
much near the top there. If we
look at when Trim was either
turned off, unavailable because
[00:33:05.23]
[00:33:07.57]
operating system or the bus that
I was using for testing the
drive. Uh, it was pretty much
[00:33:13.27]
the case where a lot more of a
level of playing field where
yes, your Intel and your Crucial
[00:33:19.57]
here were pretty similar to what
they were before at the bottom.
And even your OCZ I guess that
[00:33:26.33]
is kind of an anomaly, because
of how it was behaving and sh...
it ended up would match just
[00:33:32.83]
like the Patriot. But a lot of
these drives with Trim off
behave a lot closer to a normal
[00:33:38.43]
hard drive. So not quite the
same. There's still gonna be
cases where data goes away but
[00:33:43.47]
it's definitely a case where you
see lower recoverability
significantly lower
[00:33:49.10]
recoverability with trim turned
on an opposed to off. This does
combine quick format and non
[00:33:55.67]
quick format. So uh... Wanted to
look at file deletion first and
then we'll look at formatting.
[00:34:00.67]
[00:34:04.27]
So this takes away all of the
formatting tests and just looks
at I had a file and I deleted it
[00:34:09.90]
regardless of trim being on or
off or anything like that. So
you will see across the board
[00:34:14.90]
[00:34:16.97]
your Crucial, Intel, and lesser
degree your Samsung are lower
recoverability. Your Seagate and
[00:34:22.80]
your Patriot and your Super
Talent basically act like normal
hard drives. So we basically
[00:34:27.03]
have two different classes of
drives where something that
behaves very close to normal
[00:34:31.03]
hard drive and something that
behaves very different. But
really looks more blatant and
[00:34:36.47]
more obvious is the quick format
recoverability. And we know on a
normal hard drive you can pretty
[00:34:41.17]
much recover data if its quick
formatted. Uh on an SSD, uh if
you have a Patriot, you're good.
[00:34:46.33]
Uh and your Super Talent, your
good. But if you have one of the
other drives, if you quick
[00:34:51.83]
format the drive, it's pretty
much gone. And that's a really a
fundamental change for how we
[00:34:57.13]
view forensics and so... if
you're dealing with a drive
that's been quick formatted and
[00:35:02.50]
it's one of these ones that
offers very low recoverability,
your chance of getting evidence
[00:35:06.13]
from that is gonna be pretty
low. So really, general
observations for here is that
[00:35:11.13]
[00:35:13.63]
the solid state drives, they
cant be considered to behave the
same or identically to
[00:35:18.47]
traditional hard drives from a
forensics perspective. Deleted
file recoverability does vary
[00:35:24.30]
significantly and it really
depends on the type of drive and
the controller. And as we saw
[00:35:30.73]
different factors, you know, if
trim is on or off, how the drive
is connected, what operating
[00:35:34.87]
system. Uh, what types of files
are being deleted? Is there a
quick format in play? Those
[00:35:39.80]
really impact the likelihood of
recovering data. So that kind of
leads to my contributions to my
[00:35:45.77]
research. I really think that
this would be something that can
be referenced. Uh when
[00:35:50.83]
attempting to recover deleted
files from an SSD and really to
help understand uh what
[00:35:55.67]
situations lead to successful
recovery. Recoverability. So if
you're someone who needs to face
[00:36:01.90]
a solid state drive in a
forensics investigation this
framework for some of the
[00:36:05.63]
testing can really be used to
understand what you need to do
to see before you spend a lot of
[00:36:10.80]
time trying to recover data off
of that drive, if it's something
that's worth while or not. And
[00:36:15.13]
really the impact of trim, has
been something that's bee
published on a lot lesser scale.
[00:36:20.13]
[00:36:22.30]
Uh this is really the biggest
demonstration that I could find
of trim actually having an
[00:36:26.40]
impact on forensics
recoverability uh across a large
range of drives whether or not
[00:36:32.37]
it's turned on or off. So really
to look at the conclusions that
I have for this. Uh as a
[00:36:37.37]
[00:36:39.50]
forensics investigator, you
really need to be acutely aware
of the drive differences. You
[00:36:43.73]
need to understand, yes, first
and foremost that you're dealing
with an SSD. Uh you need to
[00:36:48.03]
understand that the drives are
gonna behave differently and how
to recover data from them. And
[00:36:53.43]
that SSDs do negatively impact
the recoverability. And then we
also have to look at modifying
[00:36:59.00]
our forensics practices to
recognize these drives and
understand what they can be
[00:37:03.13]
done. Well what can be done with
them in order to get data back.
Or if there's any way that we we
[00:37:07.50]
have to look at this at a more
deeper level. Uh if we look at
the flash, bypassing the
[00:37:12.13]
controller. Is that something
that might be forensically
practical? It's... one of the
[00:37:16.07]
things that I think is a great
opportunity for some future
work. Uh so definitely if when
[00:37:21.10]
we're looking at the future work
here. Uh bypassing the flash
controller is something that is
[00:37:26.20]
definitely interesting and way
beyond my technical capabilities
to understand how that electrons
[00:37:31.23]
works. So some of you I know are
smart enough to do that.
Definitely something to look
[00:37:34.77]
at.uh looking at how an
operating system makes a
difference. If the operating
[00:37:38.83]
system's running looking at more
forensics data. That could
definitely be something that is
[00:37:43.23]
an opportunity to look at as
well. So I'd like to acknowledge
a couple of people who helped
[00:37:48.80]
make this presentation possible.
Uh, three professors at RIT uh
dealt with me for way too long,
[00:37:55.40]
for getting this research done.
It took years. Uh I'd like to
thank them, Dr. Pan, Dr. Mishra
[00:37:59.83]
and Professor Stackpole. And
then Bill Mathews at Hurricane
Labs for supporting me through
[00:38:03.37]
this research. With that I'd
like to open the floor for any
questions. And I will leave my
[00:38:07.47]
contact info up here if anyone
wants to stop by. Thank you very
much. [Applause] So, yes over
[00:38:12.47]
[00:38:14.60]
there. [Inaudible question from
audience] Okay so the question
was, is there a way to tell when
[00:38:19.60]
[00:38:24.60]
you get the computer whether the
trim state is on or off. So that
is something thats configured in
[00:38:29.60]
[00:38:38.80]
the operating system. Uh so you
would have to look at the
operating system first in order
[00:38:43.60]
to do that. Uh which of course
has forensics implications as
well. Uh so one of the ways that
[00:38:50.53]
I would suggest maybe doing this
is to image the drive first and
then you can actually use the
[00:38:55.93]
image copy as a way to
interrogate what's going on. Uh
that way if you modify it,
[00:39:00.87]
you're not causing any problems
with the original evidence. >>
Is it in like registry..? >> Is
[00:39:06.73]
it something that you can run
just a command on like a Windows
system. Uh there's just uh
[00:39:11.50]
either a PowerShell or a Command
Line. It'll just print the value
of that whether it's on or off.
[00:39:17.50]
I believe it's stored in the
emergency as well when you make
that change. Next. >> Um, I was
[00:39:22.50]
[00:39:25.47]
wondering uh do you have any
indication of how other busses
like M2 or PCI Express or Stats
[00:39:32.20]
would respond? >> Yes, so M2 and
PCIE for sata type drives uh
basically function the same way.
[00:39:37.20]
[00:39:42.23]
So it it is passing a sata type
bus like an M sata disk. It
looks like a PCIE slot but it is
[00:39:48.37]
sata. So that would behave the
same way as I was saying. M2 is
the same idea or it's the sata
[00:39:53.57]
bus. Now on SAS a lot of times
that's gonna be behind a raid
controller. And the raid
[00:39:59.70]
controller often does not pass
the trim through. Although some
of the new raid controllers will
[00:40:04.07]
actually do that. Uh so that
would be a little bit of a
different behavior. Now there
[00:40:09.43]
was one other bus that you
mentioned? >> SAS, uh PCIE
and... >> Okay, so we covered
[00:40:14.03]
everything. >> Um, one more
quick question. >> Yeah, sure.
>> What, what about Enterprise
[00:40:17.17]
grade drives that have over
capacity? >> So, every SSD in
some way has some kind of over
[00:40:22.93]
capacity. >> Okay. >> So you
look at your hundred twenty gig
SSD typically has 128 gigs of
[00:40:28.20]
flash. Uh the Enterprise ones
could be you know a hundred gig
disk that has hundred twenty
[00:40:34.30]
eight gigs of flash. So the
behavior should be very similar
because the over provision flash
[00:40:39.33]
is still gonna be there.
Alright. And one of the tests
that I did actually simulated
[00:40:44.30]
over provisioning the flash by
partitioning the drive a lot
smaller which is the same
[00:40:48.50]
fundamental idea. And basically
I observed the same behavior
that I expected. >> Thank you.
[00:40:54.73]
>> Uh we have another line over
here. >> Yeah, sure. >> Um so
you mentioned uh bypassing the
[00:41:01.07]
uh, the FTL. Um some studies
have already been done on that
by UC San Diego and the results
[00:41:07.17]
were published in I believe in
2011. Um so... >> Yes, I uh
actually looked into some of
[00:41:11.53]
that research and I, I know
exactly what paper you're
talking about. Yeah. >> Yeah,
[00:41:15.30]
using the, the custom FPGA
controller to bypass the
behavior of the FTL. Um.. So
[00:41:20.30]
[00:41:23.00]
yeah, just wanted to make sure
that everybody else is aware of
that as well. >> No, yeah, that
[00:41:26.30]
research is definitely
interesting and I think there's
you, that a lot of that was
[00:41:30.70]
looking at the contents of the
flash cell itself and trying to
assemble that would be really
[00:41:36.47]
interesting. Like how do we pull
actual files off of that because
uh definitely if you're looking
[00:41:42.43]
for a string of attacks that's a
great way to find that data. If
you're looking for images or
[00:41:47.50]
larger files, trying to piece
all that together. Definitely
something that is more work can
[00:41:52.90]
be done. >> One more question.
>> Yeah, no I get one more
question, okay. So you guys we
[00:41:57.90]
[00:42:00.60]
can talk afterwards but uh I
will take one from this way. >>
Hey boss, how's it going today?
[00:42:06.80]
>> It's pretty good. >> So I
actually work for a storage
manufacturing company and I was
[00:42:11.27]
wondering if you've given any
consideration or if you can talk
to the impact of Bit rot on SSDs
[00:42:13.47]
uh for forensics investigations
especially since bit rot on SSDs
will occur so much faster than
[00:42:18.57]
on a hard... hard drive. So in
terms of bit rot, bit.. in terms
of bit rot, what he's referring
[00:42:23.57]
[00:42:25.73]
to is the fact the flash
actually degrades over time. So
a normal hard drive, there's
[00:42:30.83]
some magnetic decomposition as
times goes on. On a flash cell
it's quicker. Just to make sure.
[00:42:36.30]
>> yeah, no, exactly right. >.
yeah so that sort of thing, from
a forensics case, a lot of that
[00:42:42.43]
is gonna be not necessarily
identified through the
controller because the
[00:42:47.27]
controller it's gonna either see
that and it's gonna be there or
it's not. Now the controller
[00:42:51.43]
might proactively move that just
somewhere else, or refresh the
contents of flash to basically
[00:42:56.00]
reenergize the cells. But from a
forensics perspective, I don't
know if that's gonna be
[00:43:01.20]
something that's necessarily
gonna be available throughout eh
controller. >> Well the concern
[00:43:05.47]
is is that uh in the past, that
we've actually been approached
by some people that we're trying
[00:43:09.10]
to forensically analyze some of
the SSDs in some of our
products. And the problem is
[00:43:13.30]
that you uh achieve a warrant or
similar you take this equipment
out of it's source location. As
[00:43:17.83]
soon as you power it down, the
controllers no longer have the
ability to scrub the SSDs. So
[00:43:22.47]
you, if that stuff is sitting in
a forensics labs, you know two,
three months before it's
[00:43:27.27]
analyzed, by the time you get to
analyzing it, three, four months
some SSDs will already start to
[00:43:32.00]
lose data. >> Yeah, so one of
the things that I noticed as
part of this research is pretty
[00:43:37.63]
much all of the cleaning of the
data happened almost instantly.
So in terms of deleting data I
[00:43:44.40]
think it's gonna be pretty much
gone regardless. Now in terms of
actual stored data on there, uh
[00:43:50.10]
I didn't see any cases where if
I left these drives sit for you
know months on end that the data
[00:43:54.90]
would be gone or anything like
that. Uh now, it's I guess it's
possible where that bit rot
[00:44:01.77]
could happen in a period of
time. I, it's just not something
I saw happen as part of this
[00:44:06.10]
research. >> Alright. Beautiful.
[Applause] >> yeah. Alright
thank you very much. I will be
[00:44:10.23]
outside uh for any other
questions you guys might have.
Thank you very much, you've been
[00:44:14.67]
a great audience, I really
appreciate it.
[00:44:16.57]