Hi,
Is there anything that can wreck havoc on NVMe to check how well the code copes with it?
Something like a script or utility that runs in background and resets PCIe controller,
detaches disks and anything else that could theoretically happen to NVMe.
Thanks,
E.

Hi,
SPDK is tested in “tiers” and nvme is no different other than it may have more robust
tests than some of the other modules due to its maturity:
- unit tests: Look in /spdk/test/unit/lib/nvme and you’ll find “per file” unit tests that
use C-unit. These are, of course, SW only tests focused around exercising the public API
in each file geared towards catching regressions. Every new public function added is
required to have a new unit test. Coverage reports are available on the CI page at spdk.io
and there are clearly opportunities there for anyone that wants to beef it up.
- component level tests attempt to isolate just a pustular module (sometimes a few other
are required to be involved) and use either real hardware or run on a VM with emulated
NVMe devices. They are in spdk/test/lib/nvme mainly via nvme.sh however you’ll find a few
standalone focused tests that operate solely on the component, like sgl, in cases where we
really want to throw strange things at complicated code (like sgl and prp creation).
You’ll find a few other focused tets in here like hotplug.sh for hotplug testing and aer
for Async Event Requests and overhead for latency measurements
- system: there are bunch of other tests that use NVMe as a back end but are focused on
system level applications that are part of SPDK like the NVMe-oF target tests, the vHost
tests. iSCSI tests, etc.
- performance: these are pretty manual right now so there’s not much to point at but as
the name implies they’re really about performance stress testing.
Basically there’s a TON of tests in the repo. There’s a lot of activity right now ongoing
to beef up the automated testing, to move our CI over to Jenkins and to rework the dozens
of bash scripts that control the various tests. The goal there is to make it easier for
anyone to come in and find what they are interested in to either understand testing, help
beef it up or reproduce a failure locally that they encountered on a patch in CI.
After you dig around a little feel free to ask more questions and someone will jump in and
provide more detail. As the effort I just mentioned makes progress there will be new docs
added to http://spdk.io that explain the test strategy and hierarchy in a clear way - we
hope ☺
-Paul
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of E.W.Z.
Sent: Wednesday, February 7, 2018 9:14 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org&gt;
Subject: [SPDK] Code robustness test
Hi,
Is there anything that can wreck havoc on NVMe to check how well the code copes with it?
Something like a script or utility that runs in background and resets PCIe controller,
detaches disks and anything else that could theoretically happen to NVMe.
Thanks,
E.

Hi Paul,
Thanks for your reply. I wasn’t clear enough with my question. Sure you have tests in
SPDK, but I was asking for something that may assist me in testing my code. I’m writing
something that uses NVMe driver. I’m trying to handle errors where applicable. Now I want
to simulate real world problems which may occur (like controller reset) and see if my code
handles such a situation.
Thanks,
E.
From: Luse, Paul E
Sent: Thursday, February 8, 2018 1:37 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] Code robustness test
Hi,
SPDK is tested in “tiers” and nvme is no different other than it may have more robust
tests than some of the other modules due to its maturity:
- unit tests: Look in /spdk/test/unit/lib/nvme and you’ll find “per file” unit tests that
use C-unit. These are, of course, SW only tests focused around exercising the public API
in each file geared towards catching regressions. Every new public function added is
required to have a new unit test. Coverage reports are available on the CI page at spdk.io
and there are clearly opportunities there for anyone that wants to beef it up.
- component level tests attempt to isolate just a pustular module (sometimes a few other
are required to be involved) and use either real hardware or run on a VM with emulated
NVMe devices. They are in spdk/test/lib/nvme mainly via nvme.sh however you’ll find a few
standalone focused tests that operate solely on the component, like sgl, in cases where we
really want to throw strange things at complicated code (like sgl and prp creation).
You’ll find a few other focused tets in here like hotplug.sh for hotplug testing and aer
for Async Event Requests and overhead for latency measurements
- system: there are bunch of other tests that use NVMe as a back end but are focused on
system level applications that are part of SPDK like the NVMe-oF target tests, the vHost
tests. iSCSI tests, etc.
- performance: these are pretty manual right now so there’s not much to point at but as
the name implies they’re really about performance stress testing.
Basically there’s a TON of tests in the repo. There’s a lot of activity right now ongoing
to beef up the automated testing, to move our CI over to Jenkins and to rework the dozens
of bash scripts that control the various tests. The goal there is to make it easier for
anyone to come in and find what they are interested in to either understand testing, help
beef it up or reproduce a failure locally that they encountered on a patch in CI.
After you dig around a little feel free to ask more questions and someone will jump in and
provide more detail. As the effort I just mentioned makes progress there will be new docs
added to http://spdk.io that explain the test strategy and hierarchy in a clear way - we
hope ☺
-Paul
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of E.W.Z.
Sent: Wednesday, February 7, 2018 9:14 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org&gt;
Subject: [SPDK] Code robustness test
Hi,
Is there anything that can wreck havoc on NVMe to check how well the code copes with it?
Something like a script or utility that runs in background and resets PCIe controller,
detaches disks and anything else that could theoretically happen to NVMe.
Thanks,
E.

Hi,
For reset you can use our error injection framework. You can find it in this location:
spdk\lib\bdev\error . You can configure it using RPC commands. What it basically does is
to add another layer on top of existing one (like NVMe) and simulate errors. It can
simulate resets. Let me know if this is something that you were looking for.
Regards,
Maciek
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Ernest Zaslavsky
Sent: Thursday, February 8, 2018 6:57 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org&gt;
Subject: Re: [SPDK] Code robustness test
Hi Paul,
Thanks for your reply. I wasn’t clear enough with my question. Sure you have tests in
SPDK, but I was asking for something that may assist me in testing my code. I’m writing
something that uses NVMe driver. I’m trying to handle errors where applicable. Now I want
to simulate real world problems which may occur (like controller reset) and see if my code
handles such a situation.
Thanks,
E.
From: Luse, Paul E<mailto:paul.e.luse@intel.com>
Sent: Thursday, February 8, 2018 1:37 AM
To: Storage Performance Development Kit<mailto:spdk@lists.01.org>
Subject: Re: [SPDK] Code robustness test
Hi,
SPDK is tested in “tiers” and nvme is no different other than it may have more robust
tests than some of the other modules due to its maturity:
- unit tests: Look in /spdk/test/unit/lib/nvme and you’ll find “per file” unit tests that
use C-unit. These are, of course, SW only tests focused around exercising the public API
in each file geared towards catching regressions. Every new public function added is
required to have a new unit test. Coverage reports are available on the CI page at spdk.io
and there are clearly opportunities there for anyone that wants to beef it up.
- component level tests attempt to isolate just a pustular module (sometimes a few other
are required to be involved) and use either real hardware or run on a VM with emulated
NVMe devices. They are in spdk/test/lib/nvme mainly via nvme.sh however you’ll find a few
standalone focused tests that operate solely on the component, like sgl, in cases where we
really want to throw strange things at complicated code (like sgl and prp creation).
You’ll find a few other focused tets in here like hotplug.sh for hotplug testing and aer
for Async Event Requests and overhead for latency measurements
- system: there are bunch of other tests that use NVMe as a back end but are focused on
system level applications that are part of SPDK like the NVMe-oF target tests, the vHost
tests. iSCSI tests, etc.
- performance: these are pretty manual right now so there’s not much to point at but as
the name implies they’re really about performance stress testing.
Basically there’s a TON of tests in the repo. There’s a lot of activity right now ongoing
to beef up the automated testing, to move our CI over to Jenkins and to rework the dozens
of bash scripts that control the various tests. The goal there is to make it easier for
anyone to come in and find what they are interested in to either understand testing, help
beef it up or reproduce a failure locally that they encountered on a patch in CI.
After you dig around a little feel free to ask more questions and someone will jump in and
provide more detail. As the effort I just mentioned makes progress there will be new docs
added to http://spdk.io that explain the test strategy and hierarchy in a clear way - we
hope ☺
-Paul
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of E.W.Z.
Sent: Wednesday, February 7, 2018 9:14 AM
To: Storage Performance Development Kit
<spdk@lists.01.org<mailto:spdk@lists.01.org>>
Subject: [SPDK] Code robustness test
Hi,
Is there anything that can wreck havoc on NVMe to check how well the code copes with it?
Something like a script or utility that runs in background and resets PCIe controller,
detaches disks and anything else that could theoretically happen to NVMe.
Thanks,
E.

Sounds like just what the doctor ordered. Will check it out. Thanks!
Sent from Mail for Windows 10
From: Szwed, Maciej
Sent: Thursday, February 8, 2018 9:12 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] Code robustness test
Hi,
For reset you can use our error injection framework. You can find it in this location:
spdk\lib\bdev\error . You can configure it using RPC commands. What it basically does is
to add another layer on top of existing one (like NVMe) and simulate errors. It can
simulate resets. Let me know if this is something that you were looking for.
Regards,
Maciek
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Ernest Zaslavsky
Sent: Thursday, February 8, 2018 6:57 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org&gt;
Subject: Re: [SPDK] Code robustness test
Hi Paul,
Thanks for your reply. I wasn’t clear enough with my question. Sure you have tests in
SPDK, but I was asking for something that may assist me in testing my code. I’m writing
something that uses NVMe driver. I’m trying to handle errors where applicable. Now I want
to simulate real world problems which may occur (like controller reset) and see if my code
handles such a situation.
Thanks,
E.
From: Luse, Paul E
Sent: Thursday, February 8, 2018 1:37 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] Code robustness test
Hi,
SPDK is tested in “tiers” and nvme is no different other than it may have more robust
tests than some of the other modules due to its maturity:
- unit tests: Look in /spdk/test/unit/lib/nvme and you’ll find “per file” unit tests that
use C-unit. These are, of course, SW only tests focused around exercising the public API
in each file geared towards catching regressions. Every new public function added is
required to have a new unit test. Coverage reports are available on the CI page at spdk.io
and there are clearly opportunities there for anyone that wants to beef it up.
- component level tests attempt to isolate just a pustular module (sometimes a few other
are required to be involved) and use either real hardware or run on a VM with emulated
NVMe devices. They are in spdk/test/lib/nvme mainly via nvme.sh however you’ll find a few
standalone focused tests that operate solely on the component, like sgl, in cases where we
really want to throw strange things at complicated code (like sgl and prp creation).
You’ll find a few other focused tets in here like hotplug.sh for hotplug testing and aer
for Async Event Requests and overhead for latency measurements
- system: there are bunch of other tests that use NVMe as a back end but are focused on
system level applications that are part of SPDK like the NVMe-oF target tests, the vHost
tests. iSCSI tests, etc.
- performance: these are pretty manual right now so there’s not much to point at but as
the name implies they’re really about performance stress testing.
Basically there’s a TON of tests in the repo. There’s a lot of activity right now ongoing
to beef up the automated testing, to move our CI over to Jenkins and to rework the dozens
of bash scripts that control the various tests. The goal there is to make it easier for
anyone to come in and find what they are interested in to either understand testing, help
beef it up or reproduce a failure locally that they encountered on a patch in CI.
After you dig around a little feel free to ask more questions and someone will jump in and
provide more detail. As the effort I just mentioned makes progress there will be new docs
added to http://spdk.io that explain the test strategy and hierarchy in a clear way - we
hope ☺
-Paul
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of E.W.Z.
Sent: Wednesday, February 7, 2018 9:14 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org&gt;
Subject: [SPDK] Code robustness test
Hi,
Is there anything that can wreck havoc on NVMe to check how well the code copes with it?
Something like a script or utility that runs in background and resets PCIe controller,
detaches disks and anything else that could theoretically happen to NVMe.
Thanks,
E.