Hi Shahar,
Can you describe why you require starting/stopping the reactor on demand?
Thanks,
-Jim
On 10/8/18, 1:07 AM, "SPDK on behalf of Shahar Salzman"
<spdk-bounces(a)lists.01.org on behalf of shahar.salzman(a)kaminario.com&gt; wrote:
Hi,
I am now working with my passthrough bdev directly from my application, and require
adding/removing them dynamically. In addition, I also require starting/stopping the
reactor on demand without tearing down the entire system (e.g. dpdk stays initialized).
This mostly works out of the box, but there are a few globals and bdev internal fields
which require re-init or tear down.
In addition, the copy engine did not io_unregister itself, so I also added this in the
patchset, allowing me the following application life span:
- env init (init config, spdk_env_init)
- reactors init
- while (application is alive)
subsystem_init + reactor start
...
Do some NVMeF stuff
...
subsysem_fini + reactors stop
- reactors_fini
- rpc_finish
I submitted the following patches for review:
https://review.gerrithub.io/c/spdk/spdk/+/428305https://review.gerrithub.io/c/spdk/spdk/+/428304https://review.gerrithub.io/c/spdk/spdk/+/428303https://review.gerrithub.io/c/spdk/spdk/+/428302
WDYT?
Shahar
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

I can't comment on Shahar's use case but it would be helpful in cases where
I want to flex out reactors based on the # of devices which are not fixed.
In my use case, I don't really want to have reactors running if I don't
need them since they do incur some cost as far as I understand it (maybe
max_delay_us could be used).
For example, it may be good enough to have 1 reactor serving 2
vhost_scsi_controllers starting out. However, if I need to add more
controllers (since I want to add more devices), then I'd like to start
another reactor for the new vhost_scsi_controllers. Reducing controllers is
tricky but I'm not sure this happens very often in my use case.
-- Michael
On Mon, Oct 8, 2018 at 11:00 AM Harris, James R <james.r.harris(a)intel.com&gt;
wrote:

Hi Shahar,
Can you describe why you require starting/stopping the reactor on demand?
Thanks,
-Jim
On 10/8/18, 1:07 AM, "SPDK on behalf of Shahar Salzman" <
spdk-bounces(a)lists.01.org on behalf of shahar.salzman(a)kaminario.com&gt;
wrote:
Hi,
I am now working with my passthrough bdev directly from my
application, and require adding/removing them dynamically. In addition, I
also require starting/stopping the reactor on demand without tearing down
the entire system (e.g. dpdk stays initialized). This mostly works out of
the box, but there are a few globals and bdev internal fields which require
re-init or tear down.
In addition, the copy engine did not io_unregister itself, so I also
added this in the patchset, allowing me the following application life span:
- env init (init config, spdk_env_init)
- reactors init
- while (application is alive)
subsystem_init + reactor start
...
Do some NVMeF stuff
...
subsysem_fini + reactors stop
- reactors_fini
- rpc_finish
I submitted the following patches for review:
https://review.gerrithub.io/c/spdk/spdk/+/428305https://review.gerrithub.io/c/spdk/spdk/+/428304https://review.gerrithub.io/c/spdk/spdk/+/428303https://review.gerrithub.io/c/spdk/spdk/+/428302
WDYT?
Shahar
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Hi Michael,
The reactors definitely have a cost if they are polling with no real work to do.
Ben Walker has been working on a patch series [1] that would effectively decouple and
“spdk_thread” from an “spdk_reactor”. Currently the relationship is always one-to-one.
Vishal Verma pushed some patches earlier this year [2] that track how much real work a
reactor is doing by tracking whether a poller did any real work. Combining these two
would enable dynamic load balancing. It sounds like that might meet the vhost use case
you’ve described.
This is why I’m interested in understanding more about Shahar’s request to start and stop
reactors and reinitialize the bdev layer, and whether it is also based on a desire to do
this kind of load balancing.
Thanks,
-Jim
[1] Patch series starts here: https://review.gerrithub.io/#/c/spdk/spdk/+/417784/
[2] Main patch is: https://review.gerrithub.io/#/c/spdk/spdk/+/412695/
On 10/8/18, 12:24 PM, "SPDK on behalf of Michael Haeuptle"
<spdk-bounces(a)lists.01.org on behalf of michaelhaeuptle(a)gmail.com&gt; wrote:
I can't comment on Shahar's use case but it would be helpful in cases where
I want to flex out reactors based on the # of devices which are not fixed.
In my use case, I don't really want to have reactors running if I don't
need them since they do incur some cost as far as I understand it (maybe
max_delay_us could be used).
For example, it may be good enough to have 1 reactor serving 2
vhost_scsi_controllers starting out. However, if I need to add more
controllers (since I want to add more devices), then I'd like to start
another reactor for the new vhost_scsi_controllers. Reducing controllers is
tricky but I'm not sure this happens very often in my use case.
-- Michael
On Mon, Oct 8, 2018 at 11:00 AM Harris, James R <james.r.harris(a)intel.com&gt;
wrote:

Hi Shahar,
Can you describe why you require starting/stopping the reactor on demand?
Thanks,
-Jim
On 10/8/18, 1:07 AM, "SPDK on behalf of Shahar Salzman" <
spdk-bounces(a)lists.01.org on behalf of shahar.salzman(a)kaminario.com&gt;
wrote:
Hi,
I am now working with my passthrough bdev directly from my
application, and require adding/removing them dynamically. In addition, I
also require starting/stopping the reactor on demand without tearing down
the entire system (e.g. dpdk stays initialized). This mostly works out of
the box, but there are a few globals and bdev internal fields which require
re-init or tear down.
In addition, the copy engine did not io_unregister itself, so I also
added this in the patchset, allowing me the following application life span:
- env init (init config, spdk_env_init)
- reactors init
- while (application is alive)
subsystem_init + reactor start
...
Do some NVMeF stuff
...
subsysem_fini + reactors stop
- reactors_fini
- rpc_finish
I submitted the following patches for review:
https://review.gerrithub.io/c/spdk/spdk/+/428305https://review.gerrithub.io/c/spdk/spdk/+/428304https://review.gerrithub.io/c/spdk/spdk/+/428303https://review.gerrithub.io/c/spdk/spdk/+/428302
WDYT?
Shahar
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Hi,
My use case is a distributed highly available storage appliance with features such as
dedup, snapshots etc.
We have both FC and iSCSI targets implemented in the kernel, and basically using the same
cores as SPDK for receiving IO. Reactor polling comes at a cost which I pay happily as
long as there is work for the NVMeF target. If the customer does not use the system with
NVMeF, this is just taking away from FC/iSCSI performance.
With the patches bellow, I can start/stop without tearing down memory etc.
It should be noted that I am not using the spdk app, since I require this fine
granularity, so I embedded the init/run/destroy flow in our app.
Shahar
________________________________
From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Harris, James R
<james.r.harris(a)intel.com&gt;
Sent: Tuesday, October 9, 2018 3:28:13 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] Multiple start/stop of reactor and add/remove of bdev
Hi Michael,
The reactors definitely have a cost if they are polling with no real work to do.
Ben Walker has been working on a patch series [1] that would effectively decouple and
“spdk_thread” from an “spdk_reactor”. Currently the relationship is always one-to-one.
Vishal Verma pushed some patches earlier this year [2] that track how much real work a
reactor is doing by tracking whether a poller did any real work. Combining these two
would enable dynamic load balancing. It sounds like that might meet the vhost use case
you’ve described.
This is why I’m interested in understanding more about Shahar’s request to start and stop
reactors and reinitialize the bdev layer, and whether it is also based on a desire to do
this kind of load balancing.
Thanks,
-Jim
[1] Patch series starts here: https://review.gerrithub.io/#/c/spdk/spdk/+/417784/
[2] Main patch is: https://review.gerrithub.io/#/c/spdk/spdk/+/412695/
On 10/8/18, 12:24 PM, "SPDK on behalf of Michael Haeuptle"
<spdk-bounces(a)lists.01.org on behalf of michaelhaeuptle(a)gmail.com&gt; wrote:
I can't comment on Shahar's use case but it would be helpful in cases where
I want to flex out reactors based on the # of devices which are not fixed.
In my use case, I don't really want to have reactors running if I don't
need them since they do incur some cost as far as I understand it (maybe
max_delay_us could be used).
For example, it may be good enough to have 1 reactor serving 2
vhost_scsi_controllers starting out. However, if I need to add more
controllers (since I want to add more devices), then I'd like to start
another reactor for the new vhost_scsi_controllers. Reducing controllers is
tricky but I'm not sure this happens very often in my use case.
-- Michael
On Mon, Oct 8, 2018 at 11:00 AM Harris, James R <james.r.harris(a)intel.com&gt;
wrote:

Hi Shahar,
Can you describe why you require starting/stopping the reactor on demand?
Thanks,
-Jim
On 10/8/18, 1:07 AM, "SPDK on behalf of Shahar Salzman" <
spdk-bounces(a)lists.01.org on behalf of shahar.salzman(a)kaminario.com&gt;
wrote:
Hi,
I am now working with my passthrough bdev directly from my
application, and require adding/removing them dynamically. In addition, I
also require starting/stopping the reactor on demand without tearing down
the entire system (e.g. dpdk stays initialized). This mostly works out of
the box, but there are a few globals and bdev internal fields which require
re-init or tear down.
In addition, the copy engine did not io_unregister itself, so I also
added this in the patchset, allowing me the following application life span:
- env init (init config, spdk_env_init)
- reactors init
- while (application is alive)
subsystem_init + reactor start
...
Do some NVMeF stuff
...
subsysem_fini + reactors stop
- reactors_fini
- rpc_finish
I submitted the following patches for review:
https://review.gerrithub.io/c/spdk/spdk/+/428305https://review.gerrithub.io/c/spdk/spdk/+/428304https://review.gerrithub.io/c/spdk/spdk/+/428303https://review.gerrithub.io/c/spdk/spdk/+/428302
WDYT?
Shahar
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Hi Shahar,
Thanks for the clarifications – specifically that you are embedding the library init/fini
calls directly in your app.
I looked at your patches – overall they seem reasonable. I’ve posted some comments on
them in GerritHub – we should be able to get them merged pretty easily.
Thanks,
-Jim
On 10/9/18, 12:47 AM, "SPDK on behalf of Shahar Salzman"
<spdk-bounces(a)lists.01.org on behalf of shahar.salzman(a)kaminario.com&gt; wrote:
Hi,
My use case is a distributed highly available storage appliance with features such as
dedup, snapshots etc.
We have both FC and iSCSI targets implemented in the kernel, and basically using the
same cores as SPDK for receiving IO. Reactor polling comes at a cost which I pay happily
as long as there is work for the NVMeF target. If the customer does not use the system
with NVMeF, this is just taking away from FC/iSCSI performance.
With the patches bellow, I can start/stop without tearing down memory etc.
It should be noted that I am not using the spdk app, since I require this fine
granularity, so I embedded the init/run/destroy flow in our app.
Shahar
________________________________
From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Harris, James R
<james.r.harris(a)intel.com&gt;
Sent: Tuesday, October 9, 2018 3:28:13 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] Multiple start/stop of reactor and add/remove of bdev
Hi Michael,
The reactors definitely have a cost if they are polling with no real work to do.
Ben Walker has been working on a patch series [1] that would effectively decouple and
“spdk_thread” from an “spdk_reactor”. Currently the relationship is always one-to-one.
Vishal Verma pushed some patches earlier this year [2] that track how much real work a
reactor is doing by tracking whether a poller did any real work. Combining these two
would enable dynamic load balancing. It sounds like that might meet the vhost use case
you’ve described.
This is why I’m interested in understanding more about Shahar’s request to start and
stop reactors and reinitialize the bdev layer, and whether it is also based on a desire to
do this kind of load balancing.
Thanks,
-Jim
[1] Patch series starts here: https://review.gerrithub.io/#/c/spdk/spdk/+/417784/
[2] Main patch is: https://review.gerrithub.io/#/c/spdk/spdk/+/412695/
On 10/8/18, 12:24 PM, "SPDK on behalf of Michael Haeuptle"
<spdk-bounces(a)lists.01.org on behalf of michaelhaeuptle(a)gmail.com&gt; wrote:
I can't comment on Shahar's use case but it would be helpful in cases
where
I want to flex out reactors based on the # of devices which are not fixed.
In my use case, I don't really want to have reactors running if I don't
need them since they do incur some cost as far as I understand it (maybe
max_delay_us could be used).
For example, it may be good enough to have 1 reactor serving 2
vhost_scsi_controllers starting out. However, if I need to add more
controllers (since I want to add more devices), then I'd like to start
another reactor for the new vhost_scsi_controllers. Reducing controllers is
tricky but I'm not sure this happens very often in my use case.
-- Michael
On Mon, Oct 8, 2018 at 11:00 AM Harris, James R <james.r.harris(a)intel.com&gt;
wrote:

Hi Shahar,
Can you describe why you require starting/stopping the reactor on demand?
Thanks,
-Jim
On 10/8/18, 1:07 AM, "SPDK on behalf of Shahar Salzman" <
spdk-bounces(a)lists.01.org on behalf of shahar.salzman(a)kaminario.com&gt;
wrote:
Hi,
I am now working with my passthrough bdev directly from my
application, and require adding/removing them dynamically. In addition, I
also require starting/stopping the reactor on demand without tearing down
the entire system (e.g. dpdk stays initialized). This mostly works out of
the box, but there are a few globals and bdev internal fields which require
re-init or tear down.
In addition, the copy engine did not io_unregister itself, so I also
added this in the patchset, allowing me the following application life span:
- env init (init config, spdk_env_init)
- reactors init
- while (application is alive)
subsystem_init + reactor start
...
Do some NVMeF stuff
...
subsysem_fini + reactors stop
- reactors_fini
- rpc_finish
I submitted the following patches for review:
https://review.gerrithub.io/c/spdk/spdk/+/428305https://review.gerrithub.io/c/spdk/spdk/+/428304https://review.gerrithub.io/c/spdk/spdk/+/428303https://review.gerrithub.io/c/spdk/spdk/+/428302
WDYT?
Shahar
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Hi Jim,
thanks for the reply! These patches actually solve another issue that we're
currently having. We have a cooperative threading environment and I had to
modify the current reactor loop to insert a yield to give other threads a
chance to run. Making the poller and message processing functions
accessible allows me to write our own reactor loop more easily.
As you stated correctly, my use case is to load balance based on the
current # of devices that are configured by a user, which changes over
time. Once the patches are merged, I need to take a closer look and do some
experiments.
-- Michael
On Mon, Oct 8, 2018 at 6:28 PM Harris, James R <james.r.harris(a)intel.com&gt;
wrote:

Hi Michael,
The reactors definitely have a cost if they are polling with no real work
to do.
Ben Walker has been working on a patch series [1] that would effectively
decouple and “spdk_thread” from an “spdk_reactor”. Currently the
relationship is always one-to-one. Vishal Verma pushed some patches
earlier this year [2] that track how much real work a reactor is doing by
tracking whether a poller did any real work. Combining these two would
enable dynamic load balancing. It sounds like that might meet the vhost
use case you’ve described.
This is why I’m interested in understanding more about Shahar’s request to
start and stop reactors and reinitialize the bdev layer, and whether it is
also based on a desire to do this kind of load balancing.
Thanks,
-Jim
[1] Patch series starts here:
https://review.gerrithub.io/#/c/spdk/spdk/+/417784/
[2] Main patch is: https://review.gerrithub.io/#/c/spdk/spdk/+/412695/
On 10/8/18, 12:24 PM, "SPDK on behalf of Michael Haeuptle" <
spdk-bounces(a)lists.01.org on behalf of michaelhaeuptle(a)gmail.com&gt; wrote:
I can't comment on Shahar's use case but it would be helpful in cases
where
I want to flex out reactors based on the # of devices which are not
fixed.
In my use case, I don't really want to have reactors running if I don't
need them since they do incur some cost as far as I understand it
(maybe
max_delay_us could be used).
For example, it may be good enough to have 1 reactor serving 2
vhost_scsi_controllers starting out. However, if I need to add more
controllers (since I want to add more devices), then I'd like to start
another reactor for the new vhost_scsi_controllers. Reducing
controllers is
tricky but I'm not sure this happens very often in my use case.
-- Michael
On Mon, Oct 8, 2018 at 11:00 AM Harris, James R <
james.r.harris(a)intel.com&gt;
wrote:
> Hi Shahar,
>
> Can you describe why you require starting/stopping the reactor on
demand?
>
> Thanks,
>
> -Jim
>
>
>
>
> On 10/8/18, 1:07 AM, "SPDK on behalf of Shahar Salzman" <
> spdk-bounces(a)lists.01.org on behalf of shahar.salzman(a)kaminario.com&gt;
> wrote:
>
> Hi,
>
>
> I am now working with my passthrough bdev directly from my
> application, and require adding/removing them dynamically. In
addition, I
> also require starting/stopping the reactor on demand without tearing
down
> the entire system (e.g. dpdk stays initialized). This mostly works
out of
> the box, but there are a few globals and bdev internal fields which
require
> re-init or tear down.
>
> In addition, the copy engine did not io_unregister itself, so I
also
> added this in the patchset, allowing me the following application
life span:
>
>
> - env init (init config, spdk_env_init)
>
> - reactors init
>
> - while (application is alive)
>
> subsystem_init + reactor start
>
> ...
>
> Do some NVMeF stuff
>
> ...
>
> subsysem_fini + reactors stop
>
> - reactors_fini
>
> - rpc_finish
>
>
>
> I submitted the following patches for review:
>
> https://review.gerrithub.io/c/spdk/spdk/+/428305
>
> https://review.gerrithub.io/c/spdk/spdk/+/428304
>
> https://review.gerrithub.io/c/spdk/spdk/+/428303
>
> https://review.gerrithub.io/c/spdk/spdk/+/428302
>
>
> WDYT?
>
>
> Shahar
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk