*[PATCH v12 0/4] iommu/arm-smmu: Add runtime pm/sleep support@ 2018-07-08 17:34 Vivek Gautam
2018-07-08 17:34 ` [PATCH v12 1/4] iommu/arm-smmu: Add pm_runtime/sleep ops Vivek Gautam
` (3 more replies)0 siblings, 4 replies; 31+ messages in thread
From: Vivek Gautam @ 2018-07-08 17:34 UTC (permalink / raw)
To: joro, robh+dt, mark.rutland, robin.murphy, will.deacon, iommu,
devicetree, linux-kernel
Cc: alex.williamson, rjw, robdclark, linux-pm, freedreno, sboyd,
tfiga, sricharan, m.szyprowski, architt, linux-arm-msm, jcrouse,
vivek.gautam
This series provides the support for turning on the arm-smmu's
clocks/power domains using runtime pm. This is done using the
recently introduced device links patches, which lets the smmu's
runtime to follow the master's runtime pm, so the smmu remains
powered only when the masters use it.
As not all implementations support clock/power gating, we are checking
for a valid 'smmu->dev's pm_domain' to conditionally enable the runtime
power management for such smmu implementations that can support it.
With addition of a new device link flag DL_FLAG_AUTOREMOVE_SUPPLIER [11]
the device links created between arm-smmu and its clients will be
automatically purged when arm-smmu driver unbinds from its device.
This series also adds support for Qcom's arm-smmu-v2 variant that
has different clocks and power requirements.
Took some reference from the exynos runtime patches [1].
With conditional runtime pm now, we avoid touching dev->power.lock
in fastpaths for smmu implementations that don't need to do anything
useful with pm_runtime.
This lets us to use the much-argued pm_runtime_get_sync/put_sync()
calls in map/unmap callbacks so that the clients do not have to
worry about handling any of the arm-smmu's power.
Previous version of this patch series is @ [5].
Note: This series is now based on the device link changes [11] for
adding new flag - DL_FLAG_AUTOREMOVE_SUPPLIER.
Rafael will pull in the device link changes and these patches have
to be pulled in based on Rafael's branch. As Rafael said -
"I would prefer to apply them myself to be honest and put them on an
public git branch for you to pull from."
[v12]
* Use new device link's flag introduced in [11] -
DL_FLAG_AUTOREMOVE_SUPPLIER. With this devices links are automatically
purged when arm-smmu driver unbinds.
* Using pm_runtime_force_suspend() instead of pm_runtime_disable() to
avoid following warning from arm_smmu_device_remove()
[295711.537507] ------------[ cut here ]------------
[295711.544226] Unpreparing enabled smmu_mdp_ahb_clk
[295711.549099] WARNING: CPU: 0 PID: 1 at ../drivers/clk/clk.c:697
clk_core_unprepare+0xd8/0xe0
...
[295711.674073] Call trace:
[295711.679454] clk_core_unprepare+0xd8/0xe0
[295711.682059] clk_unprepare+0x28/0x40
[295711.685964] clk_bulk_unprepare+0x28/0x40
[295711.689701] arm_smmu_device_remove+0x88/0xd8
[295711.693692] arm_smmu_device_shutdown+0xc/0x18
[295711.698120] platform_drv_shutdown+0x20/0x30
[v11]
* Some more cleanups for device link. We don't need an explicit
delete for device link from the driver, but just set the flag
DL_FLAG_AUTOREMOVE.
device_link_add() API description says -
"If the DL_FLAG_AUTOREMOVE is set, the link will be removed
automatically when the consumer device driver unbinds."
* Addressed the comments for 'smmu' in arm_smmu_map/unmap().
* Dropped the patch [10] that introduced device_link_del_dev() API.
[v10]
* Introduce device_link_del_dev() API to delete the link between
given consumer and supplier devices. The users of device link
do not need to store link pointer to delete the link later.
They can straightaway use this API by passing consumer and
supplier devices.
* Made corresponding changes to arm-smmu driver patch handling the
device links.
* Dropped the patch [9] that was adding device_link_find() API to
device core layer. device_link_del_dev() serves the purpose to
directly delete the link between two given devices.
[v9]
* Removed 'rpm_supported' flag, instead checking on pm_domain
to enable runtime pm.
* Creating device link only when the runtime pm is enabled, as we
don't need a device link besides managing the power dependency
between supplier and consumer devices.
* Introducing a patch to add device_link_find() API that finds
and existing link between supplier and consumer devices.
Also, made necessary change to device_link_add() to use this API.
* arm_smmu_remove_device() now uses this device_link_find() to find
the device link between smmu device and the master device, and then
delete this link.
* Dropped the destroy_domain_context() fix [8] as it was rather,
introducing catastrophically bad problem by destroying
'good dev's domain context.
* Added 'Reviwed-by' tag for Tomasz's review.
[v8]
* Major change -
- Added a flag 'rpm_supported' which each platform that supports
runtime pm, can enable, and we enable runtime_pm over arm-smmu
only when this flag is set.
- Adding the conditional pm_runtime_get/put() calls to .map, .unmap
and .attach_dev ops.
- Dropped the patch [6] that exported pm_runtim_get/put_suupliers(),
and also dropped the user driver patch [7] for these APIs.
* Clock code further cleanup
- doing only clk_bulk_enable() and clk_bulk_disable() in runtime pm
callbacks. We shouldn't be taking a slow path (clk_prepare/unprepare())
from these runtime pm callbacks. Thereby, moved clk_bulk_prepare() to
arm_smmu_device_probe(), and clk_bulk_unprepare() to
arm_smmu_device_remove().
- clk data filling to a common method arm_smmu_fill_clk_data() that
fills the clock ids and number of clocks.
* Addressed other nits and comments
- device_link_add() error path fixed.
- Fix for checking negative error value from pm_runtime_get_sync().
- Documentation redo.
* Added another patch fixing the error path in arm_smmu_attach_dev()
to destroy allocated domain context.
[v7]
* Addressed review comments given by Robin Murphy -
- Added device_link_del() in .remove_device path.
- Error path cleanup in arm_smmu_add_device().
- Added pm_runtime_get/put_sync() in .remove path, and replaced
pm_runtime_force_suspend() with pm_runtime_disable().
- clk_names cleanup in arm_smmu_init_clks()
* Added 'Reviewed-by' given by Rob H.
[V6]
* Added Ack given by Rafael to first patch in the series.
* Addressed Rob Herring's comment for adding soc specific compatible
string as well besides 'qcom,smmu-v2'.
[V5]
* Dropped runtime pm calls from "arm_smmu_unmap" op as discussed over
the list [3] for the last patch series.
* Added a patch to export pm_runtime_get/put_suppliers() APIs to the
series as agreed with Rafael [4].
* Added the related patch for msm drm iommu layer to use
pm_runtime_get/put_suppliers() APIs in msm_mmu_funcs.
* Dropped arm-mmu500 clock patch since that would break existing
platforms.
* Changed compatible 'qcom,msm8996-smmu-v2' to 'qcom,smmu-v2' to reflect
the IP version rather than the platform on which it is used.
The same IP is used across multiple platforms including msm8996,
and sdm845 etc.
* Using clock bulk APIs to handle the clocks available to the IP as
suggested by Stephen Boyd.
* The first patch in v4 version of the patch-series:
("iommu/arm-smmu: Fix the error path in arm_smmu_add_device") has
already made it to mainline.
[V4]
* Reworked the clock handling part. We now take clock names as data
in the driver for supported compatible versions, and loop over them
to get, enable, and disable the clocks.
* Using qcom,msm8996 based compatibles for bindings instead of a generic
qcom compatible.
* Refactor MMU500 patch to just add the necessary clock names data and
corresponding bindings.
* Added the pm_runtime_get/put() calls in .unmap iommu op (fix added by
Stanimir on top of previous patch version.
* Added a patch to fix error path in arm_smmu_add_device()
* Removed patch 3/5 of V3 patch series that added qcom,smmu-v2 bindings.
[V3]
* Reworked the patches to keep the clocks init/enabling function
separately for each compatible.
* Added clocks bindings for MMU40x/500.
* Added a new compatible for qcom,smmu-v2 implementation and
the clock bindings for the same.
* Rebased on top of 4.11-rc1
[V2]
* Split the patches little differently.
* Addressed comments.
* Removed the patch #4 [2] from previous post
for arm-smmu context save restore. Planning to
post this separately after reworking/addressing Robin's
feedback.
* Reversed the sequence to disable clocks than enabling.
This was required for those cases where the
clocks are populated in a dependent order from DT.
[1] https://lkml.org/lkml/2016/10/20/70
[2] https://patchwork.kernel.org/patch/9389717/
[3] https://patchwork.kernel.org/patch/10204925/
[4] https://patchwork.kernel.org/patch/10102445/
[5] https://lkml.org/lkml/2018/3/22/191
[6] https://patchwork.kernel.org/patch/10204945/
[7] https://patchwork.kernel.org/patch/10204925/
[8] https://patchwork.kernel.org/patch/10254105/
[9] https://patchwork.kernel.org/patch/10277975/
[10] https://patchwork.kernel.org/patch/10281613/
[11] https://patchwork.kernel.org/patch/10491481/
Sricharan R (3):
iommu/arm-smmu: Add pm_runtime/sleep ops
iommu/arm-smmu: Invoke pm_runtime during probe, add/remove device
iommu/arm-smmu: Add the device_link between masters and smmu
Vivek Gautam (1):
iommu/arm-smmu: Add support for qcom,smmu-v2 variant
.../devicetree/bindings/iommu/arm,smmu.txt | 42 +++++
drivers/iommu/arm-smmu.c | 178 +++++++++++++++++++--
2 files changed, 210 insertions(+), 10 deletions(-)
--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation
^permalinkrawreply [flat|nested] 31+ messages in thread

*Re: [PATCH v12 1/4] iommu/arm-smmu: Add pm_runtime/sleep ops
2018-07-16 10:11 ` Vivek Gautam@ 2018-07-16 10:23 ` Rafael J. Wysocki0 siblings, 0 replies; 31+ messages in thread
From: Rafael J. Wysocki @ 2018-07-16 10:23 UTC (permalink / raw)
To: Vivek Gautam
Cc: Rafael J. Wysocki, Tomasz Figa, Mark Rutland,
open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS,
Linux PM, Stephen Boyd, Will Deacon, Rafael J. Wysocki,
Linux Kernel Mailing List, list@263.net:IOMMU DRIVERS,
Joerg Roedel, Rob Herring, linux-arm-msm, freedreno
Hi,
On Mon, Jul 16, 2018 at 12:11 PM, Vivek Gautam
<vivek.gautam@codeaurora.org> wrote:
> HI Rafael,
>
>
>
> On 7/16/2018 2:21 PM, Rafael J. Wysocki wrote:
>>
>> On Thu, Jul 12, 2018 at 12:57 PM, Vivek Gautam
>> <vivek.gautam@codeaurora.org> wrote:
[cut]
>>>> Although, given the PM
>>>> subsystem internals, the suspend function wouldn't be called on SMMU
>>>> implementation needed power control (since they would have runtime PM
>>>> enabled) and on others, it would be called but do nothing (since no
>>>> clocks).
>>>>
>>>>> Honestly, I just don't know. :-)
>>>>>
>>>>> It just looks odd the way it is done. I think the clock should be
>>>>> gated during system-wide suspend too, because the system can spend
>>>>> much more time in a sleep state than in the working state, on average.
>>>>>
>>>>> And note that you cannot rely on runtime PM to always do it for you,
>>>>> because it may be disabled at a client device or even blocked by user
>>>>> space via power/control in sysfs and that shouldn't matter for
>>>>> system-wide PM.
>>>>
>>>> User space blocking runtime PM through sysfs is a good point. I'm not
>>>> 100% sure how the PM subsystem deals with that in case of system-wide
>>>> suspend. I guess for consistency and safety, we should have the
>>>> suspend callback.
>>>
>>> Will add the following suspend callback (same as
>>> arm_smmu_runtime_suspend):
>>>
>>> static int __maybe_unused arm_smmu_pm_suspend(struct device *dev)
>>> {
>>> struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>>>
>>> clk_bulk_disable(smmu->num_clks, smmu->clks);
>>>
>>> return 0;
>>> }
>>
>> I think you also need to check if the clock has already been disabled
>> by runtime PM. Otherwise you may end up disabling it twice in a row.
>
>
> Should I rather call a pm_runtime_put() in suspend callback?
That wouldn't work as runtime PM may be effectively disabled by user
space via sysfs. That's one of the reasons why you need the extra
system-wide suspend callback in the first place. :-)
> Or an expanded form something similar to:
> https://elixir.bootlin.com/linux/v4.18-rc5/source/drivers/slimbus/qcom-ctrl.c#L695
Yes, you can do something like that, but be careful to make sure that
the state of the device after system-wide resume is consistent with
its runtime PM status in all cases.
^permalinkrawreply [flat|nested] 31+ messages in thread

*Re: [PATCH v12 1/4] iommu/arm-smmu: Add pm_runtime/sleep ops
2018-07-23 5:59 ` Marek Szyprowski@ 2018-07-23 11:05 ` Rafael J. Wysocki0 siblings, 0 replies; 31+ messages in thread
From: Rafael J. Wysocki @ 2018-07-23 11:05 UTC (permalink / raw)
To: Marek Szyprowski
Cc: Rafael J. Wysocki, Tomasz Figa, Vivek Gautam, Rafael J. Wysocki,
list@263.net:IOMMU DRIVERS, Joerg Roedel, Rob Herring,
Mark Rutland, Robin Murphy, Will Deacon, devicetree,
Linux Kernel Mailing List, Alex Williamson, Rob Clark, Linux PM,
freedreno, Stephen Boyd, Sricharan R, Archit Taneja,
linux-arm-msm, Jordan Crouse
On Mon, Jul 23, 2018 at 7:59 AM, Marek Szyprowski
<m.szyprowski@samsung.com> wrote:
> Hi Rafael,
>
> On 2018-07-11 22:36, Rafael J. Wysocki wrote:
>> On Wed, Jul 11, 2018 at 3:40 PM, Marek Szyprowski
>> <m.szyprowski@samsung.com> wrote:
[cut]
>>> Frankly, if there are no other reasons I suggest to wire system
>>> suspend/resume to pm_runtime_force_* helpers:
>>> SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
>>> pm_runtime_force_resume).
>> Not a good idea at all IMO.
>>
>> Use PM driver flags rather I'd say.
>
> Frankly, till now I wasn't aware of the DPM_FLAG_* in struct dev_pm_info
> 'driver_flags'.
They are a relatively recent addition.
> I've briefly checked them but I don't see the equivalent
> of using SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
> pm_runtime_force_resume): keep device suspend if it was runtime suspended
> AND really call pm_runtime_suspend if it was not runtime suspended on
> system suspend.
DPM_FLAG_SMART_SUSPEND is expected to cause that to happen. If you
want the device to remain in suspend after the system-wide resume from
sleep (if possible), you can set DPM_FLAG_LEAVE_SUSPENDED for it too.
Currently a caveat is that genpd still doesn't support the flags. I
have patches for that, but I haven't got to posting them yet. If you
are interested, I can push this work forward relatively quickly, so
please let me know.
>>> This way you will have everything related to suspending and resuming in
>>> one place and you would not need to bother about all possible cases (like
>>> suspending from runtime pm active and suspending from runtime pm suspended
>>> cases as well as restoring proper device state on resume). This is
>>> especially important in recent kernel releases, where devices are
>>> system-suspended regardless their runtime pm states (in older kernels
>>> devices were first runtime resumed for system suspend, what made code
>>> simpler, but wasn't best from power consumption perspective).
>>>
>>> If you go this way, You only need to ensure that runtime resume will also
>>> restore proper device state besides enabling all the clocks. This will
>>> also prepare your driver to properly operate inside power domain, where it
>>> is possible for device to loose its internal state after runtime suspend
>>> when respective power domain has been turned off.
>> I'm not sure if you are aware of the pm_runtime_force_* limitations, though.
>
> What are those limitations?
First off, they don't work with middle-layer code implementing its own
PM callbacks that actually operate on devices (like the PCI bus type
or the generic ACPI PM domain). This means that drivers using them
may not work on systems with ACPI, for example.
Second, pm_runtime_force_resume() will always try to leave the device
in suspend during system-wide resume from sleep which may not be
desirable.
Finally, they expect the runtime PM status to be updated by
system-wide PM callbacks of devices below the one using them (eg. its
children and their children etc) which generally is not required and
may not take place unless the drivers of those devices use
pm_runtime_force_* themselves.
So careful there.
^permalinkrawreply [flat|nested] 31+ messages in thread