I've noticed that most of the cores on my FX-8350 seem to idle at 2.1 GHz even when the system is very lightly loaded. This seems odd to me; I would expect them to idle at 1.4 GHz (the lowest speed). I do see cores drop to 1.4 GHz occasionally, but it is just a core or two, for a few seconds here and there.

My FX-8320 (on Kubuntu 12.04) behaved more like I expected, with most of the cores sitting at 1.4 GHz unless there was load on the system.

Not sure if this is an FX-8350 thing, or a 14.04 thing.

I'm not particularly concerned; idle CPU temps are a very reasonable 38C, and the fans are nearly silent. But it's still puzzling. And it does mean idle CPU power usage is a little higher than I'd like.

Are the both systems using the same CPU scaling governor? Is it possible the one system has a device like a USB joystick or a different daemon that is preventing the CPU from idling?

Same governor (modulo any differences between the governor implementations in 12.04 / 14.04), and similar (though not identical) hardware configurations. (Previous system was an M5A97 R2.0, current one is an M5A99FX Pro R2.0.)

I suppose if I'm feeling ambitious I could clone the system drive and try bringing it up on the other system (it's still intact, I haven't cannibalized it for parts yet), to see if the difference is the CPU/motherboard. But my guess is that either the CPU governor got revised between 12.04 and 14.04 (and is more aggressive at ramping up CPU clocks), or there's just enough additional background CPU activity in 14.04 to keep the cores from dropping to the lowest power state.

Try powertop (https://01.org/powertop) to see if can find anything that is not allowing the CPU to reach idle. It is an intel maintained tool, but it will work on AMD CPUs as well.

Thanks. Now that you mention it, I've heard of the tool before, but it had slipped my mind.

With one browser window and Amarok open (but not playing), powertop indicated that Chrome and the JACK audio stack were generating a lot of events -- hundreds per second -- even though system load was quite low. After killing Chrome, Amarok, and JACK, those events disappeared (as expected); however, it did not have a huge effect on CPU clocks. If I left the system sitting idle for a couple of minutes, I would see up to half of the cores drop down to 1.4 GHz for short periods of time, but the rest would remain at elevated clocks.

I took a couple of screenshots of what I'm talking about. These are after killing Chrome, Amarok, and JACK. Top left window is the built-in top command. Top right window is sysmon2, a system health monitoring script I wrote which shows CPU power usage (as reported by the CPU power monitoring feature on recent AMD CPUs) and core clocks, among other things. Bottom window is powertop.

Negligible system load; 3 of the cores have throttled down, but one is running at full speed, and 4 are at intermediate clock speeds. CPU power 74W.

Slightly higher (but still trivial for an 8-core CPU) system load; all cores now at 2.1 GHz. CPU power 84W.

While this is still a relatively unscientific experiment, based on this I am starting to suspect that the "on demand" CPU governor got re-tuned between 12.04 and 14.04, and now tends to keep more cores in higher power states. If so, this is good from a latency standpoint for anything requiring quick CPU response, but kind of sucks for power management. With more emphasis being placed on real-time performance lately, it would not surprise me if they've gone down this path.

The reason I ask, not being all that familiar with the newer AMD CPUs, is that there are definitely some knobs in the ondemand governor that could be related to this behavior, most notably powersave_bias (which is different than the powersave governor).

Will probably poke at this some more tonight. I guess I need to dig into what the various tuneable parameters do. For whatever reason, the system seems to be biased towards leaving multiple cores sitting in an intermediate power state, even if system loading is such that just a couple of cores would probably be sufficient.

* Have you tried seeing what sustained load across all cores does? Run Handbrake on a test video and see how the cores behave.

Have not checked what 100% load (800% load in Linux-speak, since it's an 8-core CPU...) does; my expectation is that all of the cores will spool up to 4 GHz, and power usage will peg at ~125W. Will verify this.

* Are you running the latest AMD firmware for the CPU? I know it wasn't installed and running by default in 14.04.

No idea. Will look into this. What firmware are you talking about? Microcode patches? Isn't that something that is normally handled by the BIOS/EFI?

Typically yes, though if you check the Additional Drivers utility in 16.04 (and at least the two prior versions, IIRC) you'll find AMD microcode updates squirreled away. My guess is that it downloads them as part of a firmware bundle and applies them early in the kernel boot process. From my personal experience it made a significant difference in games that thumped plural CPUs hard, but outside of that use case it didn't affect too much. It still might be worth applying here just to ensure good behavior.

You'll most likely be fine with the Asus defaults, though undervolting certainly helped my average power use and kept peak voltage down substantially; running the numbers on an online TDP calculator, it thumped it down to ~100 watts under full load. I definitely noticed a difference in both noise and heat after doing it.

* Are you running the latest AMD firmware for the CPU? I know it wasn't installed and running by default in 14.04.

No idea. Will look into this. What firmware are you talking about? Microcode patches? Isn't that something that is normally handled by the BIOS/EFI?

Typically yes, though if you check the Additional Drivers utility in 16.04 (and at least the two prior versions, IIRC) you'll find AMD microcode updates squirreled away. My guess is that it downloads them as part of a firmware bundle and applies them early in the kernel boot process. From my personal experience it made a significant difference in games that thumped plural CPUs hard, but outside of that use case it didn't affect too much. It still might be worth applying here just to ensure good behavior.

Ahh, interesting. Will definitely do this.

Concupiscence wrote:

You'll most likely be fine with the Asus defaults, though undervolting certainly helped my average power use and kept peak voltage down substantially; running the numbers on an online TDP calculator, it thumped it down to ~100 watts under full load. I definitely noticed a difference in both noise and heat after doing it.

I'm not too worried about full load behavior; the CPU fan and case ventilation are sufficient to keep the noise level reasonable even at 125W. I have run it up to 100% to make sure the core temperature stays within limits; I just didn't specifically look at the CPU clocks when I did that.

It would be nice to get the idle usage consistently below 50W... that seems like a reasonable goal to me.

Like you said, the BIOS/EFI will definitely have a microcode patch, but whether or not it's the latest one for your processor is at the mercy of the motherboard manufacturer.

So, like Concupiscence noted, it's good to install the package that will update the microcode straight from AMD. It'll apply during the boot process, and it'll either be the same as the one the board firmware previously applied during the boot or even newer.

So I guess the obvious thing to do would be switch to the conservative governor?

Possibly. Haven't had time to dig further.

Using the conservative governor results in behavior that is closer to what I saw in 12.04 with the ondemand governor, but may over-compensate in the other direction (cores tend to stay at 1.4 GHz even when there's moderate load). Provided it does not result in video/audio stuttering, this may be acceptable. I plan to run with the conservative governor for a while to see how it goes. If it is deemed unacceptable, I guess the next step is to start playing around with the detailed settings for the governor.

CPU power usage under light load appears to be reduced by around 30-40%. I'm seeing readings in the 40-50W range with JACK running and a few tabs open in Chrome.

If I intentionally run 8+ CPU intensive threads the cores do all spool up to 4 GHz (as they should), and CPU power usage hovers around 122W.

The RPM tachometer for my case exhaust fan is apparently a little glitchy. When I cranked the CPU load up, I was seeing occasional readings in excess of 150,000 RPM!

Edit: The fact that you're allowed to specify power management governors per core leads to an interesting question. If you specify (say) conservative for half of your cores and ondemand for the other half, will the CPU scheduler prefer cores which are being less aggressively power managed when deciding where to schedule a process?

OK, no video or audio glitches that I noticed, but "conservative" just feels a little laggy. Not terrible, but I think it is taking too long to ramp up clocks when something requires a burst of CPU, e.g. rendering a new web page or launching a program when you click something. It's set back to "ondemand" for now and I'll play with the detailed settings another day.

All righty then... I was in a hacking mood tonight, so I decided to go DIY on this. Rather than picking one of the stock CPU power management governors and tuning it, I concluded that what I really want is a meta-governor, which switches between governors based on system activity and load.

I wrote a simple daemon (as a bash script), which watches the desktop environment for keyboard and mouse activity. Any input device activity causes the "ondemand" governor (which favors responsiveness over power saving) to be used. After a certain number of seconds of inactivity, we fall back to the "conservative" governor, and eventually to the "powersave" governor (which pins clocks to their minimum frequency). But wait... if we have a resource intensive background job (e.g. media encode) running, we don't want the "powersave" governor to kick in, since that would slow things down. So we also watch system load, and if it exceeds a certain threshold, we hold off on enabling the "powersave" governor.

I've initially got the thresholds set to 15 seconds (for the drop to "conservative"), 15 minutes (to drop to "powersave"), and a system load load of 2.00 to keep the system out of "powersave"). The daemon polls every 0.5 second, so that's the maximum latency for switching out of the lower-power governor states back to "ondemand".

Gonna run with this for a while, and see if I experience any issues.

The script runs in a "debug" mode if executed as a non-root user, where it merely logs what changes it would make to the CPU power management state if it could.

#!/bin/bash## cpugovmon - change CPU governor based on inactivity time and system load# Copyright (C) 2016 Michael Uchima## This program is free software: you can redistribute it and/or modify# it under the terms of the GNU General Public License as published by# the Free Software Foundation, either version 3 of the License, or# (at your option) any later version.## This program is distributed in the hope that it will be useful,# but WITHOUT ANY WARRANTY; without even the implied warranty of# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the# GNU General Public License for more details.## See <http://www.gnu.org/licenses/> for a copy of the GNU General# Public License.

# Tunable parametersSECS_CONSERVATIVE=15 # secs of DE inactivity for conservative governorSECS_POWERSAVE=900 # secs of DE inactivity for powersave governorLOAD_NOSAVE=200 # sys load (x100) that bumps us up from powersave to conservative

All righty then... I was in a hacking mood tonight, so I decided to go DIY on this. Rather than picking one of the stock CPU power management governors and tuning it, I concluded that what I really want is a meta-governor, which switches between governors based on system activity and load.

I wrote a simple daemon (as a bash script), which watches the desktop environment for keyboard and mouse activity. Any input device activity causes the "ondemand" governor (which favors responsiveness over power saving) to be used. After a certain number of seconds of inactivity, we fall back to the "conservative" governor, and eventually to the "powersave" governor (which pins clocks to their minimum frequency). But wait... if we have a resource intensive background job (e.g. media encode) running, we don't want the "powersave" governor to kick in, since that would slow things down. So we also watch system load, and if it exceeds a certain threshold, we hold off on enabling the "powersave" governor.

I've initially got the thresholds set to 15 seconds (for the drop to "conservative"), 15 minutes (to drop to "powersave"), and a system load load of 2.00 to keep the system out of "powersave"). The daemon polls every 0.5 second, so that's the maximum latency for switching out of the lower-power governor states back to "ondemand".

Gonna run with this for a while, and see if I experience any issues.

The script runs in a "debug" mode if executed as a non-root user, where it merely logs what changes it would make to the CPU power management state if it could.

#!/bin/bash## cpugovmon - change CPU governor based on inactivity time and system load# Copyright (C) 2016 Michael Uchima## This program is free software: you can redistribute it and/or modify# it under the terms of the GNU General Public License as published by# the Free Software Foundation, either version 3 of the License, or# (at your option) any later version.## This program is distributed in the hope that it will be useful,# but WITHOUT ANY WARRANTY; without even the implied warranty of# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the# GNU General Public License for more details.## See <http://www.gnu.org/licenses/> for a copy of the GNU General# Public License.

# Tunable parametersSECS_CONSERVATIVE=15 # secs of DE inactivity for conservative governorSECS_POWERSAVE=900 # secs of DE inactivity for powersave governorLOAD_NOSAVE=200 # sys load (x100) that bumps us up from powersave to conservative

That's some cool work, but I think dropping to powersave might be excessive. Conservative is going to keep your cores at low frequencies already.

Yeah, perhaps; and it did add the complication of needing to monitor load average to accommodate CPU-intensive background jobs. However, there was one specific scenario I had in mind when I did it -- the case where you've left a web browser up, and there's a badly coded page that pegs a CPU core. Even with the conservative governor that core will ramp up to full speed; with the support for powersave mode, that core will still get throttled down if you haven't touched the system for 15 minutes.

It is possible dropping all the way to powersave mode might cause stuttering issues for video playback, in which case the SECS_POWERSAVE value could be increased to a couple of hours (to accommodate full-length movies) if desired. Or disable powersave mode entirely by setting SECS_POWERSAVE to a very large value.

Also, a caveat I should've mentioned: Current Intel CPUs (Sandy Bridge and later, IIRC) use a different power management scheme, which probably won't work with this script. Indications are that Intel's power management does a decent job without further assistance anyway. So don't bother unless you're running Linux on AMD or an older Intel processor.

#!/bin/bash## wattbar - show CPU power usage in a terminal window as a bargraph# Copyright (C) 2016 Michael Uchima## Uses AMD's on-die CPU power sensor, available in Bulldozer and# later cores. Sensors package must be installed and properly# configured for this to work!## Top bar is instantaneous power, bottom bar is 10 second rolling# average. Samples 5x/second, updates display once a second.## Note: For systems which are very lightly loaded, it is possible# that this script may account for a non-trivial fraction of the# system load!## This program is free software: you can redistribute it and/or modify# it under the terms of the GNU General Public License as published by# the Free Software Foundation, either version 3 of the License, or# (at your option) any later version.## This program is distributed in the hope that it will be useful,# but WITHOUT ANY WARRANTY; without even the implied warranty of# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the# GNU General Public License for more details.## See <http://www.gnu.org/licenses/> for a copy of the GNU General# Public License.

As an aside for the Linux CLI newbies, the line that sets the WATTS variable is a good example of using a shell pipeline to pick apart the output of a Linux CLI tool. Yeah, it's a little cryptic (sed pipelines tend to be that way), but it is stuff like this that makes it possible to quickly hack simple tools together. I wrote that line by repeatedly trying it in a CLI window, tweaking the sed expression until I got it right, then copy-pasting from the CLI window into the editor. It only took a couple of minutes to figure that part out; the rest was straightforward bash scripting.