cpupower utility is a collection of tools for power efficiency of processor.
frequency-set sub-command controls settings for processor frequency.
"-g [governor]" specifies a policy to select processor frequency.
The performance governor statically sets frequency of the processor cores specified
by "-c" option to the highest possible for maximum performance.

nohz_full

This kernel option sets adaptive tick mode (NOHZ_FULL) to specified processors.
Since the number of interrupts is reduced to ones per second, latency-sensitive
applications can take advantage of it.

Enabling this option which is the default allows the processor to transmit to its
minimum frequency when entering the power state C1. If the switch is disabled the
CPU stays at its maximum frequency in C1. Because of the increase of power
consumption users should only select this option after performing application
benchmarking to verify improved performance in their environment. The default is "Enabled".

DCU Streamer Prefetcher

This BIOS switch allows 2 options: "Enabled" and "Disabled". The default is "Enabled".

This prefetcher is a L1 data cache prefetcher, which detects multiple loads from the same cache line
done within a time limit, in order to then prefetch the next line from the L2 cache or the main memory
into the L1 cache based on the assumption that the next cache line will also be needed.

Hardware P-States

This BIOS switch allows 4 options: "Native Mode", "Disabled", "Out of Band Mode"
and "Native Mode with No legacy Support". The default is "Native Mode".

With Hardware Power Management(HWPM) the processors provides a flexible interface
between Hardware and Platform for performance management and improving energy efficiency.

In Native Mode the HWPM operates cooperatively with the OS via a software interface
to provide constraints and hints.

When disabled, system does not use HWPM.

Power Supply Mode (Default = Balanced Performance)

This BIOS switch allows 4 options: "Balanced performance", "Performance",
"Balanced Energy" and "Energy Efficient". The default is "Balanced Performance"
optimized to maximum power savings with minimal impact on performance.
"Performance" disables all power management options with any impact on performance.
"Balanced Energy" is optimized for power efficiency and "Energy Efficient" for
power savings. The BIOS switch is only selectable if the BIOS switch
"Power Technology" is set to "Custom".

The two options "Balanced Performance" and "Balanced Energy" should always be the
first choice as both options optimize the efficiency of the system. In cases where
the performance is not sufficient or the power consumption is too high the two
options "Performance" or "Energy Efficient" could be an alternative.

IMC Interleaving

This BIOS option controls the interleaving between the Integrated Memory Controllers (IMCs).
There are two IMCs per socket in Skylake Server. If IMC Interleaving is set to 2-way,
addresses will be interleaved between the two IMCs. If IMC Interleaving is set to 1-way,
there will be no interleaving. If SNC is disabled, IMC Interleaving should be set to 2-way.
If SNC is enabled, IMC Interleaving should be set to 1-way.
Default setting is "Auto".

Hyper-Threading [all]

This BIOS option enables or disables additional hardware thread which shares same
physical core. Generally "Enabled" is recommended but disabling it makes sense
for the application which requires the shortest possible response times.
Default setting is "Enabled".

Link Frequency Select

This switch allows the configuration of the Intel Ultra Path Interconnect (UPI)
link speed. Default is auto, which configures the optimal link speed automatically.
It can be set "9.6 GT/s", "10.4 GT/s" or "Auto".

LLC dead line alloc

This BIOS switch allows 2 options: "Enabled" and "Disabled". The default is "Enabled".
In the Skylake non-inclusive cache scheme, the mid-level cache (MLC) evictions are filled
into the last-level cache (LLC). When lines are evicted from the MLC, the core can flag them
as "dead" (i.e., not likely to be read again).
The LLC has the option to drop dead lines and not fill them in the LLC.
If the Dead Line LLC Alloc feature is disabled, dead lines will always be dropped and will
never fill into the LLC. This can help save space in the LLC and prevent the LLC from evicting
useful data. However, if the Dead Line LLC Alloc feature is enabled, the LLC can
opportunistically fill dead lines into the LLC if there is free space available.

Package C State

This BIOS option allows 6 options: "C0/C1", "C2", "C6(non Retention)", "C6(Retention)", "No Limit" and "Auto".
The default setting is "Auto".
Package C-states is one of energy-saving options of the processor, which not only allow
the individual cores of a processor, but the entire processor chip to be put into a type
of sleep state. As a result, power consumption is even further reduced.
But the "waking-up time" that is required to change from the lower package C-states to
the active (C0) state is even longer in comparison with the CPU or core C-states.
If the "C0" setting is made in the BIOS, the processor chip always remains active.
It can improve the performance of latency sensitive workloads.

Patrol Scrub

This BIOS option enables or disables the so-called memory scrubbing, which cyclically
accesses the main memory of the system in the background regardless of the operating
system in order to detect and correct memory errors in a preventive way. The time of
this memory test cannot be influenced and can under certain circumstances result in
losses in performance. The disabling of the Patrol Scrub option increases the
probability of discovering memory errors in case of active accesses by the operating
system. Until these errors are correctable, the ECC technology of the memory modules
ensures that the system continues to run in a stable way. However, too many correctable
memory errors increase the risk of discovering non-correctable errors, which then
result in a system standstill.

Stale AtoS

This BIOS switch allows 2 options: "Enabled" and "Disabled". The default is "Disabled".

The in-memory directory has three states: I, A, and S.
I (invalid) state means the data is clean and does not exist in any other socket's cache.
A (snoopAll) state means the data may exist in another socket in exclusive or modified state.
S (Shared) state means the data is clean and may be shared across one or more socket's caches.

When doing a read to memory, if the directory line is in the A state we must snoop all the
other sockets because another socket may have the line in modified state. If this is the case,
the snoop will return the modified data. However, it may be the case that a line is read in
A state and all the snoops come back a miss. This can happen if another socket read the line
earlier and then silently dropped it from its cache without modifying it.

If Stale AtoS feature is enabled, in the situation where a line in A state returns only snoop
misses, the line will transition to S state. That way, subsequent reads to the line will
encounter it in S state and not have to snoop, saving latency and snoop bandwidth.
Stale AtoS may be beneficial in a workload where there are many cross-socket reads.

Sub NUMA Cluster

Sub NUMA Cluster (SNC) breaks up the last-level cache (LLC) into two disjoint clusters
based on address range, with each cluster bound to one memory controller. SNC improves
average latency to the LLC/memory and is a replacement for the "Cluster On Die" (COD)
feature found in previous processor families.
For a multi-socketed system, all SNC clusters are mapped to unique NUMA domains.
IMC Interleaving must be set to the correct value to correspond with SNC enable/disable.
If SNC and IMC Interleave are both set to Auto, the result will be SNC disabled (only one
cluster per socket) with 2-way IMC interleave. If SNC is set to Enable, IMC Interleave
should be set to 1-way, which will result in two clusters per socket.
The BIOS switch "Sub NUMA Clustering" allows 3 options: "auto", "enabled" and "disabled".
The default setting is "disabled".

Intel VT for Directed I/O (VT-d)

This BIOS option enables or disables I/O virtualization functions of the CPU.
If the server is not used for virtualization, this option should be set to "Disabled".
Default setting is "Enabled".

VMX

This BIOS option enables or disables additional virtualization functions of the CPU.
If the server is not used for virtualization, this option should be set to "Disabled".
This can result in energy savings. Default setting is "Enabled".

This option configures the processor Xtended Prediciton Table (XPT) prefetch feature.
The XPT prefetcher exists on top of other prefetchers that that can prefetch data in the
core DCU, MLC, and LLC. The XPT prefetcher will issue a speculative DRAM read request in
parallel to an LLC lookup. This prefetch bypasses the LLC, saving latency. In some cases,
setting this option to disabled can improve performance. In some cases, setting this option
to disabled can improve performance. Typically, setting this option to enable provides
better performance. This option must be enabled when Sub-NUMA Clustering is enabled. Values
for this BIOS option can be:
Enabled: Allows a read request sent to the LLC to speculatively issue a copy of the read to
the memory controller requesting the prefetch.
Disabled: Does not allow the LLC to speculatively issue copies of reads. Disabling this will
also disables Sub-NUMA Cluster (SNC).