Performance issues-Solaris

If its comes to performance issue in Unix operating systems, everybody will be behind on CPU.But not all the time CPU will cause for the performance problem. Because sometimes CPU may waiting for resources like Disk I/O, Swapping ,Network.As sysadmin you should able to pin point where the problem.vmstat is very useful command to determined where it the problem.

After introduction Solaris zones, sysadmin job is become bit difficult to isolate the problems from other local zones and global zone. prstat is very useful command to find local zones CPU usage from global zone.
Solaris Support for Process Binding and Processor Sets

The fields of vmstat’s display are procs
r in run queue
b blocked for resources I/O, paging etc.
w swapped

memory (in Kbytes)
swap – amount of swap space currently available
free – size of the free list page ( in units per second).
re page reclaims – see -S option for how this field is modified.
mf minor faults – see -S option for how this field is modified.
pi kilobytes paged in
po kilobytes paged out
fr kilobytes freed
de anticipated short-term memory shortfall (Kbytes)
sr pages scanned by clock algorithmdisk ( operations per second )

There are slots for up to four disks, labeled with a single letter and number.

The letter indicates the type of disk (s = SCSI, i = IPI, etc) . The number is

the logical unit number. faults
in (non clock) device interrupts
sy system calls
cs CPU context switches cpu – breakdown of percentage usage of CPU time. On multiprocessors this is an a

verage across all processors.
us user time
sy system time
id idle time

Problem symptoms:

1.) If the number of processes in run queue (procs r) are consistently greater than the number of CPUs on the system it will slow down system as there are more processes then available CPUs .
2.) if this number is more than four times the number of available CPUs in the system then system is facing shortage of cpu power and will greatly slow down the processess on the system.
3.) If the idle time (cpu id) is consistently 0 and if the system time (cpu sy) is double the user time (cpu us) system is facing shortage of CPU resources.

Resolution :

Resolution to these kind of issues involves tuning of application procedures to make efficient use of cpu and as a last resort increasing the cpu power or adding more cpu to the system

Top:
Top is tool which is commonly available all the Unix/Linux.IN Solaris we need to install .To see top 10 process which are utilizing high CPU time,use below command.

Prstat:
Prstat is in-built Solaris performance monitoring tool.Its very useful to find zone wise CPU utilization and also we can monitor amount of time threads wait for a processor using prstat -mL command in the LAT coloum.

The below command will CPU statistics for every 5 seconds for five times.We can add SAR in crontab to monitor CPU usage all the time.The load averages can be monitored extended time periods by looking at run queue lengths and the amount of time that the run queue is occupied using sar -q. High number of system calls per second per processor can be monitored using sar -c.

iostat ( Input Output statistics )
iostat reports terminal and disk I/O activity and CPU utilization. The first line of output is for the time period since boot & each subsequent line is for the prior interval . Kernel maintains a number of counters to keep track of the values.
iostat’s activity class options default to tdc (terminal, disk, and CPU). If any other option/s are specified, this default is completely overridden i.e. iostat -d will report only statistics about the disks.

synctax : iostat <options> interval count
option – let you specify the device for which information is needed like disk , cpu or terminal. (-d , -c , -t or -tdc ) . x options gives the extended statistics .
interval – is time period in seconds between two samples . iostat 4 will give data at each 4 seconds interval.
count – is the number of times the data is needed . iostat 4 5 will give data at 4 seconds interval 5 times

disk name of the disk
r/s reads per second
w/s writes per second
Kr/s kilobytes read per second
Kw/s kilobytes written per second
wait average number of transactions waiting for service (Q length)
actv average number of transactions actively being serviced
(removed from the queue but not yet completed)
%w percent of time there are transactions waiting
for service (queue non-empty)
%b percent of time the disk is busy (transactions in progress)
svc_t Service time (svc_t)

Results and Solutions:
The values to look from the iostat output are:
Reads/writes per second (r/s , w/s)
Percentage busy (%b)
Service time (svc_t)
If a disk shows consistently high reads/writes along with , the percentage busy (%b) of the disks is greater than 5 percent, and the average service time (svc_t) is greater than 30 milliseconds, then one of the following action needs to be taken
1.)Tune the application to use disk i/o more efficiently by modifying the disk queries and using available cache facilities of application servers .
2.) Spread the file system of the disk on to two or more disk using disk striping feature of volume manager /disksuite etc.
3.) Increase the system parameter values for inode cache , ufs_ninode , which is Number of inodes to be held in memory. Inodes are cached globally (for UFS), not on a per-file system basis
4.) Move the file system to another faster disk /controller or replace existing disk/controller to a faster
one.

On multicpu systems , vmstat averages the number of CPUs into the output. For per-process statistics .Without options, vmstat displays a one-line summary of the virtual memory activity since the system was booted.

syntax:
Basic synctax is vmstat <options> interval count
option – let you specify the type of information needed such as paging -p , cache -c ,.interrupt -i etc.
if no option is specified information about process , memory , paging , disk ,interrupts & cpu is displayed .
interval – is time period in seconds between two samples . vmstat 4 will give data at each 4 seconds interval.
count – is the number of times the data is needed . vmstat 4 5 will give data at 4 seconds interval 5
times.
Example
The following command displays a summary of what the system
is doing every five seconds.

page ( in units per second).
re page reclaims – see -S option for how this field is modified.
mf minor faults – see -S option for how this field is modified.
pi kilobytes paged in
po kilobytes paged out
fr kilobytes freed
de anticipated short-term memory shortfall (Kbytes)
sr pages scanned by clock algorithm

disk ( operations per second )
There are slots for up to four disks, labeled with a single letter and number.
The letter indicates the type of disk (s = SCSI, i = IPI, etc) . The number is
the logical unit number.

cpu – breakdown of percentage usage of CPU time. On multiprocessors this is an average across all processors.
us user time
sy system time
id idle time

Results and Solutions:
A. CPU issues:
Following columns has to be watched to determine if there is any cpu issue
Processes in the run queue (procs r)
User time (cpu us)
System time (cpu sy)
Idle time (cpu id)
procs cpu
r b w us sy id
0 0 0 4 14 82
0 0 1 3 35 62
0 0 1 3 33 64
0 0 1 1 21 78
Problem symptoms:
1.) If the number of processes in run queue (procs r) are consistently greater than the number of CPUs on the system it will slow down system as there are more processes then available CPUs .
2.) if this number is more than four times the number of available CPUs in the system then system is facing shortage of cpu power and will greatly slow down the processess on the system.
3.) If the idle time (cpu id) is consistently 0 and if the system time (cpu sy) is double the user time (cpu us) system is facing shortage of CPU resources.

Resolution :
Resolution to these kind of issues involves tuning of application procedures to make efficient use of cpu and as a last resort increasing the cpu power or adding more cpu to the system.

B. Memory Issues:
Memory bottlenecks are determined by the scan rate (sr) . The scan rate is the pages scanned by the clock algorithm per second. If the scan rate (sr) is continuously over 200 pages per second then there is a memory shortage.

Resolution :
1. Tune the applications & servers to make efficient use of memory and cache.
2. Increase system memory .
3. Implement priority paging in s in pre solaris 8 versions by adding line “set priority paging=1” in
/etc/system. Remove this line if upgrading from Solaris 7 to 8 & retaining old /etc/system file.