Softpanorama

May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

iostat

News Recommended Books Recommended Links Linux Solaris

AIX

Examples
vmstat netstat iostat uptime command sar

Unix top command

 Linux Performance Tuning
watch     Horror Stories Unix History Humor Etc

iostat   is a low overhead tool that can be automated and provides useful report about I/O activity on the server. The command is pretty simple to use. It provides the defalt report if invoked without any parameters.  You can also specify several parameters

iostat -d -x interval count

The default report provides an overview of CPU and disk I/O statistics:

Linux 2.6.9-5.0.3.EL_blb_P4 (foobar.example.com)     02/23/2005
 
avg-cpu:  %user   %nice    %sys %iowait   %idle
           0.14    0.00    0.22    2.37   97.27
 
Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
hda               0.74        33.25        99.74    2344531    7031970
hdb               2.08       214.19       288.00   15100778   20304528
hdd               2.03       200.41       288.00   14128983   20304528
md0              39.28       414.59       287.81   29229074   20290864
sda               0.00         0.01         0.00        408          0

Below the first line (which contains the system's kernel version and hostname, along with the current date), iostat  displays an overview of the system's average CPU utilization since the last reboot. The CPU utilization report includes the following percentages:

Below the CPU utilization report is the device utilization report. This report contains one line for each active disk device on the system and includes the following information:

This is just a one report that can be obtained using iostat. Different reports can be obtained using different options.

Since different application processes have different I/O patterns,  I/O bottlenecks may appear on different disks at different times during each day. It is important to remember that these bottlenecks are transient in nature. A disk may experience an I/O bottleneck for a very short period, and this short-duration bottleneck may repeat itself many times each day. Summarizing I/O by the hour does not provide adequate information about transient bottlenecks as I/O spikes with duration of say 10 min or less disappeared in the hourly average.

To get the most accurate results, you should collect I/O statistics at more frequent intervals—preferably no more than ten minutes between samples—over a representative time period, such as a week. Because individual application processes have different I/O patterns, bottlenecks may appear on different disks at various times during each day. And because Oracle transactions happen very quickly, a disk may experience an I/O bottleneck for a very short period—but a short-duration bottleneck may nonetheless repeat itself thousands of times each day. If you make the mistake of summarizing I/O by the hour, as many DBAs do, you won't see these bottlenecks because the I/O spikes will not be evident in the hourly average.

The goal of load balancing is to distribute the files across disks so as to achieve a single static optimal I/O throughput. Once the I/O subsystem is balanced, the files will not need to be moved unless new processes change the I/O pattern for the disks.

The goal is to find the optimal file placement where overall load balance is achieved for all of the many variations of disk access. Load balancing is essentially the identification of hot disks, and the movement of datafiles to less-used cool disks. As such, disk load balancing is an iterative process since it is possible that relocating a datafile may relieve contention for one process, only to cause I/O contention for an unrelated process. Also, for databases placed on a small number of disks, it is possible that I/O contention cannot be avoided. Consider a 30GB database spread across disks with 20 competing processes for data. On average, ten processes would be queued waiting for I/O from each of the two disks. Clearly, these types of systems will always experience I/O contention.

After using data collected by iostat to identify a hot disk, you would use data collected by the Oracle utilities to identify which mount point and file contain the table causing the excessive I/O activity.

Identifying the hot disk is only the beginning of the quest. We must then see what mount point on the disk is causing the problem, which datafile on the mount point, and finally, what Oracle table is causing the excessive I/O. Only with this approach can the Oracle administrator fully understand how to perform disk load balancing. With that in mind, let’s look at the first method for collecting Oracle I/O statistics. We will then move on to look at collecting UNIX I/O statistics.


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

[Jul 10, 2020] I-O reporting from the Linux command line by Tyler Carrigan

Jul 10, 2020 | www.redhat.com

I/O reporting from the Linux command line Learn the iostat tool, its common command-line flags and options, and how to use it to better understand input/output performance in Linux.

Posted: July 9, 2020 | by Tyler Carrigan (Red Hat)

Image

Image by Pexels

More Linux resources

If you have followed my posts here at Enable Sysadmin, you know that I previously worked as a storage support engineer. One of my many tasks in that role was to help customers replicate backups from their production environments to dedicated backup storage arrays. Many times, customers would contact me concerned about the speed of the data transfer from production to storage.

Now, if you have ever worked in support, you know that there can be many causes for a symptom. However, the throughput of a system can have huge implications for massive data transfers. If all is well, we are talking hours, if not... I have seen a single replication job take months.

We know that Linux is loaded full of helpful tools for all manner of issues. For input/output monitoring, we use the iostat command. iostat is a part of the sysstat package and is not loaded on all distributions by default.

Installation and base run

I am using Red Hat Enterprise Linux 8 here and have included the install output below.

[ Want to try out Red Hat Enterprise Linux? Download it now for free. ]

NOTE : the command runs automatically after installation.

[root@rhel ~]# iostat
bash: iostat: command not found...
Install package 'sysstat' to provide command 'iostat'? [N/y] y
    
    
 * Waiting in queue... 
The following packages have to be installed:
lm_sensors-libs-3.4.0-21.20180522git70f7e08.el8.x86_64    Lm_sensors core libraries
sysstat-11.7.3-2.el8.x86_64    Collection of performance monitoring tools for Linux
Proceed with changes? [N/y] y
    
    
 * Waiting in queue... 
 * Waiting for authentication... 
 * Waiting in queue... 
 * Downloading packages... 
 * Requesting data... 
 * Testing changes... 
 * Installing packages... 
Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test)     06/17/2020     _x86_64_    (4 CPU)
    
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.17    0.05    4.09    0.65    0.00   83.03
    
Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             206.70      8014.01      1411.92    1224862     215798
sdc               0.69        20.39         0.00       3116          0
sdb               0.69        20.39         0.00       3116          0
dm-0            215.54      7917.78      1449.15    1210154     221488
dm-1              0.64        14.52         0.00       2220          0

If you run the base command without options, iostat displays CPU usage information. It also displays I/O stats for each partition on the system. The output includes totals, as well as per second values for both read and write operations. Also, note that the tps field is the total number of Transfers per second issued to a specific device.

The practical application is this: if you know what hardware is used, then you know what parameters it should be operating within. Once you combine this knowledge with the output of iostat , you can make changes to your system accordingly.

Interval runs

It can be useful in troubleshooting or data gathering phases to have a report run at a given interval. To do this, run the command with the interval (in seconds) at the end:

[root@rhel ~]# iostat -m 10
Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test)     06/17/2020     _x86_64_    (4 CPU)
    
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.94    0.05    0.35    0.04    0.00   98.62
    
Device             tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda              12.18         0.44         0.12       1212        323
sdc               0.04         0.00         0.00          3          0
sdb               0.04         0.00         0.00          3          0
dm-0             12.79         0.43         0.12       1197        329
dm-1              0.04         0.00         0.00          2          0
    
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.24    0.00    0.15    0.00    0.00   99.61
    
Device             tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda               0.00         0.00         0.00          0          0
sdc               0.00         0.00         0.00          0          0
sdb               0.00         0.00         0.00          0          0
dm-0              0.00         0.00         0.00          0          0
dm-1              0.00         0.00         0.00          0          0
    
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.20    0.00    0.18    0.00    0.00   99.62
    
Device             tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda               0.50         0.00         0.00          0          0
sdc               0.00         0.00         0.00          0          0
sdb               0.00         0.00         0.00          0          0
dm-0              0.50         0.00         0.00          0          0
dm-1              0.00         0.00         0.00          0          0

The above output is from a 30-second run.

You must use Ctrl + C to exit the run.

Easy reading

To clean up the output and make it easier to digest, use the following options:

-m changes the output to megabytes, which is a bit easier to read and is usually better understood by customers or managers.

[root@rhel ~]# iostat -m
Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test)     06/17/2020     _x86_64_    (4 CPU)
    
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.51    0.09    0.55    0.07    0.00   97.77
    
Device             tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda              22.23         0.81         0.21       1211        322
sdc               0.07         0.00         0.00          3          0
sdb               0.07         0.00         0.00          3          0
dm-0             23.34         0.80         0.22       1197        328
dm-1              0.07         0.00         0.00          2          0

-p allows you to specify a particular device to focus in on. You can combine this option with the -m for a nice and tidy look at a particularly concerning device and its partitions.

[root@rhel ~]# iostat -m -p sda
Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test)     06/17/2020     _x86_64_    (4 CPU)
    
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.19    0.07    0.45    0.06    0.00   98.24
    
Device             tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda              17.27         0.63         0.17       1211        322
sda2             16.83         0.62         0.17       1202        320
sda1              0.10         0.00         0.00          7          2
Advanced stats

If the default values just aren't getting you the information you need, you can use the -x flag to view extended statistics:

[root@rhel ~]# iostat -m -p sda -x 
Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test)     06/17/2020     _x86_64_    (4 CPU)
    
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.06    0.06    0.40    0.05    0.00   98.43
    
Device            r/s     w/s     rMB/s     wMB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sda             12.20    2.83      0.54      0.14     0.02     0.92   0.16  24.64    0.55    0.50   0.00    45.58    52.37   0.46   0.69
sda2            12.10    2.54      0.54      0.14     0.02     0.92   0.16  26.64    0.55    0.47   0.00    45.60    57.88   0.47   0.68
sda1             0.08    0.01      0.00      0.00     0.00     0.00   0.00  23.53    0.44    1.00   0.00    43.00   161.08   0.57   0.00

Some of the options to pay attention to here are:

There are other values present, but these are the ones to look out for.

Shutting down

This article covers just about everything you need to get started with iostat . If you have other questions or need further explanations of options, be sure to check out the man page or your preferred search engine. For other Linux tips and tricks, keep an eye on Enable Sysadmin!

UNIX Performance Management By Jaqui Lynch (Boston College)

The purpose of this paper is to introduce the performance analyst to some of the free tools available to monitor and manage performance on UNIX systems, and to provide a guideline on how to diagnose and fix performance problems in that environment. The paper is based on the authors experiences with AIX and will cover many of the tools available on that and other UNIX platforms. It will also provide some Rules of Thumb for analyzing the performance of UNIX systems.

As more mission critical work finds its way from the mainframe to distributed systems, performance management for those systems is becoming more important. The goal for systems management is not only to maximize system throughput, but also to reduce response time. In order to this it is necessary to not only work on the system resources, but also to work on profiling and tuning applications.

In UNIX there are 7 major resource types that need to be monitored and tuned - CPU, memory, disk space and arms, communications lines, I/O Time, Network Time and applications programs. There are also standard rules of thumb in most of these areas. From the users perspective the only one they see is total execution time so we will start by looking at that.

Total execution time from a users perspective consists of wall-clock time. At a process level this is measured by running the time command. This provides you with real time (wallclock), user code CPU and system code CPU. If user + sys > 80% then there is a good chance the system is CPU constrained. The components of total running time include:

1. User-state CPU - the actual amount of time the CPU spends running the users program in the user state. It includes time spent executing library calls, but does not include time spent in the kernel on its behalf. This value can be greatly affected by the use of optimization at compile time and by writing efficient code.

2. System-state CPU - this is the amount of time the CPU spends in the system state on behalf of this program. All I/O routines require kernel services. The programmer can affect this value by the use of blocking for I/O transfers.

3. I/O Time and Network Time - these are the amount of time spent moving data and servicing I/O requests.

4. Virtual Memory Performance - This includes context switching and swapping.

5. Time spent running other programs.

In order to measure these areas there are a multitude of tools available. The most useful are:

Other commands that will be useful include lsvg, lspv, lslv, lsps and lsdev. Each of these will be discussed below and then a general problem solving approach will be offered. It is important to note that the results and options for all of these commands may differ depending on the platform they are being run on. Most of the options discussed below are those for AIX and some of the tools are specific to AIX such as:

The first tool to be discussed is uptime. This provides the analyst with the System Load Average (SLA). It is important to note that the SLA can only be used as a rough indicator as it does not take into account scheduling priority and it counts as runnable all jobs waiting for disk I/O, including NFS I/O. However, uptime is a good place to start when trying to determine whether a bottleneck is CPU or I/O based.

When uptime is run it provides three load averages - the first is for the last minute, the second is for the last 5 minutes and the third is for the last 15 minutes. If the value is borderline but has been falling over the last 15 minutes, then it would be wise to just monitor the situation. However, a value between 4 and 7 is fairly heavy and means that performance is being negatively affected. Above 7 means the system needs serious review and below 3 means the workload is relatively light. If the system is a single user workstation then the load average should be less than 2. There is also a command called ruptime that allows you to request uptime information remotely.

The sar command provides a good alternative to uptime with the -q option. It provides statistics on the average length of the run queue, the percentage of time the run queue is occupied, the average length of the swap queue and the percentage of time the swap queue is occupied. The run queue lists jobs that are in memory and runnable, but does not include jobs that are waiting for I/O or sleeping. The run queue size should be less than 2. If the load is high and the runqocc=0 then the problem is most likely memory or I/O, not CPU. The swap queue lists jobs that are ready to run but have been swapped out.

The sar command deserves special mention as it is a very powerful command. The command is run by typing in:

sar -options int #samples

where valid options generally are:

After determining that the problem may well be CPU based it would then be necessary to move onto iostat to get more detail. Running iostat provides a great deal of information, but the values of concern here are the %user and %sys. If (%user + %sys) > 80% over a period of time then it is very likely the bottleneck is CPU. In particular it is necessary to watch for average CPU being greater than 70% with peaks above 90%. It is also possible to get similar information by running the ps -au or sar -u commands. Both of these provide information about CPU time. The sar -u command, in particular, breaks the time down into user, system, time waiting for blocked I/O (i.e. NFS, disk, ..) and idle time.

The ps -au command also provides information on the %physical memory the process is using and the current status for the process. Statuses shown are:

Status Meaning

The cron or at command can be used to automatically schedule execution of these commands to ensure snapshots are taken at the appropriate times. The atq command can be used to list what is in the at queue and the crontab -e command edits the cron table.

Once it has been determined that the problem is a CPU bottleneck there are several options. It is possible to limit the cputime a process can use by the limit command. If the problem relates to one process then it is also possible to model or profile that process using the prof, gprof or tprof command to find out whether it is possible to optimize the program code.

Prof and gprof are very similar and have several disadvantages when compared to tprof. Both prof and gprof require a recompile of the program using wither the -p or the -pg option and they impact performance of that program very badly. Tprof only needs to be recompiled in order to do source code level profiling (-qlist option). In particular tprof exhibits the following characteristics (AIX only):

So, the recommendation would be to use tprof if it is available on the chosen platform. It is also possible that the vendor will have their own equivalent to tprof.

Running the time or timex commands can also give a good indication of whether the process is CPU intensive. Compiler options have been proven to extensively affect the performance of CPU intensive programs as can be seen from the table below. It is well worth trying different options when compiling the program such as -O, -O2, -O3 and -Q (inline streams the code). Time/timex can give you an indication of how much benefit this will provide. Timex can also be run using the -s option which causes a full set of sar output to be generated for the duration of the programs execution. As can be seen from the table below, it is possible to see reductions in the order of 50% in CPU utilization by using optimization.

User CPU running Program phntest

Compiler Seconds % of CPU

Options for None

None 53.03 100%

-O 26.34 49.67%

-O -Q 25.11 47.35%

-O2 27.04 50.99%

-O2 -Q 24.92 46.99%

-O3 28.48 53.71%

-O3 -Q 26.13 49.27%

It is also possible to change the priority of the process so that other processes can gain control of the CPU ahead of it. This can be done by using the nice, renice or setpri commands. Renice is not available on all platforms. Before using these commands, it is useful to understand how the priority scheme works in UNIX.

Priorities range from 0-127 with 127 being the lowest priority. The actual priority of a task is set by the following calculation:

pri=nice+puser+(cpu/2)

Puser normally defaults to 40 and nice to 20 unless the nice or renice commands have been used against the process. On AIX a tick is 1/100th of a second and new priorities are calculated every tick as follows:

new-pri=(tick/2)+puser+nice

Every second tick is recalculated as tick=tick/2 and then new-pri is again recalculated.

Otherwise, a CPU upgrade may be the only solution if there is no other machine that the workload can be run on.

If the problem does not appear to be CPU then it becomes necessary to investigate memory and I/O as possible causes. Again, it is possible to use iostat or sar to get the information that is needed here. The iowait field shown in the iostat command is a good indicator of whether there are I/O problems. If iowait is greater than 40% then it becomes necessary to investigate the physical volumes to ensure that none of them are greater than 70% full. The lspv command can be used to determine utilization of the physical volume.

Iostat is a low overhead tool that can be automated and provides local counts for I/O data. Unlike sar, iostat does not provide timestamps in the output so it is important to make a note of start/stop times. However, iostat uses kernel data which makes it hardware specific with respect to the results obtained.

Iostat provides data on several important values for each physical disk. These include: %time the physical disk was busy, kilobytes per second to/from the disk, transfers per second to/from, kilobytes read and kilobytes written. This will help to determine if there is an imbalance of I/O amongst the physical volumes. If all appears to be normal here then the next step is to investigate which filesystems the I/O is directed at. If most of the I/O is directed at the page files then memory needs to be investigated.

Information on cylinder access and seek distances is available using the sadp command and cache statistics for disk are available using the sar -b command. Further information can be obtained by running filemon and looking to see what the most active filesystems are.

Filemon provides a list of the most active segments, the most active logical volumes and physical volumes, and detailed statistics for the busiest files as well as the segments, physical and logical volumes. Details include transfers, reads, read sizes, read times in msecs, logical seeks, write times, seek distances, throughput (kb/sec) and utilization percentages. However, it is important to note that filemon runs trace in the background which can affect performance. It is also possible to run fileplace which gives information on space efficiency and sequentiality.

This would be a good time to run lsvg, lslv and lspv to get a map of the layout of the physical and logical volumes on the system as well as the various volume groups. This will make it much simpler to get more indepth information. By running lsdev -C it is also possible to determine what kind of disk devices are installed and what size they are. By using a combination of the above commands a map can be produced for each physical volume of the filesystems and their placement on the disk. The lsattr -E -l sys0 command can be used to obtain information on system parameters such as cache sizes and other associated values.

If the bulk of the I/O (>30%) is going to a logical volume or filesystem that is not used for paging then the problem is most likely user I/O. This can be resolved by one of several options -checking fragmentation, reorganizing the filesystem, adding physical volumes or splitting the data in another manner. Adding memory may still help with the problem.

Other items to take into account when looking at I/O performance include the intra and inter policies, mirroring of disks, write verify and the scheduling policy for logical volumes. It is also important to remember that the SCSI card can only talk to one device at a time. Where multiple disks are behind one SCSI card, sequential readwrites are helped if they are spread across multiple adapters. Newer technologies such as SCSI-2, fast and wide and raid will also help improve performance. Some of the newer controllers also provide buffers for each disk and can perform two way searches.

If the bulk of the I/O is going to paging (i.e. the page LV is > 30%) then it becomes necessary to investigate further. The only options available to cure a paging problem are to write more memory efficient code, move the process to another system, add memory, reschedule the process so it doesn't contend with other memory intensive workloads, or add physical volumes or more page datasets. There are three commands that are used to investigate paging - lsps (or pstat), vmstat and svmon.

lsps -a will provide information on all of the page spaces available including physical volume and utilization. Vmstat is another low overhead tool and provides information on actual memory, free pages, processes on the I/O waitq, reclaims, pageins, pageouts, pages freed by the stealer per second, interrupts, system calls and CPU utilization. Like iostat, vmstat does not provide timestamps. Svmon -G provides similar information except it breaks memory down into work, persistent and client pages that are either in use or pinned. It is also possible to use the sar -w command.

When looking at paging it is important to note that the stealer will run whenever there are only ((2 x real) -8) pages left. So on a 32mb machine the stealer will run if there are only 56 pages left. The Rule of Thumb for page space versus real memory is generally in the order of Page = 2 x real. On some systems not all of the kernel processes are pinned so they can also be paged out. A pagein rate of >5/sec means that the system is memory constrained. Also, if fre is less than (.1(AVM)) then this may indicate that the system is real memory constrained. This depends on the way the VMM uses memory. For instance, AIX will use all memory for disk caching, etc before it reuses any so it is not unusual to see fre very low (110-120). Looking at pageins, pageouts, and the FR to SR ratio is a much more meaningful indicator for problems.

So, if at this point there is no problem with CPU and the system is not disk bound it becomes necessary to investigate the network to check whether it is remote file I/O bound. This is the last step before running the more resource heavy trace command to determine what is really happening. To look at network statistics there are three useful commands - netstat, netpmon and nfsstat.

Netstat -i shows the network interfaces along with input and output packets and errors. It also gives the number of collisions. The Mtu field shows the maximum ip packet size (transfer unit) and should be the same on all systems. In AIX it defaults to 1500. Both Oerrs (number of output errors since boot) and Ierrs (Input errors since boot) should be < 0.025. If Oerrs>0.025 then it is worth increasing the send queue size. Ierrs includes checksum errors and can also be an indicator of a hardware error such as a bad connector or terminator. The Collis field shows the number of collisions since boot and can be as high as 10%. If it is greater then it is necessary to reorganize the network as the network is obviously overloaded on that segment.

Netstat -m s used to analyze the use of mbufs in order to determine whether these are the bottleneck. The no -a command is used to see what the current values are. Values of interest are thewall, lowclust, lowmbuf and dogticks.

An mbuf is a kernel buffer that uses pinned memory and is used to service network communications. Mbufs come in two sizes - 256 bytes and 4096 bytes (clusters of 256 bytes). Thewall is the maximum memory that can be taken up for mbufs. Lowmbuf is the minimum number of mbufs to be kept free while lowclust is the minimum number of clusters to be kept free. Mb_cl_hiwat is the maximum number of free buffers to be kept in the free buffer pool and should be set to at least twice the value of lowclust to avoid thrashing.

Netstat -v is used to look at queues and other information. If Max packets on S/W transmit queue is >0 and is equal to current HW transmit queue length then the send queue size should be increased. If the No mbuf errors is large then the receive queue size needs to be increased.

Nfsstat is used to report on client and server NFS information, primarily at the daemon level. Nfsstat -c provides client information such as retrans and badxid. If badxid=retrans and retrans > 5% of calls the server is the problem, but if retrans > 5% of calls and badxid < retrans then the network is the problem. Also, if there are lots of timeouts then it is useful to increase the number of NFSDs and the qsize.

Netpmon is a further command that focuses on CPU, network adapters, remote nodes and LAN traffic. It is used to get a feeling for what is happening overall. By using a combination of the above commands it is possible to obtain a very clear view of what is happening at the network level.

At this point it is important to mention the UNIX kernel tables, as these can affect performance without any real indicators as to the cause. To find out what they are set to the pstat -T or sar -v commands can be used. Most of the values are calculated based on the value for maxusers so it is important to know what that is set to. It is often recommended that maxusers generally be determined by the following calculation:

Max Users = (2+ # users act + #NFS clients + .5 NFS exports )

In particular, attention should be paid to the following table sizes:

Process Table Size (NPROCS) - this is the maximum number of processes that can be in the system. On systems where Xwindows is heavily used this needs to be increased. If the table is full, then the process will fail to start.

Text Table Size (NTEXT) - This is the maximum number of executables that can be in the system at a time. If the table is full then the exe will not run.

Inode Table Size (NINODE) - This is a cache of the active inode entries. If this table fills up then performance slows down.

File Table Size (NFILE) - This is the maximum number of files that can be open at one time. If the table fills up the open will fail.

Callout Table Size (NCALLOUT) - This is the maximum number of timers that can be active at one time. Since timers are used heavily by device drivers to monitor I/O devices, the system will crash if this table fills up.

General Default Calculations (may be platform Specific)

Field Calculation

Nproc 20+(8*maxusers)

Ntext 36+maxusers

Ninode Nproc+80+(13*maxusers)

Nfile 16*(Nproc+16+maxusers)/10+64

Ncallout 16+Nproc

Other kernel settings that should be reviewed are the number of processes per user, maximum open files per user and maximum mounted filesystems. All of these can have unpredictable effects on performance.

If none of the above provides a reasonable clue to what is going on it is necessary to bring out the most powerful tool of all - trace. Trace will provide indepth information on everything that is going on in the system. However, it will definitely affect performance and thus, should be used judiciously.

As can be seen above there is a great deal of information that can be gleaned from the system for relatively minimal effort. Figure 1 contains some of the Rules of Thumb (ROT) that are useful along with what they apply to and the tool that best provides the information. These ROTs can then be used as follows to diagnose and fix performance problems in UNIX systems.

So to reiterate: first iostat, sar and uptime are run to determine whether it appears to be a CPU problem. If it is CPU then it is possible to try profiling, time/timex, optimization, priority changing or a CPU upgrade. If the problem is not CPU then it is necessary to investigate for possible I/O problems further using iostat, and then filemon, lsvg, lsdev, lsattr, lspv and lslv. I/O solutions include adding disk space and reorganizing filesystems.

If the I/O breakdown indicates the problem is with paging (page lv>30%) then svmon, lsps or pstat should be used. Possible solutions include adding memory or disk space. If the system does not appear to be disk bound then it is time to check for remote file I/O problems using nfsstat, netpmon and netstat. Finally, if none of these identify the problem it is time to resort to trace.

By taking such a structured approach to problem diagnosis it is possible to rapidly isolate the problem area. Taking these measurements when the system is behaving normally is also a useful option as this provides a baseline to compare future measurements with.

To do performance measurement properly it is helpful to automate the reporting process using a scripting language (such as perl) combined with scheduling commands such as at or cron. These languages can als be used to create graphical representations of the output from the tools.

By using the above mentioned tools and methodology, it is possible to diagnose performance problems on most UNIX systems, using non-proprietary tools that come standard with the system.

References:

1. System Performance Tuning, Mike Loukides, O'Reilly and Associates

2. SC23-2365, Performance Monitoring and Tuning Guide, IBM

Figure 1: Rules of Thumb

Type ROT Tool
CPU (%user+%sys)>80% iostat
Memory fre < .1(avm) or >=110-120 vmstat
Paging 5 pageins/sec
Disk PV util > 80%

IOWait > 40%

Tm_act > 70%

lspv

iostat

iostat

Network retrans > 5% (total calls) nfsstat
CPU Runq runqsz <2 is good sar -q
System Load Average (CPU) > 10 - very bad

4-7 - fairly heavy

<3 - light load

Single user station <2

[Mar 14, 2010] Performance Management Guide - Using the vmstat, iostat, netstat, and sar Commands

The vmstat, iostat, netstat, and sar commands have functional characteristics that make them useful for continuous monitoring of system performance. These commands can:

The following example shows samples of the periodic reports produced by these programs.

# vmstat 5 2
kthr     memory             page              faults        cpu
----- ----------- ------------------------ ------------ -----------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa
 0  1 75318   142   0   0   0   0    0   0 299 1845 315  5  2 93  0
 0  1 75318   141   0   0   0   0    0   0 626 4949 842  8  6 87  0

See The vmstat Command (CPU), The vmstat Command (Memory), and Assessing Disk Performance with the vmstat Command for detailed discussions of the vmstat command.

# iostat 5 2
 
tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0          0.5               5.2      1.8       92.7       0.3
                " Disk history since boot not available. "
 
 
tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0         41.4               8.1      5.5       84.2       2.2
 
Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0           7.1      28.4       6.9          0       144
hdisk1           0.0       0.0       0.0          0         0
cd0              0.0       0.0       0.0          0         0

The system maintains a history of disk activity. If the history is disabled (smitty chgsys -> Continuously maintain DISK I/O history [false]) you see this message when running the iostat command: Disk history since boot not available. The interval disk I/O statistics are unaffected by this. For detailed discussion of this command, see The iostat Command and Assessing Disk Performance with the iostat Command.

# netstat -I tr0 5
    input   (tr0)      output           input   (Total)    output
 packets  errs  packets  errs colls  packets  errs  packets  errs colls
  725227     0   445748     0     0   799996     0   520517     0     0
       0     0        0     0     0        0     0        0     0     0
       2     0        0     0     0        2     0        0     0     0
CTRL C

Other useful netstat command options to use are -s and -v. See The netstat Command for details.

# sar -P ALL 5 2
 
AIX rugby 3 4 00058033A100    12/01/99
 
11:17:41 cpu    %usr    %sys    %wio   %idle
11:17:46  0        0       0       0     100
          1        0       0       0     100
          2        0       0       0     100
          3        0       0       0     100
          -        0       0       0     100
11:17:51  0        0       0       0     100
          1        0       0       0     100
          2        0       0       0     100
          3        0       0       0     100
          -        0       0       0     100
 
Average   0        0       0       0     100
          1        0       0       0     100
          2        0       0       0     100
          3        0       0       0     100
          -        0       0       0     100

For details on the sar command, see The sar Command and Assessing Disk Performance with the sar Command.

Remember that the first report from the vmstat, iostat, and netstat commands is for cumulative activity since the last system boot. The second report shows activity for the first 5-second interval. The sar command does not report the cumulative activity since the last system boot.

These commands are the basic foundation on which to construct a performance-monitoring mechanism. Shell scripts can be written to perform data reduction on the command output and warn of performance problems or record data on the status of the system when a problem is occurring. For example, a shell script could test the CPU idle percentage for zero and execute another shell script when that CPU-saturated condition occurred. A script such as:

# ps -ef | egrep -v "STIME|$LOGNAME" | sort +3 -r | head -n 15

would record the 15 active processes that had consumed the most CPU time recently (other than the processes owned by the user of the script).

Depending on the required level of sophistication, creating such a family of shell scripts can be a substantial project. Fortunately, there are packages available that require less development and setup and have considerably more function than the typical installation would want to implement locally.

Reference

Linux

iostat - Report Central Processing Unit (CPU) statistics and input/out- put statistics for devices and partitions.
       iostat [ -c | -d ] [ -k ] [ -t ] [ -V ] [ -x ] [ { device [ ... ] | ALL
       } ] [ -p [ { device | ALL } ] ] [ interval [ count ] ]

The iostat command is used for monitoring system input/output device loading by observing the time the devices are active in relation to their average transfer rates. The iostat command generates reports that can be used to change system configuration to better balance the input/output load between physical disks.

The first report generated by the iostat command provides statistics concerning the time since the system was booted. Each subsequent report covers the time since the previous report. All statistics are reported each time the iostat command is run. The report consists of a CPU header row followed by a row of CPU statistics. On multiprocessor sys- tems, CPU statistics are calculated system-wide as averages among all processors. A device header row is displayed followed by a line of statistics for each device that is configured.

The interval parameter specifies the amount of time in seconds between each report. The first report contains statistics for the time since system startup (boot). Each subsequent report contains statistics col- lected during the interval since the previous report. The count parame- ter can be specified in conjunction with the interval parameter. If the count parameter is specified, the value of count determines the number of reports generated at interval seconds apart. If the interval parame- ter is specified without the count parameter, the iostat command gener- ates reports continuously.

The iostat command generates two types of reports, the CPU Utilization report and the Device Utilization report.

CPU Utilization Report

The first report generated by the iostat command is the CPU Uti- lization Report. For multiprocessor systems, the CPU values are global averages among all processors. The report has the fol- lowing format:

Device Utilization Report

The second report generated by the iostat command is the Device Utilization Report. The device report provides statistics on a per physical device or partition basis. Block devices for which statistics are to be displayed may be entered on the command line. Partitions may also be entered on the command line provid- ing that option -x is not used. If no device nor partition is entered, then statistics are displayed for every device used by the system, and providing that the kernel maintains statistics for it. If the ALL keyword is given on the command line, then statistics are displayed for every device defined by the system, including those that have never been used. The report may show the following fields, depending on the flags used:

OPTIONS

The iostat command takes into account the following environment variable:

S_TIME_FORMAT If this variable exists and its value is ISO then the current locale will be ignored when printing the date in the report header. The iostat command will use the ISO 8601 format (YYYY- MM-DD) instead.

Recommended Links

2.5.4.1. The iostat command

Linux System Administrator's Guide - 7.1.2. The iostat command.

Adam's Tech Blog " How to read the iostat command

Solaris Performance Monitoring & Tuning – iostat, vmstat, netstat

Using the UNIX iostat utility to find disk bottlenecks in Oracle

The Approach to Locating Hot Disks - Locating Hot Disks - Hot Disks

How to Install iostat Using yum

Performance Management Guide - Monitoring Disk I/O

> Hunting I/O Bottlenecks with iostat | Linux Magazine

Examples

iostat Display a single history since boot report for all CPU and Devices.

iostat -d 2 Display a continuous device report at two second intervals.

iostat -d 2 6 Display six reports at two second intervals for all devices.

iostat -x hda hdb 2 6 Display six reports of extended statistics at two second inter- vals for devices hda and hdb.

iostat -p sda 2 6 Display six reports at two second intervals for device sda and all its partitions (sda1, etc.)

AIX

Help -

Reports Central Processing Unit (CPU) statistics, asynchronous input/output (AIO) and input/output statistics for the entire system, adapters, TTY devices, disks CD-ROMs, tapes and file systems.

Syntax

iostat [ -a ] [ -l ] [ -s ] [ -t ] [ -T ] [ -V] [ -z ] [ { -A [ -P ] [ -q | -Q ] } | { [ -d | -p] -D [ -R ] }[ -m ] ] [ { -f | -F } [ filesystems,... ] ][ -S power] [ -@ wparname | ALL | Global ] [ drives ... ] [ interval] [ count ]

Restriction: The -a, -t, -z, -A, -P, -q, -Q, -p, -R, -m flags and the wparname parameter are restricted inside workload partitions.

Description

The iostat command is used to monitor system input/output (I/O) devices (physical and logical) that are loaded, by observing the time for which these devices are active. The iostat command also generates reports that can be used to change system configuration to better balance the I/O load between file systems, physical volumes, and adapters.

The iostat command generates various utilization and throughput reports based on the options that you specify. On multiprocessor systems, CPU statistics are calculated system-wide as averages among all processors.

A report generated by the iostat command consists of system configuration information and various utilization and throughput reports. The system configuration row displays at the start of the iostat command and whenever there is a change in monitored configuration. In addition to system configuration, WPAR configuration is also displayed for the WPAR that has enforced resource limits when the -@ flag is used.

The system configuration and WPAR configuration information includes the following values:

lcpu
Indicates the number of logical CPUs.
drives
Indicates the number of disks (including CDs). This information is displayed only when adapters, disks, or CDs are monitored.
tapes
Indicates the number of tapes. This information is displayed only when adapters or tapes are monitored.
ent
Indicates the entitled capacity. This information is displayed only when the partition is running with shared processor.
vdisk
Indicates the number of virtual disks. This information is displayed only when adapters, disks, or CDs are monitored.
wpars
Indicates the number of active system workload partitions. This information is displayed only when you specify the -@ flag.
maxserver
Indicates the maximum number of AIO servers that can serve slow-path IOs. This is a system-wide value. It is displayed only if asynchronous I/O is monitored.
cpulim
Indicates the processor-resource limit for a WPAR in terms of processor units. This information is displayed only for WPARs with enforced processor-resource limit.
rset
Indicates the resource-set type (regular or exclusive) that is associated with the WPAR. This information is displayed only when there is a resource set that is associated with the WPAR.

The Interval parameter specifies the amount of time in seconds between each report. If the Interval parameter is not specified, the iostat command generates a single report containing statistics for the time since system startup (boot). The Count parameter can be specified in conjunction with the Interval parameter. If the Count parameter is specified, the value of count determines the number of reports generated at Interval seconds apart. If the Interval parameter is specified without the Count parameter, the iostat command generates reports continuously.

The iostat command is useful in determining whether a physical volume is becoming a performance bottleneck and if there is potential to improve the situation. The % utilization field for the physical volumes indicates how evenly the file activity is spread across the drives. A high % utilization on a physical volume is a good indication that there may be contention for this resource. Since the CPU utilization statistics are also available with the iostat report, the percentage of time the CPU is in I/O wait can be determined at the same time. Consider distributing data across drives if the I/O wait time is significant and the disk utilization is not evenly distributed across volumes.

Beginning with AIX 5.3, the iostat command reports number of physical processors consumed (physc) and the percentage of entitlement consumed (% entc) in Micro-Partitioning environments. These metrics will only be displayed on Micro-Partitioning environments.

Note:

Some system resource is consumed in maintaining disk I/O history for the iostat command. Use the sysconfig subroutine, or the SMIT to stop history accounting. While the iostat command is running for Count of iterations and if there is a change in system configuration that affects the output of iostat command, it prints a warning message about the configuration change. It then continues the output after printing the updated system configuration information and the header.

If you specify the -a flag, the information is displayed in a report in the following order:

If the Drive parameter is specified, only those names specified are displayed. One or more alphabetic or alphanumeric values can be specified for Drives. If you specify the Drive parameter, the TTY and CPU reports are displayed and the disk or tape report contains statistics for the specified drives. If a drive name that you specified is not found, the report lists that name and displays the message Drive Not Found. If you did not configure drives on the system, no disk or tape report is generated.

Restriction: The first character in the Drive parameter cannot be numeric.

Tape utilization report is generated only if you specified the -p or -a flag.

Note:

The -@ option is not supported when executed within a workload partition.

Reports

The iostat command generates four types of reports, the TTY and CPU utilization report, the disk/tape utilization report, the file system utilization report, the system throughput report and the adapter throughput report.

Tips:

TTY and CPU Utilization Report

The first report generated by the iostat command is the TTY and CPU utilization report. For multiprocessor systems, the CPU values are global averages among all processors. Also, the I/O wait state is defined system-wide and not per processor. The report has the following format:

Column Description
tin Shows the total number of characters read by the system for all TTYs.
tout Shows the total number of characters written by the system to all TTYs.
% user Shows the percentage of CPU utilization that occurred while executing at the user level (application).
% sys Shows the percentage of CPU utilization that occurred while executing at the system level (kernel).
% idle Shows the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.
% iowait Shows the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
physc Shows the number or the fraction of physical processors consumed, displayed only if the partition is running with shared processor.
% entc Shows the percentage of entitled capacity consumed, which is displayed only if the partition is running with shared processor. Because the time base over which this data is computed can vary, the entitled capacity percentage can sometimes exceed 100%. This excess is noticeable only with small sampling intervals.
% rc Shows the percentage of the consumed processor resource. The information is displayed only for WPARs with enforced processor-resource limit.

This information is updated at regular intervals by the kernel (typically sixty times per second). The TTY report provides a collective account of characters per second received from all terminals on the system as well as the collective count of characters output per second to all terminals on the system.

Methods Used to Compute CPU Disk I/O Wait Time

Operating system version 4.3.3 and later contain enhancements to the method used to compute the percentage of CPU time spent waiting on disk I/O (wio time). The method used in AIX 4.3.2 and earlier versions of the operating system can, under certain circumstances, give an inflated view of wio time on SMPs. The wio time is reported by the commands sar (%wio), vmstat (wa) and iostat (% iowait).

The method used in AIX 4.3.2 and earlier versions is as follows: At each clock interrupt on each processor (100 times a second per processor), a determination is made as to which of the four categories (usr/sys/wio/idle) to place the last 10 ms of time. If the CPU was busy in usr mode at the time of the clock interrupt, then usr gets the clock tick added into its category. If the CPU was busy in kernel mode at the time of the clock interrupt, then the sys category gets the tick. If the CPU was not busy, a check is made to see if any I/O to disk is in progress. If any disk I/O is in progress, the wio category is incremented. If no disk I/O is in progress and the CPU is not busy, the idle category gets the tick. The inflated view of wio time results from all idle CPUs being categorized as wio regardless of the number of threads waiting on I/O. For example, systems with just one thread doing I/O could report over 90 percent wio time regardless of the number of CPUs it has.

The method used in AIX 4.3.3 and later is as follows: The change in operating system version 4.3.3 is to only mark an idle CPU as wio if an outstanding I/O was started on that CPU. This method can report much lower wio times when just a few threads are doing I/O and the system is otherwise idle. For example, a system with four CPUs and one thread doing I/O will report a maximum of 25 percent wio time. A system with 12 CPUs and one thread doing I/O will report a maximum of 8 percent wio time. NFS client reads/writes go through the VMM, and the time that biods spend in the VMM waiting for an I/O to complete is now reported as I/O wait time.

Disk/Tape Utilization Report

The second report generated by the iostat command is the disk/tape utilization report. By default, the disk utilization report is displayed, and you must specify the -p flag to display the tape utilization report.

When you specify the -m flag, the path utilization report is displayed.

The disk report provides statistics on a per-physical-disk basis, and tape utilization report provides statistics on a per-tape-basis. The default report has the following format:

% tm_act Indicates the percentage of time the physical disk/tape was active (bandwidth utilization for the drive).
Kbps Indicates the amount of data transferred (read or written) to the drive in KB per second.
tps Indicates the number of transfers per second that were issued to the physical disk/tape. A transfer is an I/O request to the physical disk/tape. Multiple logical requests can be combined into a single I/O request to the disk. A transfer is of indeterminate size.
Kb_read The total number of KB read.
Kb_wrtn The total number of KB written.

If you specify the -D flag, the report has the following metrics for disk/tape. Extended metrics for disk are displayed by default and users need to specify the -p option for tape utilization report:

Metrics related to transfers (xfer):
% tm_act Indicates the percentage of time the physical disk/tape was active (bandwidth utilization for the drive).
bps Indicates the amount of data transferred (read or written) per second to the drive. Different suffixes are used to represent the unit of transfer. Default is in bytes per second.
tps Indicates the number of transfers per second that were issued to the physical disk/tape. A transfer is an I/O request to the physical disk/tape. Multiple logical requests can be combined into a single I/O request to the disk. A transfer is of indeterminate size.
bread Indicates the amount of data read per second, from the drive. Different suffixes are used to represent the unit of transfer. Default is in bytes per second.
bwrtn Indicates the amount of data written per second, to the drive. Different suffixes are used to represent the unit of transfer. Default is in bytes per second.

Read Service Metrics (read):
rps Indicates the number of read transfers per second.
avgserv Indicates the average service time per read transfer. Different suffixes are used to represent the unit of time. Default is in milliseconds.
minserv Indicates the minimum read service time. Different suffixes are used to represent the unit of time. Default is in milliseconds.
maxserv Indicates the maximum read service time. Different suffixes are used to represent the unit of time. Default is in milliseconds.
timeouts Indicates the number of read timeouts per second.
fails Indicates the number of failed read requests per second.

Write Service Metrics (write):
wps Indicates the number of write transfers per second.
avgserv Indicates the average service time per write transfer. Different suffixes are used to represent the unit of time. Default is in milliseconds.
minserv Indicates the minimum write service time. Different suffixes are used to represent the unit of time. Default is in milliseconds.
maxserv Indicates the maximum write service time. Different suffixes are used to represent the unit of time. Default is in milliseconds.
timeouts Indicates the number of write timeouts per second.
fails Indicates the number of failed write requests per second.

Wait Queue Service Metrics (queue): Restriction: These metrics are not applicable for tapes.
avgtime Indicates the average time spent by a transfer request in the wait queue. Different suffixes are used to represent the unit of time. Default is in milliseconds.
mintime Indicates the minimum time spent by a transfer request in the wait queue. Different suffixes are used to represent the unit of time. Default is in milliseconds.
maxtime Indicates the maximum time spent by a transfer request in the wait queue. Different suffixes are used to represent the unit of time. Default is in milliseconds.
avgwqsz Indicates the average wait queue size.
avgsqsz Indicates the average service queue size.
sqfull Indicates the number of times the service queue becomes full (that is, the disk is not accepting any more service requests) per second.

Legend of suffixes representing different units of representation
Suffix Description
K 1000 bytes
M 1 000 000 bytes if displayed in xfer metrics. Minutes, if displayed in read/write/wait service metrics.
G 1 000 000 000 bytes.
T 1 000 000 000 000 bytes.
S Seconds.
H Hour.
Note:

For drives that do not support service time metrics, read, write and wait queue service metrics will not be displayed.

Statistics for CD-ROM devices are also reported.

System Throughput Report

This report is generated if the -s flag is specified. This report provides statistics for the entire system. This report has the following format:

Kbps Indicates the amount of data transferred (read or written) in the entire system in KB per second.
tps Indicates the number of transfers per second issued to the entire system.
Kb_read The total number of KB read from the entire system.
Kb_wrtn The total number of KB written to the entire system.

Tip: The -s flag, when used with the -@ or -f flag, displays logical and physical volume throughput, which corresponds to File Systems and Disks respectively.

Adapter Throughput Report

This report is generated if the -a flag is specified. This report provides statistics on an adapter-by-adapter basis (for both physical and virtual adapters). This report has the following format for a physical adapter report:

Kbps Indicates the amount of data transferred (read or written) in the adapter in KB per second.
tps Indicates the number of transfers per second issued to the adapter.
Kb_read The total number of KB read from the adapter.
Kb_wrtn The total number of KB written to the adapter.

The virtual adapter's default throughput report has the following format:

Kbps Indicates the amount of data transferred (read or written) in the adapter in KB per second.
tps Indicates the number of transfers per second issued to the adapter.
bkread Number of blocks received per second from the hosting server to this adapter.
bkwrtn Number of blocks per second sent from this adapter to the hosting server.
partition-id The partition ID of the hosting server, which serves the requests sent by this adapter.

The virtual adapter's extended throughput report (-D option) has the following format:

Metrics related to transfers (xfer:)
Kbps Indicates the amount of data transferred (read or written) in the adapter in KB per second.
tps Indicates the number of transfers per second issued to the adapter.
bkread Number of blocks received per second from the hosting server to this adapter.
bkwrtn Number of blocks per second sent from this adapter to the hosting server.
partition-id The partition ID of the hosting server, which serves the requests sent by this adapter.

Adapter Read Service Metrics (read:)
rps Indicates the number of read requests per second.
avgserv Indicates the average time to receive a response from the hosting server for the read request sent. Different suffixes are used to represent the unit of time. Default is in milliseconds.
minserv Indicates the minimum time to receive a response from the hosting server for the read request sent. Different suffixes are used to represent the unit of time. Default is in milliseconds.
maxserv Indicates the maximum time to receive a response from the hosting server for the read request sent. Different suffixes are used to represent the unit of time. Default is in milliseconds.

Adapter Write Service Metrics (write:)
wps Indicates the number of write requests per second.
avgserv Indicates the average time to receive a response from the hosting server for the write request sent. Different suffixes are used to represent the unit of time. Default is in milliseconds.
minserv Indicates the minimum time to receive a response from the hosting server for the write request sent. Different suffixes are used to represent the unit of time. Default is in milliseconds.
maxserv Indicates the maximum time to receive a response from the hosting server for the write request sent. Different suffixes are used to represent the unit of time. Default is in milliseconds.

Adapter Wait Queue Metrics (queue:)
avgtime Indicates the average time spent by a transfer request in the wait queue. Different suffixes are used to represent the unit of time. Default is in milliseconds.
mintime Indicates the minimum time spent by a transfer request in the wait queue. Different suffixes are used to represent the unit of time. Default is in milliseconds.
maxtime Indicates the maximum time spent by a transfer request in the wait queue. Different suffixes are used to represent the unit of time. Default is in milliseconds.
avgwqsz Indicates the average wait queue size.
avgsqsz Indicates the average service queue size.
sqfull Indicates the number of times the service queue becomes full (that is, the hosting server is not accepting any more service requests) per second.

Legend of suffixes representing different units of representation
Suffix Description
K 1000 bytes.
M 1 000 000 bytes if displayed in xfer metrics. Minutes, if displayed in read/write/wait service metrics.
G 1 000 000 000 bytes.
T 1 000 000 000 000 bytes.
S Seconds.
H Hours.
Asynchronous I/O Report

The asynchronous I/O report has the following column headers :

avgc Average global AIO request count per second for the specified interval.
avfc Average fastpath request count per second for the specified interval.
maxgc Maximum global AIO request count since the last time this value was fetched.
maxfc Maximum fastpath request count since the last time this value was fetched.
serv Number of active AIO servers.
File System Utilization Report

The file system utilization report provides statistics on a per-file-system basis. The default report has the following format:

Filesystem Indicates the file system name.
% tm_act Indicates the percentage of time the file system is active.
Kbps Indicates the amount of data transferred (read or written) to the file system in KB per second.
Tps Indicates the number of transfers per second that are issued to the file system. A transfer is of indeterminate size.
Kb_read The total number of KBs read.
Kb_wrtn The total number of KBs written.

Important: You must specify the disk names before you invoke the -f or -F flag. If you specify the -f or -F flag, separate file system names to be monitored by commas.

Flags

-a Displays the adapter throughput report. The -a flag can be specified with the -A flag, but not when the -q or -Q flag is specified. The -a flag is mutually exclusive with the -f or -F flag.
-A Displays the asynchronous IO utilization report, and turns off the display of TTY utilization report.
-d Turns off the display of TTY utilization report or CPU utilization report. If you do not specify the -d or -p flag, then by default the -d flag is turned on. The -t and -d flags together turn off both disks and TTY or CPU statistics, allowed only with the -a or -s flags. The -d flag is mutually exclusive with the -t flag unless you specify the -a or -s flag, too. The -d flag is mutually exclusive with the -p flag unless you specify the -a or -s flag, too.
-D Displays the extended tape/drive utilization report. Use the -D flag with the -d or -p flag. The -D flag is mutually exclusive with the -t flag unless you specify the -a or -s flag, too. The -D flag is mutually exclusive with the -f or -F flag.
-f Displays the file system utilization report. The -f flag is mutually exclusive with the -a or -D flag. The -f flag can be specified with the -A flag, but not when the -q or -Q flag is specified.
-F Displays the file system utilization report, and turns off other utilization reports. The -F flag is mutually exclusive with the -a or -D flag. The -F flag can be specified with the -A flag, but not when the -q or -Q flag is specified.
-l Displays the output in long listing mode.
-m Displays the path utilization report. The -m flag is mutually exclusive with the -t flag.
-p Displays the tape utilization report. The -p flag is mutually exclusive with the -d flag unless you specify the -a or -s flag, too.

Note:

Only the Atape device utilization is reported.
-P Displays the adapter throughput report, using the POSIX AIO calls.
-q Specifies AIO queues and their request counts. The -q flag can be specified only with -A or -P flag.
-Q Displays a list of all the mounted file systems and the associated queue numbers with their request counts. The -Q flag can be specified only with -A or -P flag.
-R Specifies that the reset of min* and max* values should happen at each interval. The default is to reset the values once when iostat is started. The -R flag can be specified only with the -D flag.
-s Specifies the system throughput report. You can specify the -a flag with the -A flag, but not when you have specified the -q or -Q flag. Inside a workload partition, you can specify the -s flag only with the -f or -F flag.
-S power Displays the processor statistics that are multiplied by a value of 10 power. The default value of the power parameter is 0. The following fields are scaled:
  • % user
  • % sys
  • % idle
  • % iowait
  • physc
  • entc
Note:

By default, the %user, %sys, %idle, and %iowait fields are relative to the processor consumption of a WPAR. When you specify the -S flag with a nonzero power, the %user, %sys, %idle, and %iowait fields are relative to system-wide processor consumption.

Note:

The value of power can only be between 0 and 3.

-t Turns off the display of disk utilization report. The -t and -d flags together turn off both disks and TTY or CPU statistics, allowed only with the -a or -s flags. The -t flag is mutually exclusive with the -d flag unless you specify the -a or -s flag, too. The -t flag is mutually exclusive with the -D flag unless you specify the -a or -s flag, too. The -t flag is mutually exclusive with the -m flag.
-T Displays the time stamp.
-V Displays valid nonzero statistics.
-z Resets the disk input/output statistics. Only root users can use this option.
-@ Reports I/O activities of a workload partition:
  • Specify -@ ALL to display the activity for the global environment and all workload partitions in the system.
  • Specify the -@ flag with a list of workload partition names to display the activity for that workload partition.
  • Specify -@ Global to display the activity for the global environment only.
  • Specify the -@ flag inside a WPAR to display system-wide statistics along with WPAR statistics.

The -@ flag can be specified only with –d and –D, -f or -F flags. All possible combinations of the -s, -T, -f, -F, -d, -D and -l flags are allowed.

Restriction: The -@ flag is mutually exclusive with -a, -t, -z, -A, -P, -q, -Q, and the -m flag.

Security

Attention RBAC users and Trusted AIX users: This command can perform privileged operations. Only privileged users can run privileged operations. For more information about authorizations and privileges, see Privileged Command Database in Security. For a list of privileges and the authorizations associated with this command, see the lssecattr command or the getcmdattr subcommand.

Examples

  1. To display a single history since boot report for all TTY, CPU, and Disks, enter the following command:

    iostat
  2. To display a continuous disk report at two second intervals for the disk with the logical name disk1, enter the following command:

    iostat -d disk1 2
  3. To display six reports at two second intervals for the disk with the logical name disk1, enter the following command:

    iostat disk1 2 6
  4. To display six reports at two second intervals for all disks, enter the following command:

    iostat -d 2 6
  5. To display six reports at two second intervals for three disks named disk1, disk2, disk3, enter the following command:

    iostat disk1 disk2 disk3 2 6
  6. To print the System throughput report since boot, enter the following command:

     
    iostat -s
  7. To print the adapter throughput reports at 5-second intervals, enter the following command:

    iostat -a 5
  8. To print 10 system and adapter throughput reports at 20-second intervals, with only the TTY and CPU report (no disk reports), enter the following command:

      
    iostat -sat 20 10
  9. To print the system and adapter throughput reports with the disk utilization reports of hdisk0 and hdisk7 every 30 seconds, enter the following command:

    iostat -sad hdisk0 hdisk7 30
  10. To display time stamp next to each line of output of iostat, enter the following command:
    iostat -T 60
  11. To display 6 reports at 2-second intervals on AIO, enter the following command:
    iostat -A 2 6 
  12. To display AIO statistics since boot for queues associated with all mounted file systems, enter the following command:
    iostat -A -Q
  13. To display extended drive report for all disks, enter the following command:
    iostat -D 
  14. To display extended drive report for all tapes, enter the following command:
    iostat -Dp
  15. To display extended drive report for a specific disk, enter the following command:
    iostat –D hdisk0 
  16. To reset the disk input/output statistics, enter the following command:
    iostat –z 
  17. To display only file system statistics for all workload partitions, enter the following command:
    iostat -F -@ ALL 
  18. To display system throughput of all workload partitions along with the system, enter the following command:
    iostat -f -s -@ ALL
  19. To display file system statistics that are appended with default O/P, enter the following command:
    iostat -f
  20. To display logical and physical system throughput, enter the following command:
    iostat -s -f
  21. To display throughput for user-specified drives and file systems, enter the following command:
    iostat hdisk0 hdisk1 -f /dev/fslv00 /dev/fslv01 /dev/fslv02
  22. To display the processor statistics that are multiplied by a factor of 10, enter the following command:
    iostat -S 1

File

/usr/bin/iostat Contains the iostat command.

Related Information

The vmstat command.

The iostadd Kernel Service.

Monitoring disk I/O in Performance management

The Input and Output Handling Programmer's Overview in AIX Version 6.1 General Programming Concepts: Writing and Debugging Programs describes the files, commands, and subroutines used for low-level, stream, terminal, and asynchronous I/O interfaces.



Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D


Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to to buy a cup of coffee for authors of this site

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: July, 10, 2020