|
Home | Switchboard | Unix Administration | Red Hat | TCP/IP Networks | Neoliberalism | Toxic Managers |
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and bastardization of classic Unix |
News | Recommended Books | Recommended Links | Linux | Solaris | Examples | |
vmstat | netstat | iostat | uptime command | sar | Linux Performance Tuning | |
watch | Horror Stories | Unix History | Humor | Etc |
|
iostat is a low overhead tool that can be automated and provides useful report about I/O activity on the server. The command is pretty simple to use. It provides the defalt report if invoked without any parameters. You can also specify several parameters
iostat -d -x interval count
- -d : Display the device utilization report (d == disk)
- -x : Display extended statistics including disk utilization
- interval : It is time period in seconds between two samples . iostat 2 will give data at each 2 seconds interval.
- count : It is the number of times the data is needed . iostat 2 5 will give data at 2 seconds interval 5 times
|
The default report provides an overview of CPU and disk I/O statistics:
Linux 2.6.9-5.0.3.EL_blb_P4 (foobar.example.com) 02/23/2005 avg-cpu: %user %nice %sys %iowait %idle 0.14 0.00 0.22 2.37 97.27 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hda 0.74 33.25 99.74 2344531 7031970 hdb 2.08 214.19 288.00 15100778 20304528 hdd 2.03 200.41 288.00 14128983 20304528 md0 39.28 414.59 287.81 29229074 20290864 sda 0.00 0.01 0.00 408 0
Below the first line (which contains the system's kernel version and hostname, along with the current date), iostat displays an overview of the system's average CPU utilization since the last reboot. The CPU utilization report includes the following percentages:
Below the CPU utilization report is the device utilization report. This report contains one line for each active disk device on the system and includes the following information:
This is just a one report that can be obtained using iostat. Different reports can be obtained using different options.
Since different application processes have different I/O patterns, I/O bottlenecks may appear on different disks at different times during each day. It is important to remember that these bottlenecks are transient in nature. A disk may experience an I/O bottleneck for a very short period, and this short-duration bottleneck may repeat itself many times each day. Summarizing I/O by the hour does not provide adequate information about transient bottlenecks as I/O spikes with duration of say 10 min or less disappeared in the hourly average.
To get the most accurate results, you should collect I/O statistics at more frequent intervals—preferably no more than ten minutes between samples—over a representative time period, such as a week. Because individual application processes have different I/O patterns, bottlenecks may appear on different disks at various times during each day. And because Oracle transactions happen very quickly, a disk may experience an I/O bottleneck for a very short period—but a short-duration bottleneck may nonetheless repeat itself thousands of times each day. If you make the mistake of summarizing I/O by the hour, as many DBAs do, you won't see these bottlenecks because the I/O spikes will not be evident in the hourly average.
The goal of load balancing is to distribute the files across disks so as to achieve a single static optimal I/O throughput. Once the I/O subsystem is balanced, the files will not need to be moved unless new processes change the I/O pattern for the disks.
The goal is to find the optimal file placement where overall load balance is achieved for all of the many variations of disk access. Load balancing is essentially the identification of hot disks, and the movement of datafiles to less-used cool disks. As such, disk load balancing is an iterative process since it is possible that relocating a datafile may relieve contention for one process, only to cause I/O contention for an unrelated process. Also, for databases placed on a small number of disks, it is possible that I/O contention cannot be avoided. Consider a 30GB database spread across disks with 20 competing processes for data. On average, ten processes would be queued waiting for I/O from each of the two disks. Clearly, these types of systems will always experience I/O contention.
After using data collected by iostat to identify a hot disk, you would use data collected by the Oracle utilities to identify which mount point and file contain the table causing the excessive I/O activity.
Identifying the hot disk is only the beginning of the quest. We must then see what mount point on the disk is causing the problem, which datafile on the mount point, and finally, what Oracle table is causing the excessive I/O. Only with this approach can the Oracle administrator fully understand how to perform disk load balancing. With that in mind, let’s look at the first method for collecting Oracle I/O statistics. We will then move on to look at collecting UNIX I/O statistics.
|
Switchboard | ||||
Latest | |||||
Past week | |||||
Past month |
Jul 10, 2020 | www.redhat.com
I/O reporting from the Linux command line Learn the iostat tool, its common command-line flags and options, and how to use it to better understand input/output performance in Linux.
Posted: July 9, 2020 | by Tyler Carrigan (Red Hat)
Image by Pexels More Linux resources
- Download Now: Linux Commands Cheat Sheet
- Advanced Linux Commands Cheat Sheet for Developers
- Download Red Hat Enterprise Linux Server 8 Trial
- Linux System Administration Skills Assessment
If you have followed my posts here at Enable Sysadmin, you know that I previously worked as a storage support engineer. One of my many tasks in that role was to help customers replicate backups from their production environments to dedicated backup storage arrays. Many times, customers would contact me concerned about the speed of the data transfer from production to storage.
Now, if you have ever worked in support, you know that there can be many causes for a symptom. However, the throughput of a system can have huge implications for massive data transfers. If all is well, we are talking hours, if not... I have seen a single replication job take months.
We know that Linux is loaded full of helpful tools for all manner of issues. For input/output monitoring, we use the
Installation and base runiostat
command.iostat
is a part of thesysstat
package and is not loaded on all distributions by default.I am using Red Hat Enterprise Linux 8 here and have included the install output below.
[ Want to try out Red Hat Enterprise Linux? Download it now for free. ]
NOTE : the command runs automatically after installation.
[root@rhel ~]# iostat bash: iostat: command not found... Install package 'sysstat' to provide command 'iostat'? [N/y] y * Waiting in queue... The following packages have to be installed: lm_sensors-libs-3.4.0-21.20180522git70f7e08.el8.x86_64 Lm_sensors core libraries sysstat-11.7.3-2.el8.x86_64 Collection of performance monitoring tools for Linux Proceed with changes? [N/y] y * Waiting in queue... * Waiting for authentication... * Waiting in queue... * Downloading packages... * Requesting data... * Testing changes... * Installing packages... Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test) 06/17/2020 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 2.17 0.05 4.09 0.65 0.00 83.03 Device tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 206.70 8014.01 1411.92 1224862 215798 sdc 0.69 20.39 0.00 3116 0 sdb 0.69 20.39 0.00 3116 0 dm-0 215.54 7917.78 1449.15 1210154 221488 dm-1 0.64 14.52 0.00 2220 0If you run the base command without options,
iostat
displays CPU usage information. It also displays I/O stats for each partition on the system. The output includes totals, as well as per second values for both read and write operations. Also, note that the tps field is the total number of Transfers per second issued to a specific device.The practical application is this: if you know what hardware is used, then you know what parameters it should be operating within. Once you combine this knowledge with the output of
Interval runsiostat
, you can make changes to your system accordingly.It can be useful in troubleshooting or data gathering phases to have a report run at a given interval. To do this, run the command with the interval (in seconds) at the end:
[root@rhel ~]# iostat -m 10 Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test) 06/17/2020 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 0.94 0.05 0.35 0.04 0.00 98.62 Device tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 12.18 0.44 0.12 1212 323 sdc 0.04 0.00 0.00 3 0 sdb 0.04 0.00 0.00 3 0 dm-0 12.79 0.43 0.12 1197 329 dm-1 0.04 0.00 0.00 2 0 avg-cpu: %user %nice %system %iowait %steal %idle 0.24 0.00 0.15 0.00 0.00 99.61 Device tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 0.00 0.00 0.00 0 0 sdc 0.00 0.00 0.00 0 0 sdb 0.00 0.00 0.00 0 0 dm-0 0.00 0.00 0.00 0 0 dm-1 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 0.20 0.00 0.18 0.00 0.00 99.62 Device tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 0.50 0.00 0.00 0 0 sdc 0.00 0.00 0.00 0 0 sdb 0.00 0.00 0.00 0 0 dm-0 0.50 0.00 0.00 0 0 dm-1 0.00 0.00 0.00 0 0The above output is from a 30-second run.
You must use Ctrl + C to exit the run.
Easy readingTo clean up the output and make it easier to digest, use the following options:
-m
changes the output to megabytes, which is a bit easier to read and is usually better understood by customers or managers.[root@rhel ~]# iostat -m Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test) 06/17/2020 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.51 0.09 0.55 0.07 0.00 97.77 Device tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 22.23 0.81 0.21 1211 322 sdc 0.07 0.00 0.00 3 0 sdb 0.07 0.00 0.00 3 0 dm-0 23.34 0.80 0.22 1197 328 dm-1 0.07 0.00 0.00 2 0
-p
allows you to specify a particular device to focus in on. You can combine this option with the-m
for a nice and tidy look at a particularly concerning device and its partitions.[root@rhel ~]# iostat -m -p sda Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test) 06/17/2020 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.19 0.07 0.45 0.06 0.00 98.24 Device tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 17.27 0.63 0.17 1211 322 sda2 16.83 0.62 0.17 1202 320 sda1 0.10 0.00 0.00 7 2Advanced statsIf the default values just aren't getting you the information you need, you can use the
-x
flag to view extended statistics:[root@rhel ~]# iostat -m -p sda -x Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test) 06/17/2020 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.06 0.06 0.40 0.05 0.00 98.43 Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sda 12.20 2.83 0.54 0.14 0.02 0.92 0.16 24.64 0.55 0.50 0.00 45.58 52.37 0.46 0.69 sda2 12.10 2.54 0.54 0.14 0.02 0.92 0.16 26.64 0.55 0.47 0.00 45.60 57.88 0.47 0.68 sda1 0.08 0.01 0.00 0.00 0.00 0.00 0.00 23.53 0.44 1.00 0.00 43.00 161.08 0.57 0.00Some of the options to pay attention to here are:
- avgqu-sz - average queue length of a request issued to the device
- await - average time for I/O requests issued to the device to be served (milliseconds)
- r_await - average time for read requests to be served (milliseconds)
- w_await - average time for write requests to be served (milliseconds)
There are other values present, but these are the ones to look out for.
Shutting downThis article covers just about everything you need to get started with
iostat
. If you have other questions or need further explanations of options, be sure to check out the man page or your preferred search engine. For other Linux tips and tricks, keep an eye on Enable Sysadmin!
The purpose of this paper is to introduce the performance analyst to some of the free tools available to monitor and manage performance on UNIX systems, and to provide a guideline on how to diagnose and fix performance problems in that environment. The paper is based on the authors experiences with AIX and will cover many of the tools available on that and other UNIX platforms. It will also provide some Rules of Thumb for analyzing the performance of UNIX systems.
As more mission critical work finds its way from the mainframe to distributed systems, performance management for those systems is becoming more important. The goal for systems management is not only to maximize system throughput, but also to reduce response time. In order to this it is necessary to not only work on the system resources, but also to work on profiling and tuning applications.
In UNIX there are 7 major resource types that need to be monitored and tuned - CPU, memory, disk space and arms, communications lines, I/O Time, Network Time and applications programs. There are also standard rules of thumb in most of these areas. From the users perspective the only one they see is total execution time so we will start by looking at that.
Total execution time from a users perspective consists of wall-clock time. At a process level this is measured by running the time command. This provides you with real time (wallclock), user code CPU and system code CPU. If user + sys > 80% then there is a good chance the system is CPU constrained. The components of total running time include:
1. User-state CPU - the actual amount of time the CPU spends running the users program in the user state. It includes time spent executing library calls, but does not include time spent in the kernel on its behalf. This value can be greatly affected by the use of optimization at compile time and by writing efficient code.
2. System-state CPU - this is the amount of time the CPU spends in the system state on behalf of this program. All I/O routines require kernel services. The programmer can affect this value by the use of blocking for I/O transfers.
3. I/O Time and Network Time - these are the amount of time spent moving data and servicing I/O requests.
4. Virtual Memory Performance - This includes context switching and swapping.
5. Time spent running other programs.
In order to measure these areas there are a multitude of tools available. The most useful are:
- cron Process scheduling
- nice/renice Change priorities
- setpri Set priorities
- netstat Network statistics
- nfsstat NFS statistics
- time/timex Process CPU Utilization
- uptime System Load Average
- ps Process Statistics
- iostat BSD tool for I/O
- sar Bulk System Activity
- vmstat BSD tool for V. Memory
- gprof Call Graph profiling
- prof Process Profiling
- trace Used to get more depth
Other commands that will be useful include lsvg, lspv, lslv, lsps and lsdev. Each of these will be discussed below and then a general problem solving approach will be offered. It is important to note that the results and options for all of these commands may differ depending on the platform they are being run on. Most of the options discussed below are those for AIX and some of the tools are specific to AIX such as:
- tprof CPU Usage
- svmon Memory Usage
- filemon Filesystem, LV .. activity
- netpmon Network resources
The first tool to be discussed is uptime. This provides the analyst with the System Load Average (SLA). It is important to note that the SLA can only be used as a rough indicator as it does not take into account scheduling priority and it counts as runnable all jobs waiting for disk I/O, including NFS I/O. However, uptime is a good place to start when trying to determine whether a bottleneck is CPU or I/O based.
When uptime is run it provides three load averages - the first is for the last minute, the second is for the last 5 minutes and the third is for the last 15 minutes. If the value is borderline but has been falling over the last 15 minutes, then it would be wise to just monitor the situation. However, a value between 4 and 7 is fairly heavy and means that performance is being negatively affected. Above 7 means the system needs serious review and below 3 means the workload is relatively light. If the system is a single user workstation then the load average should be less than 2. There is also a command called ruptime that allows you to request uptime information remotely.
The sar command provides a good alternative to uptime with the -q option. It provides statistics on the average length of the run queue, the percentage of time the run queue is occupied, the average length of the swap queue and the percentage of time the swap queue is occupied. The run queue lists jobs that are in memory and runnable, but does not include jobs that are waiting for I/O or sleeping. The run queue size should be less than 2. If the load is high and the runqocc=0 then the problem is most likely memory or I/O, not CPU. The swap queue lists jobs that are ready to run but have been swapped out.
The sar command deserves special mention as it is a very powerful command. The command is run by typing in:
sar -options int #samples
where valid options generally are:
- -g or -p Paging
- -q Average Q length
- -u CPU Usage
- -w Swapping and Paging
- -y Terminal activity
- -v State of kernel tables
After determining that the problem may well be CPU based it would then be necessary to move onto iostat to get more detail. Running iostat provides a great deal of information, but the values of concern here are the %user and %sys. If (%user + %sys) > 80% over a period of time then it is very likely the bottleneck is CPU. In particular it is necessary to watch for average CPU being greater than 70% with peaks above 90%. It is also possible to get similar information by running the ps -au or sar -u commands. Both of these provide information about CPU time. The sar -u command, in particular, breaks the time down into user, system, time waiting for blocked I/O (i.e. NFS, disk, ..) and idle time.
The ps -au command also provides information on the %physical memory the process is using and the current status for the process. Statuses shown are:
Status Meaning
- R Runnable
- T Stopped
- P Waiting on Pagein
- D Waiting on I/O
- S Sleeping < 20 secs
- I Idle - sleeping >20 secs
- Z Zombie or defunct
- W Process is swapped out
- > Mem. soft limit exceeded
- N Niced Process (low pri)
- < Niced Process (high pri)
The cron or at command can be used to automatically schedule execution of these commands to ensure snapshots are taken at the appropriate times. The atq command can be used to list what is in the at queue and the crontab -e command edits the cron table.
Once it has been determined that the problem is a CPU bottleneck there are several options. It is possible to limit the cputime a process can use by the limit command. If the problem relates to one process then it is also possible to model or profile that process using the prof, gprof or tprof command to find out whether it is possible to optimize the program code.
Prof and gprof are very similar and have several disadvantages when compared to tprof. Both prof and gprof require a recompile of the program using wither the -p or the -pg option and they impact performance of that program very badly. Tprof only needs to be recompiled in order to do source code level profiling (-qlist option). In particular tprof exhibits the following characteristics (AIX only):
- No count of routine calls
- No call graph
- Source statement profiling
- Summary of all CPU usage
- No recompile needed for routine level profiling
- No increase in User CPU
- Prof/gprof differ as follows:
- Generally available
- Count of routine calls
- Call graph (gprof)
- Routine level profiling only
- Single Process CPU usage
- Recompile needed
- 10-300% increase in User CPU
So, the recommendation would be to use tprof if it is available on the chosen platform. It is also possible that the vendor will have their own equivalent to tprof.
Running the time or timex commands can also give a good indication of whether the process is CPU intensive. Compiler options have been proven to extensively affect the performance of CPU intensive programs as can be seen from the table below. It is well worth trying different options when compiling the program such as -O, -O2, -O3 and -Q (inline streams the code). Time/timex can give you an indication of how much benefit this will provide. Timex can also be run using the -s option which causes a full set of sar output to be generated for the duration of the programs execution. As can be seen from the table below, it is possible to see reductions in the order of 50% in CPU utilization by using optimization.
User CPU running Program phntest
Compiler Seconds % of CPU
Options for None
None 53.03 100%
-O 26.34 49.67%
-O -Q 25.11 47.35%
-O2 27.04 50.99%
-O2 -Q 24.92 46.99%
-O3 28.48 53.71%
-O3 -Q 26.13 49.27%
It is also possible to change the priority of the process so that other processes can gain control of the CPU ahead of it. This can be done by using the nice, renice or setpri commands. Renice is not available on all platforms. Before using these commands, it is useful to understand how the priority scheme works in UNIX.
Priorities range from 0-127 with 127 being the lowest priority. The actual priority of a task is set by the following calculation:
pri=nice+puser+(cpu/2)
Puser normally defaults to 40 and nice to 20 unless the nice or renice commands have been used against the process. On AIX a tick is 1/100th of a second and new priorities are calculated every tick as follows:
new-pri=(tick/2)+puser+nice
Every second tick is recalculated as tick=tick/2 and then new-pri is again recalculated.
Otherwise, a CPU upgrade may be the only solution if there is no other machine that the workload can be run on.
If the problem does not appear to be CPU then it becomes necessary to investigate memory and I/O as possible causes. Again, it is possible to use iostat or sar to get the information that is needed here. The iowait field shown in the iostat command is a good indicator of whether there are I/O problems. If iowait is greater than 40% then it becomes necessary to investigate the physical volumes to ensure that none of them are greater than 70% full. The lspv command can be used to determine utilization of the physical volume.
Iostat is a low overhead tool that can be automated and provides local counts for I/O data. Unlike sar, iostat does not provide timestamps in the output so it is important to make a note of start/stop times. However, iostat uses kernel data which makes it hardware specific with respect to the results obtained.
Iostat provides data on several important values for each physical disk. These include: %time the physical disk was busy, kilobytes per second to/from the disk, transfers per second to/from, kilobytes read and kilobytes written. This will help to determine if there is an imbalance of I/O amongst the physical volumes. If all appears to be normal here then the next step is to investigate which filesystems the I/O is directed at. If most of the I/O is directed at the page files then memory needs to be investigated.
Information on cylinder access and seek distances is available using the sadp command and cache statistics for disk are available using the sar -b command. Further information can be obtained by running filemon and looking to see what the most active filesystems are.
Filemon provides a list of the most active segments, the most active logical volumes and physical volumes, and detailed statistics for the busiest files as well as the segments, physical and logical volumes. Details include transfers, reads, read sizes, read times in msecs, logical seeks, write times, seek distances, throughput (kb/sec) and utilization percentages. However, it is important to note that filemon runs trace in the background which can affect performance. It is also possible to run fileplace which gives information on space efficiency and sequentiality.
This would be a good time to run lsvg, lslv and lspv to get a map of the layout of the physical and logical volumes on the system as well as the various volume groups. This will make it much simpler to get more indepth information. By running lsdev -C it is also possible to determine what kind of disk devices are installed and what size they are. By using a combination of the above commands a map can be produced for each physical volume of the filesystems and their placement on the disk. The lsattr -E -l sys0 command can be used to obtain information on system parameters such as cache sizes and other associated values.
If the bulk of the I/O (>30%) is going to a logical volume or filesystem that is not used for paging then the problem is most likely user I/O. This can be resolved by one of several options -checking fragmentation, reorganizing the filesystem, adding physical volumes or splitting the data in another manner. Adding memory may still help with the problem.
Other items to take into account when looking at I/O performance include the intra and inter policies, mirroring of disks, write verify and the scheduling policy for logical volumes. It is also important to remember that the SCSI card can only talk to one device at a time. Where multiple disks are behind one SCSI card, sequential readwrites are helped if they are spread across multiple adapters. Newer technologies such as SCSI-2, fast and wide and raid will also help improve performance. Some of the newer controllers also provide buffers for each disk and can perform two way searches.
If the bulk of the I/O is going to paging (i.e. the page LV is > 30%) then it becomes necessary to investigate further. The only options available to cure a paging problem are to write more memory efficient code, move the process to another system, add memory, reschedule the process so it doesn't contend with other memory intensive workloads, or add physical volumes or more page datasets. There are three commands that are used to investigate paging - lsps (or pstat), vmstat and svmon.
lsps -a will provide information on all of the page spaces available including physical volume and utilization. Vmstat is another low overhead tool and provides information on actual memory, free pages, processes on the I/O waitq, reclaims, pageins, pageouts, pages freed by the stealer per second, interrupts, system calls and CPU utilization. Like iostat, vmstat does not provide timestamps. Svmon -G provides similar information except it breaks memory down into work, persistent and client pages that are either in use or pinned. It is also possible to use the sar -w command.
When looking at paging it is important to note that the stealer will run whenever there are only ((2 x real) -8) pages left. So on a 32mb machine the stealer will run if there are only 56 pages left. The Rule of Thumb for page space versus real memory is generally in the order of Page = 2 x real. On some systems not all of the kernel processes are pinned so they can also be paged out. A pagein rate of >5/sec means that the system is memory constrained. Also, if fre is less than (.1(AVM)) then this may indicate that the system is real memory constrained. This depends on the way the VMM uses memory. For instance, AIX will use all memory for disk caching, etc before it reuses any so it is not unusual to see fre very low (110-120). Looking at pageins, pageouts, and the FR to SR ratio is a much more meaningful indicator for problems.
So, if at this point there is no problem with CPU and the system is not disk bound it becomes necessary to investigate the network to check whether it is remote file I/O bound. This is the last step before running the more resource heavy trace command to determine what is really happening. To look at network statistics there are three useful commands - netstat, netpmon and nfsstat.
Netstat -i shows the network interfaces along with input and output packets and errors. It also gives the number of collisions. The Mtu field shows the maximum ip packet size (transfer unit) and should be the same on all systems. In AIX it defaults to 1500. Both Oerrs (number of output errors since boot) and Ierrs (Input errors since boot) should be < 0.025. If Oerrs>0.025 then it is worth increasing the send queue size. Ierrs includes checksum errors and can also be an indicator of a hardware error such as a bad connector or terminator. The Collis field shows the number of collisions since boot and can be as high as 10%. If it is greater then it is necessary to reorganize the network as the network is obviously overloaded on that segment.
Netstat -m s used to analyze the use of mbufs in order to determine whether these are the bottleneck. The no -a command is used to see what the current values are. Values of interest are thewall, lowclust, lowmbuf and dogticks.
An mbuf is a kernel buffer that uses pinned memory and is used to service network communications. Mbufs come in two sizes - 256 bytes and 4096 bytes (clusters of 256 bytes). Thewall is the maximum memory that can be taken up for mbufs. Lowmbuf is the minimum number of mbufs to be kept free while lowclust is the minimum number of clusters to be kept free. Mb_cl_hiwat is the maximum number of free buffers to be kept in the free buffer pool and should be set to at least twice the value of lowclust to avoid thrashing.
Netstat -v is used to look at queues and other information. If Max packets on S/W transmit queue is >0 and is equal to current HW transmit queue length then the send queue size should be increased. If the No mbuf errors is large then the receive queue size needs to be increased.
Nfsstat is used to report on client and server NFS information, primarily at the daemon level. Nfsstat -c provides client information such as retrans and badxid. If badxid=retrans and retrans > 5% of calls the server is the problem, but if retrans > 5% of calls and badxid < retrans then the network is the problem. Also, if there are lots of timeouts then it is useful to increase the number of NFSDs and the qsize.
Netpmon is a further command that focuses on CPU, network adapters, remote nodes and LAN traffic. It is used to get a feeling for what is happening overall. By using a combination of the above commands it is possible to obtain a very clear view of what is happening at the network level.
At this point it is important to mention the UNIX kernel tables, as these can affect performance without any real indicators as to the cause. To find out what they are set to the pstat -T or sar -v commands can be used. Most of the values are calculated based on the value for maxusers so it is important to know what that is set to. It is often recommended that maxusers generally be determined by the following calculation:
Max Users = (2+ # users act + #NFS clients + .5 NFS exports )
In particular, attention should be paid to the following table sizes:
Process Table Size (NPROCS) - this is the maximum number of processes that can be in the system. On systems where Xwindows is heavily used this needs to be increased. If the table is full, then the process will fail to start.
Text Table Size (NTEXT) - This is the maximum number of executables that can be in the system at a time. If the table is full then the exe will not run.
Inode Table Size (NINODE) - This is a cache of the active inode entries. If this table fills up then performance slows down.
File Table Size (NFILE) - This is the maximum number of files that can be open at one time. If the table fills up the open will fail.
Callout Table Size (NCALLOUT) - This is the maximum number of timers that can be active at one time. Since timers are used heavily by device drivers to monitor I/O devices, the system will crash if this table fills up.
General Default Calculations (may be platform Specific)
Field Calculation
Nproc 20+(8*maxusers)
Ntext 36+maxusers
Ninode Nproc+80+(13*maxusers)
Nfile 16*(Nproc+16+maxusers)/10+64
Ncallout 16+Nproc
Other kernel settings that should be reviewed are the number of processes per user, maximum open files per user and maximum mounted filesystems. All of these can have unpredictable effects on performance.
If none of the above provides a reasonable clue to what is going on it is necessary to bring out the most powerful tool of all - trace. Trace will provide indepth information on everything that is going on in the system. However, it will definitely affect performance and thus, should be used judiciously.
As can be seen above there is a great deal of information that can be gleaned from the system for relatively minimal effort. Figure 1 contains some of the Rules of Thumb (ROT) that are useful along with what they apply to and the tool that best provides the information. These ROTs can then be used as follows to diagnose and fix performance problems in UNIX systems.
So to reiterate: first iostat, sar and uptime are run to determine whether it appears to be a CPU problem. If it is CPU then it is possible to try profiling, time/timex, optimization, priority changing or a CPU upgrade. If the problem is not CPU then it is necessary to investigate for possible I/O problems further using iostat, and then filemon, lsvg, lsdev, lsattr, lspv and lslv. I/O solutions include adding disk space and reorganizing filesystems.
If the I/O breakdown indicates the problem is with paging (page lv>30%) then svmon, lsps or pstat should be used. Possible solutions include adding memory or disk space. If the system does not appear to be disk bound then it is time to check for remote file I/O problems using nfsstat, netpmon and netstat. Finally, if none of these identify the problem it is time to resort to trace.
By taking such a structured approach to problem diagnosis it is possible to rapidly isolate the problem area. Taking these measurements when the system is behaving normally is also a useful option as this provides a baseline to compare future measurements with.
To do performance measurement properly it is helpful to automate the reporting process using a scripting language (such as perl) combined with scheduling commands such as at or cron. These languages can als be used to create graphical representations of the output from the tools.
By using the above mentioned tools and methodology, it is possible to diagnose performance problems on most UNIX systems, using non-proprietary tools that come standard with the system.
References:
1. System Performance Tuning, Mike Loukides, O'Reilly and Associates
2. SC23-2365, Performance Monitoring and Tuning Guide, IBM
Figure 1: Rules of Thumb
Type ROT Tool CPU (%user+%sys)>80% iostat Memory fre < .1(avm) or >=110-120 vmstat Paging 5 pageins/sec Disk PV util > 80% IOWait > 40%
Tm_act > 70%
lspv iostat
iostat
Network retrans > 5% (total calls) nfsstat CPU Runq runqsz <2 is good sar -q System Load Average (CPU) > 10 - very bad 4-7 - fairly heavy
<3 - light load
Single user station <2
The vmstat, iostat, netstat, and sar commands have functional characteristics that make them useful for continuous monitoring of system performance. These commands can:
- Produce reports indefinitely at a fixed interval
- Report on activity that varies with different types of load
- Report on activity since the last previous report (except the sar command), so changes in activity are easy to detect
The following example shows samples of the periodic reports produced by these programs.
# vmstat 5 2 kthr memory page faults cpu ----- ----------- ------------------------ ------------ ----------- r b avm fre re pi po fr sr cy in sy cs us sy id wa 0 1 75318 142 0 0 0 0 0 0 299 1845 315 5 2 93 0 0 1 75318 141 0 0 0 0 0 0 626 4949 842 8 6 87 0See The vmstat Command (CPU), The vmstat Command (Memory), and Assessing Disk Performance with the vmstat Command for detailed discussions of the vmstat command.
# iostat 5 2 tty: tin tout avg-cpu: % user % sys % idle % iowait 0.0 0.5 5.2 1.8 92.7 0.3 " Disk history since boot not available. " tty: tin tout avg-cpu: % user % sys % idle % iowait 0.0 41.4 8.1 5.5 84.2 2.2 Disks: % tm_act Kbps tps Kb_read Kb_wrtn hdisk0 7.1 28.4 6.9 0 144 hdisk1 0.0 0.0 0.0 0 0 cd0 0.0 0.0 0.0 0 0The system maintains a history of disk activity. If the history is disabled (smitty chgsys -> Continuously maintain DISK I/O history [false]) you see this message when running the iostat command: Disk history since boot not available. The interval disk I/O statistics are unaffected by this. For detailed discussion of this command, see The iostat Command and Assessing Disk Performance with the iostat Command.
# netstat -I tr0 5 input (tr0) output input (Total) output packets errs packets errs colls packets errs packets errs colls 725227 0 445748 0 0 799996 0 520517 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 2 0 0 0 0 CTRL COther useful netstat command options to use are -s and -v. See The netstat Command for details.
# sar -P ALL 5 2 AIX rugby 3 4 00058033A100 12/01/99 11:17:41 cpu %usr %sys %wio %idle 11:17:46 0 0 0 0 100 1 0 0 0 100 2 0 0 0 100 3 0 0 0 100 - 0 0 0 100 11:17:51 0 0 0 0 100 1 0 0 0 100 2 0 0 0 100 3 0 0 0 100 - 0 0 0 100 Average 0 0 0 0 100 1 0 0 0 100 2 0 0 0 100 3 0 0 0 100 - 0 0 0 100For details on the sar command, see The sar Command and Assessing Disk Performance with the sar Command.
Remember that the first report from the vmstat, iostat, and netstat commands is for cumulative activity since the last system boot. The second report shows activity for the first 5-second interval. The sar command does not report the cumulative activity since the last system boot.
These commands are the basic foundation on which to construct a performance-monitoring mechanism. Shell scripts can be written to perform data reduction on the command output and warn of performance problems or record data on the status of the system when a problem is occurring. For example, a shell script could test the CPU idle percentage for zero and execute another shell script when that CPU-saturated condition occurred. A script such as:
# ps -ef | egrep -v "STIME|$LOGNAME" | sort +3 -r | head -n 15would record the 15 active processes that had consumed the most CPU time recently (other than the processes owned by the user of the script).
Depending on the required level of sophistication, creating such a family of shell scripts can be a substantial project. Fortunately, there are packages available that require less development and setup and have considerably more function than the typical installation would want to implement locally.
iostat [ -c | -d ] [ -k ] [ -t ] [ -V ] [ -x ] [ { device [ ... ] | ALL } ] [ -p [ { device | ALL } ] ] [ interval [ count ] ]
The iostat command is used for monitoring system input/output device loading by observing the time the devices are active in relation to their average transfer rates. The iostat command generates reports that can be used to change system configuration to better balance the input/output load between physical disks.
The first report generated by the iostat command provides statistics concerning the time since the system was booted. Each subsequent report covers the time since the previous report. All statistics are reported each time the iostat command is run. The report consists of a CPU header row followed by a row of CPU statistics. On multiprocessor sys- tems, CPU statistics are calculated system-wide as averages among all processors. A device header row is displayed followed by a line of statistics for each device that is configured.
The interval parameter specifies the amount of time in seconds between each report. The first report contains statistics for the time since system startup (boot). Each subsequent report contains statistics col- lected during the interval since the previous report. The count parame- ter can be specified in conjunction with the interval parameter. If the count parameter is specified, the value of count determines the number of reports generated at interval seconds apart. If the interval parame- ter is specified without the count parameter, the iostat command gener- ates reports continuously.
The iostat command generates two types of reports, the CPU Utilization report and the Device Utilization report.
CPU Utilization Report
The first report generated by the iostat command is the CPU Uti- lization Report. For multiprocessor systems, the CPU values are global averages among all processors. The report has the fol- lowing format:
Device Utilization Report
The second report generated by the iostat command is the Device Utilization Report. The device report provides statistics on a per physical device or partition basis. Block devices for which statistics are to be displayed may be entered on the command line. Partitions may also be entered on the command line provid- ing that option -x is not used. If no device nor partition is entered, then statistics are displayed for every device used by the system, and providing that the kernel maintains statistics for it. If the ALL keyword is given on the command line, then statistics are displayed for every device defined by the system, including those that have never been used. The report may show the following fields, depending on the flags used:
The iostat command takes into account the following environment variable:
S_TIME_FORMAT If this variable exists and its value is ISO then the current locale will be ignored when printing the date in the report header. The iostat command will use the ISO 8601 format (YYYY- MM-DD) instead.
Linux System Administrator's Guide - 7.1.2. The iostat command.
Adam's Tech Blog " How to read the iostat command
Solaris Performance Monitoring & Tuning – iostat, vmstat, netstat
Using the UNIX iostat utility to find disk bottlenecks in Oracle
The Approach to Locating Hot Disks - Locating Hot Disks - Hot Disks
How to Install iostat Using yum
Performance Management Guide - Monitoring Disk I/O> Hunting I/O Bottlenecks with iostat | Linux Magazine
iostat Display a single history since boot report for all CPU and Devices.
iostat -d 2 Display a continuous device report at two second intervals.
iostat -d 2 6 Display six reports at two second intervals for all devices.
iostat -x hda hdb 2 6 Display six reports of extended statistics at two second inter- vals for devices hda and hdb.
iostat -p sda 2 6 Display six reports at two second intervals for device sda and all its partitions (sda1, etc.)
Reports Central Processing Unit (CPU) statistics, asynchronous input/output (AIO) and input/output statistics for the entire system, adapters, TTY devices, disks CD-ROMs, tapes and file systems.
iostat [ -a ] [ -l ] [ -s ] [ -t ] [ -T ] [ -V] [ -z ] [ { -A [ -P ] [ -q | -Q ] } | { [ -d | -p] -D [ -R ] }[ -m ] ] [ { -f | -F } [ filesystems,... ] ][ -S power] [ -@ wparname | ALL | Global ] [ drives ... ] [ interval] [ count ]
Restriction: The -a, -t, -z, -A, -P, -q, -Q, -p, -R, -m flags and the wparname parameter are restricted inside workload partitions.
The iostat command is used to monitor system input/output (I/O) devices (physical and logical) that are loaded, by observing the time for which these devices are active. The iostat command also generates reports that can be used to change system configuration to better balance the I/O load between file systems, physical volumes, and adapters.
The iostat command generates various utilization and throughput reports based on the options that you specify. On multiprocessor systems, CPU statistics are calculated system-wide as averages among all processors.
A report generated by the iostat command consists of system configuration information and various utilization and throughput reports. The system configuration row displays at the start of the iostat command and whenever there is a change in monitored configuration. In addition to system configuration, WPAR configuration is also displayed for the WPAR that has enforced resource limits when the -@ flag is used.
The system configuration and WPAR configuration information includes the following values:
The Interval parameter specifies the amount of time in seconds between each report. If the Interval parameter is not specified, the iostat command generates a single report containing statistics for the time since system startup (boot). The Count parameter can be specified in conjunction with the Interval parameter. If the Count parameter is specified, the value of count determines the number of reports generated at Interval seconds apart. If the Interval parameter is specified without the Count parameter, the iostat command generates reports continuously.
The iostat command is useful in determining whether a physical volume is becoming a performance bottleneck and if there is potential to improve the situation. The % utilization field for the physical volumes indicates how evenly the file activity is spread across the drives. A high % utilization on a physical volume is a good indication that there may be contention for this resource. Since the CPU utilization statistics are also available with the iostat report, the percentage of time the CPU is in I/O wait can be determined at the same time. Consider distributing data across drives if the I/O wait time is significant and the disk utilization is not evenly distributed across volumes.
Beginning with AIX 5.3, the iostat command reports number of physical processors consumed (physc) and the percentage of entitlement consumed (% entc) in Micro-Partitioning environments. These metrics will only be displayed on Micro-Partitioning environments.
Note:
Some system resource is consumed in maintaining disk I/O history for the iostat command. Use the sysconfig subroutine, or the SMIT to stop history accounting. While the iostat command is running for Count of iterations and if there is a change in system configuration that affects the output of iostat command, it prints a warning message about the configuration change. It then continues the output after printing the updated system configuration information and the header.
If you specify the -a flag, the information is displayed in a report in the following order:
If the Drive parameter is specified, only those names specified are displayed. One or more alphabetic or alphanumeric values can be specified for Drives. If you specify the Drive parameter, the TTY and CPU reports are displayed and the disk or tape report contains statistics for the specified drives. If a drive name that you specified is not found, the report lists that name and displays the message Drive Not Found. If you did not configure drives on the system, no disk or tape report is generated.
Restriction: The first character in the Drive parameter cannot be numeric.
Tape utilization report is generated only if you specified the -p or -a flag.
Note:The -@ option is not supported when executed within a workload partition.
The iostat command generates four types of reports, the TTY and CPU utilization report, the disk/tape utilization report, the file system utilization report, the system throughput report and the adapter throughput report.
Tips:
The first report generated by the iostat command is the TTY and CPU utilization report. For multiprocessor systems, the CPU values are global averages among all processors. Also, the I/O wait state is defined system-wide and not per processor. The report has the following format:
This information is updated at regular intervals by the kernel (typically sixty times per second). The TTY report provides a collective account of characters per second received from all terminals on the system as well as the collective count of characters output per second to all terminals on the system.
Operating system version 4.3.3 and later contain enhancements to the method used to compute the percentage of CPU time spent waiting on disk I/O (wio time). The method used in AIX 4.3.2 and earlier versions of the operating system can, under certain circumstances, give an inflated view of wio time on SMPs. The wio time is reported by the commands sar (%wio), vmstat (wa) and iostat (% iowait).
The method used in AIX 4.3.2 and earlier versions is as follows: At each clock interrupt on each processor (100 times a second per processor), a determination is made as to which of the four categories (usr/sys/wio/idle) to place the last 10 ms of time. If the CPU was busy in usr mode at the time of the clock interrupt, then usr gets the clock tick added into its category. If the CPU was busy in kernel mode at the time of the clock interrupt, then the sys category gets the tick. If the CPU was not busy, a check is made to see if any I/O to disk is in progress. If any disk I/O is in progress, the wio category is incremented. If no disk I/O is in progress and the CPU is not busy, the idle category gets the tick. The inflated view of wio time results from all idle CPUs being categorized as wio regardless of the number of threads waiting on I/O. For example, systems with just one thread doing I/O could report over 90 percent wio time regardless of the number of CPUs it has.
The method used in AIX 4.3.3 and later is as follows: The change in operating system version 4.3.3 is to only mark an idle CPU as wio if an outstanding I/O was started on that CPU. This method can report much lower wio times when just a few threads are doing I/O and the system is otherwise idle. For example, a system with four CPUs and one thread doing I/O will report a maximum of 25 percent wio time. A system with 12 CPUs and one thread doing I/O will report a maximum of 8 percent wio time. NFS client reads/writes go through the VMM, and the time that biods spend in the VMM waiting for an I/O to complete is now reported as I/O wait time.
The second report generated by the iostat command is the disk/tape utilization report. By default, the disk utilization report is displayed, and you must specify the -p flag to display the tape utilization report.
When you specify the -m flag, the path utilization report is displayed.
The disk report provides statistics on a per-physical-disk basis, and tape utilization report provides statistics on a per-tape-basis. The default report has the following format:
If you specify the -D flag, the report has the following metrics for disk/tape. Extended metrics for disk are displayed by default and users need to specify the -p option for tape utilization report:
Metrics related to transfers (xfer): | |
---|---|
% tm_act | Indicates the percentage of time the physical disk/tape was active (bandwidth utilization for the drive). |
bps | Indicates the amount of data transferred (read or written) per second to the drive. Different suffixes are used to represent the unit of transfer. Default is in bytes per second. |
tps | Indicates the number of transfers per second that were issued to the physical disk/tape. A transfer is an I/O request to the physical disk/tape. Multiple logical requests can be combined into a single I/O request to the disk. A transfer is of indeterminate size. |
bread | Indicates the amount of data read per second, from the drive. Different suffixes are used to represent the unit of transfer. Default is in bytes per second. |
bwrtn | Indicates the amount of data written per second, to the drive. Different suffixes are used to represent the unit of transfer. Default is in bytes per second. |
Read Service Metrics (read): | |
---|---|
rps | Indicates the number of read transfers per second. |
avgserv | Indicates the average service time per read transfer. Different suffixes are used to represent the unit of time. Default is in milliseconds. |
minserv | Indicates the minimum read service time. Different suffixes are used to represent the unit of time. Default is in milliseconds. |
maxserv | Indicates the maximum read service time. Different suffixes are used to represent the unit of time. Default is in milliseconds. |
timeouts | Indicates the number of read timeouts per second. |
fails | Indicates the number of failed read requests per second. |
Write Service Metrics (write): | |
---|---|
wps | Indicates the number of write transfers per second. |
avgserv | Indicates the average service time per write transfer. Different suffixes are used to represent the unit of time. Default is in milliseconds. |
minserv | Indicates the minimum write service time. Different suffixes are used to represent the unit of time. Default is in milliseconds. |
maxserv | Indicates the maximum write service time. Different suffixes are used to represent the unit of time. Default is in milliseconds. |
timeouts | Indicates the number of write timeouts per second. |
fails | Indicates the number of failed write requests per second. |
Wait Queue Service Metrics (queue): | Restriction: These metrics are not applicable for tapes. |
avgtime | Indicates the average time spent by a transfer request in the wait queue. Different suffixes are used to represent the unit of time. Default is in milliseconds. |
mintime | Indicates the minimum time spent by a transfer request in the wait queue. Different suffixes are used to represent the unit of time. Default is in milliseconds. |
maxtime | Indicates the maximum time spent by a transfer request in the wait queue. Different suffixes are used to represent the unit of time. Default is in milliseconds. |
avgwqsz | Indicates the average wait queue size. |
avgsqsz | Indicates the average service queue size. |
sqfull | Indicates the number of times the service queue becomes full (that is, the disk is not accepting any more service requests) per second. |
Legend of suffixes representing different units of representation | |
---|---|
Suffix | Description |
K | 1000 bytes |
M | 1 000 000 bytes if displayed in xfer metrics. Minutes, if displayed in read/write/wait service metrics. |
G | 1 000 000 000 bytes. |
T | 1 000 000 000 000 bytes. |
S | Seconds. |
H | Hour. |
For drives that do not support service time metrics, read, write and wait queue service metrics will not be displayed.
Statistics for CD-ROM devices are also reported.
This report is generated if the -s flag is specified. This report provides statistics for the entire system. This report has the following format:
Kbps | Indicates the amount of data transferred (read or written) in the entire system in KB per second. |
tps | Indicates the number of transfers per second issued to the entire system. |
Kb_read | The total number of KB read from the entire system. |
Kb_wrtn | The total number of KB written to the entire system. |
Tip: The -s flag, when used with the -@ or -f flag, displays logical and physical volume throughput, which corresponds to File Systems and Disks respectively.
This report is generated if the -a flag is specified. This report provides statistics on an adapter-by-adapter basis (for both physical and virtual adapters). This report has the following format for a physical adapter report:
Kbps | Indicates the amount of data transferred (read or written) in the adapter in KB per second. |
tps | Indicates the number of transfers per second issued to the adapter. |
Kb_read | The total number of KB read from the adapter. |
Kb_wrtn | The total number of KB written to the adapter. |
The virtual adapter's default throughput report has the following format:
Kbps | Indicates the amount of data transferred (read or written) in the adapter in KB per second. |
tps | Indicates the number of transfers per second issued to the adapter. |
bkread | Number of blocks received per second from the hosting server to this adapter. |
bkwrtn | Number of blocks per second sent from this adapter to the hosting server. |
partition-id | The partition ID of the hosting server, which serves the requests sent by this adapter. |
The virtual adapter's extended throughput report (-D option) has the following format:
Metrics related to transfers (xfer:) | |
---|---|
Kbps | Indicates the amount of data transferred (read or written) in the adapter in KB per second. |
tps | Indicates the number of transfers per second issued to the adapter. |
bkread | Number of blocks received per second from the hosting server to this adapter. |
bkwrtn | Number of blocks per second sent from this adapter to the hosting server. |
partition-id | The partition ID of the hosting server, which serves the requests sent by this adapter. |
Adapter Read Service Metrics (read:) | |
---|---|
rps | Indicates the number of read requests per second. |
avgserv | Indicates the average time to receive a response from the hosting server for the read request sent. Different suffixes are used to represent the unit of time. Default is in milliseconds. |
minserv | Indicates the minimum time to receive a response from the hosting server for the read request sent. Different suffixes are used to represent the unit of time. Default is in milliseconds. |
maxserv | Indicates the maximum time to receive a response from the hosting server for the read request sent. Different suffixes are used to represent the unit of time. Default is in milliseconds. |
Adapter Write Service Metrics (write:) | |
---|---|
wps | Indicates the number of write requests per second. |
avgserv | Indicates the average time to receive a response from the hosting server for the write request sent. Different suffixes are used to represent the unit of time. Default is in milliseconds. |
minserv | Indicates the minimum time to receive a response from the hosting server for the write request sent. Different suffixes are used to represent the unit of time. Default is in milliseconds. |
maxserv | Indicates the maximum time to receive a response from the hosting server for the write request sent. Different suffixes are used to represent the unit of time. Default is in milliseconds. |
Adapter Wait Queue Metrics (queue:) | |
---|---|
avgtime | Indicates the average time spent by a transfer request in the wait queue. Different suffixes are used to represent the unit of time. Default is in milliseconds. |
mintime | Indicates the minimum time spent by a transfer request in the wait queue. Different suffixes are used to represent the unit of time. Default is in milliseconds. |
maxtime | Indicates the maximum time spent by a transfer request in the wait queue. Different suffixes are used to represent the unit of time. Default is in milliseconds. |
avgwqsz | Indicates the average wait queue size. |
avgsqsz | Indicates the average service queue size. |
sqfull | Indicates the number of times the service queue becomes full (that is, the hosting server is not accepting any more service requests) per second. |
Legend of suffixes representing different units of representation | |
---|---|
Suffix | Description |
K | 1000 bytes. |
M | 1 000 000 bytes if displayed in xfer metrics. Minutes, if displayed in read/write/wait service metrics. |
G | 1 000 000 000 bytes. |
T | 1 000 000 000 000 bytes. |
S | Seconds. |
H | Hours. |
The asynchronous I/O report has the following column headers :
avgc | Average global AIO request count per second for the specified interval. |
avfc | Average fastpath request count per second for the specified interval. |
maxgc | Maximum global AIO request count since the last time this value was fetched. |
maxfc | Maximum fastpath request count since the last time this value was fetched. |
serv | Number of active AIO servers. |
The file system utilization report provides statistics on a per-file-system basis. The default report has the following format:
Filesystem | Indicates the file system name. |
% tm_act | Indicates the percentage of time the file system is active. |
Kbps | Indicates the amount of data transferred (read or written) to the file system in KB per second. |
Tps | Indicates the number of transfers per second that are issued to the file system. A transfer is of indeterminate size. |
Kb_read | The total number of KBs read. |
Kb_wrtn | The total number of KBs written. |
Important: You must specify the disk names before you invoke the -f or -F flag. If you specify the -f or -F flag, separate file system names to be monitored by commas.
Attention RBAC users and Trusted AIX users: This command can perform privileged operations. Only privileged users can run privileged operations. For more information about authorizations and privileges, see Privileged Command Database in Security. For a list of privileges and the authorizations associated with this command, see the lssecattr command or the getcmdattr subcommand.
iostat
iostat -d disk1 2
iostat disk1 2 6
iostat -d 2 6
iostat disk1 disk2 disk3 2 6
iostat -s
iostat -a 5
iostat -sat 20 10
iostat -sad hdisk0 hdisk7 30
iostat -T 60
iostat -A 2 6
iostat -A -Q
iostat -D
iostat -Dp
iostat –D hdisk0
iostat –z
iostat -F -@ ALL
iostat -f -s -@ ALL
iostat -f
iostat -s -f
iostat hdisk0 hdisk1 -f /dev/fslv00 /dev/fslv01 /dev/fslv02
iostat -S 1
/usr/bin/iostat | Contains the iostat command. |
The vmstat command.
The iostadd Kernel Service.
Monitoring disk I/O in Performance management
The Input and Output Handling Programmer's Overview in AIX Version 6.1 General Programming Concepts: Writing and Debugging Programs describes the files, commands, and subroutines used for low-level, stream, terminal, and asynchronous I/O interfaces.
Society
Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy
Quotes
War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes
Bulletin:
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
History:
Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history
Classic books:
The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor
The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D
Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...
|
You can use PayPal to to buy a cup of coffee for authors of this site |
Disclaimer:
The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.
Last modified: July, 10, 2020