One-stop performance analysis using atop

Linux system administrators often receive complaints about the performance of their systems. It can be rather difficult to track down these problems and to find why, when, and how often they happen. Being able to zoom in on the processes that are responsible, and to see what has happened in the past, is very valuable. The atop utility was written with just these things in mind.

Performance analysis tools

Linux has a rich set of tools for performance analysis, but each has its own capabilities and limitations. In developing atop, the following list was considered to be desirable features for the tool:

• The tool should obviously be able to show the current situation. However many resource problems don't occur "now". Often complaints will come in about the system performance "last night" or "last week". Therefore the tool must be able to look in the past. Being able to look in the future would be a "nice to have" but was deemed too difficult to implement.

• It should show the load of the four main resources on a system level: CPU, memory, disk I/O, and network usage.

• The four main resources are consumed by or on behalf of processes, so the tool should be able to show which processes (over)load the four resources.

• A monitoring tool takes snapshots of the system, using a certain interval. If a process used resources since the last snapshot but has exited before the current snapshot, the tool should still be able to show which processes loaded which resources. In other words: the sum of resource usage by the processes should be equal to the system wide reported resource usage.

Looking at this list of requirements, none of the existing standard analysis tools meets the bill. sar shows extensive data regarding CPU, memory, disk and network usage from the past and the present. However, it cannot "zoom in" on processes: it only shows resource usage on a system level. vmstat and iostat can only show CPU, memory and disk usage on a system level; they cannot show usage data from the past. Finally top, one of the most used performance monitors, does show CPU and memory usage on a system level and on a process level. However, it only shows the current situation, it cannot show usage data from the past. It also does not show the resource consumption for exited processes, so with top it is possible that on a system level the CPU is shown as 90% busy, while the sum of all CPU consumption on a process level is only 40% (the other 50% might have been used by processes that exited between the previous and the current snapshot).

This chart compares the characteristics of these other analysis tools with atop:

atop is free software, and can be downloaded from the web site, though many Linux distributions include atop in their repositories. After installing atop, the command atopsar is also available. It can be compared to sar but references the same log files that are generated and used by atop.

Characteristics of atop

atop was created mainly because the other tools don't report about processes that exit between snapshots. When using "process accounting", the kernel writes a record to a log file for every process that exits. atop will use these records to make a process activity list that is as complete as possible, including processes that exited since the last snapshot.

atop shows the load of the CPUs, memory, disks, and network on a system level. Apart from the network, atop also shows which processes consume these resources (for network utilization per process, a kernel patch is provided). By default, atop shows generic information about processes (like PID, name, CPU utilization, memory utilization, disk utilization, and status). However, more information about the process's memory usage, disk I/O, and scheduling characteristics is available by using single-character keystrokes (for example, s for scheduling characteristics).

Users can always override the default sorting order that atop uses. For example, for more information about a process's memory usage, the M subcommand sorts the processes in descending order of their resident memory usage. But, these processes can also be sorted on their disk I/O usage by using the D subcommand. Typing A will let atop determine what the most sensible sorting order would be given the most heavily used resource at the moment. In the system overview (the top half of the screen) a line will be highlighted if that particular resource is overloaded.

Obviously, not all data about all resources can be shown on the screen at once. Therefore, if the window is resized, atop will automatically show more (or less) data depending on the room available. Configurable priorities are used to determine what data is no longer shown if there is too little space.

Using atop on a system level

The default screen of atop looks like this:

On a system level, not only CPU and memory statistics are visible, but also disk and network usage data. In the example above, the line with label CPU shows a total of (27+61+25+214+73)=400% CPU capacity, so there are 4 CPUs in this system. The lines labeled cpu show the individual CPUs (each rated at 100%). The CPUs are listed in order of busyness. The CPU is considered busy when it is in system mode, user mode, or handling interrupts, and it is idle when idle or waiting for I/O (wait). Therefore, in this case the sort order for the CPUs is 3, 1, 2, 0 as shown in the last column, just in front of the wait percentage "w".

The header line shows that ten seconds have elapsed since the previous snapshot. During this time, four CPUs provide a total of 40 seconds of computing capacity. The line with label PRC shows the sum of the CPU time used by processes, i.e. 5.20 seconds in system mode (corresponds to 26% sys + 25% irq in the line labeled CPU) and 6.20 in user mode (corresponds to 61% user in the line labeled CPU).

A line labeled DSK gives information about a physical disk that has been active in the past interval. It shows the name of the disk, the I/O busy percentage, the number of read and write requests, and the average service time per request (avio). By making the screen wider, more data is shown: the disk bandwidth for reads and writes (in MiB/s) and the average queue length for that disk.

If the system uses LVM or MD software RAID volumes, the same information is shown for each active logical volume of MD volume. In the figure below, it is clear that several writes (111 plus 1) to logical volumes may be combined to fewer writes to the physical disk (54). The combined transfers are larger and therefore use a higher service time per transfer.

In the same way, data for memory availability, usage, paging, and page scanning is shown. The last lines in the system overview show network related data, per interface, on IP-layer level and on transport level.

Using atop on a process level

It is useful to be able to see how busy the system is, but if a system is too busy, the tool needs to be able to zoom in to find the culprit. This is where atop shines. atop tries to make sure the books balance. Other tools do not take processes that exited into account. For example, with top it is possible that on a system level the CPUs are 99% busy, even though top shows only two active processes that together have used only 5% of the CPU in the sample period. One notorious example of this is a kernel compilation: lots of short-lived processes that eat up all of the CPU, but only a few of them show up in top's output.

atop uses process accounting to take into account processes that have exited. In the first full screenshot shown at system level, we can see bzip2 having used 61% of the CPU time. atop lists it in angle brackets, to show that the process has exited. In addition, the exit code (column EXC) is shown. It is interesting to see if a process is eating up CPU power in system mode (system call and interrupt handling) or in user mode. Fortunately, atop shows you both, per process.

The Linux process scheduler determines which process gets the CPU. Scheduling information can be seen by using the s subcommand:

One can see how many threads are in the "running" state (TRUN), "sleeping interruptible" (TSLPI), or "sleeping uninterruptible" (TSLPU). The scheduling policy (normal, round-robin realtime, fifo realtime, etc.) is also shown.

CPU is not the only scarce resource in the system. Therefore, atop can also show per-process usage statistics for memory, disk bandwidth, and network bandwidth. Zooming in on disk statistics (using the d subcommand), atop shows the following:

Recent kernels are often configured with the option "Per-task storage I/O accounting", so the kernel keeps track of how much data is passed by the write and read system calls related to disk I/O (WRDSK and RDDSK respectively). However, not all write system calls will lead to physical writes on disk. For example, if a region of a file is written and then overwritten before the data was flushed from the page cache to disk, the first writes are shown by atop even though they have never been written to disk. In this case, the column WCANCL shows the amount of data whose physical write was canceled. In the example above, the actions of tar canceled 172KiB worth of writes.

Extra information with patches

The kernel does not register network bandwidth usage per process. Patches are available that make the kernel keep track of network usage per process. After receiving the n subcommand, atop will show network related data per process:

The TCPRCV/TCPSND and UDPRCV/UDPSND columns show the number of packets being received and sent per process by these transport layers. The RAWRCV and RAWSND show the number of "raw" packets received and sent. These are packets that go directly from the application to the IP layer, not passing through TCP or UDP. For example the ping program sends ICMP ECHO REQUEST packets directly through the ICMP layer to receive ICMP ECHO REPLY packets.

The TCPSASZ column shows the average send transfer size. If the screen is wide enough, the average receive transfer size is also shown, both for TCP and for UDP.

Unfortunately, these patches are not part of the mainline kernel. In 2008 an attempt was made to merge them, but the modifications conflicted with other new features (like cgroups) that were under development at the time.

Back to the future

atop is useful as a tool for the here and now. But what if the system was slow in the past? The normal installation of the atop package starts an atop daemon nightly. This daemon takes snapshots and writes them to a log file (/var/log/atop/atop_YYYYMMDD). The default snapshot interval for a logging atop is 10 minutes, but obviously this is configurable. Every logfile is preserved for a month (also configurable), so performance events a full month back can still be observed.

The log file can be viewed using atop -r log_filename. The subcommand t forwards to the next sample in the atop logfile (i.e. 10 minutes by default), subcommand T rewinds one sample. Subcommand b branches to a specific time in the current logfile. All other subcommands to zoom in on specific resources also work. The logfile that the atop daemon creates can also be viewed using a sar-like interface using the command atopsar.

One-stop analysis

Performance analysis is a cyclic process of measuring, drawing conclusions, measuring in more detail, drawing more detailed conclusions, and so on until you can really pinpoint the aching spot. Performance problems do not only occur in the here and now, but also in the past, so a performance analysis tool should be able give information in both situations. As described in this article, atop has been designed to be a complete one-stop tool to guide you at least through the first cycles (system and process level for the four most critical resources: CPU, memory, disk I/O and network).

More information about atop can be found on the atop website, as well as in the atop man page [PDF]. There is also a case study [PDF] available that shows how atop can be used to analyze a problem with processes leaking memory.