User: Password:
|
|
Subscribe / Log in / New account

Development

One-stop performance analysis using atop

May 12, 2010

This article was contributed by Jan Christiaan van Winkel & Gerlof Langeveld

Linux system administrators often receive complaints about the performance of their systems. It can be rather difficult to track down these problems and to find why, when, and how often they happen. Being able to zoom in on the processes that are responsible, and to see what has happened in the past, is very valuable. The atop utility was written with just these things in mind.

Performance analysis tools

Linux has a rich set of tools for performance analysis, but each has its own capabilities and limitations. In developing atop, the following list was considered to be desirable features for the tool:

  • The tool should obviously be able to show the current situation. However many resource problems don't occur "now". Often complaints will come in about the system performance "last night" or "last week". Therefore the tool must be able to look in the past. Being able to look in the future would be a "nice to have" but was deemed too difficult to implement.

  • It should show the load of the four main resources on a system level: CPU, memory, disk I/O, and network usage.

  • The four main resources are consumed by or on behalf of processes, so the tool should be able to show which processes (over)load the four resources.

  • A monitoring tool takes snapshots of the system, using a certain interval. If a process used resources since the last snapshot but has exited before the current snapshot, the tool should still be able to show which processes loaded which resources. In other words: the sum of resource usage by the processes should be equal to the system wide reported resource usage.

Looking at this list of requirements, none of the existing standard analysis tools meets the bill. sar shows extensive data regarding CPU, memory, disk and network usage from the past and the present. However, it cannot "zoom in" on processes: it only shows resource usage on a system level. vmstat and iostat can only show CPU, memory and disk usage on a system level; they cannot show usage data from the past. Finally top, one of the most used performance monitors, does show CPU and memory usage on a system level and on a process level. However, it only shows the current situation, it cannot show usage data from the past. It also does not show the resource consumption for exited processes, so with top it is possible that on a system level the CPU is shown as 90% busy, while the sum of all CPU consumption on a process level is only 40% (the other 50% might have been used by processes that exited between the previous and the current snapshot).

This chart compares the characteristics of these other analysis tools with atop:

[Comparison chart]

atop is free software, and can be downloaded from the web site, though many Linux distributions include atop in their repositories. After installing atop, the command atopsar is also available. It can be compared to sar but references the same log files that are generated and used by atop.

Characteristics of atop

atop was created mainly because the other tools don't report about processes that exit between snapshots. When using "process accounting", the kernel writes a record to a log file for every process that exits. atop will use these records to make a process activity list that is as complete as possible, including processes that exited since the last snapshot.

atop shows the load of the CPUs, memory, disks, and network on a system level. Apart from the network, atop also shows which processes consume these resources (for network utilization per process, a kernel patch is provided). By default, atop shows generic information about processes (like PID, name, CPU utilization, memory utilization, disk utilization, and status). However, more information about the process's memory usage, disk I/O, and scheduling characteristics is available by using single-character keystrokes (for example, s for scheduling characteristics).

Users can always override the default sorting order that atop uses. For example, for more information about a process's memory usage, the M subcommand sorts the processes in descending order of their resident memory usage. But, these processes can also be sorted on their disk I/O usage by using the D subcommand. Typing A will let atop determine what the most sensible sorting order would be given the most heavily used resource at the moment. In the system overview (the top half of the screen) a line will be highlighted if that particular resource is overloaded.

Obviously, not all data about all resources can be shown on the screen at once. Therefore, if the window is resized, atop will automatically show more (or less) data depending on the room available. Configurable priorities are used to determine what data is no longer shown if there is too little space.

Using atop on a system level

The default screen of atop looks like this:

[atop screen]

Just like top, the screen is divided in two parts: the top half shows system-wide data, and the bottom half is used for per-process data.

On a system level, not only CPU and memory statistics are visible, but also disk and network usage data. In the example above, the line with label CPU shows a total of (27+61+25+214+73)=400% CPU capacity, so there are 4 CPUs in this system. The lines labeled cpu show the individual CPUs (each rated at 100%). The CPUs are listed in order of busyness. The CPU is considered busy when it is in system mode, user mode, or handling interrupts, and it is idle when idle or waiting for I/O (wait). Therefore, in this case the sort order for the CPUs is 3, 1, 2, 0 as shown in the last column, just in front of the wait percentage "w".

The header line shows that ten seconds have elapsed since the previous snapshot. During this time, four CPUs provide a total of 40 seconds of computing capacity. The line with label PRC shows the sum of the CPU time used by processes, i.e. 5.20 seconds in system mode (corresponds to 26% sys + 25% irq in the line labeled CPU) and 6.20 in user mode (corresponds to 61% user in the line labeled CPU).

A line labeled DSK gives information about a physical disk that has been active in the past interval. It shows the name of the disk, the I/O busy percentage, the number of read and write requests, and the average service time per request (avio). By making the screen wider, more data is shown: the disk bandwidth for reads and writes (in MiB/s) and the average queue length for that disk.

If the system uses LVM or MD software RAID volumes, the same information is shown for each active logical volume of MD volume. In the figure below, it is clear that several writes (111 plus 1) to logical volumes may be combined to fewer writes to the physical disk (54). The combined transfers are larger and therefore use a higher service time per transfer.

[LVM disk output]

In the same way, data for memory availability, usage, paging, and page scanning is shown. The last lines in the system overview show network related data, per interface, on IP-layer level and on transport level.

Using atop on a process level

It is useful to be able to see how busy the system is, but if a system is too busy, the tool needs to be able to zoom in to find the culprit. This is where atop shines. atop tries to make sure the books balance. Other tools do not take processes that exited into account. For example, with top it is possible that on a system level the CPUs are 99% busy, even though top shows only two active processes that together have used only 5% of the CPU in the sample period. One notorious example of this is a kernel compilation: lots of short-lived processes that eat up all of the CPU, but only a few of them show up in top's output.

atop uses process accounting to take into account processes that have exited. In the first full screenshot shown at system level, we can see bzip2 having used 61% of the CPU time. atop lists it in angle brackets, to show that the process has exited. In addition, the exit code (column EXC) is shown. It is interesting to see if a process is eating up CPU power in system mode (system call and interrupt handling) or in user mode. Fortunately, atop shows you both, per process.

The Linux process scheduler determines which process gets the CPU. Scheduling information can be seen by using the s subcommand:

[scheduling info]

One can see how many threads are in the "running" state (TRUN), "sleeping interruptible" (TSLPI), or "sleeping uninterruptible" (TSLPU). The scheduling policy (normal, round-robin realtime, fifo realtime, etc.) is also shown.

CPU is not the only scarce resource in the system. Therefore, atop can also show per-process usage statistics for memory, disk bandwidth, and network bandwidth. Zooming in on disk statistics (using the d subcommand), atop shows the following:

[disk usage info]

Recent kernels are often configured with the option "Per-task storage I/O accounting", so the kernel keeps track of how much data is passed by the write and read system calls related to disk I/O (WRDSK and RDDSK respectively). However, not all write system calls will lead to physical writes on disk. For example, if a region of a file is written and then overwritten before the data was flushed from the page cache to disk, the first writes are shown by atop even though they have never been written to disk. In this case, the column WCANCL shows the amount of data whose physical write was canceled. In the example above, the actions of tar canceled 172KiB worth of writes.

Extra information with patches

The kernel does not register network bandwidth usage per process. Patches are available that make the kernel keep track of network usage per process. After receiving the n subcommand, atop will show network related data per process:

[network usage info]

The TCPRCV/TCPSND and UDPRCV/UDPSND columns show the number of packets being received and sent per process by these transport layers. The RAWRCV and RAWSND show the number of "raw" packets received and sent. These are packets that go directly from the application to the IP layer, not passing through TCP or UDP. For example the ping program sends ICMP ECHO REQUEST packets directly through the ICMP layer to receive ICMP ECHO REPLY packets.

The TCPSASZ column shows the average send transfer size. If the screen is wide enough, the average receive transfer size is also shown, both for TCP and for UDP.

Unfortunately, these patches are not part of the mainline kernel. In 2008 an attempt was made to merge them, but the modifications conflicted with other new features (like cgroups) that were under development at the time.

Back to the future

atop is useful as a tool for the here and now. But what if the system was slow in the past? The normal installation of the atop package starts an atop daemon nightly. This daemon takes snapshots and writes them to a log file (/var/log/atop/atop_YYYYMMDD). The default snapshot interval for a logging atop is 10 minutes, but obviously this is configurable. Every logfile is preserved for a month (also configurable), so performance events a full month back can still be observed.

The log file can be viewed using atop -r log_filename. The subcommand t forwards to the next sample in the atop logfile (i.e. 10 minutes by default), subcommand T rewinds one sample. Subcommand b branches to a specific time in the current logfile. All other subcommands to zoom in on specific resources also work. The logfile that the atop daemon creates can also be viewed using a sar-like interface using the command atopsar.

One-stop analysis

Performance analysis is a cyclic process of measuring, drawing conclusions, measuring in more detail, drawing more detailed conclusions, and so on until you can really pinpoint the aching spot. Performance problems do not only occur in the here and now, but also in the past, so a performance analysis tool should be able give information in both situations. As described in this article, atop has been designed to be a complete one-stop tool to guide you at least through the first cycles (system and process level for the four most critical resources: CPU, memory, disk I/O and network).

More information about atop can be found on the atop website, as well as in the atop man page [PDF]. There is also a case study [PDF] available that shows how atop can be used to analyze a problem with processes leaking memory.

Comments (12 posted)

Brief items

Emacs 23.2 released

The Emacs 23.2 release is out. There's a lot of new stuff, but, arguably, the most significant change is the incorporation of the Collection of Emacs Development Environment Tools (CEDET), which brings a project management utility, function and variable name completion, a code generation mechanism, and, naturally, a mode for manipulating UML diagrams within the editor. See the NEWS file for a full list of additions and changes.

Full Story (comments: 83)

Jato v0.1.0 released

The Jato v0.1.0 release is out. "Jato is an open source, JIT-only virtual machine for Java that aims to support the JVM specification version 2 including 1.5 amendments. It is not complete but you can run some Java applications with it on 32-bit x86 Linux machines. Jato uses Boehm GC as its garbage collector and GNU Classpath to provide essential Java run-time libraries."

Full Story (comments: none)

The Koha "Harley" release

As noted by LWN reader "hackerb9," LibLime has announced the "Harley" release of the Koha library management system. "This release includes all functionality in Koha as of October 2009, plus many new features not yet available. Harley is compatible with both 3.0x and upcoming 3.2 versions. The code for the individual new features included in this version has also been made available for download from the GIT repository." This release is an important step toward the resolution of the disagreement between LibLime and the Koha development community we reported on last week.

Comments (1 posted)

LLVM gets its own C++ standard library

The LLVM libc++ project has announced its existence. "libc++ is an implementation of the C++ Standard Library, with a focus on standards compliance, highly efficient generated code, and with an aim to support C++'0x when the standard is ratified." The project is said to be "approximately 85% complete at this point."

Comments (42 posted)

MythTV 0.23 released

The MythTV 0.23 release is out. "MythTV 0.23 brings a new event system, brand new python bindings, the beta MythNetvision internet video plugin, new audio code and surround sound upmixer, several new themes (Arclight and Childish), newly resynced ffmpeg libraries, and fixes for analog scanning, among many others." More information can be found in the release notes.

Comments (none posted)

py.test-1.3.0 released

Version 1.3.0 of py.test - an automated testing tool for Python2 and Python3 programs - has been released. There's a long list of new features; click below for the changelog.

Full Story (comments: none)

Ryzom source released as free software

Back in 2008, LWN covered the story of Ryzom, a popular multiplayer role-playing game which was owned by a company in bankruptcy. An attempt to acquire and liberate the source at that time failed, but good things come (sometimes) to those who wait: Ryzom is now available under the AGPLv3 license. The artwork, too, is available under the Creative Commons Attribution-ShareAlike license. "By freeing Ryzom code, Winch Gate is transforming the MMORPG marketplace and is setting a precedent for how gaming software should evolve--in freedom. The source code released totals over two (2) million lines of source code and over 20,000 high quality textures and thousands of 3D objects."

Comments (24 posted)

Newsletters and articles

Development newsletters from the last week

Comments (none posted)

Aknin: Guido's Python

Yaniv Aknin has launched into a series of articles intended to describe how Python works from the inside; thus far, the introductory and Objects 101 installments are available. "I want to see Python more like Guido does, so I'm starting out on what should develop to a series on Python's internals. On the curriculum is mainly CPython, mainly py3k, mainly bytecode evaluation (I'm not a big compilation fan) - but practically everything around executing Python and Python-like code (Unladen Swallow, Jython, Cython, etc) might turn out to be fair game in this series."

Comments (none posted)

Firefox 4: fast, powerful, and empowering

Mike Beltzner, Mozilla's Director of Firefox, shares some early plans for Firefox 4. "The primary goals for Firefox 4 will be making a browser: * Fast: making Firefox super-duper fast * Powerful: enabling new open, standard Web technologies (HTML5 and beyond!), * Empowering: putting users in full control of their browser, data, and Web experience."

Comments (33 posted)

News In The Linux Audio World (Linux Journal)

Dave Phillips has a brief report "about a new Linux audio blog, music made by particle acceleration, how to use a laptop as a virtual music stand, synth emulation from the terminal command prompt, and watching the Linux Audio Conference on-line."

Comments (none posted)

Hammond: Looking Back on Review Board

On his blog, Christian Hammond looks back at three-and-a-half years of development on the code review tool, Review Board. "Attempt #3. I decided to build our own diff parser and generator from scratch. What a project. I knew nothing about diff generation and hardly knew where to start. I spent probably a good month or so just trying to work on this new diff code, and was so close to giving up so many times. It ended up being completely worth it, though, as we ended up with a very nice, extensible diff parser. [...] Without that third attempt, we'd be in the stone age. Review Board would not be as nice to use. We wouldn't have inter-line diffs (where we highlight what changed in a replace line), syntax highlighting, move detection (coming in 1.5), or function/class headers (where we show which function/class the part of the diff is in — also coming in 1.5)."

Comments (1 posted)

Williams: Eat Burgers on the Short Bus

Dan Williams, developer of NetworkManager, looks at D-Bus on his blog. In particular, he has found himself periodically giving mini-lessons about D-Bus; he captures that in this explanation of D-Bus concepts along with a concrete example of the interprocess communication (IPC) mechanism. "service: a program that responds to requests from clients. Each service is identified by a 'bus name' which clients use to find the service and send requests to it. The bus name usually looks like org.foobar.Baz. A program can claim more than one bus name; NM claims org.freedesktop.NetworkManager and org.freedesktop.NetworkManagerSystemSettings, each is a unique service which provides different functionality to clients."

Comments (33 posted)

Page editor: Jonathan Corbet
Next page: Announcements>>


Copyright © 2010, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds