Linux system administrators often receive complaints about the
performance of their systems. It can be rather difficult to track
these problems and to find why, when, and how often they happen.
able to zoom in on the processes that are responsible, and to see what
has happened in the past, is very valuable. The atop utility was
written with just these things in mind.
Performance analysis tools
Linux has a rich set of tools for performance analysis, but each has its
own capabilities and limitations. In developing atop, the
following list was considered to be desirable features for the tool:
The tool should obviously be able to show the current situation.
However many resource problems don't occur "now". Often complaints
will come in about the system performance "last night" or
"last week". Therefore the tool must be able to look in the past. Being
able to look in the future would be a "nice to have" but was deemed too
difficult to implement.
It should show the load of the four main resources on a system
level: CPU, memory, disk I/O, and network usage.
The four main resources are consumed by or on behalf of processes,
so the tool should be able to show which processes (over)load the
A monitoring tool takes snapshots of the system, using a certain
interval. If a process used
resources since the last snapshot but has exited before the
current snapshot, the tool should still be able to show which processes
loaded which resources. In other words: the sum of resource usage by the
processes should be equal to the system wide reported resource usage.
Looking at this list of requirements, none of the existing standard
analysis tools meets the bill.
sar shows extensive data regarding CPU, memory, disk
and network usage from the past and the present. However, it
cannot "zoom in" on processes:
it only shows resource usage on a system level.
vmstat and iostat
can only show CPU, memory and disk usage on a system level; they cannot
show usage data from the past.
Finally top, one of the most used performance monitors,
does show CPU and memory usage
on a system level and on a process level. However, it only shows the current
situation, it cannot show usage data from the past. It also
does not show the resource consumption for exited processes, so
with top it is possible that on a system level the CPU is
shown as 90% busy, while the sum of all CPU consumption on a process level
is only 40% (the other 50% might have been used by processes that exited
between the previous and the current snapshot).
This chart compares the characteristics of these other analysis
tools with atop:
atop is free software, and can be downloaded from the
though many Linux distributions include atop in their repositories.
After installing atop, the command atopsar
is also available. It can be compared to sar but references
the same log files that are generated and used by atop.
Characteristics of atop
atop was created mainly because the other tools don't report about
processes that exit between snapshots. When using "process accounting",
the kernel writes a record to a log file for
every process that exits. atop will use these
records to make a process activity list that is as complete as possible,
including processes that exited since the last snapshot.
atop shows the load of the CPUs, memory, disks, and network
on a system level. Apart from the network, atop also shows
which processes consume these resources (for network utilization
per process, a kernel patch is provided).
By default, atop shows generic information about processes (like
PID, name, CPU utilization, memory utilization, disk utilization, and
status). However, more information about the process's memory
usage, disk I/O, and scheduling characteristics is available by using
keystrokes (for example, s for scheduling characteristics).
Users can always
default sorting order that atop uses. For example, for more information
about a process's memory usage, the M subcommand sorts
the processes in descending order of their resident memory
usage. But, these processes can also be sorted on their
disk I/O usage by using the D subcommand. Typing A will
let atop determine what the most sensible sorting order would be
given the most heavily used resource at the moment.
In the system overview (the top half of the screen) a line will be
highlighted if that particular resource is overloaded.
Obviously, not all data about all resources can be shown on the screen at once.
Therefore, if the window is resized, atop will automatically show
more (or less) data depending on the room available. Configurable
priorities are used to
determine what data is no longer shown if there is too little space.
Using atop on a system level
The default screen of atop looks like this:
Just like top
, the screen is divided in two parts:
the top half
shows system-wide data, and the bottom half is used for per-process
On a system level, not only CPU and memory statistics are visible, but
also disk and network usage data.
In the example above,
the line with label CPU shows a total of (27+61+25+214+73)=400% CPU
capacity, so there are 4 CPUs in this system. The lines labeled cpu
show the individual CPUs (each rated at 100%). The CPUs are
listed in order of busyness. The CPU is considered busy when it is in
system mode, user mode, or handling interrupts, and it is idle when idle or
waiting for I/O (wait). Therefore, in this case the sort order for the
CPUs is 3, 1, 2, 0
as shown in the
last column, just in front of the wait percentage "w".
The header line shows that ten seconds have elapsed since
the previous snapshot. During this time, four CPUs provide a total of 40 seconds
of computing capacity.
The line with label PRC shows the sum of the CPU time
used by processes, i.e. 5.20 seconds in system mode
(corresponds to 26% sys + 25% irq in the line
labeled CPU) and 6.20 in user mode (corresponds to 61% user
in the line labeled CPU).
A line labeled DSK gives information about a physical disk
that has been active in the past interval. It shows the name of the disk,
the I/O busy percentage, the number of read and write requests, and the average
service time per request (avio). By making the screen wider,
more data is shown: the disk bandwidth for reads and writes (in MiB/s)
and the average queue length for that disk.
If the system uses LVM or MD software RAID volumes, the same
information is shown for each active logical volume of MD volume. In
the figure below, it is clear that several writes (111 plus 1)
to logical volumes may be combined to fewer writes to the physical
disk (54). The combined transfers are larger and therefore use a
higher service time per transfer.
In the same way, data for memory availability, usage, paging, and page
scanning is shown. The last lines in the system overview show network
related data, per interface, on IP-layer level and on transport level.
Using atop on a process level
It is useful to be able to see how busy the system is, but if a system is
too busy, the tool needs to be able to zoom in to find the culprit.
This is where
atop shines. atop tries to make sure the
books balance. Other tools do not take processes that exited into
account. For example, with top it is possible that on a system level
the CPUs are 99% busy, even though top shows only two active
processes that together have used only 5% of the CPU in the sample
period. One notorious example of this is a kernel compilation: lots
of short-lived processes that eat up all of the CPU, but only a
few of them show up in top's output.
atop uses process accounting to take into account processes that have
exited. In the first full screenshot
shown at system level,
we can see bzip2 having used 61% of the CPU time. atop
in angle brackets, to show that the process has exited.
In addition, the exit code (column EXC) is shown.
It is interesting to see if a process is eating up CPU power in
system mode (system call and interrupt handling) or in user mode.
Fortunately, atop shows you both, per process.
The Linux process scheduler determines which process gets the CPU. Scheduling information can be seen
by using the s subcommand:
One can see how many threads are in the "running" state (TRUN),
"sleeping interruptible" (TSLPI), or
"sleeping uninterruptible" (TSLPU). The scheduling
policy (normal, round-robin realtime, fifo realtime, etc.) is also shown.
CPU is not the only scarce resource in the system. Therefore,
atop can also show per-process usage statistics for memory,
disk bandwidth, and network bandwidth. Zooming in on disk statistics
(using the d subcommand), atop shows the following:
Recent kernels are often configured with the option
"Per-task storage I/O accounting", so
the kernel keeps track of how much data is passed by the
write and read system calls related to disk I/O
(WRDSK and RDDSK respectively). However, not all
write system calls will lead to physical writes on disk. For example, if a
region of a file is written and then overwritten before the data was
flushed from the page cache to disk, the first writes are shown
by atop even though they have never been written to disk. In
this case, the column
WCANCL shows the amount of data whose physical write was
canceled. In the example above, the actions
of tar canceled 172KiB worth of writes.
Extra information with patches
The kernel does not register network bandwidth usage per process.
are available that make the kernel keep track of network usage
per process. After receiving the n subcommand, atop will show
network related data per process:
The TCPRCV/TCPSND and UDPRCV/UDPSND
columns show the number of packets being received and sent per process by
these transport layers. The RAWRCV and
RAWSND show the number of "raw" packets received and sent.
These are packets that go directly from the application to the IP
layer, not passing through TCP or UDP. For example the ping
program sends ICMP ECHO REQUEST packets directly through the ICMP
layer to receive ICMP ECHO REPLY packets.
The TCPSASZ column shows the average send transfer size.
If the screen is wide enough, the average receive transfer size
is also shown, both for TCP and for UDP.
Unfortunately, these patches are not part of the mainline kernel.
In 2008 an attempt was made to merge them, but the modifications conflicted
with other new features (like cgroups) that were under development at the time.
Back to the future
atop is useful as a tool for the here and now. But what if
the system was slow in the past?
The normal installation of the atop package starts an
atop daemon nightly.
This daemon takes snapshots and writes them to a log file
(/var/log/atop/atop_YYYYMMDD). The default
snapshot interval for a logging atop is 10 minutes, but obviously
this is configurable. Every logfile is preserved for a month (also
configurable), so performance events a full month back can still be
The log file can be viewed using atop -r log_filename.
The subcommand t forwards to the next sample in the atop
logfile (i.e. 10 minutes by default),
subcommand T rewinds one sample. Subcommand b
branches to a
specific time in the current logfile. All other subcommands to zoom in on
specific resources also work.
The logfile that the atop daemon creates can also be viewed using
a sar-like interface using the command
Performance analysis is a cyclic process of measuring, drawing conclusions,
measuring in more detail, drawing more detailed conclusions, and so on until
you can really pinpoint the aching spot. Performance problems do not
only occur in the here and now, but also in the past, so a performance
analysis tool should be able give information in both situations.
As described in this article,
atop has been designed to be a complete one-stop tool to guide you
at least through the first cycles (system and process level for the
four most critical resources: CPU, memory, disk I/O and network).
More information about atop can be found on the
atop website, as well as in the atop man page
[PDF]. There is also a
case study [PDF]
available that shows
how atop can be used to analyze a problem with processes
Comments (12 posted)
The Emacs 23.2 release is out. There's a lot of new stuff, but, arguably,
the most significant change is the incorporation of the Collection of Emacs Development
Environment Tools (CEDET)
, which brings a project management utility,
function and variable name completion, a code generation mechanism, and,
naturally, a mode for manipulating UML diagrams within the editor. See the NEWS file
full list of additions and changes.
Full Story (comments: 83)
The Jato v0.1.0 release is out. "Jato is an open source, JIT-only virtual machine for Java that aims to support
the JVM specification version 2 including 1.5 amendments. It is not complete
but you can run some Java applications with it on 32-bit x86 Linux machines.
Jato uses Boehm GC as its garbage collector and GNU Classpath to provide
essential Java run-time libraries.
Full Story (comments: none)
by LWN reader
"hackerb9," LibLime has announced
the "Harley" release
of the Koha library management system.
"This release includes all functionality in Koha as of October 2009,
plus many new features not yet available. Harley is compatible with both
3.0x and upcoming 3.2 versions. The code for the individual new features
included in this version has also been made available for download from the
" This release is an important step toward the
resolution of the disagreement between LibLime and the Koha development
community we reported on
Comments (1 posted)
The LLVM libc++ project
. "libc++ is an implementation of the C++ Standard
Library, with a focus on standards compliance, highly efficient generated
code, and with an aim to support C++'0x when the standard is
" The project is said to be "approximately 85%
complete at this point
Comments (42 posted)
The MythTV 0.23 release
is out. "MythTV 0.23 brings a new event system, brand new python bindings, the beta MythNetvision internet video plugin, new audio code and surround sound upmixer, several new themes (Arclight and Childish), newly resynced ffmpeg libraries, and fixes for analog scanning, among many others.
" More information can be found in the release notes
Comments (none posted)
Version 1.3.0 of py.test - an automated testing tool for Python2 and
Python3 programs - has been released. There's a long list of new features;
click below for the changelog.
Full Story (comments: none)
Back in 2008, LWN covered the
story of Ryzom
, a popular multiplayer role-playing game which was owned
by a company in bankruptcy. An attempt to acquire and liberate the source
at that time failed, but good things come (sometimes) to those who wait: Ryzom is now available under the AGPLv3
. The artwork, too, is available under the Creative Commons
Attribution-ShareAlike license. "By freeing Ryzom code, Winch Gate
is transforming the MMORPG marketplace and is setting a precedent for how
gaming software should evolve--in freedom. The source code released totals
over two (2) million lines of source code and over 20,000 high quality
textures and thousands of 3D objects.
Comments (24 posted)
Newsletters and articles
Comments (none posted)
Yaniv Aknin has launched into a series of articles intended to describe how
Python works from the inside; thus far, the introductory
and Objects 101
installments are available. "I want to see Python more like Guido
does, so I'm starting out on what should develop to a series on Python's
internals. On the curriculum is mainly CPython, mainly py3k, mainly
bytecode evaluation (I'm not a big compilation fan) - but practically
everything around executing Python and Python-like code (Unladen Swallow,
Jython, Cython, etc) might turn out to be fair game in this series.
Comments (none posted)
Mike Beltzner, Mozilla's Director of Firefox, shares
some early plans for Firefox 4. "The primary goals for Firefox 4 will be making a browser: * Fast: making Firefox super-duper fast * Powerful: enabling new open, standard Web technologies (HTML5 and beyond!), * Empowering: putting users in full control of their browser, data, and Web experience.
Comments (33 posted)
Dave Phillips has a brief
"about a new Linux audio blog, music made by particle acceleration, how to use a laptop as a virtual music stand, synth emulation from the terminal command prompt, and watching the Linux Audio Conference on-line.
Comments (none posted)
On his blog, Christian Hammond looks back
at three-and-a-half years of development on the code review tool, Review Board
. "Attempt #3. I decided to build our own diff parser and generator from scratch. What a project. I knew nothing about diff generation and hardly knew where to start. I spent probably a good month or so just trying to work on this new diff code, and was so close to giving up so many times. It ended up being completely worth it, though, as we ended up with a very nice, extensible diff parser.
Without that third attempt, we'd be in the stone age. Review Board would not be as nice to use. We wouldn't have inter-line diffs (where we highlight what changed in a replace line), syntax highlighting, move detection (coming in 1.5), or function/class headers (where we show which function/class the part of the diff is in — also coming in 1.5).
Comments (1 posted)
Dan Williams, developer of NetworkManager, looks at D-Bus
on his blog. In particular, he has found himself periodically giving mini-lessons about D-Bus; he captures that in this explanation of D-Bus concepts along with a concrete example of the interprocess communication (IPC) mechanism.
"service: a program that responds to requests from clients. Each service is identified by a 'bus name' which clients use to find the service and send requests to it. The bus name usually looks like org.foobar.Baz. A program can claim more than one bus name; NM claims org.freedesktop.NetworkManager and org.freedesktop.NetworkManagerSystemSettings, each is a unique service which provides different functionality to clients.
Comments (33 posted)
Page editor: Jonathan Corbet
Next page: Announcements>>