![[LWN Logo]](/images/lwn.banner.gif)
To: linux-smp@vger.rutgers.edu
Subject: [FAQ] v0.14, 10 april 1998
From: David Mentre <David.Mentre@irisa.fr>
Date: 10 Apr 1998 14:57:38 +0200
Hello all Linux SMP users/hackers,
Here is the latest Linux SMP FAQ. As previously stated, I'm open to any
criticism, remarks, etc. This FAQ is yours. If you want to add a
section, just email me the relevant text.
Linux SMP FAQ
David Mentre', David.Mentre@irisa.fr
v0.14, 10 april 1998
This FAQ review main issues (and I hope solutions) related to SMP con-
figuration under Linux.
______________________________________________________________________
Table of Contents
1. Introduction
2. Questions related to any architectures
2.1 Kernel side
2.2 User side
3. Intel architecture specific questions
3.1 Why it doesn't work on my machine?
3.2 Possible causes of crash
3.3 Motherboard specific information
3.3.1 Motherboards with known problems
3.3.2 Motherboards with no known problems
4. Useful pointers
4.1 Various
4.2 SMP specific patches
5. Glossary
6. List of contributors
______________________________________________________________________
1. Introduction
Linux can work on SMP (Symetric Multi-Processors) machines. SMP
support has started with the 2.0 family and has been improved in the
2.1 (future 2.2) saga.
FAQ maintained by David Mentr (David.Mentre@irisa.fr). The latest
edition of this FAQ can be found at
http://www.irisa.fr/prive/mentre/smp-faq/.
If you want to contribute to this FAQ, I would prefer a diff against
the SGML version <http://www.irisa.fr/prive/mentre/smp-faq/smp-
faq.sgml> of this document, but any remarks (in plain text) will be
greatly appreciated.
This FAQ is an improvement of a first draft made by Chris Pirih.
All information contained in this FAQ is provided "as is." All
warranties, expressed, implied or statutory, concerning the accuracy
of the information of the suitability for any particular use are
hereby specifically disclaimed. While every effort has been taken to
ensure the accuracy of the information contained in this FAQ, the
authors assume(s) no responsibility for errors or omissions, or for
damages resulting from the use of the information contained herein.
2. Questions related to any architectures
2.1. Kernel side
1. Does Linux support multi-threading? If I start two or more
processes, will they be distributed among the available CPUs?
Yes.
2. What kind of architectures are supported in SMP?
As far as I (David Mentr) know, only Intel Pentium and UltraSparc
architectures are supported.
3. How do I make a Linux SMP kernel?
Uncomment the SMP=1 line in the main Makefile
(/usr/src/linux/Makefile).
AND
enable "RTC support" (from Robert G. Brown). Note that inserting
RTC support actually doesn't afaik prevent drift, but according to
a discussion [Robert G. Brown] remember from a year ago or so it
can prevent lockup when the clock is read at boot time.
AND
do not enable APM! APM and SMP are not compatible, and your system
will almost certainly crash under boot if APM is enabled. (Jakob
Oestergaard)
4. How do I make a Linux non-SMP kernel?
Comment the SMP=1 line in the Makefile (and not set SMP to 0).
5. How can I tell if it worked?
cat /proc/cpuinfo
Typical output (dual PentiumII):
______________________________________________________________________
processor : 0
cpu : 686
model : 3
vendor_id : GenuineIntel
stepping : 3
fdiv_bug : no
hlt_bug : no
fpu : yes
fpu_exception : yes
cpuid : yes
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic 11 mtrr pge mca cmov mmx
bogomips : 267.06
processor : 1
cpu : 686
model : 3
vendor_id : GenuineIntel
stepping : 3
fdiv_bug : no
hlt_bug : no
fpu : yes
fpu_exception : yes
cpuid : yes
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic 11 mtrr pge mca cmov mmx
bogomips : 267.06
______________________________________________________________________
6. What is the status of converting the kernel toward finer grained
locking and multithreading?
2.1.x has signal handling, interrupts and some I/O stuff fine grain
locked. The rest is gradually migrating. All the scheduling is SMP
safe
7. Does Linux SMP support processor affinity?
No and Yes. There is no way to force a process onto specific CPU's
but the linux scheduler has a processor bias for each process,
which tends to keep processes tied to a specific CPU.
2.2. User side
1. Do I really need SMP?
If you have to ask, you probably don't. :)
2. How does one display mutiple cpu performance?
Thanks to Samuel S. Chessman, here is some useful utilities:
Character based:
http://www.cs.inf.ethz.ch/~rauch/procps.html
Basically, it's procps v1.12.2 (top, ps, et. al.) and some
patches to support SMP.
Graphic:
xosview-1.5.1 supports SMP. And kernels above 2.1.85 (included)
have the /proc/stat/cpuX entry.
The official homepage for xosview is:
http://lore.ece.utexas.edu/~bgrayson/xosview.html
The various forissie kernel patches are at: http://www-
isia.cma.fr/~forissie/smp_kernel_patch/
3. How can I program to use two (or more CPUs) ?
Use a kernel-thread library. A good library, the pthread library
made by Xavier Leroy
<http://pauillac.inria.fr/~xleroy/linuxthreads/>;.
LinuxThread is now integrated with glibc2 (aka libc6).
From Jakob Oestergaard: Also consider using MPI. It's the industry
standard message passing interface. It doesn't give you shared
memory like threads, but it allows you to use your program in a
cluster too.
4. What has changed in the threads packages, linuxthread, etc.
Glibc is the big change. glibc is threadsafe and includes
linuxthreads Posix.4 threads by default. Real time signals are also
in glibc so POSIX AIO should also be in glibc2.1 (I hope).
5. How can I enable more than 1 process for my kernel compile?
use:
___________________________________________________________________
# make [modules|zImage|bzImages] MAKE="make -jX"
where X=max number of processes.
WARNING: This won't work for "make dep".
___________________________________________________________________
With a 2.1.x like kernel, see also the file
/usr/src/linux/Documentation/smp for specific instruction.
6. Why the time given by the time command is false ? (from Joel
Marchand)
In the 2.0 series, the result given by the time command is false.
The sum user+system is right *but* the spreading between user and
system time is false.
This bug in corrected in 2.1 series.
7. How will my application perform under SMP?
Look at SMP Performance of Linux
<http://www.interlog.com/~mackin/linux-smp.html>; which gives useful
hints how to bench a specific machine (from a post made by Cameron
MacKinnon).
8. Where can I found more information about parallel programming?
Look at the Linux Parallel Processing HOWTO
<http://yara.ecn.purdue.edu/~pplinux/PPHOWTO/pphowto.html>;
3. Intel architecture specific questions
3.1. Why it doesn't work on my machine?
1. Can I use my Cyrix/AMD/non-Intel CPU in SMP?
Short answer: no.
Long answer: Intel claims ownership to the APIC SMP scheme, and
unless a company licenses it from Intel they may not use it. There
are currently no companies that have done so. (This of course can
change in the future) FYI - Both Cyrix and AMD support the non-
proprietary OpenPIC SMP standard but currently there are no
motherboards that use it.
2. Why doesn't my old Compaq work?
Put it into MP1.1/1.4 compliant mode.
3. Why doesnt my ALR work?
Beats us too (see Alan Cox for this answer).
4. Why does SMP go so slowly?
If one of your CPU's is reporting a very low bogomips value your
vendor probably provides a buggy BIOS. Get the patch to work around
this or better yet send it back and buy a board from a competent
supplier.
5. I've heard IBM machines have problems
Some IBM machines have the MP1.4 bios block in the EBDA, allowed
but not supported by <2.1.80. Please update to the right kernel.
There is an old 486SLC based IBM SMP box. Linux/SMP requires
hardware FPU support.
6. Is there any advantage of Intel MP 1.4 over 1.1 specification?
Nope (according to Alan :) ), 1.4 is just a stricker specs of 1.1.
7. Why does the clock drift so rapidly when I run linux SMP?
This is known problem with IRQ handling and long kernel locks in
the 2.0 series kernels. Consider upgrading to a later 2.1 kernel
(not garenteed to work).
From Jakob Oestergaard: Or, consider running xntpd. That should
keep your clock right on time. (I think that I've heard that
enabling RTC in the kernel also fixes the clock drift. It works for
me! but I'm not sure whether that's general or I'm just being
lucky)
8. Why are my CPU's numbered 0 and 2 instead of 0 and 1 (or some other
odd numbering)?
The CPU number is assigned by the MB manufacturer and doesn't mean
anything. Ignore it.
9. Why does one CPU show a very low bogomips value while the first one
is normal?
The cache is not enabled on the second CPU. Check your BIOS
configuration for problems. If all else fails report the problem to
linux-smp. (I'm not clear on the proper answer to this one).
10.
My SMP system is locking up all the time. Black screen, nothing in
the logs. Help!
If you're running a 2.0 kernel, consider upgrading to later 2.0.32+
kernels or apply Leonard Zubkoff's deadlock patch. If you still
have deadlocks, apply Ingo Molnar's deadlock detection patch and
post the results (against your System.map) to linux-smp or linux-
kernel. You might also consider running a 2.1 kernel.
3.2. Possible causes of crash
You'll find in this section some possible reasons for a crash of an
SMP machine (credits are due to Jakob Oestergaard for this part). As
far as I (david) know, theses problems are Intel specific.
o Cooling problems
Get a huge casing with lots of fans.
o Bad memory
Don't buy too cheap RAM and don't use mixed RAM modules on a
motherboard that is picky about it.
Especially Tyan motherboards are known to be picky about RAM speed.
o Bad combination of different stepping CPUs
Check /proc/cpuinfo to see that your CPUs are same stepping.
o You are running 2.0.33 aren't you ?
If you run 2.0.31 or 2.1.xx you can't be sure that SMP is stable.
2.0.33 is the right kernel for a production system. 2.1.xx kernels
perform better, but they are development releases and should NOT be
considered stable!
o If your system is unstable, then DON'T overclock it!
Overclocking works great for a lot of people, but if your system is
unstable, overclocking may very well be the reason.
o 2.0.x kernel and fast ethernet (from Robert G. Brown)
2.0.X kernels on high performance fast ethernet systems have
significant (and known) problems with a race/deadlock condition in
the networking interrupt handler.
The solution is to get the latest 100BT development drivers from
CESDIS (ones that define SMPCHECK).
o A bug in the 440FX chipset (from Emil Briggs)
If you had a system using the 440FX chipset then your problem with
the lockups was possibly due to a documented errata in the chipset.
Here is a reference
References: Intel 440FX PCIset 82441FX (PMC) and 82442FX (DBX)
Specification Update. pg. 13
http://www.intel.com/design/pcisets/specupdt/297654.htm
The problem can be fixed with a bios workaround (Or a kernel patch)
and in fact David Wragg wrote a patch that's included with Richard
Gooch's mttr patch. For more information and a fix look here.
http://nemo.physics.ncsu.edu/~briggs/vfix.html
Some hardware is also known to cause problems. This includes:
o Adaptec SCSI controllers
Don't buy them, Adaptec is unsupportive to the linux developers.
This is not a SMP problem, but a general high-performance Linux
problem.
It also seems that aic7xxx driver is broken under SMP (from Robert
Hyatt).
(from Doug Ledford, author of the Adaptec driver) Just a quick
note, the 5.0.11 version of my driver for 2.0.33 is the one I
[Doug] personally recommend for SMP and/or PII systems. It's what
I use here on a PII/266 dual system, although I'm running 2.1.92
right now instead of 2.0.33. Second note, the patch will not go
into 2.0.34-pre6 cleanly, but can be used, and it has not been
submitted for any of the 34pre kernels because I don't think it's
had enough testing yet.
o 3Com 3c905 cards
Some work, some don't. Try disabling busmastering if your system is
unstable.
3.3. Motherboard specific information
Some more specific information can be found with the survey of SMP
motherboards <http://styx.phy.vanderbilt.edu/smp/mainboards.html>;
3.3.1. Motherboards with known problems
o Gigabyte
Solution: BIOS upgrade
o SuperMicro
Solution: BIOS upgrade
o EPoX KP6-LS (Christopher Allen Wing, 16 march 1998)
It appears to have the same BIOS related BogoMIPS problem as other
motherboards. (one CPU only gives about 3 BogoMIPS, the other gives
the full amount) All 2.0.x kernels lock up soon after booting, late
2.1.x kernels run slowly but don't seem to lock up. There is no
BIOS upgrade available (yet). I wrote the manufacturer but have not
received a reply.
o Tyan
Tyan motherboards are known to be picky about RAM speed (Jakob
Oestergaard).
From Doug Ledford about the onboard aic-7895 SCSI controller (for
which he wrote the driver): "BTW, make sure you have at least BIOS
version 1.16 on that Tyan motherboard. The 1.15 and below BIOS
versions have a bug relating to IRQ allocation for the 7895 SCSI
controller" (submitted by Szakacsits Szabolcs).
o GA686DLX (Andrew Crane)
Same BIOS related BogoMIPS problem as other motherboards.
Solution from Alan Cox: Congratulations, send the bill for your
hair damage to the supplier. You have yet another SMP box with
faulty bios. There is a patch for 2.0.x on www.uk.linux.org and
there are people working on generic MTRR handling for 2.1.x
3.3.2. Motherboards with no known problems
o AIR P6NDP and P6NDI (Leonard N. Zubkoff)
My primary production machine is based on an AIR P6NDP and one of
my test machines uses a P6NDI. Both seem to be fine motherboards
in my experience. The P6NDI BIOS is a little conservative in its
programming of the Natoma chipset for 50ns EDO, but a minor tweek
to one register in rc.local took care of that.
o AIR 54CDP (Chris Mauritz)
You can also list the following motherboard as working with no
problems:
AIR 54CDP motherboard / EISA/PCI / onboard aic7870 / dual P120 /
Redhat 5.0 (2.0.32 and 2.0.33 kernels)
4. Useful pointers
4.1. Various
o Parallel Processing using Linux
<http://yara.ecn.purdue.edu/~pplinux/>;
o (outdated) Linux SMP home page
<http://www.uk.linux.org/SMP/title.html>;
o linux-smp mailing list
To subscribe, send subscribe linux-smp in the message body at
majordomo@vger.rutgers.edu
To unsubscribe, send unsubscribe linux-smp in the message body at
majordomo@vger.rutgers.edu
o pthread library made by Xavier Leroy
<http://pauillac.inria.fr/~xleroy/linuxthreads/>;
o Linux SMP archives <http://www.linuxhq.com/lnxlists/linux-smp/>;
o Survey of SMP motherboards
<http://styx.phy.vanderbilt.edu/smp/mainboards.html>;
o procps <http://www.cs.inf.ethz.ch/~rauch/procps.html>;
o xosview <http://lore.ece.utexas.edu/~bgrayson/xosview.html>;
o Pentium Pro Optimized BLAS and FFTs for Intel Linux
<http://www.cs.utk.edu/~ghenry/distrib/>;
o SMP Performance of Linux <http://www.interlog.com/~mackin/linux-
smp.html>
o Multithreaded programs on linux
<http://www.e.kth.se/~e94_bek/mthread.html>;
o Linux Parallel Processing HOWTO
<http://yara.ecn.purdue.edu/~pplinux/PPHOWTO/pphowto.html>;
4.2. SMP specific patches
o Forissie kernel patches <http://www-
isia.cma.fr/~forissie/smp_kernel_patch/>
o Patch for a bug in the 440FX chipset
<http://nemo.physics.ncsu.edu/~briggs/vfix.html>;
o MTRR patch (latest version: 1.9)
<http://www.atnf.csiro.au/~rgooch/kernel-patches.html>;
5. Glossary
o SMP Symetric Multi-Processors
o APIC Advanced Programmable Interrupt Controler
o thread A thread is a processor activity in a process. The same
process can have multiple threads. Those threads share the process
address space and can therefore share data.
o pthread Posix thread, threads defined by the Posix standard.
6. List of contributors
Many thanks to those who help me to maintain this FAQ.
o Emil Briggs
o Robert G. Brown
o Samuel S. Chessman
o Alan Cox
o Andrew Crane
o Jocelyne Erhel
o Byron Faber
o Robert Hyatt
o Tony Kocurko
o Doug Ledford
o Cameron MacKinnon
o Joel Marchand
o Chris Mauritz
o Jakob Oestergaard
o Jean-Michel Rouet
o Sumit Roy
o Szakacsits Szabolcs
o El Warren
o Christopher Allen Wing
o Leonard N. Zubkoff
d.
--
David.Mentre@irisa.fr -- http://www.irisa.fr/prive/dmentre/
Linux SMP FAQ: http://www.irisa.fr/prive/dmentre/smp-faq/
== GNU & Linux: Change _our_ world ==