Intel Skylake/Kaby Lake processors: broken hyper-threading [LWN.net]

Henrique de Moraes Holschuh has posted an advisory about a processor/microcode defect recently identified on Intel Skylake and Intel Kaby Lake processors with hyper-threading enabled. "TL;DR: unfixed Skylake and Kaby Lake processors could, in some situations, dangerously misbehave when hyper-threading is enabled. Disable hyper-threading immediately in BIOS/UEFI to work around the problem. Read this advisory for instructions about an Intel-provided fix."

From:		Henrique de Moraes Holschuh <hmh-AT-debian.org>
To:		debian-user-AT-lists.debian.org, debian-devel-AT-lists.debian.org
Subject:		[WARNING] Intel Skylake/Kaby Lake processors: broken hyper-threading
Date:		Sun, 25 Jun 2017 09:19:36 -0300
Message-ID:		<20170625121936.GA7714@khazad-dum.debian.net>

This warning advisory is relevant for users of systems with the Intel
processors code-named "Skylake" and "Kaby Lake".  These are: the 6th and
7th generation Intel Core processors (desktop, embedded, mobile and
HEDT), their related server processors (such as Xeon v5 and Xeon v6), as
well as select Intel Pentium processor models.

TL;DR: unfixed Skylake and Kaby Lake processors could, in some
situations, dangerously misbehave when hyper-threading is enabled.
Disable hyper-threading immediately in BIOS/UEFI to work around the
problem.  Read this advisory for instructions about an Intel-provided
fix.


SO, WHAT IS THIS ALL ABOUT?
---------------------------

This advisory is about a processor/microcode defect recently identified
on Intel Skylake and Intel Kaby Lake processors with hyper-threading
enabled.  This defect can, when triggered, cause unpredictable system
behavior: it could cause spurious errors, such as application and system
misbehavior, data corruption, and data loss.

It was brought to the attention of the Debian project that this defect
is known to directly affect some Debian stable users (refer to the end
of this advisory for details), thus this advisory.

Please note that the defect can potentially affect any operating system
(it is not restricted to Debian, and it is not restricted to Linux-based
systems).  It can be either avoided (by disabling hyper-threading), or
fixed (by updating the processor microcode).

Due to the difficult detection of potentially affected software, and the
unpredictable nature of the defect, all users of the affected Intel
processors are strongly urged to take action as recommended by this
advisory.


DO I HAVE AN INTEL SKYLAKE OR KABY LAKE PROCESSOR WITH HYPER-THREADING?
-----------------------------------------------------------------------

The earliest of these Intel processor models were launched in September
2015.  If your processor is older than that, it will not be an Skylake
or Kaby Lake processor and you can just ignore this advisory.

If you don't know the model name of your processor(s), the command below
will tell you their model names.  Run it in a command line shell (e.g.
xterm):

    grep name /proc/cpuinfo | sort -u

Once you know your processor model name, you can check the two lists
below:

  * List of Intel processors code-named "Skylake":
    http://ark.intel.com/products/codename/37572/Skylake

  * List of Intel processors code-named "Kaby Lake":
    http://ark.intel.com/products/codename/82879/Kaby-Lake

Some of the processors in these two lists are not affected because they
lack hyper-threading support.  Run the command below in a command line
shell (e.g. xterm), and it will output a message if hyper-threading is
supported/enabled:

  grep -q '^flags.*[[:space:]]ht[[:space:]]' /proc/cpuinfo && \
	echo "Hyper-threading is supported"

Alternatively, use the processor lists above to go to that processor's
information page, and the information on hyper-threading will be there.

If your processor does not support hyper-threading, you can ignore this
advisory.


WHAT SHOULD I DO IF I DO HAVE SUCH PROCESSORS?
----------------------------------------------

Kaby Lake:

Users of systems with Intel Kaby Lake processors should immediately
*disable* hyper-threading in the BIOS/UEFI configuration.  Please
consult your computer/motherboard's manual for instructions, or maybe
contact your system vendor's support line.

The Kaby Lake microcode updates that fix this issue are currently only
available to system vendors, so you will need a BIOS/UEFI update to get
it.  Contact your system vendor: if you are lucky, such a BIOS/UEFI
update might already be available, or undergoing beta testing.

You want your system vendor to provide a BIOS/UEFI update that fixes
"Intel processor errata KBL095, KBW095 or the similar one for my Kaby
Lake processor".

We strongly recommend that you should not re-enable hyper-threading
until you install a BIOS/UEFI update with this fix.


Skylake:

Users of systems with Intel Skylake processors may have two choices:

1. If your processor model (listed in /proc/cpuinfo) is 78 or 94, and
   the stepping is 3, install the non-free "intel-microcode" package
   with base version 3.20170511.1, and reboot the system.  THIS IS
   THE RECOMMENDED SOLUTION FOR THESE SYSTEMS, AS IT FIXES OTHER
   PROCESSOR ISSUES AS WELL.

   Run this command in a command line shell (e.g. xterm) to know the
   model numbers and steppings of your processor.  All processors must
   be either model 78 or 94, and stepping 3, for the intel-microcode fix
   to work:

         grep -E 'model|stepping' /proc/cpuinfo | sort -u

   If you get any lines with a model number that is neither 78 or 94, or
   the stepping is not 3, you will have to disable hyper-threading as
   described on choice 2, below.

   Refer to the section "INSTALLING THE MICROCODE UPDATES FROM NON-FREE"
   for instructions on how to install the intel-microcode package.

2. For other processor models, disable hyper-threading in BIOS/UEFI
   configuration.  Please consult your computer/motherboard's manual for
   instructions on how to do this.  Contact your system vendor for a
   BIOS/UEFI update that fixes "Intel erratum SKW144, SKL150, SKX150,
   SKZ7, or the similar one for my Skylake processor".

NOTE: If you did not have the intel-microcode package installed on your
Skylake system before, it is best if you check for (and install) any
BIOS/UEFI updates *first*.  Read the wiki page mentioned below.


INSTALLING THE MICROCODE UPDATES FROM NON-FREE:
-----------------------------------------------

Instructions are available at:

    https://wiki.debian.org/Microcode

Updated intel-microcode packages are already available in non-free for:
unstable, testing, Debian 9 "stretch" (stable), and Debian 8 *backports*
(jessie-backports).

THE MICROCODE PACKAGES FROM THE RECENT STABLE RELEASE (June 17th, 2017)
ALREADY HAVE THE SKYLAKE FIX, BUT YOU MAY HAVE TO INSTALL THEM.

Updated intel-microcode packages in non-free for Debian 8 "jessie"
(oldstable) are waiting for approval and will likely be released in the
next non-free oldstable point release.  They are the same as the
packages in non-free jessie-backports, with a change to the version
number.

The wiki page above has instructions on how to enable "contrib" and
"non-free", so as to be possible to install the intel-microcode package.

Users of "jessie" (oldstable) might want to enable jessie-backports to
get *this* intel-microcode update faster.  This is also explained in the
wiki page above.


MORE DETAILS ABOUT THE PROCESSOR DEFECT:
----------------------------------------

On 2017-05-29, Mark Shinwell, a core OCaml toolchain developer,
contacted the Debian developer responsible for the intel-microcode
package with key information about a Intel processor issue that could be
easily triggered by the OCaml compiler.

The issue was being investigated by the OCaml community since
2017-01-06, with reports of malfunctions going at least as far back as
Q2 2016.  It was narrowed down to Skylake with hyper-threading, which is
a strong indicative of a processor defect.  Intel was contacted about
it, but did not provide further feedback as far as we know.

Fast-forward a few months, and Mark Shinwell noticed the mention of a
possible fix for a microcode defect with unknown hit-ratio in the
intel-microcode package changelog.  He matched it to the issues the
OCaml community were observing, verified that the microcode fix indeed
solved the OCaml issue, and contacted the Debian maintainer about it.

Apparently, Intel had indeed found the issue, *documented it* (see
below) and *fixed it*.  There was no direct feedback to the OCaml
people, so they only found about it later.

The defect is described by the SKZ7/SKW144/SKL150/SKX150/KBL095/KBW095
Intel processor errata.  As described in official public Intel
documentation (processor specification updates):

  Errata:   SKZ7/SKW144/SKL150/SKX150/SKZ7/KBL095/KBW095
            Short Loops Which Use AH/BH/CH/DH Registers May Cause
            Unpredictable System Behavior.

  Problem:  Under complex micro-architectural conditions, short loops
	    of less than 64 instructions that use AH, BH, CH or DH
	    registers as well as their corresponding wider register
	    (e.g. RAX, EAX or AX for AH) may cause unpredictable
	    system behavior. This can only happen when both logical
	    processors on the same physical processor are active.

  Implication: Due to this erratum, the system may experience
	    unpredictable system behavior.

We do not have enough information at this time to know how much software
out there will trigger this specific defect.

One important point is that the code pattern that triggered the issue in
OCaml was present on gcc-generated code.  There were extra constraints
being placed on gcc by OCaml, which would explain why gcc apparently
rarely generates this pattern.

The reported effects of the processor defect were: compiler and
application crashes, incorrect program behavior, including incorrect
program output.


What we know about the microcode updates issued by Intel related to
these specific errata:

Fixes for processors with signatures[1] 0x406E3 and 0x506E3 are
available in the Intel public Linux microcode release 20170511.  This
will fix only Skylake processors with model 78 stepping 3, and model 94
stepping 3.  The fixed microcode for these two processor models reports
revision 0xb9/0xba, or higher.

Apparently, these errata were fixed by microcode updates issued in early
April/2017.  Based on this date range, microcode revision 0x5d/0x5e (and
higher) for Kaby Lake processors with signatures 0x806e9 and 0x906e9
*might* fix the issue.  We do not have confirmation about which
microcode revision fixes Kaby Lake at this time.

Related processor signatures and microcode revisions:
Skylake   : 0x406e3, 0x506e3 (fixed in revision 0xb9/0xba and later,
                              public fix in linux microcode 20170511)
Skylake   : 0x50654          (no information, erratum listed)
Kaby Lake : 0x806e9, 0x906e9 (defect still exists in revision 0x48,
                              fix available as a BIOS/UEFI update)


References:
https://caml.inria.fr/mantis/view.php?id=7452
http://metadata.ftp-master.debian.org/changelogs/non-free...
https://www.intel.com/content/www/us/en/processors/core/d...
https://www.intel.com/content/www/us/en/processors/core/7...
https://www.intel.com/content/www/us/en/processors/xeon/x...
https://www.intel.com/content/www/us/en/processors/xeon/x...
https://www.intel.com/content/www/us/en/products/processo...

[1] iucode_tool -S will output your processor signature.  This tool is
    available in the *contrib* repository, package "iucode-tool".

to post comments

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 26, 2017 22:18 UTC (Mon) by SEJeff (guest, #51588) [Link] (2 responses)

I wonder if this is a problem in RHEL / Fedora that has been already addressed, or still needs to be.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 27, 2017 7:57 UTC (Tue) by TimSmall (guest, #96681) [Link]

No fix on CentOS 7 as of the time of writing. A quick hack is to copy over /lib/firmware/intel-ucode/ from a Debian box, followed by:

# verify processors and microcode versions
grep -E 'model|stepping|code' /proc/cpuinfo | sort -u
# request microcode reload
echo 1 > /sys/devices/system/cpu/microcode/reload
# verify upgrade
dmesg | grep microcode

... this works with the stock CentOS 7 3.10 kernel, but may not work on newer kernels depending on kernel options IIRC. In general there is a small risk to doing this on a booted system, because CPU features which running software are using could suddenly disappear if the microcode update disables them. This is why this is normally done very early in the boot sequence by the initramfs, assuming this is the case on CentOS 7, then the initramfs will probably need to be rebuilt with "dracut -f" to get this fix to persist across reboots (and/or apply it in a safer way in general).

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 27, 2017 13:41 UTC (Tue) by tdz (subscriber, #58733) [Link]

I checked yesterday on my Skylake-based system with Fedora 25. The system had the patched microcode and the BIOS was from Nov last year. So I guess the BIOS didn't fix is yet and chances are that Fedora has the patch already.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 26, 2017 23:05 UTC (Mon) by antiphase (subscriber, #111993) [Link] (21 responses)

I don't doubt that there is a flaw here but given that the oldest of these processors are approaching 2 years old and have been manufactured and used in huge volumes since, and this is apparently the first warning of their DANGEROUS MISBEHAVIOUR, is this possibly not a bit sensationalist?

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 26, 2017 23:12 UTC (Mon) by k8to (guest, #15413) [Link] (4 responses)

Hard to evaluate. Sometimes problems go undetected for some time even when affecting people, if the expression of the problem isn't obvious enough, or is scattered enough to not be tied to the hardware.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 27, 2017 1:48 UTC (Tue) by areilly (guest, #87829) [Link] (3 responses)

When you think about the nature of the issue (mixed byte and word access specifically involving the high byte of the four registers that have one, in a tight loop), it seems very unlikely that there will be much compiled code that has this particular pattern in it. Indeed, I couldn't find any on a quick scan over my system. On modern x86_64, compilers will be using the %[abcd]l registers if they ever want bytes, because those correspond to the only available byte-chunks of the %r[8-15]l registers. A write to one of the %...h bytes in a tight loop. Can't think of a use-case, myself. Makes me wonder what the OCaml folk were doing with it, to find the pattern.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 27, 2017 4:36 UTC (Tue) by eru (subscriber, #2753) [Link] (2 responses)

Would it not be a reasonable translation of char b = i >> 8? (with no other usage of the value i >> 8 nearby).

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 27, 2017 5:11 UTC (Tue) by areilly (guest, #87829) [Link]

Yes, and I've just verified that clang (at least) will generate movb %dh, 0(%rax) in at least one test case. Put it in a loop though and it'll happily vectorise it into shifts and shuffles.

That use doesn't feel like one that would upset a modern (threaded) pipeline though, because there is no chance of a write-after-read style problem. I think that the issue is much more likely to involve writing to one of the [abcd]h registers, which would need to be unpacked into a read-modify-update sequence of micro-ops. For "make an int from left-shifted bytes" style code, I've only seen the compiler generate byte loads followed by left-shifts, rather than loading byte values directly into high reg bytes.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 27, 2017 17:46 UTC (Tue) by pbonzini (subscriber, #60935) [Link]

Yes, but the compiler likely would only generate it at -Os (optimize for size).

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 26, 2017 23:18 UTC (Mon) by jgg (subscriber, #55211) [Link] (3 responses)

Probably.. The remarkable thing here is that Intel's internal randomised testing wasn't able to uncover this in the last two years. Usually once a major CPU is produced in silicon there is extensive randomised A/B testing performed on instruction sequences to look for variations relative to the golden architectural model, so this must be a remarkably unlikely combination of things if testing of two generations of CPU failed to uncover it.. Often this kind of testing finds all kinds of crazy and unlikely stuff.

The truth is new CPU architectures are usually pretty buggy in the first couple of steppings. The are full of disable bits and configuration stuff that the CPU vendor recommends to the BIOS to setup..

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 27, 2017 6:35 UTC (Tue) by ncm (guest, #165) [Link] (2 responses)

This should not be surprising. All modern CPUs are filled with bugs, just as all useful programs (excepting DKK's and DJB's) are filled with bugs and security holes. You only worry about the ones causing problems right now, because there isn't enough worrying capacity in the world to cover them all. You are much more likely to die from a car accident or cancer than to suffer much from a CPU bug.

But wouldn't catch me depending on one, no sirree.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 30, 2017 8:44 UTC (Fri) by marcH (subscriber, #57642) [Link] (1 responses)

> You are much more likely to die from a car accident or cancer than to suffer much from a CPU bug.

Soon we'll die from a car accident *and* a CPU bug.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Dec 19, 2017 23:57 UTC (Tue) by immibis (subscriber, #105511) [Link]

And then a car accident *caused by* a CPU bug.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 27, 2017 1:58 UTC (Tue) by NightMonkey (subscriber, #23051) [Link] (1 responses)

My Dell Precision 5510 laptop has a matching signature. Don't laptops often lag behind servers and desktops in processor chipset deployments?

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jul 2, 2017 14:59 UTC (Sun) by anton (subscriber, #25547) [Link]

No, laptops are sometimes first (e.g., Kaby Lake, Broadwell, Carrizo).

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 27, 2017 3:45 UTC (Tue) by rsidd (guest, #2582) [Link]

Silent data corruption, even if unlikely but possible, would be seen by some of us as sufficiently dangerous to warrant this warning. Unfortunately, particularly in Windows-land, people are likely to attribute it to things other than a CPU bug, especially if it is rare.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 27, 2017 9:51 UTC (Tue) by edeloget (subscriber, #88392) [Link] (1 responses)

Considering that abnormal behaviors are likely to be exploited by malicious users (at least to perform some kind of DOS attack), and considering that many servers out there feature these processors, I would not call that "sensationalism".

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 27, 2017 12:11 UTC (Tue) by nix (subscriber, #2304) [Link]

Skylake and Kaby Lake are too new to appear in any but very expensive server-class processors until very recently (the last few months). v4 and even v3 (Broadwell and older) are still on sale in fairly expensive servers. Server users are fairly conservative, often because of, well, bugs like this...

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 27, 2017 14:32 UTC (Tue) by rahvin (guest, #16953) [Link]

It's not overblown if your computer is the one that keeps crashing.

Does it need immediate patching? That would depend I guess on whether you've experienced a bunch of unexplained crashes or corrupted data. From some of what I read it's a pretty difficult bug to trigger but if you're the guy with the code that's triggering it you probably want a solution. One thing I can say, it's pretty friggen cool that Linux saw the patch so fast, it'll be a month before windows gets it.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 27, 2017 17:48 UTC (Tue) by dlang (guest, #313) [Link]

I remember the Pentium floating point bug. It took a while for it to be detected and tracked down as well.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 29, 2017 9:03 UTC (Thu) by Sesse (subscriber, #53779) [Link] (4 responses)

Yes, it's completely overblown. There are hundreds of similar CPU bugs in any given modern CPU, so there's no reason why this specific one (triggered only by code written in a niche programming language) should be specially publicized.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 29, 2017 9:21 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

Hundreds?!? Can you name a few?

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 29, 2017 10:03 UTC (Thu) by Sesse (subscriber, #53779) [Link]

https://www.intel.com/content/dam/www/public/us/en/docume... lists 173 issues in Haswell alone. Of things that can be hit by an unprivileged user:

HSD45. Certain Combinations of AVX Instructions May Cause Unpredictable System Behavior (sounds familiar?)
HSD136. Software Using Intel(R) TSX May Result In Unpredictable System Behavior (widely published, eventually was so bad that Intel had to rip out TSX from all of Haswell)
HSD142. Interactions Between Multiple Unaligned Memory Accesses And Locked Instructions May Lead to a Machine Check (try to write lock-free code? expect a crash)
HSD145. Operand-Size Override Prefix Causes 64-bit Operand Form of MOVBE Instruction to Cause a #UD (try to load big-endian data? expect a crash)
HSD162. MOVNTDQA From WC Memory May Pass Earlier Locked Instructions (try to use non-temporal loads from GPU memory? your serialization guarantees are off)

This isn't unique to Intel. See e.g. the case a while back where Ryzen would overheat and die if you ran a bunch of AVX muladds in parallel without ever touching memory.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 29, 2017 9:28 UTC (Thu) by jem (subscriber, #24231) [Link] (1 responses)

Hundreds of similar bugs? Do you have a source for this? Yes, CPUs are buggy, but the bugs are often shortcomings in supervisor level units (like the MMU or cache handling), that can be worked around in the operating system code. This time we are talking about "unpredictable behaviour" that can be triggered by a simple sequence of user level instructions.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 29, 2017 11:09 UTC (Thu) by excors (subscriber, #95769) [Link]

Depends what you mean by "similar", and "hundreds" might be an exaggeration if you only count a particular type of bug, though I think it's still slightly worrying to see what a fragile foundation our software is running on (and amazing that in practice it works as well as it does).

ARM seems to avoid publishing errata publicly itself, but e.g. http://www.nxp.com/docs/pcn_attachments/15997_IMX6DQCE.pdf lists about 130 errata for an SoC, including many CPU bugs (Cortex-A9 in this case), of which several sound like they can be triggered by userspace code, including:

"ARM: 743622 - Faulty logic in the Store Buffer may lead to data corruption", triggered by a particular sequence of writes. (Workaround is to disable some Store Buffer optimisation.)

"ARM: 751473 - Under very rare circumstances, Automatic Data prefetcher can lead to deadlock or data corruption". (Workaround is to disable automatic data prefetcher (with minimal effect on performance).)

"ARM: 743623 - Bad interaction between a minimum of seven PLDs and one Non-Cacheable LDM can lead to a deadlock". (Workaround is to not do that, or to globally disable PLD instructions (with significant effect on performance).)

There are also things like https://gcc.gnu.org/ml/gcc-patches/2014-10/msg00906.html (Cortex-A53 multiply-accumulate can give incorrect result) and https://sourceware.org/ml/binutils/2015-03/msg00446.html (Cortex-A53 can get incorrect result from an ADRP instruction near the end of a 4KB page), where the workarounds involve patching the toolchain and rebuilding all the software in the entire world (which is still much cheaper than recalling all the chips).

That's not hundreds, but it seems unsurprising to find several user-triggerable bugs like this in each new CPU. This Intel one seems relatively mild since it can be fixed with a microcode update, no rebuilding or recalling needed.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 27, 2017 8:00 UTC (Tue) by cpanceac (guest, #80967) [Link] (4 responses)

Is there a way to reproduce this? I'm interested in testing it on another processor(s) where we experience "file not found" in Yocto builds when the file is actually there.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 27, 2017 12:14 UTC (Tue) by nix (subscriber, #2304) [Link]

Honestly that doesn't sound like a hyperthreading bug to me, but more like a software problem. A thousand things can cause that symptom from filesystem bugs through networking bugs through obscure dcache bugs through things like the combination of 32-bit code compiled without -D_FILE_OFFSET_BITS=64, filesystems with 64-bit inodes, and code that assumes that stat() failures must be -EEXIST and that e.g. -EOVERFLOW is impossible.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 27, 2017 13:15 UTC (Tue) by chris.sykes (guest, #54374) [Link]

When I've encountered bugs like this in my Yocto builds, they've almost always been due to race conditions exposed by parallel builds.

A favourite is concurrent recipe and recipe-native builds.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 27, 2017 22:34 UTC (Tue) by tialaramex (subscriber, #21167) [Link] (1 responses)

Three things, in order:

1. Human error in your specific case. Like, you wrote "foo.jpg" in one place and "foo.jpeg" in another place and those aren't the same even though in your _head_ they seem like the same. Or you copy-pasted an example you found on Stack Overflow without understanding why the circumstances it was written for differ from yours...

2. Software error: Software is also written by humans, but other humans besides you use it, so they'd be somewhat likely to find bugs first. Statistically you don't see the bugs. But only statistically. Example: the build software has a bug.

3. Hardware error: The computer doesn't actually compute correctly.

Always assume the problem is (1), because this assumption will virtually always prove correct. Your comment suggests you haven't embraced this assumption, which is going to be embarrassing for you over and over until you learn.

Intel Skylake/Kaby Lake processors: broken hyper-threading

Posted Jun 28, 2017 0:35 UTC (Wed) by excors (subscriber, #95769) [Link]

On the other hand, if you state very publicly and with great confidence that you are sure there is a bug in someone else's software/hardware/etc because you've checked extremely carefully and there's no possible way the problem could have been caused by your own code, and especially if you imply they are idiots for making such mistakes, then within a couple of hours (just slightly too late to retract your accusations) you are highly likely to discover the blindingly obvious bug in your own code. The universe seems to have a tendency to maximise embarrassment like that.

If you're willing to put up with the embarrassment, it's a very quick way to get your problems solved.