LWN: Comments on "The dark side of expertise"

Re: Diversity in decision-making bodies

Nemo_bis — Wed, 15 Apr 2020 12:31:00 +0000

That's one of the central arguments of James Surowiecki's The Wisdom of Crowds.
https://en.wikipedia.org/wiki/The_Wisdom_of_Crowds#Five_e...

The dark side of expertise

akupries — Sat, 01 Feb 2020 00:48:16 +0000

There is also a dynamic variant of this illusion:

https://www.giannisarcone.com/Muller_lyer_illusion.html

The dark side of expertise

Wol — Fri, 24 Jan 2020 19:55:15 +0000

> When people walked across the bridge, it began to sway laterally.

And ALL the literature is (or was) about bridges BOUNCING VERTICALLY.

THAT - was the problem, the engineers knew all about vertical bounce and lateral sway never crossed their minds.

Cheers,
Wol

The dark side of expertise

jerojasro — Fri, 24 Jan 2020 18:22:46 +0000

This reminds me of a blog post ( https://danluu.com/wat/ ) that discusses a somewhat similar phenomenon: normalization of deviance, accepting broken things and rationalizing their continued existence.

The dark side of expertise

gerdesj — Fri, 24 Jan 2020 09:53:17 +0000

The Millennium Bridge failure was pinned on people walking in lock-step. The failure was still due to resonance. This one was even more embarrassing than the Tacoma Narrows disaster. Everyone + dog knows that soldiers are told to break stride when crossing a bridge. I think that effect was known about from the 19th C.

When people walked across the bridge, it began to sway laterally. This swaying makes people start to sync up their walking rhythm. You will almost certainly have experienced this somewhere on a small scale. The bridge now has several pistons attached to damp down the lateral movement. Any physicist will be able to describe and quantify the damped oscillation that they learned about at school, aged about 17!

The dark side of expertise

Wol — Thu, 23 Jan 2020 21:16:25 +0000

> Resonance was still the fault. You call it flutter and that was the input but not the cause. The wind made it shake a bit, then a bit more, then a lot more then it shook itself to bits in a quite horrible failure.

Except we probably need to check Snopes. I remember seeing some program that actually investigated the failure, and wind alone could not cause what happened.

Yes it shook itself to bits, but there was some prior failure without which it would not have been anywhere near as bad.

Cheers,
Wol

When the experts say "trust us"

Wol — Thu, 23 Jan 2020 21:09:48 +0000

PANIC !!!

I was aware of some studies on "expert disasters" that said that the one thing in common with nearly all of them was that there were no "outsiders" on the driving committee.

And the converse - adding a small number of people who weren't experts to the steering committee made a noticeable improvement in the committee's decision making.

Cheers,
Wol

The dark side of expertise

Wol — Thu, 23 Jan 2020 20:48:53 +0000

Yup, I remember the Millenium Bridge. A classic example.

As I remember it, all the textbooks talk about VERTICAL resonance. And nobody thought of HORIZONTAL resonance. Plus, this same problem had recurred at maybe 20-year intervals, and each time it was rather hushed up out of embarrassment that the problem wasn't spotted. With the result that it didn't get into the literature, didn't get into the engineering consciousness, and got repeated again and again.

Cheers,
Wol

The dark side of expertise

ringerc — Wed, 22 Jan 2020 01:53:22 +0000

If you enjoy this, read Bruce Schneier's work like Beyond Fear. Also check out the amazing blog/book "You are not so Smart" (https://youarenotsosmart.com/) .

Questioning your perceptions and reasoning is powerful.

The dark side of expertise

ssmith32 — Tue, 21 Jan 2020 04:57:34 +0000

> That's not to say that expertise won't occasionally lead you astray while troubleshooting, of course.

Or when running / interpreting social psychology experiments ;)

The dark side of expertise

mirabilos — Tue, 21 Jan 2020 00:22:53 +0000

+1, Funny

The dark side of expertise

emk — Sun, 19 Jan 2020 16:51:08 +0000

There's a nice summary of priming research here, and just how much of it has been recently been cast into doubt: https://replicationindex.com/2017/02/02/reconstruction-of...

I suppose this leads to another failure mode of expertise: Trusting what you find in journals too uncritically.

That's not to say that expertise won't occasionally lead you astray while troubleshooting, of course.

The dark side of expertise

mtaht — Sat, 18 Jan 2020 09:52:13 +0000

This was a really great talk that made me deeply reflect on multiple times "expertise" had screwed me up.

One example was not long in following... in my talk at this conference ("how congestion control really works in the bufferbloated age", blatant plug: https://www.youtube.com/watch?v=ZeCIbCzGY6k ), I'd set up people to act as packets, and, being american, assumed that the sending queue was going to line up on the right side, not the left, and being that we were in australia, all the participants naturally wanted to line up on the left, instead, to start with...

much "packet re-ordering" (and hilarity) ensued before we straightened it out.

The dark side of expertise

smitty_one_each — Fri, 17 Jan 2020 21:01:41 +0000

Calls for a celebratory haiku:

Somehow and somewhere
Something has gone wrong. I can
Be less specific.

The dark side of expertise

jcm — Fri, 17 Jan 2020 07:43:41 +0000

“The tests were designed by "evil psychologists" to send your brain down the wrong solution path, he said; once that happens, "you cannot stop it"”

Ok, someone has to say it. Speculative execution side-channel attack. Thanks. Byeee.

The dark side of expertise

madhatter — Fri, 17 Jan 2020 07:37:38 +0000

Very true. Once upon a time my favourite Oracle DBA turned to me, head in hands, and said "Somehow, somewhere, something has gone wrong". It's still one of the best problem reports I've ever received, since it embedded no assumptions, but instead provided me an opportunity to ask questions to home in on the precise nature, and then the likely causes, of the fault.

When I left that job, I had three t-shirts made up, for me, him, and the Windows admin, with those six words on, just to remind us of how helpful a non-misleading problem report can be.

The dark side of expertise

smitty_one_each — Fri, 17 Jan 2020 05:00:46 +0000

When I look at an error report, I try to keep in mind the non-zero likelihood that whatever the system returned as an error may or may not have anything to do with the actual problem.

It's just so easy to get tunnel vision and head off in the wrong direction when troubleshooting.

The dark side of expertise

gerdesj — Fri, 17 Jan 2020 01:31:32 +0000

As you say, not soldiers, in the case of the Tacoma Narrows bridge it was wind. I didn't mention that because it was self evident in my head!

Resonance was still the fault. You call it flutter and that was the input but not the cause. The wind made it shake a bit, then a bit more, then a lot more then it shook itself to bits in a quite horrible failure.

As a result the cross section of suspension bridges was significantly changed. I can't quite remember but I would imagine that they now have negative lift ie the faster the wind, the more down force. Extra 5 mm of steel all round to account for the extra force (I made 5 mm up but probably not too far off).

The dark side of expertise

Cyberax — Fri, 17 Jan 2020 01:16:46 +0000

> The Tacoma Narrows bridge breakup is a classic. I was a Civ Eng student at Plymouth Polytechnic (Devon, UK) in 1990ish and it was taught as a lesson pretty early on. The basic lesson is resonance - it shook itself to bits.
Not quite. It was caused by flutter, a self-reinforcing behavior when a shape can conform to aerodynamic forces, and enter a mode where the forces can start to amplify.

It was not a simple resonance, no amount of marching soldiers could have collapsed the bridge.

The dark side of expertise

gerdesj — Fri, 17 Jan 2020 00:55:41 +0000

Jake - cracking write up. Obviously something piqued your inner journo here.

My favourite Civ Eng screw up to add to the Struct Eng buggeration mentioned already is this:

The Tacoma Narrows bridge breakup is a classic. I was a Civ Eng student at Plymouth Polytechnic (Devon, UK) in 1990ish and it was taught as a lesson pretty early on. The basic lesson is resonance - it shook itself to bits. When a large military force marches across a bridge, they break step. Resonance - the synchronised steps of the soldiers can soon cause structures like bridges of various designs to fail in ways that the designer never thought of.

The London Millenium bridge - https://en.wikipedia.org/wiki/Millennium_Bridge,_London was an accident waiting to happen that all of the designers of the bloody thing would have been taught about or heard of.

I only studied Civ Eng. I ended up in IT but I clearly remember seeing the new bridge opening on TV and thinking it will suffer from lateral stability issues of some sort. How the hell can a near amateur see what the experts couldn't. Don't get me wrong, the design is a pretty clever spin on the suspension bridge by putting the suspension structure off to the side but that means that you lose vertical control, which also often determines lateral control. I've made up those terms because it is quite hard to describe how a 3D structure works. You sort of see it in your head and know when it should work. The actual numbers need quantifying but you can see obvious oddities. Despite being an IT bod I can still chase a 3D frame and see the stresses and strains in my head in terms of direction and a taste of magnitude.

Now I come to look at it that is probably not too normal. Are there any real Structural or Civil Engineers out there that can provide some insight?

The dark side of expertise

monty55 — Thu, 16 Jan 2020 23:52:19 +0000

See also "educated incapacity", originally "trained incapacity":

https://www.hudson.org/research/2219-the-expert-and-educa...

The dark side of expertise

jake — Thu, 16 Jan 2020 20:14:59 +0000

> but am I the only one who sees the lines with inward-pointing arrows as longer?

oops, no ... not sure how that slipped through, but I have adjusted the text.

thanks,

jake

The dark side of expertise

am — Thu, 16 Jan 2020 19:47:33 +0000

No, that's how most people perceive it. From Wikipedia:

The line segment forming the shaft of the arrow with two tails is perceived to be longer than that forming the shaft of the arrow with two heads.

The dark side of expertise

kid_meier — Thu, 16 Jan 2020 17:09:00 +0000

No I see the line with both inward pointing arrows as longer too.

The dark side of expertise

kmweber — Thu, 16 Jan 2020 16:27:24 +0000

Beside the point, I know, but am I the only one who sees the lines with inward-pointing arrows as longer?

The dark side of expertise

mathstuf — Thu, 16 Jan 2020 15:26:52 +0000

Some of the episodes in Cautionary Tales[1] are relevant to this (particularly the Piano episode). I can recommend all of them in any case. I imagine his book(s) are also relevant, but I haven't read them.

[1] http://timharford.com/articles/cautionarytales/

The dark side of expertise

rahvin — Thu, 16 Jan 2020 15:19:56 +0000

One of the things we do to counter something like this in Civil Engineering is an independent technical review. Basically a review by a participant from outside the project that is subject to none of the "institutional project knowledge" that may bias the design unintentionally. These reviews can be especially helpful where the design specifications have changed multiple times because the outside reviewer can act as a check on each of those decisions.

Good talk, and an excellent summary, Forensic Engineering is an interesting field to me.

The dark side of expertise

mikapfl — Thu, 16 Jan 2020 13:44:55 +0000

Thank you for these links, very insightful. Especially the first paper is really interesting, because it does find an effect, but it is mediated via the experimentator, which is also pretty mind-boggling. (%

The dark side of expertise

Archimedes — Thu, 16 Jan 2020 12:30:35 +0000

In germany there is an interesting double booking of the word "kompetenz" (competence) in the public sector.
In agencies this word is used defining the authority of your "job", not if you have the expertise to fullfil this "job".

The Hartford disaster/incident shows also a failure of the "other meaning", as the authority of each entity was limited it ended up in a back and forth loop, as nobody took the authority to exit this loop. (The exit was time/finished building something broken)
This does not imply that if that exit of that loop would have existed the problem would not have happened, there only would have been a chance in so to say a reshuffled set of expertise and authority that the end result would have been the same.

I have seen quite a few of projects where each entity was locally correct. None of the entities had all the information to check (or even in a retrospective) if they are also globally correct(ish) and none was able to/was allowed to/or took the initiative. In the end the project failed on its initial goal, but was "saved" by exactly an "out of the box"/"out of project" way to solve or circumvent it. By "other expertise" or "other authority" or of course some mixture of both of them.

It is kind of possible to read this article with using the german kompetenz in mind or the authority and ending up in the same conclusions (only the examples don't fit of course)

P.S.
I like people which honestly can play chess (or similar) against themselves and win and loose in the end, as these can (somewhat) honestly can look at both sides of a solution/problem/situation. So kind of the ones which see screws even when they only have a hammer ...

The dark side of expertise

fcrozat — Thu, 16 Jan 2020 12:29:28 +0000

For people interested, this keynote recording is already available at https://www.youtube.com/watch?v=Yv4tI6939q0

The dark side of expertise

marcH — Thu, 16 Jan 2020 08:38:37 +0000

Thanks! That explains why we all think we know what to optimize before having even started to measure anything :-)

The dark side of expertise

me@jasonclinton.com — Wed, 15 Jan 2020 23:18:35 +0000

Yes, second this so much! Such a great book!

The dark side of expertise

saffroy — Wed, 15 Jan 2020 22:42:50 +0000

On the same topic, I would recommend the recent book "Range" by David Epstein, which gives more reasons for paying attention to the value of non-experts (it also uses the example of the Mann Gulch fire, with many others).

The dark side of expertise

dmiller — Wed, 15 Jan 2020 22:35:20 +0000

Subsequent attempts with larger sample sizes have failed to replicate both of the priming studies mentioned [0][1], which casts considerable doubt on their findings.

[0] https://journals.plos.org/plosone/article?id=10.1371/jour...
[1] https://journals.sagepub.com/doi/abs/10.1177/174569161875...

The dark side of expertise

admalledd — Wed, 15 Jan 2020 22:18:45 +0000

For reasons I have been exposed to that word-priming game before, and even here in pure text as the author puts it I can get stuck far longer than I would hope. Going into it knowing what is about to happen (and even seeing the same examples again dang it!) still doesn't always help you.

The dark side of expertise

smoogen — Wed, 15 Jan 2020 22:16:21 +0000

Thank you for that detailed summary of the talk. It was very helpful and got me to look up some new things.

The dark side of expertise

butcher — Wed, 15 Jan 2020 21:45:01 +0000

+1!

EVERY single failure investigation I've worked in my 18 years of aerospace has gone a different way than what I originally surmised. I believe experience helps in the effective structuring of an investigation, but only observation and measurement will tell you the truth...