josh's clients aren't stupid, they're smart [LWN.net]

josh's clients aren't stupid, they're smart

Posted May 7, 2011 17:40 UTC (Sat) by b7j0c (guest, #27559)
In reply to: josh's clients aren't stupid, they're smart by aggelos
Parent article: Scale Fail (part 1)

> Management that a) does not know how to hire competent technical people b) actively avoids hiring competent (and probably more expensive) technical people c) consistently rewards people for finding a hacky workaround after an all-nighter instead of investigating and potentially fixing the actual problem, just deserves to fail

you've just described 95% of the www, including the inauspicious starts of many of the top ten leaders in traffic.

having built part of a top-ten site (yahoo, and my contribution, yahoo news is still number one in its category, worldwide), i'm going to arrogantly state that most people here have a wholly unrealistic view of the forethought, planning, architecture and staffing of a rapidly growing service. i am 100% confident that if you poll individuals who built very high traffic sites from an early stage, you will find a group of individuals who are utterly honest about their inability to do it right the first time, or even know how to

to post comments

josh's clients aren't stupid, they're smart

Posted May 7, 2011 22:38 UTC (Sat) by aggelos (subscriber, #41752) [Link]

It might be so, I'm not even interested on arguing on that since it's beside the point of this discussion. FWIW, you seem to imply something that is completely unbased, i.e. that ignorance and unwillingness to hire and take advantage of people with technical know-how is somehow related to success. Don't think I should bother to refute that.

Our discussion was on whether management in these companies is making an informed trade-off or if they feel safer stabbing in the dark, rather than going for the light switch. I don't think you've brought anything new to the table, other than your degree of confidence in your convictions which, I'm sure you'll agree, is not something one can argue against :-)

Besides, I'm pretty sure noone contested that it takes a lot of effort to build a highly scalable site, even if you do know what you're doing. Still not relevant to the discussion in this subthread though.

josh's clients aren't stupid, they're smart

Posted May 7, 2011 23:03 UTC (Sat) by cmccabe (guest, #60281) [Link] (35 responses)

You bring up a good point. As engineers, we naturally tend to overestimate the importance of engineering and underestimate the value of things like marketing, good management, and intelligent business deals.

However, there are more sites out there that failed because of bad engineering than you think. For example, Friendster could have been what Facebook is today if they had been able to scale the site properly.

Over-planning and over-architecting are often problems, but I think most organizations tend to under-measure rather than over-measure. If you don't know what "the problem" is, it's lot harder to throw money at it to make it go away. And if you can't tell the good engineers and the good consultants from the bad, then you really are doomed, no matter how much you have in the bank.

P.S.
It is funny to hear Josh complain about stupid clients. After all, if they were smart, he wouldn't have a job.

josh's clients aren't stupid, they're smart

Posted May 7, 2011 23:51 UTC (Sat) by wahern (subscriber, #37304) [Link] (33 responses)

Technical merit, good management, and intelligent business deals aren't good predictors of success. If they were, and if those things were accurately and reliably quantifiable (which they must be to have any substance to them) then predicting success would be incredibly more common than it is.

The secret ingredient to success is pure luck. It's hard to swallow. If you disagree, then show me your billion dollar stock portfolio. The accuracy and reliability would only need to be moderately significant in order to extract obscene amounts of wealth from the market.

People who seem to be relatively good predictors of success, such as Warren Buffett, are incapable of articulating their thinking process. If they could articulate it, then there'd be many more copy cats. This suggests the possibility that the emergence of people like Warren Buffett is also a chance occurrence, and that whatever "it factor" they possess is entirely anomalous.

Hard work and fair play don't lead to success, they lead to marginal improvements in wealth and increasingly tolerable living conditions over long spans of time. That they lead to individual success is a noble lie we tell ourselves in order for society to extract those meager gains.

Now, hard work and fair play may lead to personal fulfillment. Maybe that's more important than any kind of economic wealth.

josh's clients aren't stupid, they're smart

Posted May 8, 2011 0:26 UTC (Sun) by cmccabe (guest, #60281) [Link] (32 responses)

> Technical merit, good management, and intelligent business deals aren't
> good predictors of success. If they were, and if those things were
> accurately and reliably quantifiable (which they must be to have any
> substance to them) then predicting success would be incredibly more common

First of all, there is a middle ground between things being completely random, and completely predetermined by known factors. Second of all, a lot of the most important predictors of success are things that we haven't learned to quantify yet.

Warren Buffet has articulated his stock market strategy many times. Do your research; only invest in things that you understand; invest for the long term.

> Now, hard work and fair play may lead to personal fulfillment. Maybe
> that's more important than any kind of economic wealth.

Is personal fulfillment "accurately and reliably quantifiable"? If not, by your own argument, it has no "substance" to it. Or maybe, just maybe, science hasn't learned to quantify a lot of the most important things in life.

josh's clients aren't stupid, they're smart

Posted May 8, 2011 1:13 UTC (Sun) by tialaramex (subscriber, #21167) [Link] (22 responses)

We have psychometric tests for this stuff. So yes, we can tell whether people believe they are fulfilled, and for this purpose that's the same as knowing whether they are in fact fulfilled.

And yes, luck is a massive factor. Warren's strategy sounds good, but it's the same strategy lots of people have, without getting his success. We might as well listen to the anomalous 110 year old who tells us he puts it down to a long walk every afternoon. No doubt walking doesn't hurt, but it's not why he's 110, that's just blind luck.

Survivorship bias is a huge problem. If 500 people all pick one of twenty strategies at random, and all but one of the 500 fails, we would be wrong to assume that therefore the strategy chosen by that one person works and the other nineteen do not. But that's exactly what survivorship bias causes us to assume.

josh's clients aren't stupid, they're smart

Posted May 9, 2011 7:55 UTC (Mon) by cmccabe (guest, #60281) [Link] (21 responses)

Just for fun, here's a puzzle.

Are these digits random or not? And if not, what is the pattern?
69804177583220909702029165734725158290463091035903784297757265172087724

josh's clients aren't stupid, they're smart

Posted May 9, 2011 16:02 UTC (Mon) by dskoll (subscriber, #1630) [Link] (10 responses)

Of course those digits are random. So are these digits:

1111111111111111111111111111111111111111111111111111111111111111

(In other words: Your question is meaningless.)

josh's clients aren't stupid, they're smart

Posted May 10, 2011 6:17 UTC (Tue) by cmccabe (guest, #60281) [Link] (8 responses)

I'm just asking if there is an interesting pattern in the digits or not. That question isn't meaningless.

josh's clients aren't stupid, they're smart

Posted May 10, 2011 10:45 UTC (Tue) by dskoll (subscriber, #1630) [Link] (7 responses)

Of course it's meaningless. There's no such thing as a set of digits that are "random" or "non-random". You can take a sequence generator and run some statistical tests, but that doesn't prove anything. You can test a sequence of digits for compressibility, but that also doesn't prove anything. The digits in the decimal expansion of pi pass all kinds of statistical tests for randomness, but they are assuredly not "random".

As for a pattern, given a finite sequence of digits, you can construct any pattern you like. I could construct a degree-71 polynomial that fits the 71 digits you posted and say "Yes, that's the generator!"

josh's clients aren't stupid, they're smart

Posted May 10, 2011 14:20 UTC (Tue) by bronson (subscriber, #4806) [Link] (3 responses)

Wow. Did you miss the "just for fun" in cmccabe's original post? You must be a riot at parties.

josh's clients aren't stupid, they're smart

Posted May 12, 2011 1:33 UTC (Thu) by dskoll (subscriber, #1630) [Link] (2 responses)

Well, at the parties I go to, people don't usually open the conversation with "69804177583220909702029165734725158290463091035903784297757265172087724".

josh's clients aren't stupid, they're smart

Posted May 12, 2011 4:43 UTC (Thu) by bronson (subscriber, #4806) [Link] (1 responses)

That doesn't explain why why you feel a need to shout someone down here. Must everybody conform to your narrow idea of fun?

josh's clients aren't stupid, they're smart

Posted May 12, 2011 16:12 UTC (Thu) by dskoll (subscriber, #1630) [Link]

Chill out.

josh's clients aren't stupid, they're smart

Posted May 10, 2011 19:05 UTC (Tue) by cmccabe (guest, #60281) [Link] (2 responses)

> I could construct a degree-71 polynomial that fits the 71 digits you
> posted and say "Yes, that's the generator!"

That would be a pattern, but not an interesting one.

josh's clients aren't stupid, they're smart

Posted May 10, 2011 19:07 UTC (Tue) by cmccabe (guest, #60281) [Link]

Sorry, what I meant to say was, that polynomial would not represent a pattern in the sequence itself.

A non-interesting pattern would be something like the observation "this is a number!" or the observation that all digits from 0 to 9 occur.

josh's clients aren't stupid, they're smart

Posted May 12, 2011 11:39 UTC (Thu) by etienne (guest, #25256) [Link]

A more interesting pattern in PI would be this one:
http://kasmana.people.cofc.edu/MATHFICT/mf55-spoiler.html

josh's clients aren't stupid, they're smart

Posted May 19, 2011 11:03 UTC (Thu) by yeti-dn (guest, #46560) [Link]

Please go and learn about Kolmogorov complexity. The question is meaningful but your post, I am afraid, is not.

Just answering so I don't miss the answer

Posted May 10, 2011 21:25 UTC (Tue) by man_ls (guest, #15091) [Link] (1 responses)

It looks pretty random, but there are too few repeated consecutive digits: only three 77 and one 22. I give up, what is the pattern?

Just answering so I don't miss the answer

Posted May 10, 2011 23:05 UTC (Tue) by neilbrown (subscriber, #359) [Link]

71 digits, which is prime so grouping them is unlikely to help.

If you consider just the first 4 digits (0,1,2,3), then even digits occur 10 times each, odd digits 5 time each. However this pattern does not continue (4 and 8 also occur 5 times, but are even).

The longest gap between repeats is 24 between 2 '8's.
'6' and '4' see gaps of 23.

'6' is the least frequent digit, 7 is the most frequent (3 times as frequent)

Taken as a decimal integer the factors less than 100 are
2,2,3,3,23,23

Yet the whole number is not a perfect square.

My conclusion is that this is probably the "least uninteresting number" - very interesting....

josh's clients aren't stupid, they're smart

Posted May 11, 2011 6:00 UTC (Wed) by cmccabe (guest, #60281) [Link] (5 responses)

It's digits 99768 through 99838 from the base 10 decimal expansion of pi.

Every digit is completely predetermined, but it's pretty hard to spot if you don't know what you're looking for!

josh's clients aren't stupid, they're smart

Posted May 11, 2011 8:07 UTC (Wed) by ballombe (subscriber, #9523) [Link]

The only thing that is remarkable is that Google did not find it.

josh's clients aren't stupid, they're smart

Posted May 11, 2011 16:56 UTC (Wed) by tialaramex (subscriber, #21167) [Link] (2 responses)

So in other words the answer was "No" ?

We suspect (as far as I remember no-one has proved) that all possible sequences of digits occur in Pi. Assuming this is so, the fact that a particular sequence occurs in Pi is not interesting at all.

josh's clients aren't stupid, they're smart

Posted May 11, 2011 20:08 UTC (Wed) by cmccabe (guest, #60281) [Link] (1 responses)

Even if all possible sequences of digits occur in Pi, the fact that a particular sequence occurs at a particular position can be "interesting". If nothing else, it would make for a pretty strange compression algorithm (although probably not a practical one for most input data, given how large the indices are likely to be.)

josh's clients aren't stupid, they're smart

Posted May 12, 2011 8:27 UTC (Thu) by ekj (guest, #1524) [Link]

On the average, the index of where in pi a certain sequence occurs, is as long as the sequence itself, thus the average savings of this compression-scheme is zero.

josh's clients aren't stupid, they're smart

Posted May 12, 2011 16:56 UTC (Thu) by fuhchee (guest, #40059) [Link]

I'm still missing the "fun" part.

What Do These Digits Mean?

Posted May 19, 2011 12:35 UTC (Thu) by ldo (guest, #40946) [Link] (1 responses)

#!/usr/bin/python

import sys

charset = ' Wadefhlnost'
modulo = 13
s = 13682311570832829480888979137834570837851469148689544502986
num = iter(range(2, 9999))
while s != 1 :
        n = num.next()
        if s % n == 0 :
                sys.stdout.write("%s" % charset[(n - 1) % modulo])
                s /= n
        #end if
#end while
sys.stdout.write("\n")

What Do These Digits Mean?

Posted May 26, 2011 12:20 UTC (Thu) by net_benji (subscriber, #75195) [Link]

All 23 prime factors of s are of order 1, I think that's the gist of it.
http://www.wolframalpha.com/input/?i=factor+1368231157083...

Here are more working combinations:
charset = "hetfn adlsoW"
s = 7062883793966047784250125868403644804557392731274644684033611
charset = "tslofWe dnha"
s = 1029201132023087388381452825147240668180384494509643451351
...
and here's a different one, just for the fun of it:
charset = "wfyonirvmD s?ecutdpah"
modulo = 23
s = 79 * 211 * 241 * 463 * 487 * 499 * 563 * 571 * 673 * 787 * 911 * 977 * 991 * 1039 * 1249 * 1483 * 1489 * 1493 * 1601 * 1621 * 1697 * 1699 * 1889 * 2243 * 2311 * 2347 * 2459 * 2521 * 2719 * 2909 * 2953 * 3119 * 3271 * 3323 * 3359 * 3533 * 3733 * 3947 * 3967 * 4057 * 4177 * 4283 * 4289 * 4597 * 4651 * 4733 * 4933 * 4969 * 5021 * 5087 * 5261 * 5281 * 5347 * 5399 * 5449 * 5557 * 5641 * 5711 * 5807 * 5869 * 5981 * 6353 * 6359 * 6389 * 6569 * 6701 * 6791 * 6823 * 6983 * 7187 * 7309 * 7321 * 7481 * 7529 * 7673 * 7873 * 7901 * 8017

The source for the generator is there: https://github.com/benthaman/lwn-digits/blob/master/encod...

Hope Jon won't mind posting this nonsense here ;)

josh's clients aren't stupid, they're smart

Posted May 8, 2011 1:37 UTC (Sun) by wahern (subscriber, #37304) [Link] (7 responses)

> Warren Buffet has articulated his stock market strategy many times.
> Do your research; only invest in things that you understand;
> invest for the long term.

If you do those things in his books at best you'll approximate the market and not lose your shirt. And the model that best reproduces historical market behavior is a random walk. Overall market efficiency is an emergent effect from random processes, not the result of smart people being exceptional.

Look at the long line of successors Buffett has tried to groom. None could ever match his acumen by exercising his supposed methodology. Some have come close but inevitably make one or two wrong bets and their long-term gains fall far short of Buffett's. It's like gamblers who would have walked away with the bank but "bad luck" ruined their "streak". They disown the bad luck but claim the winning streak as the product of some personal attribute.

That's not to say that there aren't predictable ways to make money in the market, but they involve placing yourself in certain structural niches within the market. How you rise to those positions also seems to have little to do with intelligence or acumen by comparison with your peers. Where or how those structural market niches arise are unpredictable. Facebook didn't do anything different except chance upon the emergence of a niche at the right time.

The problem with defining success is that people look at the big winners and obvious failures and compare and contrast. If you see an impoverished alcoholic bum on the street and compare him to Buffett, you might conclude that success entails not letting alcohol control your life.

But once you have all of those DON'T DOs (i.e. how not to fail) you need to find the DO DOs. But even after you've enumerated all the DOs, you can find thousands or even millions of people who DO all the same things yet never succeed. There's a difference between not being a failure, and being successful, unless swimming instead of sinking is worthy of special praise. Success as flowing from individualized merit, therefore, is an illusion. In other words, just because you're successful doesn't mean you've done anything different than millions of other people. You're not special; you're just lucky by comparison. So there's little to be gained by analyzing and investigating what's "unique" about very successful people or companies. Anything you think is unique is either anomalous or not reproducible, and so liable to lead you astray from focusing on not failing. Focusing on not failing typically means making your product as good as necessary under the circumstances, not as good as possible.

You can certainly write a book about how to live a middle-class life, or how not to run a company into the ground. But no book will ever be able to describe how to become a millionaire, how to find the next great concept, or how to ensure that great idea will come to spectacular fruition by proper technical execution. Just as was stated earlier, many--perhaps most--of "successful" websites are ugly as hell behind the curtains. They were doing just enough to stay afloat, just like everybody else (on average), and serendipitously found themselves in charmed waters.

josh's clients aren't stupid, they're smart

Posted May 8, 2011 8:12 UTC (Sun) by cmccabe (guest, #60281) [Link] (5 responses)

If the stock market were completely random, it would be completely useless. The whole reason for having a stock market is to allocate money to companies that will use that money in productive ways. We've already tried replacing markets with central planning committees-- it was called communism. It didn't work.

Obviously, markets have a lot of problems and inefficiencies. A lot of mutual fund managers are overpaid for the small amount of research that they do. Sometimes the market over-allocates resources to one sector, like housing. But just because the market has inefficiencies, or Warren Buffet is having trouble finding a successor, doesn't somehow indicate that the market is completely random.

Despite your professed love for quantifiable data, you haven't really presented that much hard evidence for me to argue against. What I would like to argue against is your fatalistic attidue. You seem to view yourself as a helpless pawn of all-powerful external forces.

Ironically, optimism-- or at any rate, the will to keep on going-- is one of the best predictors I know of success-- and not just in business. If you look at any open source leader, any entrepreneur, successful politician-- whatever-- you will find that one thing they have in common is that they believe in themselves and their abilities.

One book you might be interested in is "The Drunkard's Walk" by Leonard Mlodinow, a physicist at Caltech. In it, he puts forth some pretty convincing arguments that life is more random than we think. Mlodinow has a statistician's focus on hard data and it's pretty interesting. The performance of mutual fund managers, the outcome of the world series, and the decisions of Hollywood producers are all revealed as being more random than we might think. He also takes a swing a Bill Gates, claiming that he was just in the right place in the right time. Personally, I disagree-- I think Gates would have been at least moderately successful in any place and time. Anyway, you might enjoy the book.

josh's clients aren't stupid, they're smart

Posted May 8, 2011 22:37 UTC (Sun) by ibukanov (subscriber, #3942) [Link] (2 responses)

> Ironically, optimism-- or at any rate, the will to keep on going-- is one of the best predictors I know of success-- and not just in business.

This is not what I have observed. The most successful in business people I know personally are pessimists. In one particular case the pessimism made the guy to prepare for the worst. That allowed for him to survive bad times and gain from the good times disproportionally.

Another example is a very optimistic guy who quieted his job as an oil engineer and went to Thailand to start a shrimp farm. It was rather successful initially. But one day a drunken worker from the factory got into a conflict with local mobs and they burn the farm down... Now the guy is back into his engineering position.

josh's clients aren't stupid, they're smart

Posted May 9, 2011 16:08 UTC (Mon) by raven667 (subscriber, #5198) [Link] (1 responses)

Are you sure you aren't arguing against your point? In the examples you gave you have one person who is successful because they had the will to keep going and were prepared, which you call pessimism and one where the person was not successful when they quit after the first setback which you call optimistic.

Success is just like cancer, you can get (un)lucky but if you keep hitting those carcinogens you improve your chances of catching it, although not guaranteed either way.

josh's clients aren't stupid, they're smart

Posted May 9, 2011 21:19 UTC (Mon) by ibukanov (subscriber, #3942) [Link]

> successful because they had the will to keep going

In the first case it was not the will but rather the cash that the company saved during the good times under assumption that those would not last. With that it was not problematic to carry on during the bad times. And when competitors that hired too many engineers as consultants for the oil industry went burst it was trivial to pick up their clients. (In Norway it is rather hard to fire a person on a permanent position so a bankruptcy often is the simplest solution for a small company.)

> the person was not successful when they quit after the first setback

Only a very optimistic person would invest a few years of savings into something he had only vague idea about. And that fire on the shrimp farm was not a setback, it was a total disaster. As far as I know the farm was not insured against that type of damaged and the guy simply had no money to try one more time.

Note that I am not arguing that pessimism is required for a business success. I just want to say that even if optimism could be a necessary trait for some kinds of entrepreneurship, it may lead to failure just as well as to success.

josh's clients aren't stupid, they're smart

Posted May 14, 2011 14:31 UTC (Sat) by jjs (guest, #10315) [Link] (1 responses)

I highly recommend John Bogle's books, including "Enough" and "Don't Count on It". He's been a professional investor since the 1950's, founded Vanguard, and those books I've read include graphs & numbers illustrating his points. Main point - trying to outguess the market is a loser's game (at best over the long term you match the market, but when disintermediation is taken into account, you lose money). Over the long term, most money managers make lots of money on transaction fees, but their customers lose.

Another good book is "The Black Swan" by Nassim Nicholas Taleb.

josh's clients aren't stupid, they're smart

Posted May 14, 2011 14:40 UTC (Sat) by jjs (guest, #10315) [Link]

One other key point - the separation of the stock market into the economic and emotional sides - and overall economy grows with the economic part, bubbles and crashes are mainly caused by the emotional investment overinvesting/underinvesting as people chase this week's numbers, not the long term health.

josh's clients aren't stupid, they're smart

Posted May 12, 2011 9:30 UTC (Thu) by ekj (guest, #1524) [Link]

It depends on how you define "success".

If you mean, do a lot better than most people who peruse the same activity, then it's almost tautological that there's no recipee for it. If there was a simple and unambigious way of outperforming the general market as an investor, for example, then all investors would be using technique, thus negatings it's effectiveness. It's matemathically obvious that investors as a group, will not be able to beat the market. Thus one person can only beat the market to PRECISELY the same degree as another underperforms the market.

But there's other ways of defining success.

There's things that on the average *do* lead to goals that people desire. It's just that it's hard to do them. If you can manage to do them though, then achieving those goals become likely. I'd call that success.

Saving 10% of your income in a low-cost index-fund will, with high probability, give you a very solid financial position in a decade or two. (but not make you a billionaire)

You're just saying, it seems to me, there's no simple, predictable, repeatable way of doing the exceptional. That's almost tautological, because if there where, it'd no longer -be- exceptional.

josh's clients aren't stupid, they're smart

Posted May 14, 2011 1:41 UTC (Sat) by rgmoore (✭ supporter ✭, #75) [Link]

Warren Buffet has articulated his stock market strategy many times. Do your research; only invest in things that you understand; invest for the long term.

That isn't an investment strategy; it's a short list of how to avoid common investment mistakes. Yes, you need to put your money into businesses you know and understand, but the sticking point is understanding business and knowing what to research when you're doing your homework. It's the knowledge necessary to do those things that makes Buffet an investment genius, and that detailed knowledge can't be encapsulated into a few pithy rules.

As investment advice those rules about as useful as Babe Ruth's advice on hitting a baseball: wait for a good pitch and hit it. It's good advice as far as it goes, but it's useless without the talent and skill to put it into practice. Similarly, it's great to come up with a list of best coding practices, but they're no replacement for judgment, skill, and experience in programming.

josh's clients aren't stupid, they're smart

Posted May 12, 2011 0:43 UTC (Thu) by jberkus (guest, #55561) [Link]

<quote>P.S.
It is funny to hear Josh complain about stupid clients. After all, if they were smart, he wouldn't have a job.</quote>

I'd still rather get paid to solve interesting problems than stupid ones.

And what's really frustrating to me as a consultant is to put dozens of hours into a real solution for a client's scalability issues, only to have my work discarded because of management problems. Whether I get paid or not.

josh's clients aren't stupid, they're smart

Posted May 12, 2011 8:17 UTC (Thu) by ekj (guest, #1524) [Link]

True. I can state from my own personal experience that the first architectures of MediaWiki where -horribly- far from being done "right" for scaling to anything even remotely resembling the level of scaling needed today.

Infact, it's -still- not done anywhere near "right", instead the limitations are worked-around.