LWN: Comments on "Imitation, not artificial, intelligence"

90% accuracy is terrible

raven667 — Thu, 01 Aug 2024 23:38:04 +0000

> The use of AI or II or however you want to market it, as a tool, when you already know what you're doing, and can indeed evaluate solutions and course correct is useful and evidently valuable. But what happens in a decade or so when the pre-LLM generation is the last who actually has knowledge, and everyone else only has "LLM interaction skills"?

I've heard of this kind of problem as well, an LLM-informed code-completion tool with rigid syntax linting could be useful to a skilled practitioner, but this stuff is being marketed to the unskilled as if they don't have to understand their problem to have the computer magically create a solution, but that isn't true, at all. The LLM has no way to divine how a module of code is supposed to fit into an overall architecture in a maintainable way, especially if the person writing the prompt doesn't have any idea themselves, the LLM can't make architectural trade-offs because it has no reasoning or understanding of the problem.

It seems reasonable that if you create code you don't know how to create and don't understand then you've immediately created tech-debt for yourself because you'll need to understand it to maintain and debug it. What is this mythical code supposed to be good for that no one needs to understand or maintain?

90% accuracy is terrible

farnz — Tue, 30 Jul 2024 10:17:53 +0000

Indeed some skills disappear, but they disappear because they are not that needed anymore. It's not the knowledge that disappears (we still know how to add, substract, multiply ...) but the fluency because we don't need it everyday. If for some reason we start needing it again, we practice and get better at it.

The problem with this is the cost of practising; a good example is nuclear reactor engineering. From the 1950s through 1970s, France had significant programmes to ensure that it had a supply of people who were able to work safely on nuclear reactors, both civilian and military, and who had enough experience to distinguish "this is a transient state, and it'll be fine" from "this is not normal, and we need to worry".

They then let those programs atrophy during the 1980s through the 2010s. They're now training up a new workforce to work on nuclear power plants to replace the people who are retiring, and they're finding that a lot of the knowledge isn't written down - it's stuff you learn from experience and being told "no, that's not right" when you're following a textbook process.

Now, we can definitely recover all of that knowledge over the same timescale it took to build it up - if we started on nuclear power plants from scratch now, and maintained a skilled workforce of nuclear technicians, we could be back in the 1970s state within 20 years. But recovering lost knowledge is often slow, even when you've got it all written down on a piece of paper.

90% accuracy is terrible

Wol — Tue, 30 Jul 2024 09:42:40 +0000

> If for some reason we start needing it again, we practice and get better at it.

The danger is that there is suddenly a big need for niche skills ...

As a (many people would say unlikely) example, imagine suddenly we lose access to fossil fuel. Okay, wind and solar will take up *some* slack, but most people will be car-less. We'll suddenly need LOTS more horses, and lots more blacksmiths, and and and. The skills are there, but scaling them up rapidly to meet demand will be pretty much out of the question.

There's some social evidence that something like this happened to destroy the Roman Empire. Historians think there was a smallpox epidemic about 200AD, that took out a lot of specialised workers and basically distorted the economy so badly the empire never recovered. Same thing about 1500 in South America. Compare that to the Black Death in the late 1300s, which seriously shrank the economy but because it took out workers pretty evenly across the economy, it didn't distort things that much. The resulting boom became the renaissance.

If anything happens to us, it'll be a Roman Empire crash, not a post-Black Death Renaissance...

Cheers,
Wol

90% accuracy is terrible

cpitrat — Tue, 30 Jul 2024 08:24:55 +0000

Socrate had similar concerns about writing:
https://newlearningonline.com/literacies/chapter-1/socrat...

"how could they possibly think that words that have been written down can do more than remind those who already know what the writing is about?"

Since then, many new technologies were faced with this kind of concerns: pocket calculators (people won't know how to count anymore), TV (people won't read anymore), computers, internet, ...

Indeed some skills disappear, but they disappear because they are not that needed anymore. It's not the knowledge that disappears (we still know how to add, substract, multiply ...) but the fluency because we don't need it everyday. If for some reason we start needing it again, we practice and get better at it.

Consider the web in 1994

anselm — Mon, 29 Jul 2024 14:28:57 +0000

because they have no feedback to tell them it IS rubbish

That's what the underpaid peons in places like Kenya are for. (One problem is that they rely on LLMs for their fact-checking because otherwise they couldn't keep up with the pressure.)

https://amycastor.com/2023/09/12/pivot-to-ai-pay-no-attention-to-the-man-behind-the-curtain/

Making information more accessible

paulj — Mon, 29 Jul 2024 10:15:59 +0000

I for one encourage my competitors to make full use of LLM code reviewers!

(Though, there will undoubtedly be regular collateral damage to society soon enough).

Consider the web in 1994

songmaster — Mon, 29 Jul 2024 01:15:06 +0000

It sounds like it might be worth asking an LLM to update those outdated tutorials based on the changes that have been made to the code since the version that was documented. We all hate to have to update our documentation, if it works maybe that’s something we can accept automated help with?

Making information more accessible

khim — Mon, 29 Jul 2024 00:46:55 +0000

> It’s really obvious from your comment you haven’t actually tried these tools.

On the contrary: at my $DAYJOB they have enabled that crap and now I'm looking on the idiotic attempts of these LLMs to create something every time I write comment during code review.

Sometimes, when comment is about something trivial, like “we should use string_view here, not string, see totw #1” it even generates sensible code. But that's rarity. Most of the time it generates patent nonsense, because it doesn't understand what it does. It couldn't, there are no brain behind what it does.

> Frontier LLMs are in fact quite good about including error handling in code, especially if you ask them.

They are pretty good at generating crap that looks sensible but doesn't work!

To the tune that when I see that change proposed is crap I just know that I have to contact privately to ask submitter to stop using that nonsense and write things by hand.

Unfortunately not every company have the rule that unreviewed code couldn't be accepted, I shudder to think about what this crazyness would lead to in companies that accept code without reviewing it.

Consider the web in 1994

Paf — Mon, 29 Jul 2024 00:30:17 +0000

I encourage you to try them - they are generally extremely good at synthesizing from sets of the partly outdated tutorials - which is what’s found online, largely -and providing something that works. Not unlike what a person might do.

Consider the web in 1994

Paf — Mon, 29 Jul 2024 00:27:16 +0000

I strongly encourage you to actually *try* these tools. It is very apparent from your comments about boilerplate code that you have not. Seriously - I’m a programmer with 12 years of pretty successful work experience, largely on an out of tree distributed file system. I wondered if they might be limited to boilerplate. While they are *extremely* good at boilerplate, they absolutely are not limited to it. Sure, they can’t really do much kernel work of length, but they are wildly good at even moderately complex scripting and helping you use unfamiliar APIs. Yes, the API part is sort of boilerplate, but it doesn’t have to be common stuff - it can be obscure ones, including kernel APIs.

Making information more accessible

Paf — Mon, 29 Jul 2024 00:17:35 +0000

It’s really obvious from your comment you haven’t actually tried these tools. Frontier LLMs are in fact quite good about including error handling in code, especially if you ask them.

The Netherlands will be fine

Wol — Sun, 28 Jul 2024 10:05:17 +0000

> The predictions of many metres of sea level rise are based on thermal expansion of the oceans. Which won't happen suddenly, but slowly (OTOH, trying to reverse such warming would be equally slow).

Sorry, wrong physics again. Warming the oceans will use (very slow) conduction. However, both the troposphere and the oceans have very efficient cooling mechanisms. For every century it takes to rise, it'll take maybe a decade to cool?

As soon as we stop filling the stratosphere with greenhouse gases and turning it into a blanket, these mechanisms will bring temperatures down quickly.

The maximum surface temperature of the ocean is about 38C. At this point the cooling mechanism called a tropical storm kicks in. That's why, as temperatures rise, storms have been getting more frequent, more severe, and moving further away from the tropics.

And the cooling mechanism for the oceans themselves is called an ocean gyre. The one I know is the North Atlantic gyre, composed of the Humboldt current taking cold Arctic water down to the equator, and the Gulf Stream bringing warm equatorial water to the Arctic.

Once the stratosphere is dumping heat, not radiating it back to the surface, we should start getting polar blizzards recreating the ice caps, and the gyres reasserting themselves (the North Altlantic gyre is in serious trouble thanks to the loss of the Arctic ice cap). At which point we could find ourselves heading rapidly into another ice age. Or not as the case may be.

Cheers,
Wol

Consider the web in 1994

DanilaBerezin — Sat, 27 Jul 2024 21:48:15 +0000

As mentioned in the article, even though LLMs are trained on generic data, they can be fed specific data and instructed to base their responses off that specific data, effectively allowing you to create a specialist LLM. So if you want an LLM that specifically explains things about Python, you can take your favorite generic model, feed it python documentation, and then prompt it to answer all your python related questions. And this generally works. I've already seen it employed at workplaces where models are locally trained on relevant codebases/documentation and employees can utilize it to answer questions that would otherwise require talking to another employee to answer or reading through everything yourself.

Consider the web in 1994

Cyberax — Sat, 27 Jul 2024 21:33:41 +0000

> The whole idea of LLMs is that bigger models and more data is supposed to get you some kind of intelligence

That's not quite right. The current crop of LLMs is inherently limited because they have a relatively small "working memory" and even less ability to learn on the fly. The current work is aimed at fixing this.

Consider the web in 1994

Wol — Sat, 27 Jul 2024 20:29:26 +0000

Third - successful learning requires feedback from outside that you've actually got it right.

LLMs spew rubbish because they have no feedback to tell them it IS rubbish.

Cheers,
Wol

The Netherlands will be fine

Wol — Sat, 27 Jul 2024 20:26:27 +0000

> The conservative IPCC estimate is only about 60 cm by 2100, but even that will be problematic.

And I seriously expect (a) that estimate is wrong, and (b) it'll be wrong as in too low.

As I said, we consistently underestimate change. Go back to 1980, rampant population growth, the EXTREMELY OPTIMISTIC forecasts of Y2K population said 8Bn. We undershot by over 2Bn I think. (The "we think it'll actually be this" estimate was 12Bn!)

Thwaites glacier, 10 years? Let me guess it'll actually be five. Quite likely less.

Cheers,
Wol

Consider the web in 1994

atnot — Sat, 27 Jul 2024 19:14:12 +0000

> an LLM done right for this particular use case would be trained on documentation and be able to transform the information on the fly into an easily digestible format a la tutorial style

That seems like wishful thinking to me for a few reasons:

The whole idea of LLMs is that bigger models and more data is supposed to get you some kind of intelligence, but the current crop of LLMs has already been trained on pretty much the sum total of human text output available today and it's still crap. In fact one reason why they're not getting better is that improvements require exponentially more data and we're already out of data to train on. Getting superior results with a fraction of that with the same technology seems unlikely.

Second, effective teaching requires both understanding the subject matter and good knowledge and experience with how humans tend to think an learn. LLMs can't remotely do either of things at all, let alone for a complex subject like programming. Summarization, sure. That just requires pattern matching the phrases humans tend to use to indicate asides and writing conventions like the three part essay structure. But that's very different.

Consider the web in 1994

DanilaBerezin — Sat, 27 Jul 2024 17:48:47 +0000

For programming specific questions? Hopefully on documentation, which is generally up-to-date and not as misleading as random youtube videos. ChatGPT can absolutely be misleading and most LLMs today are, but I think the main point is that an LLM done right for this particular use case would be trained on documentation and be able to transform the information on the fly into an easily digestible format a la tutorial style. This would enable beginners to gain all the benefits of tutorials without any of the costs.

Consider the web in 1994

LtWorf — Sat, 27 Jul 2024 16:51:12 +0000

On what do you think llm are trained?

Consider the web in 1994

DanilaBerezin — Sat, 27 Jul 2024 16:15:15 +0000

As someone who used to rely quite a bit on tutorials when I first got into programming, you'd be surprised by how large of a portion of the tutorials out there are outdated or misleading "slop" as the speaker says.

90% accuracy is terrible

passcod — Sat, 27 Jul 2024 10:40:23 +0000

If a teacher was wrong 10% of the time, all the time, they would be fired. When an LLM does that, it's unleashed onto the world.

Rhetoric aside, something I'm quite worried about but don't see talked about a lot is whether we're coming up to a learners' cliff, where a lot of beginners in various fields at the moment are learning via LLMs, which is in my view both misleading them into all kinds of wrong pathways (my primary experience is having to (re-)teach core concepts of the Rust programming language, and I've heard of literature professors expressing similar frustrations) and not teaching them how to look things up (they don't google or visit docs/help/manuals, they go to LLMs and then to various Discord channels) nor to puzzle out problems (again immediately going to LLMs).

The use of AI or II or however you want to market it, as a tool, when you already know what you're doing, and can indeed evaluate solutions and course correct is useful and evidently valuable. But what happens in a decade or so when the pre-LLM generation is the last who actually has knowledge, and everyone else only has "LLM interaction skills"?

The Netherlands will be fine

malmedal — Fri, 26 Jul 2024 23:12:47 +0000

My understanding is that sea-level rise if all ice melts is about 70 meters, and that the models say we have already emitted enough CO2 to get 3 to 10 meters. That will take hundreds of years however, so there is time to mitigate.

The scariest realistic scenario is Thwaite's glacier, which, when it collapses, can give us 65 cm all by itself in the span of maybe ten years. Those ten years could start tomorrow or in a hundred years.

The conservative IPCC estimate is only about 60 cm by 2100, but even that will be problematic.

Consider; the daily tidal forces on the ocean only amounts to about 50cm due to the moon and 25cm from the sun, so in theory a max of 75cm when they are in sync.

However the actually observed tides can be more than ten meters the Bay of Fundy and almost nothing in the Caribbean due to variations in geography.

Similarly even if we only get 60cm it will be unevenly spread, some locations will get a lot, some nothing, some may even see a decrease.

I don't think we have models good enough to confidently predict who will win and loose. If I were to guess, I'd say the biggest victim would be Bangladesh.

The Netherlands will be fine

Wol — Fri, 26 Jul 2024 22:30:24 +0000

> Potential sea level rise from ice melt is fairly limited, less than a metre I thought.

Which is why I've been saying a metre ... the danger lies in the fact that that metre rise is expected to take a century. We were shocked at how fast the Arctic melted ... and Antartica doesn't even need to melt - all it needs is to slide into the sea, and the rate of flow will only increase. Let's hope it doesn't accelerate faster than expected ... Greenland is a poster child here - the glaciers are flowing much faster now the ice shelf has gone ...

Cheers,
Wol

The Netherlands will be fine

paulj — Fri, 26 Jul 2024 22:04:07 +0000

Potential sea level rise from ice melt is fairly limited, less than a metre I thought. There's a lot of ice, but there's way, _way_ more sea.

The predictions of many metres of sea level rise are based on thermal expansion of the oceans. Which won't happen suddenly, but slowly (OTOH, trying to reverse such warming would be equally slow).

The Netherlands will be fine

Wol — Fri, 26 Jul 2024 21:58:50 +0000

> But even the worst case projections don't go that fast. The thermal mass of the ice sheet and the ocean are huge, even compared to the sun's output. Put the Antarctic ice sheet in full 24 hour sun and it would still take decades to melt appreciably.

WRONG PHYSICS.

Sorry yes we are getting a bit off topic (a bit?). But how far back do you have to go to find worst-case predictions saying we will still have an Arctic ice sheet? Just ten years? We don't like change. We consistently under-estimate it.

Rising temperatures increase the plasticity of ice. Melting isn't the problem, it's flow. And if the sea starts getting under the Weddel Ice Shelf and starts lifting and dropping it, then flow will be a *major* problem. It's happened before - it's almost certainly going to happen again - and I can see the shelf disappearing in a summer. If it does, glacier flow will be SCARY ...

At the moment, most of the Antartic ice is above sea level. It doesn't have to melt to raise sea levels - it just has to fall in.

Cheers,
Wol

The Netherlands will be fine

kleptog — Fri, 26 Jul 2024 21:12:58 +0000

> Are you sure? Are their defences capable of coping with a rise in mean sea level of one metre or more, in the space of a few months.

A metre is nothing. Tidal variation is ~2.5m and the North Sea gets way bigger waves than that. The sea defences are not the problem. Sand dunes rise together with the sea. The problem lies elsewhere.

Here the main canal that drains rainwater is at NAP-0.4m. Almost every day the low tide mark is about NAP-1m allowing collected rainwater to drain out for a few hours per day. Add (not even) a metre to the sea level and you have to start pumping. You already get situations where a high tide combined with consistant NW winds conspire to prevent draining for weeks, requiring alternative storage. Higher sea levels mean all the rivers are higher too, and this problem replicates across the country. It's all solvable, but will be very very expensive. There are ideas to pump the entire volume of the Rhine up to a higher sea level, but the energy requirements are enormous.

But even the worst case projections don't go that fast. The thermal mass of the ice sheet and the ocean are huge, even compared to the sun's output. Put the Antarctic ice sheet in full 24 hour sun and it would still take decades to melt appreciably.

We work with acceptable failure rates of "one flood every 10,000 years" so the current safety margins are more than sufficient for the time being. It's probably true that nowhere else in the world are there such large safety margins. Certainly the Americans who came to learn after hurricane Katrina thought we were nuts. Then again, we don't get hurricanes.

This is going rather off-topic though...

Consider the web in 1994

malmedal — Fri, 26 Jul 2024 20:59:59 +0000

I suppose it depends on what you mean by furniture. My grandparents had several nice-looking old cabinets, which they kept. However they threw out old sofas and chairs(or demoted them to the cabin), they wanted the stuff that actually got used to be nice and comfortable.

Surviving old furniture and inflation

Cyberax — Fri, 26 Jul 2024 20:07:54 +0000

> Good points, though I would make 1 counter point: The labour available to make quality furniture today is tiny, compared to 100 years ago.

But is it? Carpenters can be much more productive today with modern power tools, and even computer-controlled tools. Look at CNC routers, they are downright magic.

I commissioned several custom wooden products (fireplace holder and custom cabinets), I fully expect them to outlast the house. And the amount of money I paid for them is probably still less than 100 years ago.

On the other hand, there's another dimension: practicality. I grew up in a house where we had an actual solid oak wooden table and chairs. They got left behind when this house got demolished because they were completely impractical. It took several people to move the table, and the chairs were uncomfortable and also heavy. I _can_ buy solid oak wood chairs, but I much prefer IKEA chairs made of lightweight pine and birch tree. They won't last, but then they are so cheap, I can replace them without even thinking about it.

It's always about trade-offs.

The economy is screwed

LtWorf — Fri, 26 Jul 2024 17:11:12 +0000

AFAIK reddit is there to allow the USA government to run their bots. It appears that it isn't just russia doing that.

Remember the deleted post that stated that a USA airbase was the most reddit addicted city?

Consider the web in 1994

immibis — Fri, 26 Jul 2024 15:43:33 +0000

IIRC, ELIZA was taken to be intelligent by *most* people, including people who understood how it worked. Its creator found this quite shocking.

The economy is screwed

immibis — Fri, 26 Jul 2024 15:42:39 +0000

We don't have a competitive landscape any more. We have a hierarchical landscape where certain people are in charge and have locked themselves into the position of being in charge. It doesn't matter if the company will eventually fold 5 years after making a wrong decision (and it's questionable whether wrong decisions make companies more likely to fold than right decisions), because we still have to deal with the fallout of the wrong decision immediately and for the next 5 years, and then for another year until a sufficient replacement is available.

Look at all the world's information being locked on Reddit, which has just closed its doors (to every consumer of that information except for Google, who pays a lot of money to not be excluded). Reddit will surely go bankrupt due to this... in some years. It's already been over a year since Reddit started down this track. Reddit has almost never been profitable, so you could argue it's been making wrong decisions for ~20 years and still not failed. Meanwhile Facebook is occupied by almost exclusively Russian bots, and still has a high stock valuation. The markets are 99.9% disconnected from reality.

Consider the web in 1994

malmedal — Fri, 26 Jul 2024 14:57:41 +0000

Locally produced furniture range a very wide spectrum, from the most beautifully woven rattan armchair, to a bench that consists of three bamboo trunks only loosely tied down, so that if your neighbour moves your bottom gets VERY PAINFULLY pinched. The former is what the locals show off to tourists, the latter is what the ubiquitous plastic chairs are replacing.

Truly understanding programming

atnot — Fri, 26 Jul 2024 14:43:58 +0000

> it's more-or-less impossible to distinguish gibberish that ChatGPT produces from gibberish that many humans are producing!

This seems like a silly criteria. If you can't reliably distinguish the gibberish created by a brick falling onto a keyboard from the gibberish created by a cat sitting on one, that doesn't make the brick as intelligent as a cat.

Truly understanding programming

atnot — Fri, 26 Jul 2024 14:34:56 +0000

> Which is a funny point because they could extend IDE auto-complete functionality and be a useful addition but the insistence that _all_ the work needs to be done by the LLM and the lack of feedback makes it worse

Yes, I've been thinking this ever since I've seen people using github copilot. Our editors are already pretty darn good at finding out what the next valid thing can be. And they're basically guaranteed to be correct too, no "hallucination" nonsense. They're just not particularly good at sorting those options by relevance right now. There's no need to train these huge models to output code from nothing to get these advantages? Surely you could make some sort of model to score autocomplete suggestions by surrounding context that would be at least as good if not better than what microsoft is offering. And it surely wouldn't require the energy consumption of a small nation either. I'd use that in a heartbeat. But nobody seems to be interested in that sort of thing because it's not going to "replace programmers" or whatever.

Surviving old furniture and inflation

paulj — Fri, 26 Jul 2024 14:01:01 +0000

Good points, though I would make 1 counter point: The labour available to make quality furniture today is tiny, compared to 100 years ago.

The supply of expert labour has diminished to near 0, along with the demand for their well made furniture having been destroyed by cheap, hastily bolted together, (mostly fibreboard, a minority in pine, smaller amount again in better wood) stuff.

Consider the web in 1994

paulj — Fri, 26 Jul 2024 13:56:44 +0000

You're comparing the developing world today to the developed world 100+ years ago. Not quite sure it's a fair comparison.

I don't doubt there was crap back then. For sure, people were also doing things like using old tea boxes for furniture. But the middle-class market for furniture, the quality between then and now, the difference is huge. My mother still has a lot of furniture from her mother (other side of the family), and I fully expect my children will have some of it. We bought a book case ourselves ten years ago. The best quality one - and best value *by far* - we found was from a second-hand shop. The book case looks to be circa 1930s / 1940s (by comparison to my gran's furniture of that era).

There's a cottage industry of second hand furniture from pre-WWII (60s latest) cause the quality and value of that furniture is far above what is made now.

And it stands to reason: How many people in the last 40 years became expert carpenters and furniture makers, compared to 100 years ago?

Meanwhile, back in Linuxland

corbet — Fri, 26 Jul 2024 13:52:30 +0000

I think we're getting pretty far afield here ... again. Can we try to keep the focus on Linux and free software, please?

Consider the web in 1994

Wol — Fri, 26 Jul 2024 13:48:25 +0000

Are you sure? Are their defences capable of coping with a rise in mean sea level of one metre or more, in the space of a few months.

That's why the tsunami caused the nuclear disaster in Japan - their defences were designed to stop a 10m wave. But the defenses were on the plate that dropped, which is why a 9m wave went straight over the top of it ...

I've seen the Victoria Embankment with the Thames almost to the top. That was a long time ago, before the Thames Barrier. The Barrier's estimated lifetime has been about halved. If we have a rise of a meter, I think the Thames will simply flow *round* the barrier, and if the Embankment goes - well - the Strand is so called because it was the strand - the beach on the banks of The Thames. The water will probably go a LOT further than that ...

Cheers,
Wol

Consider the web in 1994

Wol — Fri, 26 Jul 2024 13:41:33 +0000

> > There exists a large amount of furniture today that are of far worse quality than the cheapest stuff you can buy in the west.

> Indigenous? Made in these developing countries locally? Hard to believe.

Locally made furniture, for local people - the economics are against crappy stuff. Get a bad reputation, you've lost your job, you go hungry. The incentive is to make things as GOOD as you can, for the cheapest materials and least time. But the sweet spot is not the crap spot. And if the customer can source it cheaply he'll make sure you have the best available materials.

Crap is only possible when market economics have destroyed the local craft industry, and all the brand names are competing to get to the bottom as fast as possible.

Cheers,
Wol

Surviving old furniture and inflation

pizza — Fri, 26 Jul 2024 13:16:58 +0000

> And if you look into history, what you find is that the market for furniture back then was limited to the people who today are happy to pay a tonne of money for furniture - it was sufficiently expensive to buy anything that most people got by with far less than we have today.

Look no further than the word "cupboard". Today it refers to an enclosed cabinet with a door, but its origin is literally a "cup board", ie a flat piece of wood you store your cups on.