LWN: Comments on "Debian LTS?"

Debian LTS?

pabs — Thu, 12 Sep 2013 09:20:01 +0000

Woops, my reply is here:

http://lists.debian.org/debian-devel/2013/08/msg00649.html

Debian LTS?

faheem — Sat, 07 Sep 2013 17:19:39 +0000

Both these links bring up the same message for me.

Debian LTS?

mathstuf — Mon, 02 Sep 2013 17:12:06 +0000

> Truth be told, most of these legacy applications are insecure anyway, but there's little incentive for attackers to target something that only has 1 installation in the world.

Unless that one installation is hosted by a highly lucrative target.

Debian LTS?

intgr — Mon, 02 Sep 2013 16:42:33 +0000

> personally, i think that the whole "LTS is good for security" idea is seriously misguided.

Yes, but you are missing the point. The real reason is "LTS is good for stability" and "we need security as well" is an afterthought.

> i wrote my own shell and perl wrappers around ssh to automate mass upgrades of multiple servers - test the upgrade on test/dev machines first

As always, it depends on what you do. This may work well if you mostly only use distro-supplied components.

If you have tons of custom legacy applications that depend on distribution components and most applications have no meaningful way to automate testing, it becomes very hard to guarantee stability with updates. And you will almost always find out too late, in production, since it would take too much human time to thoroughly test every feature of every application on the testing setup every time. I hate to use this world, but I guess that's "enterprise".

Truth be told, most of these legacy applications are insecure anyway, but there's little incentive for attackers to target something that only has 1 installation in the world.

> of course, i also have the exciting new bugs instead of the boring old ones [...] I don't have any *more* bugs, I just have *different* ones

Old bugs are good bugs. People know them and know how to work around them. It's regressions that actually cost people time.

Debian LTS?

dlang — Sun, 01 Sep 2013 03:30:44 +0000

I agree with this. I haven't run my servers where everything gets upgraded as frequently as cas, but I've done a similar upgrade cycle on the packages that provide the functionality that my servers are there to provide, including the kernel (and the other packages aren't exposed to the network, so they aren't as much of a security problem to leave on whatever version)

As cas says, this approach makes upgrades a routine thing that you get really good at rather than something that requires special planning.

another advantage is that when you do run into problems (usually on your test/non-production boxes, but occasionally on a production box for a bug that only shows up under heavy load), it's far easier to contact the upstream developers and work with them to get the bug fixed than to report it to a distro against a version that upstream no longer supports and wait for things to get fixed (I've had bugs sit for years this way)

like upgrades, if you do custom packages routinely, they become no big deal

so for the last several years, my servers have been Debian Stable + custom kerneol + custom packages for the things that I care about.

Debian LTS?

cas — Sun, 01 Sep 2013 02:53:28 +0000

personally, i think that the whole "LTS is good for security" idea is seriously misguided.

IMO, the best way to maintain a secure system is to stay ahead of the bugs, ahead of the attackers - which means running (slightly behind) the leading edge of development.

in short: a moving target is harder to hit.

i've been running all of my desktop machines, and almost all of my production servers (thousands of them over the years - physical and virtual) on debian sid since the mid 1990s - with regular, routine upgrades every 2-3 months on average. before tools like puppet came along, i wrote my own shell and perl wrappers around ssh to automate mass upgrades of multiple servers - test the upgrade on test/dev machines first (which also gives first-hand experience of any required config changes so i can write a perl or sed script to automate the config update) and then automate the rollout to production if there were no major problems. in fact, i still use a mixture of my own scripts and puppet today.

the end result is that my servers tend to be at least 6+ months ahead of the script kiddies and other attackers, and I have the features, bug-fixes and improvements of the latest (or almost-latest) versions of all the packages I'm running. of course, i also have the exciting new bugs instead of the boring old ones...but only for a little while, if the bug is a big problem I either skip that update on production servers, or it gets fixed on the next routine upgrade. I don't have any *more* bugs, I just have *different* ones...and while i have no hard data to prove it, i suspect that I actually have *fewer* bugs this way.

for some rapidly-developing programs with bug-fixes and improvements appearing weekly or even daily, this is a huge benefit. qemu, kvm, libvirt and friends have been a great example of this - the pace of change and improvements with these packages really makes keeping up-to-date worthwhile.

I really hate to tempt fate by saying this, but i've never had one of my servers compromised - but i have had to fix (i.e. backup, erase, rebuild from scratch) servers run by people who think that "stable" or "LTS" is a good security model so end up running code which has had known exploits available for several years.

Another advantage is that instead of having to deal with a huge number of changes all at once every year or two, I deal with a much smaller number of changes all the time - this is much more manageable, and far less prone to causing downtime or problems. it's easier and much less work to fix one or two little things at a time than have to fix hundreds of things on a major update.

This is also made easier by the fact that dealing with upgrades is a routine experience that I get constant practice at, so I'm much better at it than if it was an unusual event, something I only had to do once every few years. familiarity breeds competence :-).

occasionally, when there are major changes in capabilities and especially configuration and compatibilty between one version of an important (to the particular server) package, I may have to Hold a package or set of packages for a while (a few weeks or months) so that an 'apt-get dist-upgrade' will leave them alone while the rest of the system is updated - e.g. apache/php/etc on a web server, samba or nfs on a file server, postfix/amavis/spamassassin/etc on a mail server, and so on.

Debian LTS?

iq-0 — Fri, 30 Aug 2013 17:48:02 +0000

So true. Most projects don't support any older version. If there's a bug, than it's fixed in the next release. You are expected to run the latest if you expect quality from those projects.

The problem is that this model fails when you have lots of interdependencies. This is where distributions step in and try to get the correct mix that can be regarded as stable.

But in the process they have to maintain version stability (partly because people expect their things not to change (any change can change something for the enduser, often positive but the negative ones are not tollerated) and partly because it's almost impossible to oversee the consequences of the whole when something is changed).
So distributions (and the expectations people have of them) are effectively the reason for needing security updates on older versions for most FOSS out there.

Debian LTS?

anselm — Fri, 30 Aug 2013 06:13:55 +0000

This is why distributions should not focus their attention on packaging and maintaining the software themselves, but instead focus their attention on the core operating system platform and work with upstream on packaging their software, upstream, for multiple distributions.

Right. As if upstream would be interested in maintaining packages for multiple distributions for years and years.

Many upstream projects barely have the manpower to move their software forward, and new development is a hundred times more alluring than maintaining old code.

upstreams don't have the manpower

dlang — Fri, 30 Aug 2013 04:24:14 +0000

upstream projects don't have the manpower to maintail 'LTS' versions for the 5+ years that some people want, most of them don't even really support the 3 years or so that the current Debian Stable claims to provide. This isn't just small projects, the Linux Kernel doesn'treally provide support for this long.

Debian LTS?

drag — Fri, 30 Aug 2013 01:03:25 +0000

This is why distributions should not focus their attention on packaging and maintaining the software themselves, but instead focus their attention on the core operating system platform and work with upstream on packaging their software, upstream, for multiple distributions.

'Linux ecosystem' as a whole has more then enough manpower to support significant software going back many years if upstream is good to work with on this subject and the software is well enough designed.

Obviously some projects are going to be more troublesome then others to maintain, but that is just going to be something end users are going to be forced to take into consideration when they choose the software stacks that their 'enterprise' applications depend on. It is very simply not worth it for some upstream vendors to support older versions indefinitely... their software was not designed for it and often they lack the desire for it or maintaining all released versions for 5 years has zero benefit to anybody. Other projects are different and it is sane to support their major releases for a very very long time.

The way things currently stand were distributions retreat to their respective silos and try to build and support the entire available software ecosystem in perfect equality until X many years is complete lunacy.

Individual distributions lack the manpower, ability, knowledge, expertise, and just plain intimacy with each and every software package that they build and then drop off on their ftp server. Granted some packagers are the same people as upstream and some packagers participate heavily in various projects, but I can't imagine that would make up a significant amount of the software ditributions distribute. As it is now they can barely keep up with the software packages that they currently support for their latest and greatest and there is a huge amount of software that goes completely neglected by distributions.

It is never going to work the way things are and unless people are willing to change how they organize themselves then it's never going to get better.

Debian LTS?

vonbrand — Thu, 29 Aug 2013 18:56:13 +0000

How many packages does Debian have? How many packagers? I'd suspect that the brainpower needed to look after a package's security updates competently is rougly comparable to the one needed to keep up with upstream. So a few people working on this aren't enough. Not by a very long shot.

(Same applies to any distribution, obviously)

Debian LTS?

pabs — Thu, 29 Aug 2013 12:38:06 +0000

It is interesting to note that at least one company has now committed employee time to help out with Debian security as a result of this discussion. I also believe at least one existing Debian security team member is paid by his employer for doing Debian security work.

http://lists.debian.org/20130829095957.GB14146@ftbfs.de

If anyone reading the article is able to commit time to help out with Debian security, please check out my reply:

http://lists.debian.org/debian-devel/2013/08/msg00645.html