Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
Posted Aug 29, 2016 20:25 UTC (Mon) by Darkmere (subscriber, #53695)In reply to: Böck: Multiple vulnerabilities in RPM – and a rant by SEJeff
Parent article: Böck: Multiple vulnerabilities in RPM – and a rant
Directly calling various awk implementations, the OpenSSH package calling out to perl in order to re-write the configuration file (And also leading to the OpenSSH server configuration file not being listed as a configuration file. Because it's a special snowflake.)
Other massive failures like maintainer scripts failing if the user inside a root doesn't exist _outside_ it ( dpkg really doesn't do roots very well).
Posted Aug 29, 2016 20:37 UTC (Mon)
by SEJeff (guest, #51588)
[Link] (21 responses)
preseed is awful (only autoyast is worse). Canonical attempted to repair this by writing kickseed but that ultimately failed.
abrtd + faf are amazing for being able to gather any segfault / thing to drop core on thousands of nodes and have a central reporting place. I think Ubuntu has a trimmed down simplified version of this with the apport retracer bits, but faf is generic enough it would be quite trivial to write a dpkg plugin and just have one to rule them all.
rpm -V so hard! I recall several years back (5 or 6, maybe more?) Debian Packaging Policy didn't require including the bits to run debsums, so there were literally packages in the archive that didn't work with debsums. Unbelievable to me. I know there was a huge bit of work from Canonical to fix all of the packages missing them and then the policy changes to require it. However, you still can't get the information you can get from rpm -V from any of the debian tools that I'm aware of. Simply put, a silly hack like this thing I wrote for fun (http://www.digitalprognosis.com/opensource/scripts/restor...) is absolutely impossible to do on any dpkg based system.
I could literally go on quite a while, but find the Redhat tends to be a better overall distro. Yes Debian Stable is really great for certain use cases, but for what I do (HPC / Finance), Redhat based distros simply blow it out of the water.
Posted Aug 29, 2016 21:36 UTC (Mon)
by Darkmere (subscriber, #53695)
[Link] (12 responses)
Having an installed system and being unable to verify the installed packages post-installation, because they only sign the list of hashes of all packages. So even if you keep a backup copy of all packages you install, you cannot verify the content or state of them.
It requires a special kind of madness to design this system.
Posted Aug 29, 2016 22:03 UTC (Mon)
by amacater (subscriber, #790)
[Link] (11 responses)
You know too, that the installation process checks and verifies, just as on Red Hat?
Essentially, pretty much what you can do on Red Hat you can as easily do on Debian.
Posted Aug 30, 2016 10:01 UTC (Tue)
by niner (subscriber, #26151)
[Link] (10 responses)
So dpkg seems nowadays been able to do a subset of the checks rpm does though arguably it's the most important part. It even uses rpm's output format.
Good to see dpkg catching up there.
Posted Aug 30, 2016 10:06 UTC (Tue)
by Darkmere (subscriber, #53695)
[Link] (9 responses)
apt (not dpkg) only signs the list of hashes of all packages.
There is no signature on the metadata of each package, which means that the list of sizes, permissions, and checksums are not signed.
So you can't go from "I have foobar-1.0_4.deb installed", to "Is it possible to verify that the files I have are the ones that Debian shipped?"
in an RPM package, the metadata is signed for each package. In deb, dpkg doesn't even support signatures, that's all done on the APT-layer. Officially, this is to be able to revoke signature on a file.
Unofficially it seems to be designed to make it easier to backdoor Debian & Ubuntu systems.
Posted Aug 30, 2016 20:24 UTC (Tue)
by BenHutchings (subscriber, #37955)
[Link] (7 responses)
There is a small advantage that signed packages get you here, which is you could maybe pull the original package out of a cache on the suspect system and verify its signature before using it as a reference. Without package signatures, you would have to download it again from a trusted archive. Is that what you're getting at?
Posted Aug 30, 2016 20:45 UTC (Tue)
by Darkmere (subscriber, #53695)
[Link] (3 responses)
package contains a list of filenames + hashes + metadata
File contents can be validated by the hash, integrity of the list of the hashes can be verified by itself.
The problem is that on a given debian system, you cannot go from "this package is 1.1_deb33" installed, and come to the answer of the question "Have it's files been maliciously tampered with".
There is no historical list of these signatures, once a new package is uploaded to the debian mirrors, you cannot verify any older package.
There is also no policy of not fucking around with the contents of binaries on disk -after they have been installed- by either this package, or some other package. It happens quite often that post/pre-install scripts start to mess with the files on disk.
The reasoning that Debian choose to blame was the fact that "an older package that contains serious bugs should not be trusted if a new has been published". A false claim if any ( you could simply not trust the on-file signature for installation purpose. )
What frustrates me is that once a system is no longer 100% up to date, you cannot validate that the files you have installed are unmodified.
Posted Aug 30, 2016 21:22 UTC (Tue)
by BenHutchings (subscriber, #37955)
[Link]
That sounds quite good, though its usefulness depends on exactly what's being hashed. (For example, does the signed hash include the package name and version?) That's not longer true for at least Debian, since historical packages and the corresponding signed package indexes are accessible through snapshot.debian.org. However it's not a totally straightforward procedure to download and verify a binary package from there (and the debsnap tool doesn't), let alone to check all the packages supposed to be installed on a suspect system.
Posted Sep 2, 2016 14:29 UTC (Fri)
by yoe (guest, #25743)
[Link] (1 responses)
Posted Sep 4, 2016 11:48 UTC (Sun)
by Darkmere (subscriber, #53695)
[Link]
And as someone elsewhere stated, apparently dpkg does support signed binaries, just that Debian doesn't use it.
Debian, when given two choices would make two optional and a third mandatory.
Posted Aug 30, 2016 21:19 UTC (Tue)
by lsl (subscriber, #86508)
[Link] (2 responses)
There are other kinds of unwanted modification, like accidental corruption. Think of running "make install" where the Makefile unexpectedly defaults to a prefix of /usr. Or the unbounded joy of things like pip/rubygems/… happily blowing away system packages.
That's where rpm -V style verification can be very useful.
Posted Aug 30, 2016 22:03 UTC (Tue)
by BenHutchings (subscriber, #37955)
[Link] (1 responses)
Posted Aug 30, 2016 22:20 UTC (Tue)
by guillemj (subscriber, #49706)
[Link]
Posted Aug 30, 2016 22:26 UTC (Tue)
by guillemj (subscriber, #49706)
[Link]
Posted Aug 30, 2016 4:41 UTC (Tue)
by voltagex (guest, #86296)
[Link] (7 responses)
Posted Aug 30, 2016 13:56 UTC (Tue)
by imMute (guest, #96323)
[Link]
The biggest problem I had with the whole process was not actually with preseed itself, it was with repackaging the installation media. I was simply trying to take the netinstall image, slip in my own preseed file, and repackage it for use with a USB stick. I found a couple sets of instructions on how to do that and the repackage step either would use a command that doesn't work on Jessie, or would produce an image that wouldn't boot. I eventually got it mostly working using an Ubuntu 14.04 system to run the repackage command.
It was about a year ago when I attempted this, and we gave up since cloning the entire drive was a faster way anyway - things may be different these days.
Posted Aug 30, 2016 15:41 UTC (Tue)
by SEJeff (guest, #51588)
[Link] (2 responses)
On the other hand, preseed is more murky in comparison and the documentation (used to be) horrible:
# This is how to make the installer shutdown when finished, but not
The redhat equivalent (https://access.redhat.com/documentation/en-US/Red_Hat_Ent...) is halt, just like the shell command. This is a relatively obscure reference, but in general as a sysadmin to write a preseed from scratch, you have to understand a lot of how debian-installer works.
As a sysadmin to write a kickstart from scratch, you need a lightly templated shell script with a few special stanzas. It is just so much easier.
Posted Aug 31, 2016 12:05 UTC (Wed)
by pizza (subscriber, #46)
[Link] (1 responses)
You don't even need to write it from scratch -- Do a single installation with the rough (or exact) settings you want, and as part of the installation it'll generate a kickstart file that corresponds to your installation choices. Customize it to your heart's content with addtional packages and your postinstallation scripts, and go to town.
(Kickstart is a wonderful feature. I've been using it since the RHL7 days; every single system we had in production could be completely recreated automatically; stick in a floppy and come back in a couple of hours...)
Posted Aug 31, 2016 13:53 UTC (Wed)
by SEJeff (guest, #51588)
[Link]
Posted Aug 30, 2016 18:32 UTC (Tue)
by edgewood (subscriber, #1123)
[Link]
Posted Aug 30, 2016 19:05 UTC (Tue)
by dskoll (subscriber, #1630)
[Link]
Oh, hey, I love Debian. But let me say this: I hate, hate, hate preseed with a bitter, burning passion. And working with d-i is also an exercise in pain... lots of twisty shell scripts, Perl scripts, C programs, and magical run-parts invocations without any damn clue how it all fits together.
Pressed: no documentation. Extremely fiddly. Incredibly long edit-test cycle (you basically have to make new boot media or PXE images, boot the thing, see what breaks, rinse, repeat.)
And the worst part (though I think this may have been fixed... not sure) was that
some of the answers were locale-sensitive. So if you had a user who picked an
unexpected locale, all the preseed answers would be borked.
Posted Aug 30, 2016 21:30 UTC (Tue)
by seyman (subscriber, #1172)
[Link]
One of the things I've alway appreciated when using kickstart is that a kickstart file is always generated during an installation of Fedora/RHEL/Centos (found in /root/anaconda-ks.cfg). This allows you to perform an install, grab the kickstart file and be 99% done.
Posted Aug 30, 2016 4:29 UTC (Tue)
by voltagex (guest, #86296)
[Link] (12 responses)
These should fail on the package builder servers / reproducible build servers, yes?
If you've got any examples of these in stable or testing, I would have thought you could raise a bug - let me know if you need a hand.
Posted Aug 30, 2016 14:17 UTC (Tue)
by Darkmere (subscriber, #53695)
[Link] (11 responses)
Prime one is just to grep for 'mawk' rather than 'awk' in all the packages. Perl is another favourite, that will be present in all "normal" debian systems, since apt requires it, but if you build a new root (say a container) then, suddenly...
Posted Aug 30, 2016 16:25 UTC (Tue)
by juliank (guest, #45896)
[Link] (10 responses)
Policy 3.5:
Posted Aug 30, 2016 16:49 UTC (Tue)
by Darkmere (subscriber, #53695)
[Link] (4 responses)
This is where debian packaging _shines_ at being obtuse.
base-files is essential
This then causes a hilarity of issues when bootstrapping, because you need to run the preinst configure job for mawk before any others, otherwise "awk" will not point to "mawk", something that dpkg is, by and of itself, uncapable of figuring out.
Posted Aug 30, 2016 18:19 UTC (Tue)
by tao (subscriber, #17563)
[Link] (3 responses)
As far as bootstrapping goes -- debootstrap usually does its job without a hitch -- are you saying that it fails for you?
Posted Aug 30, 2016 18:23 UTC (Tue)
by Darkmere (subscriber, #53695)
[Link] (2 responses)
I _have_ read the chapters, and I've worked around these issues, but they _are_ causing a major painpoint and showing some of the haphazard decisions in the debian packaging system.
The simple fact that a lot of the pre/post-install scripts are larger than several binaries on the system should be a cause of concern for most people.
Posted Aug 30, 2016 22:13 UTC (Tue)
by guillemj (subscriber, #49706)
[Link] (1 responses)
I've covered the situation with the awk virtual package in the following debian-policy bug message <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=759491#156> (see the whole thing for more details), so this is a case of lore that has unfortunately never been transcribed into proper documentation.
Also bootstrapping Debian is outside the realms of debian-policy, it's not covered there, debian-policy assumes that the pseudo-essential set has been fully configured at least once, I've gone into more detail in the following mail <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=767999#157>.
I've also drafted in the recent past a proposal to clean this up <https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap>, which might fix your issues in case you are trying to bootstrap something that is not supported by debootstrap, this "just" needs coordination, consensus and work.
Posted Aug 31, 2016 21:45 UTC (Wed)
by Darkmere (subscriber, #53695)
[Link]
Posted Aug 31, 2016 12:35 UTC (Wed)
by nye (subscriber, #51576)
[Link] (4 responses)
This policy has always upset me. I've never seen any explanation of why it's this way, which is strongly needed because on the face of it it's clearly a splendidly terrible idea. Normally when an idea is obviously bad, I expect to see some reasoning about why it's actually less bad than the alternatives.
I presume those reasons do exist, because nearly nothing happens in Debian without a reason, but whenever I've seen people ask about this (I'm too afraid to do so myself) the conversation seems to go:
Q: So why is this policy?
Posted Aug 31, 2016 23:16 UTC (Wed)
by anselm (subscriber, #2796)
[Link]
Presumably the idea is to avoid burdening lots of packages with long and largely identical lists of dependencies to “essential” packages. These tend to bog down dependency solvers, which can be a hassle on machines that are slow(ish) and/or don't have vast amounts of RAM, like the sort of PC that was current when Debian was new in the mid-1990s.
This policy has been around for a very long time, and in 2016 we could perhaps afford the explicit dependency lists more easily than we used to in 1995, but it is safe to say that nobody is looking forward to fixing this in tens of thousands of packages.
Posted Sep 1, 2016 12:48 UTC (Thu)
by jwilk (subscriber, #63328)
[Link]
Posted Sep 3, 2016 3:02 UTC (Sat)
by guillemj (subscriber, #49706)
[Link] (1 responses)
* They are needed by dpkg itself when performing operations on packages, on itself included (although this surface is being shrinked).
Thus:
* They have to be very hard to remove, requiring force options so that they are "guaranteed" to be always present.
Hope this clarifies things a bit. :)
Posted Sep 5, 2016 12:23 UTC (Mon)
by nye (subscriber, #51576)
[Link]
Thanks, this is the part that seems to be missing: that there is no way to declare a dependency that can be satisfied by a package being unpacked, but not configured. I wonder if there's any possibility of having this explicitly spelled out in section 3.5? (I have no idea how hard it is to get an explanatory note added to the policy document; maybe it's straightforward or maybe it's the same process as changing the policy itself.)
The issue of dependency loops is the only other explanation I've seen mentioned, but in itself it doesn't seem sufficient given that it could be solved with some special case rules much simpler than the current special case prohibition on declaring them as dependencies, or possibly is already covered by the other properties of essential packages.
It's unfortunate that this special dependency status wasn't created in the first place, rather than the rule in 3.5, because at least having that information declared makes changing the essential set (or creating ultra-small derivatives like the old Emdebian Crush) a more tractable problem, rather than pretty much requiring an audit of every package. Obviously I understand that hindsight is 20:20 and that changing it now would be a mammoth job without immediate benefit.
Posted Aug 30, 2016 22:33 UTC (Tue)
by guillemj (subscriber, #49706)
[Link] (2 responses)
That should (in principle!) not happen when using either --root or --instdir, but if you have a reproducer I'm very interested in a bug report, or a short recipe, so that I can get it fixed. Thanks.
Posted Aug 31, 2016 21:58 UTC (Wed)
by Darkmere (subscriber, #53695)
[Link] (1 responses)
On a system (well, container) created with `debootstrap --include=debootstrap stable <something>`, run:
This means that the building container doesn't have `dbus` installed and the chroot inside does. Wich causes debootstrap to fail.
Posted Aug 31, 2016 22:44 UTC (Wed)
by guillemj (subscriber, #49706)
[Link]
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
"Currently the only functional check performed is an md5sum verification of the file contents against the stored value in the files database. It will only get checked if the database contains the file md5sum."
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
package contains a signature blob of above list
package manager then saves this in the database
Böck: Multiple vulnerabilities in RPM – and a rant
File contents can be validated by the hash, integrity of the list of the hashes can be verified by itself.
There is no historical list of these signatures, once a new package is uploaded to the debian mirrors, you cannot verify any older package.
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
Not a part of the package management tools.
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
dpkg has supported signed .deb for a long time (since dpkg 1.9.0) via the debsig-verify package (check also the --no-debsig dpkg option in /etc/dpkg/dpkg.cfg). You can sign them with the debsigs package.
Debian just does not use signed .deb packages, it uses signed repository indices.
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
# reboot into the installed system.
d-i debian-installer/exit/halt boolean true
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
Also not the person you were responding to, but I've been learning preseed as a way to have a reproducible install of servers, both bare metal and VMs. From my perspective, the things I don't like about preseed are:
Böck: Multiple vulnerabilities in RPM – and a rant
So basically doc and improvements to partitioning, I guess.
preseed (was Böck: Multiple vulnerabilities in RPM – and a rant)
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
Packages are not required to declare any dependencies they have on other packages which are marked Essential (see below), and should not do so unless they depend on a particular version of that package.
Böck: Multiple vulnerabilities in RPM – and a rant
libc6 is essential
base-files has Pre-Depends on awk
(m)awk is of course _not_ Essential.
libc6 of course _uses_ awk in the setup scripts.
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
>Packages are not required to declare any dependencies they have on other packages which are marked Essential (see below), and should not do so unless they depend on a particular version of that package.
A: Because it's part of our packaging requirements.
Q: So why is it part of our packaging requirements?
A: Because it's policy.
(Perhaps with a side order of "it's always been that way".)
Böck: Multiple vulnerabilities in RPM – and a rant
Um, Policy does explain it:
Böck: Multiple vulnerabilities in RPM – and a rant
Essential is needed in part to avoid unresolvable dependency loops on upgrade. If packages add unnecessary dependencies on packages in this set, the chances that there will be an unresolvable dependency loop caused by forcing these Essential packages to be configured first before they need to be is greatly increased. It also increases the chances that frontends will be unable to calculate an upgrade path, even if one exists.
Böck: Multiple vulnerabilities in RPM – and a rant
* They are needed to boot the system, at least to a point where you can run into some kind of recovery mode (although there's been talk about getting rid of this for the benefit of chroots and similar).
* They are needed by several maintainer script types, which have no other way to declare dependency satisfiability at that point; postrm and config maintainer scripts cannot rely on any of their dependencies being satisfied, and only rely on the implicit set of Essential:yes packages. preinst can only rely on Essential:yes and Pre-Depends being satisfied (but not necessarily fully configured).
* They have to be always functional, regardless of their installation state (even when not fully configured, just unpacked or other transitioning states), and regardless of system crashes/reboots/power-outage/etc midair on dpkg operations (barring filesystem/disk corruption). There's no dependency field to distinguish that relationship, because both Pre-Depends and Depends are too strong. So we'd need to add a new field just to declare that kind of relationship, which means that removing stuff from Essential would probably be similarly burdensome just in a different way, as we'd have to update thousands of packages to switch the field name. (I've not even thought how a partial upgrade from a release that has a package marked as Essential:yes to one that does not, would even work with such fictional field.)
* They are used to avoid dependency loops. This does not have much to do with hardware performance, but more to do with how dpkg handles dependency loops in presence of maintainer scripts, it just picks where to break the loop randomly, so you might get in trouble. Which means dependency loops are always an undesirable thing to get into. The above mentioned properties of Essential:yes removes this problem.
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
Böck: Multiple vulnerabilities in RPM – and a rant
`debootstrap stable newroot`
Böck: Multiple vulnerabilities in RPM – and a rant
