LWN: Comments on "Fedora considers curl-minimal"

Bootstrap problem

paulj — Thu, 09 Jun 2022 10:25:04 +0000

They are not dead. There are large companies with such setups.

IDNs

mirabilos — Sun, 20 Mar 2022 10:33:13 +0000

You are missing two things.

One, the attack surface of IDNs will be gone from cURL, which I believe is the actual point of this exercise.
Two, the presence or absence of IDN support can be done independent of cURL or other tools, and even be added to only a subset of system images.

IDNs

zdzichu — Sun, 20 Mar 2022 08:04:04 +0000

So instead of curl being linked with libidn, you will have punyencode linked with it. libidn is still on your system. In the big picture, you have not shrinked your system image (which is the point of curl-minimal exercise).

IDNs

mirabilos — Sun, 20 Mar 2022 05:29:57 +0000

Instead of having tools automatically support IDNs, with the exception of GUI tools that directly take typed user input (and even there, different solutions might be better), I’d rather have a CLI tool to punyencode things which you can then call like: curl http://$(punyencode "$domain")/path/file

Fedora considers curl-minimal

cypherpunks2 — Sat, 19 Mar 2022 01:23:34 +0000

The security argument is that the protocol implementations may have security vulnerabilities, not that the protocols themselves are not secure by design. FTP is insecure in that it is unencrypted and poorly authenticated, but it is not (necessarily) insecure in that it's easy to find an RCE in the code.

Fedora considers curl-minimal

amacater — Tue, 15 Mar 2022 15:11:37 +0000

Curl is the "Oh dear, if that's the recommended way to get components for [Kubernetes/Open Stack/any other popular program], it's bound to be a world of hurt putting it all together" moment for me.

Maybe I've been insulated by living with distributions for too long but it's also very much a "don't trust anything that asks you to curl/wget stuff from random 'Net addresses" syndrome, I'm afraid.

Fedora considers curl-minimal

Paf — Tue, 15 Mar 2022 12:29:26 +0000

Given curls use in scripts and other tools used in installers, etc, I feel comfortable saying the user survey is massively *unrepresentative* of use.

Those interested enough in curl to take the survey are vastly more likely to use weird protocols.

In essence we see it has two lives:
A basic system component which is worked in to the fabric of other things, in which role it uses HTTP, HTTPS, and FTP to get stuff from the internet
A Swiss army utility protocol fiddler/translator for developers and admins

It doesn’t seem crazy these would be separate packages, given the risks posed to the (much larger) first group, and the minor burden introduced for the second group who know how to deal with it.

Fedora considers curl-minimal

Wol — Tue, 15 Mar 2022 08:11:41 +0000

Interesting you mention "Inform 7" - especially as it was written by a Graham Nelson.

Because one of the names of the Pick data access language was "English", another was "Inform". And it may have (although I don't think so) been written by Don Nelson, one of the architects of Pick.

(It was called English, because it is similar to English, and likewise it allows pretty complex query logic. "Without gouging your eyes out" as you so eloquently put it - and as I'm now finding with my SQL programming at work ...)

Cheers,
Wol

Fedora considers curl-minimal

PengZheng — Tue, 15 Mar 2022 08:07:58 +0000

It's no surprise to me that the biggest user of curl is backend developer (page 4 of https://daniel.haxx.se/media/curl-user-poll-2021-analysis...). It will be a big surprise to most users (and programs using libcurl) that the "swiss army knife" has only 4 built-in protocols.

A safe default setting should be enough solving the mentioned problem.

Fedora considers curl-minimal

NYKevin — Tue, 15 Mar 2022 03:34:57 +0000

> programming languages do less of that or pay a price for it.

See for example Inform 7, which is specifically intended to look like English (and is therefore extremely prone to all sorts of weird parsing issues, but OTOH it has support for fairly complex English predicates, meaning you can do logic programming without gouging your eyes out).

Fedora considers curl-minimal

bagder — Mon, 14 Mar 2022 22:10:14 +0000

Let me repeat myself:

"which defaults to HTTP, HTTPS, FTP and FTPS"

Fedora considers curl-minimal

bagder — Mon, 14 Mar 2022 18:45:12 +0000

The curl project asks its users about these things in its annual survey. While that then is self-reported it certainly isn't an unquestionable truth, but probably the best what-features-in-curl-is-used numbers you can get.

The 2021 survey analysis is linked to from here: https://daniel.haxx.se/blog/2021/07/05/curl-user-survey-2...

Fedora considers curl-minimal

Paf — Mon, 14 Mar 2022 15:00:52 +0000

It’s just incredible to me that people seem to be arguing in favor of never deprecating *anything* by default in an internet facing program. Sure, the selected list seems too broad, possibly much too broad, but the idea that “respecting developer power” should trump anything else…. Or the repeated - in the comments here - detailed arguments about binary size, which the article and thread make clear is a non-concern…. It’s fun to have every protocol ever, sure, but if you want that it’s one package install away. One of the absolute best ways to reduce burden and improve security is *removing stuff*. Code that doesn’t exist has no bugs.

As for the Fedora thread, the idea that everyone is going to “unbreak Fedora by installing full curl”…. No, if the protocol list is reasonable (I agree this removal is too broad), 99+% of users (including developers) will *never notice there was a change*.

How many of you can honestly say you’ve needed curl to support something other than HTTP, HTTPS, FTP, SFTP, NTLM, brotli, and (I guess?) TFTP in the last decade? (IDN gets a pass for reasons cited in the article.) Not for fun - actually needed.

Fedora considers curl-minimal

rwmj — Mon, 14 Mar 2022 14:55:41 +0000

More likely if someone installs something not in Fedora and they have curl-minimal they'll be wondering why the package they just installed doesn't work.

Fedora considers curl-minimal

Paf — Mon, 14 Mar 2022 14:47:52 +0000

An option which no doubt is rigorously set by all users at all times. “Options” for doing the secure thing are borderline meaningless in most situations unless they’re shipped on by default.

Fedora considers curl-minimal

Paf — Mon, 14 Mar 2022 14:45:45 +0000

That’s not true at all. Curl is a regular - if horrifying - component of scripts.

And this isn’t “reducing developers power”. There’s literally going to be a “with ancient unused protocols fully intact” version *also packaged by Fedora*. For our theoretical developer, ‘fixing’ curl so it can support Gopher again is one brief command away.

Fedora considers curl-minimal

Paf — Mon, 14 Mar 2022 14:43:37 +0000

We could do that if we’re “serious about security”, or, being equally serious about security, we could do this. And then if someone ever installs something that isn’t packaged by Fedora, they would *also* benefit from this change if it doesn’t use that option correctly.

Yes, if we handle our footguns *correctly*, there’s no issue. There’s ongoing overhead and risk from their existence, but obviously, handled correctly, they’re fine. If they’re completely unnecessary - like most, though not all, of these protocols clearly are - we could also *stop shipping them by default*.

Fedora considers curl-minimal

rahulsundaram — Sun, 13 Mar 2022 12:03:51 +0000

> I love that spoken language grammar is easier to understand to us in terms of programming language syntax.

Spoken language has a lot more amguity that humans can parse based on context, programming languages do less of that or pay a price for it.

Fedora considers curl-minimal

rolandog — Sat, 12 Mar 2022 07:44:21 +0000

I love that spoken language grammar is easier to understand to us in terms of programming language syntax.

Fedora considers curl-minimal

PengZheng — Sat, 12 Mar 2022 04:24:17 +0000

CURL is mainly used by developers (or not?).
It really makes no sense to reduce developers' power.

Fedora considers curl-minimal

HenrikH — Fri, 11 Mar 2022 18:51:04 +0000

Add to this that curl is much easier to script for SFTP upload/download since one does not have to mess around with expect scripts and what not.

Fedora considers curl-minimal

bagder — Fri, 11 Mar 2022 14:11:48 +0000

> Even if a program is using a URL with an http scheme (i.e. protocol), the (possibly malicious) server could redirect to a different URL with a different protocol entirely, which would then invoke that code in curl if it is present

That was true until curl 7.65.2, released on Jul 17 2019. Since then there's a separate option for setting which protocols curl accepts redirects to, which defaults to HTTP, HTTPS, FTP and FTPS.

/ Daniel

Fedora considers curl-minimal

tao — Fri, 11 Mar 2022 13:19:39 +0000

No. "with [...] disabled"

"with" refers to the enumeration of items, "disabled" applies to that selection.

Think of it in programming terms:

"with" item in [dict, gopher, ...]; do
"disable" item
done

Fedora considers curl-minimal

jake — Fri, 11 Mar 2022 13:15:42 +0000

> `curl-minimal`+`libcurl-minimal` are compiled with various
> semi-obsolete protocols and infrequently-used features disabled:

the wording is a little confusing, as the 'disabled' at the end is somehow easy to miss (or at least i did too at first), but the minimal versions have various protocols and features *disabled*, thus not present in those builds.

jake

Fedora considers curl-minimal

andika — Fri, 11 Mar 2022 08:33:14 +0000

I'm confused

https://lwn.net/ml/fedora-devel/CA+voJeWtucz4WFZN6kLU2PKC... said initially:
The "minimal" variants provide only a subset of protocols (HTTP, HTTPS, FTP).

then later said:
`curl-minimal`+`libcurl-minimal` are compiled with various
semi-obsolete protocols and infrequently-used features disabled:
DICT, GOPHER, IMAP, LDAP, LDAPS, MQTT, NTLM, POP3, RTSP, SMB, SMTP,
SFTP, SCP, TELNET, TFTP, brotli compression, IDN2 names.

does he really mean to say:
`curl-minimal`+`libcurl-minimal` are compiled **without** various
semi-obsolete protocols and infrequently-used features disabled:
DICT, GOPHER, IMAP, LDAP, LDAPS, MQTT, NTLM, POP3, RTSP, SMB, SMTP,
SFTP, SCP, TELNET, TFTP, brotli compression, IDN2 names.

Fedora considers curl-minimal

taladar — Fri, 11 Mar 2022 08:17:55 +0000

It seems very strange to remove many protocols in active use as well as IDN "for security reasons" and yet leave the underspecified, unencrypted mess that is FTP in there.

Fedora considers curl-minimal

jalla — Fri, 11 Mar 2022 02:09:25 +0000

MS still ships with native FTPS support, feels a bit silly to leave that one out. The rest have proper clients (like scp) that should likely be preferred instead.

Bootstrap problem

zdzichu — Thu, 10 Mar 2022 15:55:24 +0000

I had such setup on one of the past workplaces. We used locally running "cntlm" proxy.

Fedora considers curl-minimal

smoogen — Thu, 10 Mar 2022 14:43:01 +0000

The problem with auditing is the same as saying upstream should make curl modular. Someone has to step up and do the work and no one has (mainly because you then have to interact with N upstreams who may not see it as something they want to do either.) This seems to be the inevitable 'ok so no one has that energy, what can we do?' compromise.

Fedora considers curl-minimal

rwmj — Thu, 10 Mar 2022 11:52:44 +0000

It's not how curl works now and curl already has an API for limiting protocols. I would suggest discussing any plans with upstream curl.

Fedora considers curl-minimal

SLi — Thu, 10 Mar 2022 11:35:50 +0000

How is determining it globally worse than having the option to have either curl-minimal or curl-full installed? To me, it seems it would provide the best of both worlds, and you could also provide an API for interested clients to say "I want to enable support for Gopher".

Bootstrap problem

james — Thu, 10 Mar 2022 10:46:50 +0000

Are non-transparent proxies with NTLM authentication really that dead? A system behind one of them is not going to be able to dnf install anything if it can't authenticate (dnf uses libcurl).

Fedora considers curl-minimal

jengelh — Thu, 10 Mar 2022 09:49:19 +0000

There is a limit to practicality. Some considerations:

* Providing both curl and curl-mini variants may reduce the container by 100KB, but now you have added at least 500KB to your mirrors.
* Some software has so many build dependencies that, indeed, it may make sense to build a -mini variant and then another standard variant. openSUSE does that to a few select packages to cut down on overall project build time and/or reducing build cycle lengths. libcurl is not among those, because it is not nearly as dependency-heavy as e.g. systemd.
* xkcd.com/1172
* The overhead of ELF is so damn high these days. ~14 KB for an "int main(){}" built with standard compilers and options plus strip. Splitting libcurl into multiple component libraries hence raises the disk usage for at least one case. Your RPM/DEB database would have to process more entries perhaps (because now libcurl4 and libcurl-gopher4). ld-linux.so would have to deal with more libraries, load times getting worse. You really don't want to end up like samba-libs either, do you!?
* I predict that most people will probably end up with the full curl installed for one reason or another, and the security argument gets weak.
* The "url" implementation of many a browser has probably a lot more fat than curl. If only firefox and chromium would use libcurl instead of handstrung solutions, that might, overall, be a better outcome. Don't try to make the small smaller, make the large smaller.

Fedora considers curl-minimal

rwmj — Thu, 10 Mar 2022 09:03:22 +0000

You cannot determine this list globally. There's already a mechanism in libcurl to limit protocols that can be used (CURLOPT_PROTOCOLS). It has to be invoked by the programs linking to it since they are the only thing that know what protocols they're expecting to use.

Fedora considers curl-minimal

rwmj — Thu, 10 Mar 2022 09:01:09 +0000

If we are really serious about attack surface we should audit Fedora programs that use libcurl and make sure they are using CURLOPT_PROTOCOLS (https://curl.se/libcurl/c/CURLOPT_PROTOCOLS.html), because that is the only way to ensure that unwanted/exploitable curl modules are not invoked by redirects.

Fedora considers curl-minimal

SLi — Thu, 10 Mar 2022 08:53:57 +0000

I'm curious, why does it seem to be about size instead of the attack surface as explicitly stated?

Fedora considers curl-minimal

SLi — Thu, 10 Mar 2022 08:52:43 +0000

It doesn't seem to even be what's compiled in but what's enabled by default. Perhaps there could be a configuration file in which you can enable protocols you need one by one (and disable ones enabled by default)?

Fedora considers curl-minimal

rwmj — Thu, 10 Mar 2022 08:48:06 +0000

This is exactly right. If curl was modular upstream then we could have an interesting discussion about which of the modules to include in base Fedora. But nothing would be broken - if a package was expecting to use, say, telnet it could depend on libcurl-telnet so installing the package would install the extra curl module.

As you say it requires work upstream. No one has stepped up to do that.

FWIW we already went through this process with fio (Jens Axboe's Flexible I/O tester) which used to link all of its backends together, but now uses modules ("engines"), and we then changed the packaging in Fedora to use it: https://src.fedoraproject.org/rpms/fio/blob/rawhide/f/fio...

Fedora considers curl-minimal

rwmj — Thu, 10 Mar 2022 08:42:23 +0000

It seems clear to me that the driver for this is some pointless competition where everyone tries to claim the crown of having the smallest base container. But container layers get cached - you only download them once - disks are cheap and networks are getting better. This isn't something we need.

Fedora considers curl-minimal

jd — Thu, 10 Mar 2022 08:40:39 +0000

Correct me if I'm wrong, but the real problem seems to be that the protocols are compiled in, as opposed to being something external that is dynamically loaded.

If they're dynamically loaded, then you would only have one version of any given binary, users can decide exactly what protocols they want to install and protocols that are known to be in an uncertain state can be labeled as such.

This moves responsibility from the distros to the curl maintainer, which means we don't have different distros running in different directions and nobody really knowing what they're installing.

Of course, that's a lot more work than just compiling the code twice, but it only has to be done the one time whereas protocol selection and recompiling would have to be done every time there's a code change. Not that this happens much with curl.