A new Debian debuginfod service

Sergio Durigan Junior has announced the availability of a debuginfod server for Debian systems. "In a nutshell, by using a debuginfod service you will not need to install debuginfo (a.k.a. dbgsym) files anymore; the symbols will be served to GDB (or any other debuginfo consumer that supports debuginfod) over the network. Ultimately, this makes the debugging experience much smoother (I myself never remember the full URL of our debuginfo repository when I need it)."

From:		Sergio Durigan Junior <sergiodj-AT-debian.org>
To:		Debian Devel Announce <debian-devel-announce-AT-lists.debian.org>
Subject:		New service: https://debuginfod.debian.net
Date:		Tue, 23 Feb 2021 22:53:14 -0500
Message-ID:		<87r1l6jf4l.fsf@paluero>
Archive-link:		Article

Hello there,

I would like to announce a new service that I have just configured for
Debian: https://debuginfod.debian.net.

debuginfod is a new-ish project whose purpose is to serve
ELF/DWARF/source-code information over HTTP.  It is developed under the
elfutils umbrella.  You can find more information about it here:

  https://sourceware.org/elfutils/Debuginfod.html

In a nutshell, by using a debuginfod service you will not need to
install debuginfo (a.k.a. dbgsym) files anymore; the symbols will be
served to GDB (or any other debuginfo consumer that supports debuginfod)
over the network.  Ultimately, this makes the debugging experience much
smoother (I myself never remember the full URL of our debuginfo
repository when I need it).

If you would like to use the service, and if the service supports the
Debian distribution you are using (see below), all you have to do is
make sure that the following environment variable is set in your shell:

  DEBUGINFOD_URLS="https://debuginfod.debian.net"

Currently, the elfutils and GDB packages in unstable and testing have
native support for using debuginfod.  I will soon propose a change to
the elfutils package in order to make it be configured with our
debuginfod instance by default, so that users will be able to use the
service transparently.

For now, debuginfod.debian.net is serving debug information symbols for
the following Debian distributions:

  - unstable

  - testing
  - testing-proposed-updates

  - stable
  - stable-backports
  - proposed-updates

In the near future I intend to expand this list and include the
debuginfo stored at snapshot.debian.org as well.

Setting up a debuginfod service for Debian has been on my TODO list for
some time now, and I finally got enough time & resources to do it.  I
would like to thank a few people for their feedback and help:

  - Héctor Orón (zumbi)
  - Jonathan Carter (highvoltage)
  - Paul Wise (pabs)

Last, but not least, you can find a wiki page about our service here:

  https://wiki.debian.org/Debuginfod

Thanks,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
https://sergiodj.net/

A new Debian debuginfod service

Posted Feb 24, 2021 18:14 UTC (Wed) by dullfire (guest, #111432) [Link] (8 responses)

With how "stable" I have seen gdb's parsers to be lately, I suspect this a code execution opportunity :/

A new Debian debuginfod service

Posted Feb 25, 2021 19:02 UTC (Thu) by lkundrak (subscriber, #43452) [Link] (7 responses)

Would this be a problem really? Shipping code that you end up executing is a large part of what Linux distributions do already.

Assuming the debugging symbols are signed in the same way as the packages are, shipping illicit debugging symbols achieves the same thing as directly shipping a backdoored package, only in a more complicated manner.

Also, Debian, much like Fedora, already ships the debugging symbols in separate packages. This merely optimizes the delivery process a bit.

A new Debian debuginfod service

Posted Feb 26, 2021 5:22 UTC (Fri) by k8to (guest, #15413) [Link] (5 responses)

I believe the suggestion is that if can pretend to be a debug info system on the network in can remote code execute on developer systems. That's a fairly different attack vector than "I put code into Debian".

A new Debian debuginfod service

Posted Feb 26, 2021 10:26 UTC (Fri) by cortana (subscriber, #24596) [Link] (4 responses)

I hope gdb correctly verifies the TLS server certificate against the hostname and against the system trust store (of course taking into account all the fiddly details such as either supporting every critical extension found in the certificate chain or aborting the connection...)

A new Debian debuginfod service

Posted Feb 26, 2021 12:25 UTC (Fri) by mjw (subscriber, #16740) [Link] (2 responses)

gdb relies on libdebuginfod which in turn relies on libcurl for https connections. libcurl performs peer SSL certificate verification by default. So it relies on the way the distro has setup how libcurl uses the CA store.

A new Debian debuginfod service

Posted Feb 26, 2021 16:57 UTC (Fri) by cortana (subscriber, #24596) [Link]

Hah, I didn't expect someone to check, thanks for that :)

I hope this will allay k8to's worries!

A new Debian debuginfod service

Posted Feb 26, 2021 16:59 UTC (Fri) by cortana (subscriber, #24596) [Link]

... that said - it seems CURLOPT_SSL_VERIFYSTATUS still defaults to 0, so clients have to opt in to revocation checking via OCSP (and whether revocation checking via CRL is even possible or practical, I have no idea).

A new Debian debuginfod service

Posted Feb 26, 2021 17:06 UTC (Fri) by floppus (guest, #137245) [Link]

Trusting a TLS certificate also requires trusting a lot more entities (debuginfo.debian.net operators, their ISPs and domain registrars, hundreds of CAs).

In contrast, when you install 'libc6-dbg' with apt, you're verifying the package against a list that is signed by a known, trusted key. You're not trusting the mirror, their ISP, your ISP, or any other third parties.

I don't know exactly how build-ids are generated, but apparently they're a SHA-1 hash of the original binary contents and some of the metadata. Setting aside the fact that SHA-1 looks rather weak nowadays, I wonder if it'd be possible to use this to verify the debug info before trying to parse it.

Or if that wouldn't work, perhaps 'strip' could be adapted to embed a strong hash of the debug info in the stripped binary.

A new Debian debuginfod service

Posted Feb 26, 2021 17:11 UTC (Fri) by floppus (guest, #137245) [Link]

> Assuming the debugging symbols are signed in the same way as the packages are

They aren't; that's the problem.

A new Debian debuginfod service

Posted Feb 24, 2021 22:38 UTC (Wed) by roc (subscriber, #30627) [Link] (5 responses)

Does this server offer symbols for all versions of all packages ever published for each distribution? Or only for the latest version of each package on each distribution? Because if the latter it's considerably less useful...

A new Debian debuginfod service

Posted Feb 25, 2021 4:32 UTC (Thu) by pabs (subscriber, #43278) [Link] (4 responses)

The announcement says it is planned to expand it to all the debug packages on snapshot.d.o later.

A new Debian debuginfod service

Posted Feb 25, 2021 6:14 UTC (Thu) by wahern (subscriber, #37304) [Link]

IIUC, conceptually all it's doing is querying the server by BuildID, which is an identifier embedded into all object files. dh_strip ensures this happens when splitting debug symbols during normal dpkg builds, but I think BuildID is injected into all object files as a matter of course by the compiler and/or linker, and this has been the case for awhile. No matter how you strip objects and create debuginfo objects, I think you actually have to put in some effort to remove the BuildID. (In fact, the existing setup for locally preinstalled symbols may already rely on BuildID. I still haven't completely wrapped my head around how gdb looks for external debug symbols.)

If I'm correct, increasing coverage is merely a matter of indexing BuildID for the relevant repositories. Everything should already be in place. If debug packages are available via normal Apt repositories, then indexing them is more a matter of legwork--i.e. including those mirrors and subdirectories in the scan. And if the packages aren't currently available (i.e. really old distribution), then the situation remains unchanged.

A new Debian debuginfod service

Posted Feb 25, 2021 12:26 UTC (Thu) by roc (subscriber, #30627) [Link] (2 responses)

Ah I see, that would be excellent. If all distros supported this it would be even more excellent.

A new Debian debuginfod service

Posted Feb 26, 2021 12:29 UTC (Fri) by mjw (subscriber, #16740) [Link] (1 responses)

Various distros do or are experimenting with having a debuginfod server.
See the public servers list at https://sourceware.org/elfutils/Debuginfod.html

You can use the https://debuginfod.elfutils.org/ server which federates debuginfo packages from those trusted servers.

A new Debian debuginfod service

Posted Feb 26, 2021 22:45 UTC (Fri) by roc (subscriber, #30627) [Link]

For Pernosco we built a home-made thing that scrapes debuginfo packages daily for various Ubuntu releases. Would be great to be able to debuginfod instead, but we need a complete archive of all Ubuntu debuginfo packages ever released.

A new Debian debuginfod service

Posted Feb 25, 2021 1:30 UTC (Thu) by tux3 (subscriber, #101245) [Link]

I hadn't been able to make elfutil's debuginfod feature work till now (not having tried very hard), but I'll be a happy user if this turns out to work reasonably smoothly. Thankee!

A new Debian debuginfod service

Posted Feb 25, 2021 21:52 UTC (Thu) by scientes (guest, #83068) [Link] (1 responses)

The protocol is really lame, the server uses a thread for each request and is susceptible to DDOS and is as scalable as apache 1.0, and it really should have just used HTTP/1.1.... sigh....

A new Debian debuginfod service

Posted Feb 25, 2021 21:56 UTC (Thu) by scientes (guest, #83068) [Link]

Ok sorry for not checking again. Everything looks OK.