DNF 3: better performance and a move to C++
It has only been a few years since DNF replaced Yum as the default Fedora package-management tool; that was done for Fedora 22 in 2015, though DNF had been available for several earlier Fedora releases. Since that time, DNF development has proceeded; it started a move from Python/C to all C in 2016 and has made multiple releases over the years. From an outsider's perspective, no major changes seem necessary, which makes the announcement of DNF 3, and a move to C++, a bit surprising to some.
For many years, Yum was the package-management front-end for many RPM-based distributions, including Fedora and RHEL. But it suffered from poor performance and high memory use; part of that was attributed to its iterative dependency solver. DNF was meant to fix those problems. It uses the libsolv dependency resolution library developed by openSUSE, by way of the hawkey library.
Though it wasn't a perfect drop-in replacement for Yum, DNF did replace it. But, even though DNF performed better, often much better, than its predecessor, the project continued to focus on making it faster. Ultimately, that's a large part of the reasons behind DNF 3.
Since the switch to DNF, it has steadily improved performance-wise, while maintaining the features and commands that users have been accustomed to since way back in the Yum days (just switching from "yum" to "dnf" on the command line). Under the covers, though, things were shifting around: hawkey got subsumed into libhif, which turned into libdnf. It is libdnf that is the main focus of the upcoming changes; DNF itself is envisioned as a small wrapper around the library.
In the announcement on the DNF blog, the focus on performance is evident.
There is a graph comparing the in-development version of libdnf with the
versions that shipped in Fedora 26 and 27. In all five of the
queries tested, the new code is faster; in some cases massively so (from 77
seconds to 2.5 and 40 to 2.4, for example). Beyond better performance, the
DNF 3 effort is also targeting consistent behavior between various
package-management components (between DNF and PackageKit, in particular),
a better defined and maintained API, and compatibility ("as much as
possible
") with the existing C and Python APIs. For a number of
reasons, the project believes that switching to C++ will help it achieve
those goals.
DNF project leader Daniel Mach posted a note about the announcement to the Fedora devel mailing list on March 22. It will probably not come as a surprise that some of the reaction was about the language choice. Matěj Cepl said:
Using Rust would require LLVM, though; that's a step too far for Neal Gompa:
Gompa was also concerned about the difficulty of programming in Rust versus
C++. Martin Kolman thinks C++ makes a
better choice, rather than simply following the most-recent trend: "it is
not bad to be a bit conservative when core
system components
are concerned
". Given that the project has already made its choice
and started development down that path, C++ is basically a fait
accompli at this point.
Marcin Juszkiewicz raised the issue of sharing repository metadata between users, which is something of a longtime sore point for some. If a regular user runs DNF to query about the availability or version of some package, DNF will go off and download the latest metadata into /tmp. To some, that's a big waste of time, to others, who might not have a speedy internet connection, it is more or less intolerable. Since there is already a copy of the metadata available (root's version), it should be used by everyone, he said.
The downside of that idea, of course, is that the metadata in
/var/cache/dnf may be out of date; regular users should not be
able to update the system copy, so there are worries about giving outdated
responses to these queries. Tom Hughes asked if Juszkiewicz had a proposal on how to
share the metadata "in a secure way
". He also wondered what
should happen if the metadata was out of date.
Juszkiewicz had a fairly
straightforward thought about how a regular
user would interact with the metadata: "Use. Not download. And inform
(like always) how old that data is.
" As he and others in the thread
pointed out, Debian's APT separates the metadata update from the query
functionality, while DNF (like Yum before it) combines the two; if a DNF
query finds
out-of-date metadata, it seeks to remedy that before proceeding. Several
thread participants seemed to prefer the APT scheme (or at least some way to
share read-only access to the metadata). It is not at all clear what, if
anything, will happen with that, but it is an annoyance,
at least, for some.
Some other thoughts and enhancements were batted around a bit, but the overall impression is one of muted approval of the idea. Most probably just hope that the changes don't really affect them at all, other than seeing faster query responses. That is probably the DNF developers' hope as well; package managers are the kinds of programs that can be mostly ignored—so long as they work.
Posted Mar 29, 2018 2:20 UTC (Thu)
by jhoblitt (subscriber, #77733)
[Link] (16 responses)
Posted Mar 29, 2018 3:22 UTC (Thu)
by rahulsundaram (subscriber, #21946)
[Link] (15 responses)
Posted Mar 29, 2018 15:15 UTC (Thu)
by jhoblitt (subscriber, #77733)
[Link] (3 responses)
My team has been moving towards using `alpine` for minimal containers but being musl based, it often doesn't play nice with 3rd party binaries. There is a `glibc` `alpine` variant I've been experimenting with.
Posted Mar 29, 2018 15:19 UTC (Thu)
by rahulsundaram (subscriber, #21946)
[Link]
Posted Mar 29, 2018 16:26 UTC (Thu)
by zdzichu (subscriber, #17118)
[Link] (1 responses)
Posted Mar 29, 2018 17:05 UTC (Thu)
by jhoblitt (subscriber, #77733)
[Link]
registry.fedoraproject.org/fedora-minimal:latest -- 115MB
Posted Apr 3, 2018 7:58 UTC (Tue)
by dmach (guest, #123441)
[Link] (10 responses)
Sooner or later, microdnf will gain most of the DNF features through libdnf.
Posted Apr 4, 2018 17:47 UTC (Wed)
by hkario (subscriber, #94864)
[Link] (1 responses)
also, I don't think removing support for python plugins is a good idea: ease of writing them lowers the barrier to entry and tools like Gimp are only better for support of them
Posted Apr 6, 2018 6:16 UTC (Fri)
by dmach (guest, #123441)
[Link]
About the plugins support - we do *not* have any plans to remove them from Python DNF.
Posted Apr 4, 2018 18:04 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link] (7 responses)
Posted Apr 4, 2018 18:16 UTC (Wed)
by jhoblitt (subscriber, #77733)
[Link] (6 responses)
Posted Apr 4, 2018 20:07 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link] (5 responses)
And some plugins (like langpack) should just be provided as core functionality.
Posted Apr 4, 2018 20:31 UTC (Wed)
by jhoblitt (subscriber, #77733)
[Link] (4 responses)
I gather that your satisfied with wrapper scripts like `apt-fast` and similar but it is arguably a poor experience for inexperienced end users. It also places more demands on project maintainers.
Posted Apr 4, 2018 21:13 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link] (3 responses)
> Why does a kernel need loadable modules?
> I gather that your satisfied with wrapper scripts like `apt-fast` and similar but it is arguably a poor experience for inexperienced end users. It also places more demands on project maintainers.
Package retrieval is basically a solved problem, the set of plugins is stable for ages. Integrate them all into the core and forget about it.
Posted Apr 4, 2018 22:46 UTC (Wed)
by rahulsundaram (subscriber, #21946)
[Link] (2 responses)
Not quite. There are some which are stable but there is plenty of experimentation in this area of packaging. A recent example is the modularity project implemented via a plugin. There are also things that apply in specific circumstances (snapper) or only one distro among the many which use dnf now (subscription-manager plugin). Several of the plugins have been integrated into core when it made sense (delta rpms in yum used to be handled via a plugin originally) but not all of them fit that way.
Posted Apr 4, 2018 23:03 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
Posted Apr 5, 2018 0:17 UTC (Thu)
by rahulsundaram (subscriber, #21946)
[Link]
That is painful compared to plugins and it only covers one of the use cases I was talking about
Posted Mar 29, 2018 3:28 UTC (Thu)
by marcH (subscriber, #57642)
[Link] (11 responses)
I had a quick look at the original thread and couldn't find any elaboration. Anyone with any clue about what the concerns are? I could imagine something like "portability matters" but it sounds tied to LLVM more specifically?
Posted Mar 29, 2018 5:16 UTC (Thu)
by edomaur (subscriber, #14520)
[Link]
Posted Mar 29, 2018 6:07 UTC (Thu)
by Lionel_Debroux (subscriber, #30014)
[Link] (3 responses)
Indeed, LLVM has both newer design and more permissive licensing, which enable the framework to get more, not less, popular over time. The Google and Apple ecosystems, which carry huge weight, have both switched to LLVM - arguably for licensing reasons, but they wouldn't have done so if it hadn't become a decent, and even good, option... Even MS packages Clang for non-Windows in modern versions of VS. Features like CFI aren't available in GCC, and GCC's sanitizers are a step below upstream's sanitizers, AFAIK.
That's not to say that the LLVM family is perfect. For a start, it's another fact that LLVM supports fewer ISAs than GCC does, but equally obviously, it supports all of the ones which really matter to the main traditional Linux distros. For these distros (and core components thereof), resisting the use of LLVM, for fuzzy / unspecified reasons, probably isn't a wise choice for the mid- and long-term... perhaps especially so for Fedora, which aims at being on the leading / bleeding edge ("First").
Posted Mar 29, 2018 6:14 UTC (Thu)
by magfr (subscriber, #16052)
[Link] (1 responses)
Posted Mar 29, 2018 8:16 UTC (Thu)
by josh (subscriber, #17465)
[Link]
Posted Mar 29, 2018 23:53 UTC (Thu)
by lsl (subscriber, #86508)
[Link]
I'd start with the level of care and attention received by the GCC and LLVM packages in Fedora. GCC developers are very active about tracking down issues that tend to occur when a new GCC release hits Fedora and the huge package collection gets rebuilt with it. This kind of symbiotic relationship with the upstream currently doesn't exist for LLVM so it's probably not a good idea to rely on it for super-critical distribution infrastructure until this has changed. Without someone suitably-motivated driving this, it won't happen though.
Posted Mar 29, 2018 12:00 UTC (Thu)
by epa (subscriber, #39769)
[Link] (5 responses)
Posted Mar 29, 2018 13:17 UTC (Thu)
by iq-0 (subscriber, #36655)
[Link] (4 responses)
Rust being build on LLVM is just an implementation detail and Rust doesn't consider LLVM to be the only backend for native code generation, even though it's likely to be the main one for the foreseeable future.
Posted Mar 29, 2018 21:28 UTC (Thu)
by epa (subscriber, #39769)
[Link] (3 responses)
That's not a restriction of Rust-the-language, but as Rust-the-programming-environment it certainly is. Some future version of Rust might work with GCC, just as some future Fedora release might build the kernel and everything else with LLVM. But right now, using Rust would mean adding a dependency on LLVM, and hence two separate compilers in the base system.
Or are you saying that LLVM is just an implementation choice, and it would be quite possible to program DNF in Rust without introducing a dependency on LLVM? If so, how?
Posted Mar 30, 2018 6:50 UTC (Fri)
by iq-0 (subscriber, #36655)
[Link] (2 responses)
No you understood correctly.
You stated that "would make LLVM a hard requirement to build the core system" as their possible motivation. But that is just backwards. LLVM is just a dependency of Rust the language. So either it's problematic of adding yet another language to the core system build requirements, which is a perfectly valid reason for them to express, or you state that you won't support some language because of some specific reason.
But saying it's "because of LLVM" is like stating that you won't use a programming language because it depends on tcmalloc instead of glibc malloc or because it uses yacc instead of bison.
> Or are you saying that LLVM is just an implementation choice
Yes that's what I'm saying.
> and it would be quite possible to program DNF in Rust without introducing a dependency on LLVM? If so, how?
And that is not what I meant.
Rust can create fully standalone binaries just fine, so there should not be a runtime dependency on LLVM.
Rust only introduces LLVM as a build dependency and even in that case someone created 'mrustc' which IIRC acts as a transpiler to C and could even be used to break that hard dependency for obscure platforms. I read somewhere that the Debian Rust maintainers were considering that for some of the more obscure platforms that don't have native LLVM support.
Posted Mar 30, 2018 13:10 UTC (Fri)
by khim (subscriber, #9252)
[Link] (1 responses)
If programming language depends on custom allocator and does not play well with glibc one then it's enough reason to avoid it. C# and Java come to mind.
mrustc sounds an interesting aolution for that problem, though. How well is it supported in Rust community?
Posted Apr 5, 2018 8:31 UTC (Thu)
by HelloWorld (guest, #56129)
[Link]
Posted Mar 29, 2018 3:37 UTC (Thu)
by marcH (subscriber, #57642)
[Link] (27 responses)
I keep hearing that C++ *can* be used in a safe, non-C way that can avoid corruption, crashes and security issues just like modern languages, it's "just the matter of sticking to the right subset", correct? Which of course almost never happens because there's not even a compiler flag to tell you whenever you leave that safe subset...
Would a team used to C be able to stick to such a safe C++ subset? Are they even interested in such an effort?
Posted Mar 29, 2018 4:16 UTC (Thu)
by roc (subscriber, #30627)
[Link] (7 responses)
Oh well. Perhaps safety isn't a big concern for dnf because it doesn't run with privileges and/or doesn't process untrusted input...
Posted Mar 29, 2018 22:19 UTC (Thu)
by k8to (guest, #15413)
[Link] (5 responses)
Posted Mar 29, 2018 22:26 UTC (Thu)
by roc (subscriber, #30627)
[Link]
Posted Mar 30, 2018 6:22 UTC (Fri)
by epa (subscriber, #39769)
[Link] (3 responses)
Posted Mar 30, 2018 6:40 UTC (Fri)
by karkhaz (subscriber, #99844)
[Link] (2 responses)
Slightly tangential, but I always thought it would have been good to have a replica of the Filesystem Hierarchy Standard in each user's home directory (or perhaps in a dotfile under the home directory), i.e. every user gets ~/.usr/bin, ~/.usr/lib, ~/.usr/var, etc. I have no idea why the XDG standard used weird names when there are already perfectly standard names; they put several things under ~/.local (bin, lib, share), but there's also the ~/.config directory which could perfectly well be called ~/.local/etc, and ~/.cache could be ~/.local/var/cache or something.
Having users' home directories be that uniform might make distro-supported user-installable packages a lot easier implementation-wise, too!
Posted Apr 1, 2018 17:38 UTC (Sun)
by epa (subscriber, #39769)
[Link]
Posted Apr 5, 2018 8:40 UTC (Thu)
by HelloWorld (guest, #56129)
[Link]
Posted Apr 5, 2018 7:39 UTC (Thu)
by HelloWorld (guest, #56129)
[Link]
Posted Mar 29, 2018 5:05 UTC (Thu)
by ncm (guest, #165)
[Link] (2 responses)
But you still have integer overflow, in C++ (as in Rust) to watch out for. And be sure to constrain your array and vector indices.
The main thing is, have fun! If coding in modern C++ isn't fun, you are doing it wrong.
Posted Mar 29, 2018 5:37 UTC (Thu)
by roc (subscriber, #30627)
[Link] (1 responses)
Sticking to "modern C++" does not necessarily avoid safety issues. Sometimes it makes them worse. See https://github.com/isocpp/CppCoreGuidelines/issues/1038 for example.
Posted Mar 29, 2018 22:21 UTC (Thu)
by k8to (guest, #15413)
[Link]
Certainly Rust having a sane default that is well-known is a significant improvement.
Posted Mar 29, 2018 6:54 UTC (Thu)
by rvfh (guest, #31018)
[Link] (5 responses)
Posted Mar 29, 2018 8:15 UTC (Thu)
by andresfreund (subscriber, #69562)
[Link] (1 responses)
Posted Mar 29, 2018 8:51 UTC (Thu)
by rvfh (guest, #31018)
[Link]
Posted Mar 29, 2018 11:58 UTC (Thu)
by roc (subscriber, #30627)
[Link] (2 responses)
Using shared pointers a lot can be really slow because of all the atomic operations you have to perform, even if you're traversing what should be a read-only data structure.
Using RAII to ensure mutex locks are paired with unlocks is a step up from raw pthread_mutex_lock. But how do you ensure that the right mutexes are held while you access the data they protect? Rust ensures this. C++ frameworks typically don't.
Posted Mar 29, 2018 15:13 UTC (Thu)
by kpfleming (subscriber, #23250)
[Link] (1 responses)
Posted Mar 29, 2018 21:22 UTC (Thu)
by roc (subscriber, #30627)
[Link]
Posted Mar 29, 2018 12:06 UTC (Thu)
by epa (subscriber, #39769)
[Link] (2 responses)
They say 'the rules are designed to be supported by an analysis tool' but I don't know whether there is currently such a tool.
Posted Mar 29, 2018 21:26 UTC (Thu)
by roc (subscriber, #30627)
[Link] (1 responses)
Posted Mar 29, 2018 21:29 UTC (Thu)
by roc (subscriber, #30627)
[Link]
Posted Mar 29, 2018 14:24 UTC (Thu)
by iq-0 (subscriber, #36655)
[Link] (3 responses)
For the most part: yes. Wrapping your data structures inside safe C++ often makes them more ergonomic and leads to less and cleaner code. The incentive is pretty strong.
> I keep hearing that C++ *can* be used in a safe, non-C way that can avoid corruption, crashes and security issues just like modern languages
This one is a bit tougher as there are 3 separate statements:
Basic corruption avoidance is greatly improved when using unique and shared smart pointers in C++. But these features only help reduce common occurrences of corruption through manual memory management. They won't fix all use after free bugs through references or guard against corruption through data races when using multi threading. Garbage collection will help guard against the first and stricter data management or alternative threading models help guard against the second.
Avoiding crashes is often tied to avoiding corruption. But a significant part of avoiding crashes is also in reducing possible API abuse. Stricter type systems will often help reduce this even further. The fact that C++ makes using C so easy also keeps many parts from still depending on C APIs, which will not help guard against basic unintended API abuse.
Avoiding security issues goes a long way beyond basic corruption avoidance and crash reduction. Most security issues aren't really down to the programming language itself (though certain aspects around basic datastructures help, like eg. having collision resistent hash tables), but have much more to do with the ecosystem surrounding it: How do you handle dependencies? How mature and/or well maintained are your dependencies?
C++ is definitely a step up from C in these aspects, but most dynamic languages and more modern languages will be a lot better than C++ in these areas.
But there are a lot more aspects being considered in this case. From resource usage (bye bye most dynamic languages), runtime dependencies (Java, Mono) to good integration with the current C code base (hello C++ and Rust) to familiarity of the current maintainers (bye Rust?). From a purely technical standpoint I'd personally recommend Rust, but from a project standpoint I can see where the choice for C++ would come from. Maintainers and contributers are a projects most valuable resource, even more valuable than all the code that was, is or will be.
Posted Mar 29, 2018 19:15 UTC (Thu)
by marcH (subscriber, #57642)
[Link] (2 responses)
Makes sense, thx. There was also this good point on the list: "I think it is not bad to be a bit conservative when core system components are concerned." etc.
I'm surprised Go wasn't even mentioned. It seems very successful in the not too distant containers sphere which seems very sensitive to performance too. Does a package manager really need to micro-manage memory? https://blog.golang.org/go15gc
Posted Mar 30, 2018 7:12 UTC (Fri)
by iq-0 (subscriber, #36655)
[Link] (1 responses)
I expect that there are multiple reasons for not being considered. I guess some of these are:
And there are of course the philosophical differences, like to C++ gives you things like a rich type system, templating and exception based error handling. Go goes almost completely the other direction: simpeler type system, no templating, no metaprogramming and written out error handling. Both have their pros and cons.
But the project was probably looking for a way to incrementally make the current code base less cumbersome/easier. Switching to Go would not be the quickest way to get there.
Posted Apr 5, 2018 0:01 UTC (Thu)
by lsl (subscriber, #86508)
[Link]
Which ones? The gc compiler (the one you get from golang.org) supports AArch64, AMD64, ARM, MIPS, PPC64 (BE and LE), S390x and i386 in its main development branch. There are further (incomplete) ports available that aren't merged upstream yet (SPARC, RISC-V).
That should cover all machines supported by Fedora (and then some). Maybe you looked at the binary distributions? Those are only provided for a subset. All arches are required to have CI builders running the testsuite on every commit, though. Results are available from https://build.golang.org.
Posted Mar 29, 2018 14:35 UTC (Thu)
by cry_regarder (subscriber, #50545)
[Link] (2 responses)
Posted Mar 29, 2018 14:45 UTC (Thu)
by marcH (subscriber, #57642)
[Link] (1 responses)
Posted Mar 29, 2018 22:36 UTC (Thu)
by k8to (guest, #15413)
[Link]
Posted Mar 29, 2018 6:56 UTC (Thu)
by meyert (subscriber, #32097)
[Link] (13 responses)
Posted Mar 29, 2018 9:15 UTC (Thu)
by muep (subscriber, #86754)
[Link]
Posted Mar 29, 2018 16:36 UTC (Thu)
by antiphase (subscriber, #111993)
[Link] (10 responses)
Posted Mar 29, 2018 17:01 UTC (Thu)
by rahulsundaram (subscriber, #21946)
[Link] (9 responses)
Moving from C to C++ can and does fix a number of bugs, improve security and performance. Many projects including say GCC has done it for similar reasons.
> think how much benefit people will see from being able to install more packages per second.
People do benefit from faster installations and upgrades.
Posted Mar 31, 2018 0:21 UTC (Sat)
by Twirrim (subscriber, #122505)
[Link] (1 responses)
I'd be really curious where shaving a few seconds off on package upgrades or installations is that advantageous. Yes speeding up yum is a nicer user experience, but this seems to be well into the territory of diminishing returns.
Posted Mar 31, 2018 2:16 UTC (Sat)
by rahulsundaram (subscriber, #21946)
[Link]
Two major places atleast, initial installation and full release upgrades. Anaconda, the Fedora installer doesn't support upgrades anymore. It is all handled through the dnf system upgrade plugin and for each of these use cases, the performance differences are substantial.
There are plenty of other places including package builds in the buildsystem or QA test systems which are done in a clean chroot, where a few seconds can add up over time.
Posted Mar 31, 2018 17:41 UTC (Sat)
by HenrikH (subscriber, #31152)
[Link] (6 responses)
Posted Mar 31, 2018 18:07 UTC (Sat)
by rahulsundaram (subscriber, #21946)
[Link] (4 responses)
It certainly can but I think you are mixing up two things. Read the announcement linked in the article. Dnf 3 includes both performance work and a move from C to C++. They are not necessarily related.
Posted Mar 31, 2018 18:23 UTC (Sat)
by rahulsundaram (subscriber, #21946)
[Link] (2 responses)
http://benchmarksgame.alioth.debian.org/u64q/cpp.html
It is not uniformally better but a lot of C programs end up duplicating parts of C++ poorly. A move to C++ can help in such circumstances.
Posted Apr 1, 2018 17:05 UTC (Sun)
by HenrikH (subscriber, #31152)
[Link] (1 responses)
Posted Apr 2, 2018 13:24 UTC (Mon)
by ncm (guest, #165)
[Link]
That power includes application of algorithms that are too finicky to code for just one use; performance tuning, likewise; and data structures, again. C has inlining now, but C++ is able to make much more effective use of it, and the optimizers understand that use, so can squeeze out what used to be called "abstraction overhead" (when there usually was some).
Furthermore: because it is easier to work with values directly in C++, rather than by reference, the optimizer has more to work with, not bothered by potential aliasing.
Posted Apr 1, 2018 16:58 UTC (Sun)
by HenrikH (subscriber, #31152)
[Link]
Posted Apr 7, 2018 9:57 UTC (Sat)
by HelloWorld (guest, #56129)
[Link]
There's also a more subtle reason: C++ allows you to abstract over things that C doesn't, hence allowing better code re-use. And you're more likely to use good algorithms and data structures if somebody else built them generically and put them into an easy-to-use library than if you have to build them yourself.
Posted Apr 3, 2018 10:19 UTC (Tue)
by jarmar (guest, #103679)
[Link]
Posted Mar 29, 2018 18:13 UTC (Thu)
by clugstj (subscriber, #4020)
[Link] (30 responses)
So the age of a language automatically makes it a poor choice?
Posted Mar 29, 2018 19:04 UTC (Thu)
by marcH (subscriber, #57642)
[Link]
Yes _when_ said age goes back to the pre-internet era when safety, security and reliability were very distant concerns if at all.
Of course C++ has continued to evolve since, but:
Posted Mar 29, 2018 21:33 UTC (Thu)
by roc (subscriber, #30627)
[Link]
Of course C++ has evolved to absorb some of those lessons too, modulo backward compatibility constraints. But using 33-year-old C++ would be truly daft.
Posted Mar 30, 2018 19:41 UTC (Fri)
by flussence (guest, #85566)
[Link] (27 responses)
From where I'm standing, ${insufferable trendy web2.0 flavour of the month} has 33 years of tooling to catch up on. (Hint: start with distcc and ccache.)
Posted Apr 3, 2018 20:12 UTC (Tue)
by cpitrat (subscriber, #116459)
[Link] (6 responses)
Posted Apr 4, 2018 1:27 UTC (Wed)
by zlynx (guest, #2285)
[Link] (3 responses)
If you use precompiled headers and -O0 on GCC it is nearly as fast at compiling as Go is. GCC is pretty slow with -flto and -O3 but Go can't do that at all.
Posted Apr 4, 2018 7:15 UTC (Wed)
by cpitrat (subscriber, #116459)
[Link] (2 responses)
Posted Apr 5, 2018 15:34 UTC (Thu)
by zlynx (guest, #2285)
[Link] (1 responses)
> real 0m7.059s
Here is github.com/coreos/etcd with go build -a -v. 1,620 Go files, 538,484 lines of code.
> real 0m11.980s
Both of those were done on a 4c/8t CPU with 8 threads.
So yes, Go builds faster. But not by a whole lot. And a build time of "minutes" is not realistic.
Posted Apr 5, 2018 17:10 UTC (Thu)
by excors (subscriber, #95769)
[Link]
That's certainly a manageable problem (you can design libraries in ways that don't involve enormous amounts of inline template code), but it takes a bit of knowledge and discipline to make sure you avoid dependencies that significantly hurt your project's compile times.
Posted Apr 4, 2018 21:21 UTC (Wed)
by flussence (guest, #85566)
[Link] (1 responses)
Go has its own coping mechanisms as a consequence - usually involving multiple nested containment OSes per app.
Posted Apr 5, 2018 20:51 UTC (Thu)
by lsl (subscriber, #86508)
[Link]
Go, of course, does not have that. It's a popular misconception.
On the language level, the strings used with the import statement are just that: opaque strings that identify a package but are not assigned any further meaning. For namespacing reasons it makes sense to somehow coordinate who owns which package names. Instead of coming up with with some Go-specific name registry, the convention is to just piggy-back on the DNS.
The build tooling uses these strings for finding source code within your file system. So if you import "a/b/c" that means to use the module located at $GOPATH/src/a/b/c. This convention is basically doing the same job that is done by pkg-config and arrangement of files within /usr/include and /usr/lib in the C world. If you really want to, you can ignore the build system and just call the compiler and linker yourself, passing the appropriate files and command line flags (-L, etc.).
Neither of those (compiler/build system) will retrieve anything from the network or otherwise execute arbitrary code. The build system has a source code generation step (for calling out to things like yacc), but you need to invoke it explicitly (go generate), it's not invoked during normal compilation.
The only thing that does talk to the network is the 'go get' tool. It's for developers, to quickly check out a source repo in the right place. It just uses your local Git program (or Mercurial, ...) under the covers.
Do you see why that "unauthenticated-curl-pipe-sh" analogy doesn't make much sense? The only way that Go tools talk to the network is by invoking git. So unless you have a habit of cloning arbitrary git repos and piping their contents into a shell ...
Posted Apr 7, 2018 15:08 UTC (Sat)
by HelloWorld (guest, #56129)
[Link] (19 responses)
Anybody who thinks that tooling is a good reason to use C++ has never seen what good tooling looks like.
Posted Apr 7, 2018 15:30 UTC (Sat)
by halla (subscriber, #14185)
[Link] (18 responses)
CMake is leagues better than autotools, of course, and Meson or QBS or Scons or Waf or whatever "alternative" system is popular with people who cannot use CMake because KDE decided to use it in 2006, have about a decade of catch-up and acquiring complications to do before it can contend.
And if they survive that long, I guess people will call it "insane crap" as well. Every real-world build system is complicated.
Posted Apr 8, 2018 11:00 UTC (Sun)
by HelloWorld (guest, #56129)
[Link] (5 responses)
Now, all this might be justifiable (although not really) if the end result was actually reasonably easy to use. So what does some simple task, like building a simple program with one library dependency, look like with CMake? Something like this:
Posted Apr 8, 2018 20:29 UTC (Sun)
by halla (subscriber, #14185)
[Link] (4 responses)
No, it isn't.
Posted Apr 8, 2018 20:35 UTC (Sun)
by HelloWorld (guest, #56129)
[Link]
Posted Apr 9, 2018 2:18 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
CMake is crap. It's a slightly better crap than most other C++ build systems but it's still crap compared to modern build systems for other languages.
Posted Apr 9, 2018 2:37 UTC (Mon)
by pizza (subscriber, #46)
[Link] (1 responses)
Different tools for different purposes; is that really so hard to comprehend?
Posted Apr 9, 2018 12:10 UTC (Mon)
by HelloWorld (guest, #56129)
[Link]
> Different tools for different purposes
Posted Apr 8, 2018 11:14 UTC (Sun)
by HelloWorld (guest, #56129)
[Link] (11 responses)
Posted Apr 8, 2018 12:07 UTC (Sun)
by pizza (subscriber, #46)
[Link] (10 responses)
s/solve problems/use capabilities/
Pretending otherwise is just willful ignorance.
Posted Apr 8, 2018 14:26 UTC (Sun)
by HelloWorld (guest, #56129)
[Link] (9 responses)
Posted Apr 8, 2018 18:14 UTC (Sun)
by pizza (subscriber, #46)
[Link] (2 responses)
The real world is messy.
Posted Apr 8, 2018 19:34 UTC (Sun)
by HelloWorld (guest, #56129)
[Link]
Posted Apr 8, 2018 19:35 UTC (Sun)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Golang compiler is written in Go with runtime at more than 99% in Go.
Heck, even Java these days is planning to use Java-based runtime.
Posted Apr 10, 2018 3:54 UTC (Tue)
by linuxrocks123 (subscriber, #34648)
[Link] (5 responses)
https://github.com/linuxrocks123/bake
I wasn't impressed with CMake when I used it.
Posted Apr 10, 2018 5:06 UTC (Tue)
by karkhaz (subscriber, #99844)
[Link] (4 responses)
The difference with my version is that I haven't written my own build tool---my "baker.py" always emits ninja syntax, rather than your custom syntax that needs its own build tool. So running baker.py is akin to running `configure`, and I then separately run ninja to build. You get parallel compilation and many other goodies for free with ninja.
If you don't like cmake, there's no need to write an entire build+metabuild system---cmake is just the meta-build aspect. As for builds, I don't think the fact that ninja is wonderful is controversial, so you should just emit ninja from your meta-build system and use that to build.
I haven't published it because it's ugly and needs a rewrite but hopefully the idea is clear w.r.t. how your system works.
Posted Apr 10, 2018 10:45 UTC (Tue)
by HelloWorld (guest, #56129)
[Link] (2 responses)
Posted Apr 10, 2018 11:00 UTC (Tue)
by karkhaz (subscriber, #99844)
[Link] (1 responses)
A big reason why ninja is so much faster than make is that it doesn't evaluate environment variables or have any kind of fancy logic (there's no support for all of the functions and expansions that make has). All of the decision-making should be done at configure time, there's no need to repeat it every time you build.
2. Multiple build directories, each with a different build configuration. With out-of-tree builds, I can have one build directory for release builds, one for debug, one for building only a sub-project, etc. You typically configure each of these directories once, e.g. by passing parameters or environment variables to the meta-build system; and then the build command is just `ninja` in each one. If the meta-build and build commands are combined, then you need to pass the parameters/variables _every single time_ you run a build, and need to remember which parameters are associated with each directory.
Posted Apr 10, 2018 12:29 UTC (Tue)
by HelloWorld (guest, #56129)
[Link]
As for your other argument, I don't buy that either. Release and debug builds should simply be different targets, and the build system should give you sufficient means of abstraction so that the common parts can be easily shared between the two.
Posted Oct 24, 2018 9:09 UTC (Wed)
by linuxrocks123 (subscriber, #34648)
[Link]
The two systems are somewhat different, probably most notably in that a Bakefile calls its clients to generate the dependency tree, and will give a previous client's output as input to a later client for modification. Of course, I haven't actually used that for anything. Also the -sub feature looks unique, though I haven't actually written a baker.py that uses it yet, so perhaps it's harder to make use of than I suspected. No harm in keeping it around, though.
(Also yeah it's months later; sorry for not seeing this at the time.)
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
docker.io/fedora:27 -- 235MB
DNF 3: better performance and a move to C++
Unfortunately it sometimes behaves differently than DNF.
That's why we're currently consolidating the whole software management stack.
What's not clear yet is how to deal with plugins, which are written in Python.
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
Sometimes it's better to report an error than ignore it and exit with 0.
That could make a false assumption that everything works perfectly.
On the other hand, it would be great to support plugins directly in libdnf,
in order to make them available to all tools based on the library.
We can't link libdnf with Python to keep the footprint and deps small.
Once the software management stack consolidation is finished,
we'll definitely look into this. Now it's early to say how the solution will look like.
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
Absolutely.
By the same token, the kernel doesn't have pluggable CPU schedulers (and there was a huge flame war about that, just ask Con Kolivas).
Why should there be "apt-fast"?!? The default apt should already be fast without any plugins.
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
Could you explain why?
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
This might be dnf 101, but I'm not finding much about dnf having privilege separation on the interwebs.
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
Actually Bjarne Stroustrup and Herb Sutter are specifying that right now, the effort goes by the name of C++ Core Guidelines. It can be found on GitHub.
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
In the same way, C++ always to not lock/unlock a mutex, but take an object that locks at construction and unlocks at destruction.
This allows for rather safe (as in: no deadlock, no memory leak, no dangling pointers) code already!
And of course there is much more...
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
- avoid corruption
- avoid crashes
- avoid security issues
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
- The languages are very different. They'd have to fully rewrite entire chunks instead of a more gradual migration.
- The use of garbage collection might be a reason why they switched away from Python. A good garbage collector often exhibits a higher baseline RSS, which might be a problem on resource constrained systems.
- Go <-> C interop isn't considered to be very efficient. So, depending on which parts they'd want to rewrite first, there is a good chance that performance will suffer before becoming better.
- Upstream Go has (very) limited platform support. Gccgo can be used to work around this, but that would mean a workaround for a number of primary supported platforms.
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
And not the quality of the tool and the plugins. E.g. I did hit bugs with dnf system upgrade for so many different releases!
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
- it doesn't really seem to provide the tools to *enforce* its users to evolve with it
- also based on comments above it doesn't seem it went as far as newer languages.
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
> user 0m14.177s
> sys 0m2.159s
> user 0m53.516s
> sys 0m4.843s
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
Seriously? For the longest part of its history, the tooling for the C and C++ ecosystem has been complete and utter garbage. People were (and are) using insane crap like autotools and CMake until something like Meson came along. And it took ages until decent editors (with code completion, jump to definition, refactoring etc.) came along, and they were clearly inspired by other languages.
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
Just saying "cmake is insane crap" doesn't make it insane crap, and it is, of course nothing of the kind
Oh, it absolutely is. Its custom programming language alone is enough of a reason not to use it. You can't even do the most basic thing a function is supposed to do: return a value, for crying out loud! Instead, what people resort to is assigning values to global variables whose name is derived from the function's parameters. Functions can't have named parameters either, so what you do is you pass certain “keywords” (for lack of a better word) interleaved with other function arguments and hope that the values you're passing don't clash with the “keywords” defined by the function. Needless to say, it doesn't have any meaningful programming features, like real lists, dictionaries or, heaven forbid, first class functions. The worst part about it is that they claim this to be for “simplicity”, yet NO OTHER LANGUAGE does things like that, making the whole thing feel weird and awkward to just about anybody who's had any exposure to programming.Every real-world build system is complicated.
Which is why you need a proper programming language, and not something insane like CMake.
project(hello-world C)
cmake_minimum_required(VERSION 2.6)
PkgConfig to detect GTK+ headers/library files
find_package(PkgConfig REQUIRED)
pkg_check_modules(GTK3 REQUIRED gtk+-3.0)
include_directories(${GTK3_INCLUDE_DIRS})
link_directories(${GTK3_LIBRARY_DIRS})
add_definitions(${GTK3_CFLAGS_OTHER})
add_executable(hello main.c)
target_link_libraries(hello ${GTK3_LIBRARIES})
This is garbage, why am I using a build system if I end up having to configure include directories, linker flags and locate source files and all that crap? It's the build system's job to do that! What does it look like in a language with at least somewhat sane tooling, like Scala and sbt?
libraryDependencies += "org.scalaz" %% "scalaz-core" % "7.2.21"
This is the complete build definition. Now you can keep pretending that CMake is somehow not insane and stupid. Meanwhile, I'll just sit here and laugh at you.
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
Since Cyberax didn't make a comparison, I guess you meant to respond to me.
Here's my question? How exactly does the purpose of CMake – building C and C++ programs – necessitate a custom programming language for your build system that has “functions” that can't return values, doesn't have any useful data structures and is generally weird and quirky?
The answer is it doesn't, hence “different purposes” don't justify anything here.
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
This is neither true (e. g. Go's compiler and runtime are implemented in Go and a little assembly) nor is it relevant.
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++
DNF 3: better performance and a move to C++