LWN: Comments on "Digging into Julia's package system"

Private repos

Lawless-M — Wed, 24 Nov 2021 08:15:35 +0000

One thing not mentioned is the ability to install via Private repos

You can use any Git Clone method available to you, for instance I pull from my company's privately hosted BitBucket repo.

@v.16] add ssh://git@mirror.bitbucket.intranet:7999/~myname/code.git

You can also use file paths - even windows ones!

@v1.6] add \\windowsUNC\packages\code.jl

With Distributed you can even install packages on remote machines

julia> @everywhere Pkg.add("ssh://git@mirror.bitbucket.intranet:7999/~myname/code.git")

Digging into Julia's package system

StefanKarpinski — Thu, 28 Oct 2021 19:31:51 +0000

The Julia client by default fetches all packages from a package server at https://pkg.juilalang.org, which implements a simple HTTP protocol for serving content-addressed immutable tarballs of package code. These source tarballs are stored persistently in S3 and replicated across a global network of servers that keep them forever. So even if GitHub were to disappear off the face of the earth, every registered package version would continue to be available to install via package servers. So no, Julia cannot get left-padded.

Digging into Julia's package system

garrison — Sat, 23 Oct 2021 22:18:14 +0000

Sorry, my link to the "motivation" should have been to https://github.com/JuliaPackaging/BinaryBuilder.jl#philos...

Digging into Julia's package system

garrison — Sat, 23 Oct 2021 22:12:17 +0000

One thing that might interest LWN readers is the great effort the Julia community has put into making packages that Just Work, even if they depend on code built in another language. Long ago, when a user would install a package, Julia would detect which Linux distribution it is running on, then run the suitable sudo apt-get install command (or equivalent) so that its dependencies would be available. This turned out to be brittle -- too much subtle breakage here and there, not to mention the requirement of root access.

Because of these issues, Julia 1.3 introduced an artifacts system that allows binary dependencies to be cross-compiled and distributed. The Yggdrasil repository contains recipes for cross compiling these binary dependencies -- the procedure is much like packaging a library for any other Linux distribution. While I was somewhat skeptical when this approach was introduced (their motivation sounds like a classic case of NIH to me), I must admit that it works remarkably well in practice. Years later, I rely on this system daily and it works flawlessly.

Digging into Julia's package system

roc — Sun, 17 Oct 2021 07:26:02 +0000

crates.io packages are stored in S3 and cached locally. S3 isn't really going to go down for technical reasons. Hopefully someone has a copy of the archive in case those S3 resources get deleted.

Digging into Julia's package system

droundy — Fri, 15 Oct 2021 02:04:51 +0000

I wish these Julia articles would give more comparison to other languages. Rather than trying to describe in detail how things are done in Julia, it would be much easier to read if it explained how it was similar and different from other languages. Python and Rust come to mind as systems with packaging systems that would provide productive comparisons.

Digging into Julia's package system

leephillips — Thu, 14 Oct 2021 12:56:16 +0000

I’m afraid I don’t understand your comment. I didn’t say “switch to a new version” anywhere.

Digging into Julia's package system

Wol — Thu, 14 Oct 2021 12:32:59 +0000

> I'm surprised Julia didn't do what Rust did -- there packages in the "package repository" are stored centrally, and unless there is a very serious issue released packages can never be removed.

What do you mean by "centrally". If you mean "on the net somewhere", what happens if that (for various meanings of "that") go down?

Or is that repository mirrored (should you so choose) on your machine, so you can ALWAYS re-install that package if you need? iiuc gentoo downloads everything, and while I've deliberately configured my system to forget it, I think it's easy enough to change that so it keeps it ...

Cheers,
Wol

Digging into Julia's package system

azumanga — Thu, 14 Oct 2021 11:14:30 +0000

To be honest, that sounds as bad as javascript!

Saying every package which has some dependency could "switch to a new version" doesn't feel helpful, you could do that in npm too if you like. I'm not really clear why it would be easier for Julia than it would be for Javascript.

I'm surprised Julia didn't do what Rust did -- there packages in the "package repository" are stored centrally, and unless there is a very serious issue released packages can never be removed. You can disable versions (by 'yanking' them), but users can still get those versions by specifying exact version number.

Digging into Julia's package system

leephillips — Wed, 13 Oct 2021 17:14:48 +0000

After reading Nathan Willis’ article about this incident at https://lwn.net/Articles/681410/, I have a few more observations.

Although Julia package development happens almost entirely on GitHub, the process is more decentralized than with npm, as contributors maintain their own forks as part of the GitHub pull request workflow. So one executive of a company deciding to pull down a package would not be so simple.

Every version of every package is identified by a unique UUID within the Manifest and Project files (an implementation detail I did not go into in the article). So switching a dependency on a particular version of a package means changing this identifier in the Manifests of the affected packages. It seems this part of the problem is much more tractable than the situation on npm.

Finally, it’s far less likely that a Julia programmer would create a dependency on a package that does what you can do in one line in Julia. I haven’t come across any public packages that are as trivial as leftpad.

Digging into Julia's package system

leephillips — Wed, 13 Oct 2021 16:47:41 +0000

The incident involves the sole maintainer of a package removing it from a repository. In that case Julia will continue to use the version stored on your machine, and when it checks for updates it won’t find anything. Someone in possession of the source, which would be anyone who had `dev`ed it, could recreate the GitHub repository, if the license allowed it. If it were a package with other contributors, there would be other forks of the project on GitHub, and the registry could be changed to point to one of these, I suppose.

Digging into Julia's package system

willy — Wed, 13 Oct 2021 16:35:22 +0000

Could you touch on how Julia's packaging system and/or development community prevents an incident like leftpad?