Distributions

News and Editorials

A look at package repository proxies

For simplicity's sake, I keep all of my general-purpose boxes running the same Linux distribution. That minimizes conflicts when sharing applications and data, but every substantial upgrade means downloading the same packages multiple times — taking a toll on bandwidth. I used to use apt-proxy to intelligently cache downloaded packages for all the machines to share, but there are alternatives: apt-cacher, apt-cacher-ng, and approx, as well as options available for RPM-based distributions. This article will take a look at some of these tools.

The generic way

Since Apt and RPM use HTTP to move data, it is possible to speed up multiple updates simply by using a caching Web proxy like Squid. A transparent proxy sitting between your LAN clients and the Internet requires no changes to the client machines; otherwise you must configure Apt and RPM to use the proxy, just as you must configure your Web browser to redirect its requests. In each case, a simple change in the appropriate configuration file is all that is required: /etc/apt/apt.conf.d/70debconf or /etc/rpmrc, for example.

Although straightforward, this technique has its drawbacks. First, a Web proxy will not recognize that two copies of a package retrieved from different URLs are identical, undermining the process for RPM-based distributions like Fedora, where the Yum update tool incorporates built-in mirroring.

Secondly, using the same cache for packages and all other HTTP traffic risks overflowing the cache. Very large upgrades — such as changing releases rather than individual package updates — can fill up the cache used by the proxy, and downloaded packages can get pushed out of the way by web traffic if your LAN upgrade process takes too much time. It is better to keep software updates and general web traffic separate.

Apt-proxy versus apt-cacher

The grand-daddy of the Apt caching proxies is apt-proxy. The current revision is written in Python and uses the Twisted framework. Complaints about apt-proxy's speed, memory usage, and stability spawned the creation of apt-cacher, a Perl-and-cURL based replacement that can run either as a stand-alone daemon or as a CGI script on a web server. Both operate by running as a service and accepting incoming Apt connections from client machines on a high-numbered TCP port: 9999 for apt-proxy, 3142 for apt-cacher.

Apt-proxy is configured in the file /etc/apt-proxy/apt-proxy-v2.conf. In this file, one sets up a section for each Apt repository that will be accessed by any of the machines using the proxy service. The syntax requires assigning a unique alias to each section along with listing one or more URLs for each repository. On each client machine, one must change the repository information in /etc/apt/sources.list, altering each line to point to the apt-proxy server and the appropriate section alias that was assigned in /etc/apt-proxy/apt-proxy-v2.conf.

For example, consider an apt-proxy server running on 192.168.1.100. If the original repository line in a client's sources.list is:

    deb http://archive.ubuntu.com/ubuntu/ intrepid main

It would instead need to read:

    deb http://192.168.1.100:9999/ubuntubackend intrepid main

The new URL points to the apt-proxy server on 192.168.1.100, port 9999, and to the section configured with the alias ubuntubackend. The apt-proxy-v2.conf file would contain an entry such as:

    [ubuntubackend]
    backends = http://archive.ubuntu.com/ubuntu/

If you find that syntax confusing, you are not alone. Apt-proxy requires detailed configuration on both the server and client sides: it forces you to invent aliases for all existing repositories, and to edit every repository line in every client's sources.list.

Apt-cacher is notably simpler in its configuration. Although there are a swath of options available in apt-cacher's server configuration file /etc/apt-cacher/apt-cacher.conf, the server does not need to know about all of the upstream Apt repositories that clients will access. Configuring the clients is enough to establish a working proxy. On the client side, there are two options: either rewrite the URLs of the repositories in each client's sources.list, or activate Apt's existing proxying in /etc/apt/apt.conf. But choose one or the other; you cannot do both.

To rewrite entries in sources.list, one merely prepends the address of the apt-cacher server to the URL. So

    deb http://archive.ubuntu.com/ubuntu/ intrepid main

becomes:

    deb http://192.168.1.100:3142/archive.ubuntu.com/ubuntu/ intrepid main

Alternatively, leave the sources.list untouched, and edit apt.conf, inserting the line:

    Acquire::http::Proxy "http://192.168.1.100:3142/";

Ease of configuration aside, the two tools are approximately equal under basic LAN conditions. Apt-cacher does offer more options for advanced usage, including restricting access to specific hosts, logging, rate-limiting, and cache maintenance. Both tools allow importing existing packages from a local Apt cache into the cache shared by all machines.

Much of the criticism of the tools observed on mailing lists or web forums revolves around failure modes, for example whether Twisted or cURL is more reliable as a network layer. But there are telling discussions from experienced users of both that highlight differences you would rather not experience firsthand.

For example, this discussion includes a description of how apt-proxy's simplistic cache maintenance can lose a cached package: If two clients download different versions of the same package, the earlier downloads will expire from the cache because apt-proxy does not realize that keeping both versions is desirable. If you routinely test unstable packages on one but not all of your boxes, such a scenario could bite you.

Other tools for Apt

Although apt-proxy and apt-cacher get the most attention, they are not the only options.

Approx is intended as a replacement for apt-proxy, written in Objective Caml and placing an emphasis on simplicity. Like apt-proxy, client-side configuration involves rewriting the repositories in sources.list. The server side configuration is simpler, however. Each repository is re-mapped to a single alias, with one entry per line.

Apt-cacher-ng is designed to serve as a drop-in replacement for apt-cacher, with the added benefits of multi-threading and HTTP pipelining lending it better speed. The server runs on the same TCP port, 3142, so transitioning from apt-cacher to apt-cacher-ng requires no changes on the client side. The server-side configuration is different, in that the configuration can be split into multiple external files and incorporate complicated remapping rules.

Apt-cacher-ng does not presently provide manpage documentation, supplying instead a 14-page PDF. Command-line fans may find that disconcerting. Neither application has supplanted the original utility it was designed to replace, but both are relatively recent projects. If apt-proxy or apt-cacher don't do the job for you, perhaps approx or apt-cacher-ng will.

Tools for RPM

The situation for RPM users is less rosy. Of course, as any packaging maven will tell you, RPM and Apt are not proper equivalents. Apt is the high-level tool for managing Debian packages with dpkg. A proper analog on RPM-based systems would be Yum. Unfortunately, the Yum universe does not yet have dedicated caching proxy packages like those prevalent for Apt. It is not because no one is interested; searching for the appropriate terms digs up threads at Linux Users' Group mailing lists, distribution web forums, and general purpose Linux help sites.

One can, of course, use Apt to manage an RPM-based system, but in most cases the RPM-based distributions assume that you will use some other tool designed for RPM from the ground up. In such a case, configuring Apt is likely to be a task left to the individual user, as opposed to a pre-configured Yum setup.

Most of the proposed workarounds for Yum involve some variation of the general-purpose HTTP proxy solution described above, using Squid or http-replicator. If you take this road, it is possible to avoid some of the pitfalls of lumping RPM and general web traffic into one cache by using the HTTP proxy only for package updates. Just make sure that plenty of space has been allocated for the cache.

Alternatively, setting up a local mirror of the entire remote repository, either with a tool such as mrepo, or piecemeal is possible. The local repository can then serve all of the clients on the LAN. Note, however, that this method will maintain a mirror of the entire remote repository, not just the packages that you download, and that you will have to update the machine hosting the mirror itself in the old-fashioned manner.

Finally, for the daring, one other interesting discussion proposes faking a caching proxy by configuring each machine to use the same Yum cache, shared via NFS. Caveat emptor.

I ultimately went with apt-cacher for this round of upgrades, on the basis of its simpler configuration and its widespread deployment elsewhere. Thus far, I have no complaints; the initial update went smoothly — Ubuntu boxes moving from 8.04 to 8.10, for the curious. The machines are now all in sync; time will tell whether or not additional package updates will reveal additional problems in the coming months. It's a good thing there are alternatives.

Comments (31 posted)

New Releases

Debian 5.0 released

It's official: Debian GNU/Linux 5.0 ("Lenny") is available; lots of details can be found in the release notes. For embedded developers, there are updated versions of Emdebian Grip (which is binary-compatible with regular Debian) and Emdebian Crush (which is not). And, for Debian developers who have been held back by the release freeze: it's now open season in -unstable as the "Squeeze" development cycle begins.

Comments (34 posted)

DebXO 0.5 release

DebXO is a customized version of Debian Lenny (5.0) customized for XO hardware. Click below for more information.

Full Story (comments: none)

Mandriva Linux 2009 Spring beta released

A beta version of Mandriva Linux 2009.1 has been announced. "The beta release of Mandriva Linux 2009 Spring (code name Margaux) is now available. This beta version provides some updates on major desktop components of the distribution, including KDE 4.2.0, GNOME 2.25.90, Xfce 4.6 RC1, X.org server 1.5, OpenOffice.Org 3.0.1, qt 4.5.0 (RC1)." See also the release notes and errata.

Comments (none posted)

Fedora Unity Announces Fedora 10 Re-spins

The Fedora Unity Project has announced the release of new ISO Re-Spins of Fedora 10. "These Re-Spin ISOs are based on the officially released Fedora 10 installation media and include all updates released as of February 10th, 2009."

Full Story (comments: none)

DragonFly Release 2.2

DragonFly BSD Release 2.2 is out. See the release notes for more information. "The HAMMER filesystem is considered production-ready in this release; It was first released in July 2008. The 2.2 release represents major stability improvements across the board, new drivers, much better pkgsrc support and integration, and a brand new release infrastructure with multiple target options." DragonFly BSD was forked from FreeBSD 4.8 in June of 2003.

Comments (none posted)

BackTrack 4 Beta Released

The Remote Exploit Development Team has announced the release of BackTrack 4 Beta. "The most significant of these changes is our expansion from the realm of a Pentesting LiveCD towards a full blown "Distribution". Now based on Debian core packages and utilizing the Ubuntu software repositories, BackTrack 4 can be upgraded in case of update. When syncing with our BackTrack repositories, you will regularly get security tool updates soon after they are released."

Full Story (comments: none)

Arch Linux 2009.02 ISO Release

Arch Linux has announced the release of version 2009.02. "It took us quite a while, but we think the result is worth it: we added some cool new things and ironed out some long-lasting imperfections."

Comments (none posted)

Distribution News

Debian GNU/Linux

Kudos to translators

The Debian Project congratulates the translation team for their work on Debian 5.0 (lenny). "Lenny comes with an installer translated in 63 languages (5 more than Etch), which also means that 150 millions additional people can install Debian in their language."

Full Story (comments: none)

Fedora

FUDCon Berlin 2009

The Fedora Project has announced the date and location of the next Fedora Users and Developers Conference (FUDCon); Berlin, Germany from June 26 - 28, 2009. "LinuxTag takes place from Wednesday June 24 - Saturday June 27, and FUDCon will take place from Friday the 26th - Sunday the 28th, meaning that there will be two days of overlap during which a large number of people who would otherwise never attend a FUDCon will have a chance to see Fedora up close, and in great detail."

Full Story (comments: none)

FUDCons and FADs

Fedora Project Leader Paul Frields blogs about Fedora events. "In the past, besides the notable trade and community shows, Fedora also started, years ago, the Fedora Users and Developers Conference, or FUDCon for short. FUDCon is not just an event that allows Fedora community members to get together, interact, share ideas, and produce results — it’s Red Hat’s gift back to the community, a way of thanking people for their help with the Fedora Project. Typically a FUDCon is several days long, and includes self-organizing hackfests around key project areas or goals, and a BarCamp-style day of technical sessions where contributors deliver talks in an informal and participation-oriented environment."

Comments (none posted)

Fedora Board Recap 2009-02-10

The Fedora Board Recap for February 10, 2009 covers Follow-up To Previous Business, Max Spevack Update, Fedora Store in EMEA, Denoting Fedora sponsorship, Community Architecture & LXDE In Fedora, FUDCon LATAM 2009 and FUDCon EMEA 2009 plans and Future Business.

Full Story (comments: none)

Gentoo Linux

Council meeting summary for 12 February 2009

The Gentoo council met February 12, 2009. Topics discussed included the need for a dedicated secretary, elections, technical issues, open bugs and non-technical issues.

Full Story (comments: none)

Distribution Newsletters

Arch Linux Newsletter for February, 2009

The Arch Linux Newsletter for February is out. "Welcome to another issue of the Arch Linux Newsletter. This newsletter covers from December to February. Not too much content, just the relevant. We have an excellent interview with Jan de Groot, the GNOME packager. This month feature many Arch in the media articles, proving the success of Arch Linux in general. Also we have a all the common sections updated with new information for your reading pleasure."

Comments (none posted)

DistroWatch Weekly, Issue 290

The DistroWatch Weekly for February 16, 2009 is out. "Without a shadow of a doubt, the biggest story of the past week was the release of Debian GNU/Linux 5.0 'Lenny'. After nearly two years of continuous development and a controversial vote or two, we finally get the chance to take a quick look at the finished product - the new live media as well as the 'netinst' network installation CD. In other news, Ubuntu announces that Jaunty will ship with Linux kernel 2.6.28, Wiley publishes OpenSolaris Bible and makes three sample chapters available for free download, openSUSE's Zypper gains Bash-completion improvements, Red Hat publishes a 'State of the Union' address, the Woof project releases version 0.0.0 with support for Arch Linux, and Cuba develops their own Gentoo-based variant distribution called Nova. Also in this issue are links to two interviews - the first with Steve MacIntyre, the head of the Debian project, and the second with Scott Ritchie, an Ubuntu community developer."

Comments (none posted)

Fedora Weekly News #163

The Fedora Weekly News for February 15, 2009 is out. "This week's issue provides some detail on the upcoming Fedora Activity Day (FAD) at Southern California Linux Expo (SCaLE), many posts from the Fedora Planet blogosphere, and selected wonderful event reports from FOSDEM. We welcome a brand new Quality Assurance beat this issue, with coverage of the latest test day focusing on iSCSI for Fedora 11, summary of the latest QA weekly meeting, and discussion of the process for critical-release bugs. In Development news, discussion of FLOSS multimedia codec support in Fedora, preview looks at F11 release notes, and the availability of CrossReport, a tool to evaluate the ease with which applications can be ported to Windows using the MinGW libraries." Plus several other topics.

Full Story (comments: none)

The Mint Newsletter - issue 75

The Linux Mint Newsletter is out. "The forum has moved to Canada - works much better now. For the time being activation and notification mail are not sent, unknown why. The XFCE Community Edition (CE) is almost ready for its final release. The KDE and Fluxbox CEs are almost ready for their first RC to be released" And much more.

Comments (none posted)

openSUSE Weekly Newsletter FOSDEM 2009 issue

openSUSE has released a special issue of its newsletter to cover FOSDEM 2009. Topics include Before FODSEM, FOSDEM LIVE, Media, Review - After FOSDEM 2009, and Credits.

Comments (none posted)

OpenSUSE Weekly News/59

This issue of the OpenSUSE Weekly News covers the special FOSDEM2009 edition, OpenOffice_org 3.0.1 final available, Jan-Christoph Bornschlegel: Product Creation with the openSUSE Build Service, Henne Vogelsang: Fosdem talk about collaboration features and Contrib, kamilsok: installing 64bit Java on openSUSE 11.1, and much more.

Comments (none posted)

Ubuntu Weekly Newsletter #129

The Ubuntu Weekly Newsletter for February 14, 2009 covers: Ubuntu LoCo Teams Meeting, New MOTU's, Rockin' LoCo Docs Day, Ubuntu Hug Day, Improved mail server stack: Testing needed, Drupal 5.x and 6.x LoCo Suite Released, Ubuntu Honduras being organized, Launchpod #17, Triage in Launchpad suite, PPA page perfomance improvements, Ubuntu Training for USA, HP Mini Mi Screenshots, Server Team Meeting Feb. 10th, and much more.

Full Story (comments: none)

Newsletters and articles of interest

Top 5 Netbook Linux Distributions (Internetling)

Internetling looks at the top 5 Linux netbook distributions. "Some of the advantages of running Linux on a sub-notebook are a smaller memory footprint, better security and tons of free applications right out of the box. If you decide to install it by yourself, you may encounter some compatibility problems here and there, therefore it is wiser to buy one of the more widely-sold netbooks such as the Eee PC or the Acer Aspire One."

Comments (none posted)

Cuba launches Gentoo Linux distro (DesktopLinux)

DesktopLinux covers the launch of Gentoo-based "Nova," Cuba's new distribution. "Developed by the Universidad de las Ciencias Informáticas (UCI), Nova was launched at the recent International Conference on Communication and Technologies, says the story in Reuters. Despite ongoing trade embargoes from the U.S., and what would seem to be a natural fit between open source technology and socialism, Cuba is still primarily a Microsoft Windows-centric country, according to the story. Yet, the government has come to believe that Windows could be a threat because it believes U.S. security agencies have access to Microsoft codes, says the story. Plus, the trade embargoes make it difficult to get legal, supported copies of the software for regular updates, says Reuters."

Comments (none posted)

Debian 5's Five Best Features (CompterWorld)

Steven J. Vaughan-Nichols shares his five favorite features of the newly released Debian 5.0 "Lenny". "1) X.org 7.3 integration. It used to be setting up your screen in Linux was a real pain-in-the-rump. With X.org 7.3 the X-server behind Linux's most common GUIs (graphical user interfaces), the program automatically take care of setting up your display resolution."

Comments (none posted)

Page editor: Rebecca Sobol
Next page: Development>>