Update in the background [LWN.net]

Update in the background

Posted Oct 12, 2005 1:46 UTC (Wed) by rvfh (subscriber, #31018)
Parent article: Review: Ubuntu 5.10 Breezy Badger (Linux.com)

I personally use my own script to upgrade my Kubuntu: a script creates a list of requested packages, posts it to my server.
Then my (Slackware) server detects that there is work to do and downloads the stuff for me (same script, different command).
When the script runs again on the client, it finds that some packages are there and adds them to the apt cache (I check the MD5SUM everytime to be safe). The apt-get dist-upgrade is still up to me to do when I ready, but then does not take long!

It supports multiple 'clients', of course, the idea being to maximise the cache of all clients.
I am still working on the 'cleaning' part, and I also want to be able to use the same system to install new packages.
Ah, and I want my clients to share whatever packages they have locally too. Pfew!

If that looks interesting to somebody, give me a shout!

(Log in to post comments)

Update in the background

Posted Oct 12, 2005 14:09 UTC (Wed) by pj (subscriber, #4506) [Link]

I do something similar with the -d (download-only) flag to apt-get. Supporting multiple clients and such can I think be done by using apt-move to move packages from your apt cache to a local mirror (which is what your script is essentially making, right?)

Update in the background

Posted Oct 12, 2005 14:34 UTC (Wed) by magi (subscriber, #4051) [Link]

just use apt-proxy

Update in the background

Posted Oct 12, 2005 17:21 UTC (Wed) by rvfh (subscriber, #31018) [Link]

I should have thought the problem had been fixed already, as always in GNU/Linux! It is a bit frustrating though.

But thanks a lot for the info: I shall start reading about those two commands.

Update in the background

Posted Oct 12, 2005 19:23 UTC (Wed) by szoth (subscriber, #14825) [Link]

I got good results by tweaking a squid configuration so that it would agressivly cache large files, then configuring apt to use that squid as its http proxy.

Update in the background

Posted Oct 13, 2005 4:07 UTC (Thu) by csamuel (subscriber, #2624) [Link]

Don't suppose you'd care to share that bit of the config for those of us
who are only average with Squid please ? :-)

Update in the background

Posted Oct 13, 2005 15:00 UTC (Thu) by maney (subscriber, #12630) [Link]

Set maximum_object_size large enough that it will cache, oh, the biggest chunk of OOo, and restart squid. Works well enough for my home/office setup that I've never gotten around to addng Ubuntu sources to the existing apt-cache setup (which I'm now using with only a single Sarge server box... well, and the other one that I'm fiddling with). You can get this working right from the start with a kernel option to the installer along the lines of http_proxy=http://cache.two14.net:3128/, though IIRC you'll need to add that to your user environment or add a parallel setting in /etc/apt/apt.conf for post-install use. AndOfCourse change the cache's address and port as needed for local conditions in both.

As best I recall (can't easily check it from here), the home/office's squid has a max size setting of 50 or 60 MB, and there's been no visible detriment to normal caching of web pages. It's got about 1G of disk space allocated, which might be a bit on the short side if you like to install all the desktops and everything you think you might ever need (viz., a Red Hat style install) you might want to allow more cache space...

Squid as a cache for APT

Posted Oct 13, 2005 17:43 UTC (Thu) by szoth (subscriber, #14825) [Link]

Set maximum object size in squid.conf to the size of the largest dpkg you wish to cache.

http://wiki.squid-cache.org/config/maximum_object_size

In my setup I had two gigs of RAM, so I also increased maximum object size
in memory.

http://wiki.squid-cache.org/config/maximum_object_size_in...

Then you probably want to use the LFUDA cache replacement policy to get a better byte hit rate.

http://squid.visolve.com/squid/squid24s1/cache_size.htm#c...

There's also a policy decision for memory replacement; I used LFUDA here too because of my large supply of RAM and willingness to suffer higher latency for some small files in return for better overall download speed.

http://squid.visolve.com/squid/squid24s1/cache_size.htm#m...

On your clients put a line like this in /etc/apt/apt.conf

Acquire::http::Proxy "http://proxy.mydomain.com:8080";

apt.conf usually doesn't exist yet, so you will have to create it, with just that line in it. Without editing apt.conf you can also use the http_proxy environment variable before running apt-get, e.g.:

export http_proxy="http://proxy.mydomain.com:8080"
apt-get update