LWN: Comments on "A systemd status update"

I/O scheduler performance

epa — Wed, 08 Sep 2010 11:15:42 +0000

So, then, the way to get fast I/O is to make asynchronous I/O calls from a single thread (so that the scheduler knows that fairness doesn't matter) rather than spawning multiple threads or processes.

Is there any way to fork subprocesses but still let CFQ know that they're all related and happy to altruistically share I/O bandwidth between them, so it doesn't try to slice up I/O requests fairly at the expense of total throughput?

A systemd status update

nix — Fri, 03 Sep 2010 17:08:13 +0000

init replacements: the new text editors?

A systemd status update

ssam — Sun, 29 Aug 2010 15:53:17 +0000

I should be fairly easy to generate the sysv script from any of the native formats. you would just drop things like the binary name into a template.

this is basically how a sysv script is written in the first place.

I/O scheduler performance: not good enough!

renox — Sun, 29 Aug 2010 10:38:02 +0000

Thanks, I should have seen this but the long URL was shortened which hid the issue.
I'll try again.

I/O scheduler performance: not good enough!

etienne — Thu, 26 Aug 2010 13:33:05 +0000

Sorry, said to compile showmap.c and gave the link to showmbr.c, the good link is:
http://www.mirrorservice.org/sites/download.sourceforge.n...

I/O scheduler performance: not good enough!

renox — Thu, 26 Aug 2010 12:48:49 +0000

Interesting, thanks.
It didn't work on my computer though:
>>
./showmap ./showmap
Boot record of size 512 bytes read successfully from './showmap'
First bytes 0x7F 0x45 0x4C, signature 0x804, FAT16 sig 0x0, FAT32 sig 0x4,
No FAT signature recognised, cannot analyse header.
Partition table: WindowsNTmarker 0x3C, Unknown 0x0, signature 0x804
0: indicator 0x0 i.e. non bootable, start 5570560 length 0
(i.e. end at 5570560); start 513/0/43 end 0/0/18 name 'empty'
1: indicator 0x0 i.e. non bootable, start 3014656 length 0
(i.e. end at 3014656); start 768/0/29 end 0/0/18 name 'empty'
2: indicator 0x0 i.e. non bootable, start 6750208 length 0
(i.e. end at 6750208); start 0/0/54 end 0/0/18 name 'empty'
3: indicator 0x0 i.e. non bootable, start 4587520 length 2438987776
(i.e. end at 2443575296); start 256/0/60 end 0/0/18 name 'empty'
<<
Probably a not recent enough version..

wrt upset

gvy — Wed, 25 Aug 2010 18:57:58 +0000

I think (and have seen others, too) that upstart is half-baked hack, and not exactly pretty one.

Hope Lennart's attitude towards humans will get back closer to ifplugd days not pulseaudio nights though...

I/O scheduler performance: not good enough!

etienne — Wed, 25 Aug 2010 11:28:56 +0000

Both FIEMAP and FIBMAP report block numbers, i.e. the position of the file on the device.
You have to guess yourself the position of the device on the hardware device, simple if they are the same of there is a simple partition table - difficult if there is LVM or MD(RAID) in between.
FIEMAP is a lot quicker than FIBMAP, noticeable when getting the mapping of ISO file images.
As an example:

$ wget http://www.mirrorservice.org/sites/download.sourceforge.n...
$ gcc -O2 showmap.c -o showmap
$ ./showmap ./showmap.c
File "./showmap.c" of size 15013 (32 blocks512) is on filesystem 0x302.
According to /proc/diskstats, file './showmap.c' is on device '/dev/hda2'
/dev/hda2: Permission denied
$ su
Password:
# ./showmap ./showmap.c
File "./showmap.c" of size 15013 (32 blocks512) is on filesystem 0x302.
According to /proc/diskstats, file './showmap.c' is on device '/dev/hda2'
Device block size: 512, FS block size: 4096, device size: 61432560 blocks
Device length: 31453470720 bytes
The device start at 61432560 sectors, C/H/S: 65535/16/63.
First FIEMAP says 1 extents
second FIEMAP success, 1 extents filled:
0: logical offset 0 (0 * 4096), physical offset 25904218112 (6324272 * 4096),
length 16384 (4 * 4096) flags 0x1
flags meaning: FIEMAP_EXTENT_LAST 0x1, FIEMAP_EXTENT_UNKNOWN 0x2,
FIEMAP_EXTENT_DELALLOC 0x4, FIEMAP_EXTENT_ENCODED 0x8,
FIEMAP_EXTENT_DATA_ENCRYPTED 0x0, FIEMAP_EXTENT_NOT_ALIGNED 0x100,
FIEMAP_EXTENT_DATA_INLINE 0x200, FIEMAP_EXTENT_DATA_TAIL 0x400,
FIEMAP_EXTENT_UNWRITTEN 0x800, FIEMAP_EXTENT_MERGED 0x1000.
FIGETBSZ: block size 4096 bytes
File (4 blocks of 4096) start at block 6324272 for 4 blocks,
last block 6324275 and file has 1 fragments.
FIBMAP succeeded after end of file, block index 4 give block 0
#

A systemd status update

mishad — Wed, 25 Aug 2010 06:48:58 +0000

And if this was done, would it affect how the boottime I/O is scheduled? (axboe suggested above that CFQ will try to maximise throughput when all requests come from same process, rather than attempting fairness which it does when requests are from separate processes.) Or are the little C tools discussed simply not responsible for much of the boottime I/O?

A systemd status update

marcH — Tue, 24 Aug 2010 22:43:46 +0000

> Ordering of the daemon startup is *not* necessary anymore, and the kernel CPU and IO scheduler will magically order the entire execution of us. And that's so amazing.

Yes, this looks awesome. Thanks for correcting me / reminding this.

I/O scheduler performance: not good enough!

mhelsley — Tue, 24 Aug 2010 21:53:07 +0000

I don't think FIEMAP provides block numbers. Block numbers would be unique on the partition/device whereas I think FIEMAP provides "extents" (and FIBMAP provides bits) which effectively describe offsets within the file -- not block numbers.

http://lwn.net/Articles/260803/

A systemd status update

njs — Tue, 24 Aug 2010 17:50:33 +0000

At least on Debian-derived systems, init scripts are marked as configuration files, so the package manager won't overwrite modified ones (and will ask the sysadmin what to do if there are both local and upstream changes).

A systemd status update

Frej — Tue, 24 Aug 2010 16:45:34 +0000

You are switching from awkard imperative langauge (shell) to a declarative language (.ini files).

In that sense you are not replacing anything with C programs, but shell programs with a very simple declarative program. In theory when systemd matures you have all the constructs you need to accomplish your job.

Also, I think a few of you are confused about lennart calling it C tools. It's not more C tool than bash is (it too is writting in C), it's just a more specialized tool that solves just one problem. It might be bad marketing, but lennart has a history of that ;)

A systemd status update

lethargo — Tue, 24 Aug 2010 16:39:11 +0000

Even if you can add sysvinit scripts of your own, it sounds like they plan to use C in place of most of Fedora's default service init scripts. As you noted, system administrators can still add their own init scripts.

However, what using C does NOT preserve (or what I worry about anyway) is the ability of a system administrator to see what the system is doing on boot, for all services, especially the default ones. I may not want to change them. However, as a system administrator one thing I like about Linux over Windows is the ability to look at exactly what is happening during boot and daemon startup scripts, instead of the more opaque Windows services control panel.

I would hate to have to install the C code (or find the correct version of it online) and read it to know the details of what is happening during boot.

Actually it kinda is...

khim — Tue, 24 Aug 2010 16:01:50 +0000

There might be good reasons to use C here, but the language itself is for sure not one of them.

Actually there are one good reason: it's language with compiler and it does not require massive support libraries. Why it's important to have language with compiler? Well, if you'll take a look above you'll see that everyone perceive "flexibility" of shell scripts as something really cool. Well, in rare cases where you must change something it's cool and I like the fact that systemd can use them. But in most cases dirty hacks in shell scripts are wrong way to fix the problem and they will regularly break something few years later (usually after system upgrade). Immutability of C solution is actually a good thing: it means it's easier to locate and neutralize shellscript hacks.

In a sense systemd is "back to the future" system: long ago /etc/init used /etc/inittab and managed processes directly. It was simple, robust but not very flexible system. Unfortunately when it was found that said system is not flexible enough instead of fixing it hacks were layered over hacks and we've got the mess called SysVInit. SystemD is just a reimplementation of venerable /etc/init on the new level - and it's awesome, like original.

A systemd status update

marcH — Tue, 24 Aug 2010 13:43:15 +0000

Then better insist on such progress rather than highlight the fact that they are implemented in the nearest language to assembly.

There might be good reasons to use C here, but the language itself is for sure not one of them.

A systemd status update

rahulsundaram — Tue, 24 Aug 2010 13:01:07 +0000

Contrived? That is a copy paste from my system. If you can present a case where you must absolutely must need a shell script, I am interested in hearing about that. 99% of it is boilerplate code and copy pasting the same stuff all over the place makes no sense.

A systemd status update

njd27 — Tue, 24 Aug 2010 12:06:55 +0000

Reducing complexity, coupling and duplication of code would be good things even if there isn't a huge performance improvement to be had.

A systemd status update

sorpigal — Tue, 24 Aug 2010 11:41:34 +0000

Your example is a bit contrived. Yes, it is possible to replace a complex shell script with something simpler. Does it follow that it is possible to replace *all* complex shell scripts with something simpler? Yes, an init script very rarely changes. Does it follow that init scripts *never* require hand-edited tweaks?

The value of init scripts is that I can know for sure what happens at init time because I can read the source and it's in a format every sysadmin is familiar with. I like a lot of things about systemd, but trying to kill shell scripts during init is not one of them. If performance gains are not really the goal, as someone said, and moving away from shell scripts didn't gain you much anyway, then why do it? It seems obviously detrimental to me, no matter what the benefits are.

A systemd status update

dgm — Tue, 24 Aug 2010 07:48:22 +0000

Of course. Ask Arjan van de Ven and he will tell you that the key to boot really fast is to boot less, not to boot in parallel.

A systemd status update

airlied — Tue, 24 Aug 2010 07:28:03 +0000

You realise most init scripts are shipped in packages, and hand editing them will make them get replaced at package upgrade time.

At least on Fedora we have /etc/sysconfig/ to add tunables etc.

Not sure how good a sysadmin is if they have to resort to hacking around in random scripts instead of providing a supportable solution, definitely not the type of sysadmin that I'd like to employ.

A systemd status update

aleXXX — Tue, 24 Aug 2010 06:53:46 +0000

> > We reimplemented almost all boot-up and shutdown scripts of the standard
> > Fedora install in much smaller, simpler and faster C utilities, or in
> > systemd itself.

Replacing script files with compiled C programs ?
In the age of multicore gigahertz CPUs we are going from high-level scripting language back to C to gain a few seconds while booting ?

What am I misunderstanding here ?

Alex

A systemd status update

aleXXX — Tue, 24 Aug 2010 06:51:10 +0000

Now this is a *really* bad idea.
It means that there is duplicated code (init scripts and service files), and usually only one path of them is used.

Just stating the obvious: the unused duplicated path (the init scripts) will be broken after a short time.

Alex

A systemd status update

Kamilion — Tue, 24 Aug 2010 06:04:19 +0000

Even upstart cleaned up a lot of things from raw shellscript:

# /etc/init/nginx.conf
# nginx - starts the nginx webserver
start on (net-device-up and local-filesystems)
stop on runlevel [016]

pre-start exec /usr/sbin/nginx -t

expect fork
respawn
exec /usr/sbin/nginx

# /etc/init/php-fastcgi.conf
# php-fastcgi - starts php-cgi as an external FASTCGI process
start on (net-device-up and local-filesystems)
stop on runlevel [!2345]

expect fork
respawn
exec /usr/bin/sudo -u www-data PHP_FCGI_CHILDREN=2 PHP_FCGI_MAX_REQUESTS=25 /usr/bin/php-cgi -q -b /tmp/php-fastcgi.socket

Pretty simple; How would this look as two systemd units?
(Feel free to specify better start/stop criteria)

A systemd status update

rvfh — Tue, 24 Aug 2010 05:53:26 +0000

I think you made your point ;)

A systemd status update

RogerOdle — Tue, 24 Aug 2010 05:25:40 +0000

It occurs to me that there is a significant time used spawning processes where these processes are for scripts or C programs. Can these C modules be linked into a single loadable module? I am not thinking about plug-ins that use run-time loading+linking but actually performing a static link of the modules to produce a fat module. This link process could be done as part of a system configuration. Parallel operation may be done by forking or with threads. How would the performance of this compare to the current design of systemd?

A systemd status update

rahulsundaram — Tue, 24 Aug 2010 01:00:48 +0000

Package maintainers already have written init scripts. They don't have to convert. If they want to take advantage of systemd native service files, they could write one as well but this is hardly going to take any time at all. Here is an actual example

NetworkManager.service file for systemd
----------------------------------------

[Unit]
Description=Network Manager
After=syslog.target

[Service]
Type=dbus
BusName=org.freedesktop.NetworkManager
ExecStart=/usr/sbin/NetworkManager --no-daemon

[Install]
WantedBy=network.target multi-user.target
Alias=dbus-org.freedesktop.NetworkManager.service

----

The equivalent sysvinit script is:
-----------------------------------

#!/bin/sh
#
# NetworkManager: NetworkManager daemon
#
# chkconfig: - 23 84
# description: This is a daemon for automatically switching network \
# connections to the best available connection.
#
# processname: NetworkManager
# pidfile: /var/run/NetworkManager/NetworkManager.pid
#
### BEGIN INIT INFO
# Provides: network_manager $network
# Required-Start: messagebus
# Required-Stop: messagebus
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: start and stop NetworkManager
# Description: NetworkManager is a tool for easily managing network connections
### END INIT INFO

prefix=/usr
exec_prefix=/usr
sbindir=/usr/sbin

NETWORKMANAGER_BIN=${sbindir}/NetworkManager

# Sanity checks.
[ -x $NETWORKMANAGER_BIN ] || exit 1

# Source function library.
. /etc/rc.d/init.d/functions

# Source network configuration
. /etc/sysconfig/network

# so we can rearrange this easily
processname=NetworkManager
servicename=NetworkManager
pidfile=/var/run/NetworkManager/NetworkManager.pid

RETVAL=0

start()
{
echo -n $"Setting network parameters... "
sysctl -e -p /etc/sysctl.conf >/dev/null 2>&1
success
echo

echo -n $"Starting NetworkManager daemon: "
daemon --pidfile $pidfile --check $servicename $processname --pid-file=$pidfile
RETVAL=$?
echo
if [ -n "${NETWORKWAIT}" ]; then
[ -z "${LINKDELAY}" ] && LINKDELAY=10
echo -n $"Waiting for network..."
nm-online -q --timeout=$LINKDELAY || nm-online -q -x --timeout=30
[ "$?" = "0" ] && success "network startup" || failure "network startup"
echo
[ -n "${NETWORKDELAY}" ] && /bin/sleep ${NETWORKDELAY}
fi
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/$servicename
}

stop()
{
echo -n $"Stopping NetworkManager daemon: "
killproc -p $pidfile $servicename
RETVAL=$?
echo
if [ $RETVAL -eq 0 ]; then
rm -f /var/lock/subsys/$servicename
rm -f $pidfile
fi
}

# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status -p $pidfile $processname
RETVAL=$?
;;
restart)
stop
start
;;
condrestart)
if [ -f /var/lock/subsys/$servicename ]; then
stop
start
fi
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart}"
;;
esac
exit $RETVAL

A systemd status update

nicooo — Tue, 24 Aug 2010 00:55:30 +0000

Just one, for whatever system their distribution uses to start daemons. It looks like for systemd it's only an ini file.

I/O scheduler performance: not good enough!

wmf — Tue, 24 Aug 2010 00:12:47 +0000

The FIEMAP ioctl does this, but in general apps probably shouldn't try to implement such optimizations; Linux isn't an exokernel.

I/O scheduler performance: not good enough!

renox — Mon, 23 Aug 2010 23:15:40 +0000

I remember being very impressed by a paper where userspace application can schedule their I/O by knowing (an approximation of) the block number for the file they use:
>>As an example, a tar of the Linux kernel tree was 82.5 seconds using GNU tar, while our modified tar completed in 17.9 seconds.<<

The paper is here:
http://simula.no/research/nd/publications/Simula.ND.399/s...

I wonder if there will be eventually a system call to know the block number of a file?

A systemd status update

jspaleta — Mon, 23 Aug 2010 23:14:31 +0000

No... four!
sysV style init
upstart native job definition
systemd native unit definition.
launchd native plist definition.

-jef

A systemd status update

mezcalero — Mon, 23 Aug 2010 23:10:18 +0000

Yes, we do see benefits already. Just not in all setups.

Note that systemd is *not* about on-demand activation. That's a misconception. While we do support that, at least in case of socket-based activation it will be used only for a minimal number of services. i.e. SSH and CUPS, but that's basically it. Implementing socket-based activation and centralizing bus-based activation in systemd enables us to do race-free parallelization. And that's what's is much more key in systemd's design: we can start providers and consumers of socket or bus services *at the same time*! Ordering of the daemon startup is *not* necessary anymore, and the kernel CPU and IO scheduler will magically order the entire execution of us. And that's so amazing.

It takes a while to wrap you head around it, but think about it and maybe reread my original posting again. But if you think systemd was about on-demand then you have not understood it. It's about parallelization.

In fact, doing things on-demand in many cases is recipe to make things slower: if it is clear that something is used anyway spawning it on-demand only means you delay its start-up to the latest moment possible, while it would be much smarter to start it is early as possible so that it is already ready to process requests when it is actually needed.

And we don't take the shell away from you. You can continue to use it in any way you want. We just rewrote a bunch of stuff that is complex in shell in simple C. And since it is simpler there's less reason to patch it later on.

A systemd status update

jond — Mon, 23 Aug 2010 23:09:00 +0000

Just for the sake of clarity, my objection to this is bitrot, which dlang has already covered.

A systemd status update

jond — Mon, 23 Aug 2010 23:07:58 +0000

So packagers and/or daemon authors have to write TWO startup scripts now?

A systemd status update

jond — Mon, 23 Aug 2010 23:05:42 +0000

> There's not doubt that systemd is the right approach

I don't know about that. I have doubts. I've seen others, too, who think that upstart has it right.

A systemd status update

nix — Mon, 23 Aug 2010 22:43:10 +0000

Quite so. I was scared by this at first, too, but the whole reason the systemd conversion of these scripts to C was so easy is that virtually all initscripts are 99% the same in any case. So an admin wanting a new script can just take an existing one, any existing one, and hack it a bit.

A systemd status update

rahulsundaram — Mon, 23 Aug 2010 22:25:37 +0000

Fedora 14 is scheduled to have have systemd available by default but the current recommendation is to just ship both or not bother converting at all yet. I have to note that once I have written the init scripts for the packages, I have almost never seen any need to update them. Therefore going by experience, the init scripts are already in a very much stagnant change. If anyone can point to init script configuration needs that are currently not addressable via systemd's native unit files, I rather fix getting systemd enhanced to fix that problem rather than fallback to init scripts in the longer run. So if you see any current issues, do talk to distributions and upstream systemd developers and let them know.

A systemd status update

dlang — Mon, 23 Aug 2010 22:19:24 +0000

yes they can, and initially they will (since many platforms will not support systemd)

however, how long will it be before the init script bitrots? especially if the package maintainer really only cares about one distro and that distro has systemd available.

A systemd status update

robert_s — Mon, 23 Aug 2010 22:18:39 +0000

True, but a sysadmin tends to start with the init script as provided and modify that, rather than reimplement an entire component from scratch.

I don't understand how you would be able to make small alterations to the logic of the provided C-implemented modules.

A systemd status update

rahulsundaram — Mon, 23 Aug 2010 22:16:06 +0000

Packages can ship *both* legacy init scripts and service files. If you delete the service file, systemd will fallback to reading the init scripts. There is no need to write them from scratch.