The "Devuan" Debian fork and the fuss about systemd

Posted Dec 4, 2014 20:54 UTC (Thu) by sjj (guest, #2020)
In reply to: The "Devuan" Debian fork and the fuss about systemd by mgb
Parent article: The "Devuan" Debian fork

Huh, we don't have megabytes of config files here:
$ cat /etc/redhat-release ; du -sh /usr/lib/systemd/system
CentOS Linux release 7.0.1406 (Core)
748K /usr/lib/systemd/system

What system has megabytes?

And poorly documented? How about apropos systemd; man systemd.unit; man systemd.service. I *wish* other parts of the system were so "poorly" documented!

As my first foray into systemd, I needed to run a special configuration script for openvpn server before it starts. Steps: copy service file from /usr/lib to /etc, add ExecStartPre=, and... done. Extra nice that since the config files are .ini format, and ansible knows how to edit those natively in-place, this is now automatic.

Sure, I could hack that into the openvpn startup script, or put in a new init.d script to run before openvpn, but that's brittle. And now I have a pattern - I can use the exact same pattern for other services instead of a service-specific hack (have you ever read an HP or Oracle etc "Enterprise" init script???).

Consistency and easy automation of changes in service initialization? This is a win-win-win for me as a sysadmin.

Please provide an example (preferably with reproducible test case or perfomance numbers) of logging bottleneck - I definitely would like to see if this could be problem.

The "Devuan" Debian fork and the fuss about systemd

Posted Dec 4, 2014 21:06 UTC (Thu) by dlang (guest, #313) [Link] (4 responses)

> Please provide an example (preferably with reproducible test case or perfomance numbers) of logging bottleneck - I definitely would like to see if this could be problem.

this again?

you want a test case, ok here's a simple test

take any largeish text file

run logger -f against that file and see how many messages per second you can process with systemd delivering the messages to syslog and going directly to syslog

to eliminate other possible distractions, I would suggest simplifying your syslog config to just write all logs to a single file. If you are using rsyslog, I would suggest using a template that has %timegenerated% in it so that you get the timestamp of when rsyslog sees the message as opposed to the timestamp that the journal puts on it.

While you do this, look at the time/cpu taken by logger to deliver the messages, and also the time/cpu used by the journal and syslog daemons to process the messages

The "Devuan" Debian fork and the fuss about systemd

Posted Dec 4, 2014 22:49 UTC (Thu) by sjj (guest, #2020) [Link] (3 responses)

This was an honest question. Since you've been banging this drum for weeks here, I would expect you to have some examples. Care to share?

I don't have time to setup much testing right now (maybe next week) - a quick test in a CentOS7 VM (KVM under Ubuntu latest on an MBAir) gives me about 12000 msgs/sec with /var/log on either tmpfs or xfs-on-KVM-on-ext4-on-SSD (this makes me wonder about the test setup...)

If a dinky little VM can do 12000 msg/sec, I'm fairly confident this isn't going to spoil my day. Even when I rebuild the central log server, but I'll be looking at the options at that time. Anybody have comparison numbers between stock RHEL/CentOS 6 vs 7 logging performance?

The "Devuan" Debian fork and the fuss about systemd

Posted Dec 8, 2014 19:21 UTC (Mon) by dlang (guest, #313) [Link] (2 responses)

a moderate VM should be able to do at least 10x the log volume that you are reporting (a couple hundred thousands of messages/sec), so I don't know what's going on with your test.

I don't have any systemd based systems to run tests on, so I can't give you a direct comparison from my own tests.

The "Devuan" Debian fork and the fuss about systemd

Posted Dec 9, 2014 2:57 UTC (Tue) by mathstuf (subscriber, #69389) [Link] (1 responses)

Just out of curiosity, what application(s) are pumping out that volume of data? What do you do with a fire hose of something like 10 megs a second (assuming 100 bytes per message)? What tools do you use to analyze the logs? Do you have a dedicated network interface for the logs? Eating up 10% of your gigabit bandwidth doesn't sound like the best use of it or the CPU work involved. Assuming that you have users which are generating the data interactively (and not batch jobs).

The "Devuan" Debian fork and the fuss about systemd

Posted Dec 9, 2014 3:25 UTC (Tue) by dlang (guest, #313) [Link]

I normally only see that sort of volume on my log aggregation boxes, not on individual boxes generating the logs. But you would be surprised at how much traffic a firewall or webserver can generate at a company that has multiple gigabit Internet connections. Yes, Gig-E throughput is a limiting factor on some of my systems (a limit I haven't hit outside of testing, yet. But it's in sight)

As far as best use of the CPU goes, it's better spent getting the logs off the box so that other systems can analyze them than trying to do all that analysis locally (not to mention that some things require analysis on logs across systems)

> Assuming that you have users which are generating the data interactively (and not batch jobs).

when you have 10's of millions of users using your services interactively, you can generate a LOT of log data.

for what it's worth, the reported rsyslog record is 1 million logs/sec processed by a single VM (on a 10G network with a good disk subsystem, but I think it was a 2-core VM)

The "Devuan" Debian fork and the fuss about systemd

Posted Dec 4, 2014 22:45 UTC (Thu) by rodgerd (guest, #58896) [Link] (2 responses)

> copy service file from /usr/lib to /etc, add ExecStartPre=, and... done.

Even better: source it and add the ExecStartPre= and not worry that upgrades will skew the source and your copy.

The "Devuan" Debian fork and the fuss about systemd

Posted Dec 4, 2014 23:08 UTC (Thu) by cortana (subscriber, #24596) [Link] (1 responses)

Oblig. even better again... create /etc/systemd/foo.service.d/local.conf in which you can put _just_ the ExecStartPre= line. :)

The "Devuan" Debian fork and the fuss about systemd

Posted Dec 5, 2014 3:09 UTC (Fri) by sjj (guest, #2020) [Link]

D'oh, of course, nice. Thanks!

The "Devuan" Debian fork and the fuss about systemd

Posted Dec 5, 2014 0:00 UTC (Fri) by mchapman (subscriber, #66589) [Link] (2 responses)

> $ cat /etc/redhat-release ; du -sh /usr/lib/systemd/system
CentOS Linux release 7.0.1406 (Core)
748K /usr/lib/systemd/system

And if you stick an --apparent-size in there as well, so you get the size of the content only, it drops by a factor of five.

That's what you get when you have lots of very small files. I'd much rather have that than a few large ones.

The "Devuan" Debian fork and the fuss about systemd

Posted Dec 5, 2014 3:13 UTC (Fri) by sjj (guest, #2020) [Link] (1 responses)

Thanks for the clarification. I had this small insistent voice at the back of my head during the drive home that something still wasn't right since this was only 200+ files.

The "Devuan" Debian fork and the fuss about systemd

Posted Dec 5, 2014 10:22 UTC (Fri) by cortana (subscriber, #24596) [Link]

You can make it even more accurate with something like:

find /lib/systemd/system /etc/systemd/system -type f -exec du --apparent-size -hc {} + | sort -h

Which comes to 107k on my system.