Multi-system administration with Func
Managing multiple computer systems can involve a lot of repetitive tasks: connecting to each, performing some update, status check, or configuration tweak, and then moving on to the next machine. These kinds of things can be scripted of course, but scripts of that nature typically need to be adjusted frequently as machines come and go or the tasks change. The Fedora Unified Network Controller (Func) is a tool that will help simplify system administration, but there is more to it than that—it is a framework for doing two-way secure communication, from the command line, scripts, or applications.
Func is written in Python, providing an API for scripts written in that language, but it can also be used from the command line. Each client machine—minion in Func-speak—runs the funcd daemon which contacts the master server or overlord. From the overlord machine, commands can then be issued to individual minions or to subsets of them. Some of the power of Func can be seen in simple commands like:
func "*" call service restart httpdwhich will restart the web server on all of the minions.
Similar kinds of tasks—but with more control—can be handled through the Python API. A somewhat contrived example from the Func website gives a sense of what can be done:
import func.overlord.client as fc results = fc.Client("*").service.status("httpd") for (host, returns) in results.iteritems(): if returns == 0: fc.Client(host).reboot.reboot()This example looks for minions that are running a web server and reboots each that it finds.
Managing keys can be a hassle when using ssh as an administrative tool, so Func uses another tool, Certmaster, to assist with keys. Certmaster provides a set of utilities and a Python API for managing SSL certificates. Clients generate certificate signing requests (CSRs), which contain their public key, that are sent to the Certmaster on the overlord. Administrators can either sign them from the command line or enable auto-signing. The minion then retrieves the signed certificate so that the overlord and minion communicate over an encrypted channel after that.
Func is not meant to replace ssh, instead it is intended to provide multi-system and scripting capabilities which are not the strengths of ssh. Like ssh, though, Func is meant to be easy to deploy—eventually ubiquitous, at least for Fedora—simple to use as well as easy to extend. It also has a pluggable architecture that allows Python modules to be integrated easily into Func, expanding the abilities of the minions. The documentation shows how to use the func-create-module command to generate template code which allows the administrator to ignore the Func requirements and concentrate on the task at hand.
There is nothing particularly Fedora-specific about Func, that's just where it was born. There are some efforts underway to add it for other distributions. Most of the work would be in creating distribution-specific analogs for things like restarting services and querying hardware configurations.
Red Hat has been releasing a steady stream of system administration tools over the last year or so. The Emerging Technology (ET) group has developed quite an ecosystem of tools to support installations with large numbers of servers that are frequently installed and upgraded. One might think they have a large infrastructure of such servers.
One of those tools that is frequently discussed in conjunction with Func is Cobbler. It is meant to simplify the configuration of a server to handle network installation and booting for a large server farm. From the web page:
Cobbler and the other tools coming out of the ET group are not just targeted at physical machines, but also virtualized environments. By using Cobbler, the puppet configuration manager, and the oVirt virtual machine manager, thousands of systems of various kinds can be managed in a centralized fashion. As would be expected, all of the code is available as free software.
These tools are quite interesting for system administrators, particularly those who use Fedora and have lots of systems to maintain. Even for small home networks, though, Func at least could come in handy. For overworked administrators—no matter the size of their domain—better tools are always welcome.
Posted Jun 19, 2008 2:26 UTC (Thu)
by Burgundavia (guest, #25172)
[Link] (6 responses)
Posted Jun 20, 2008 10:11 UTC (Fri)
by smitty_one_each (subscriber, #28989)
[Link] (1 responses)
Posted Jun 26, 2008 8:08 UTC (Thu)
by joib (subscriber, #8541)
[Link]
Posted Jun 22, 2008 13:50 UTC (Sun)
by rwmj (subscriber, #5474)
[Link] (2 responses)
they are not the easiest to
figure out how the hell to make them useful
You actually have a specific complaint, or are you just
spreading FUD? These tools are currently used by
large partner companies of Red Hat, who obviously find
them useful otherwise they wouldn't continue to use them.
Rich.
Posted Jun 27, 2008 4:27 UTC (Fri)
by Burgundavia (guest, #25172)
[Link] (1 responses)
Posted Jun 27, 2008 7:34 UTC (Fri)
by rwmj (subscriber, #5474)
[Link]
I'd really like to find out what issues you were
having. (Email me, r j o n e s (at) r e d h a t dot c o m)
I'm working on a set of command line tools for
administrators of small-scale virtual machine
deployments, so you can type commands such as
You could also start with the list of links at the
bottom of this page: http://virt-manager.org/.
However it's not intended that you should need to
compile everything from source, unless you want to
join in with development. Everything either
is or will be in Fedora, and the good bits should
eventually make it into RHEL.
Rich.
Posted Jun 22, 2008 14:28 UTC (Sun)
by ernest (guest, #2355)
[Link]
Posted Jun 24, 2008 1:40 UTC (Tue)
by clump (subscriber, #27801)
[Link]
Posted Aug 12, 2008 9:53 UTC (Tue)
by muwlgr (guest, #35359)
[Link]
Multi-system administration with Func
The single problem with func and the rest of the ET stuff is that they are not the easiest to
figure out how the hell to make them useful.
Multi-system administration with Func
Cobbler, for a quick glance, sounds like a souped-up version of the Solaris 10 Jumpstart
feature, which can make rolling out new installations somewhat easier.
This comment doesn't really add much, but I'm hoping there is someone with more Cobbler
experience who knows Jumpstart that can pontificate.
Multi-system administration with Func
I have no experience of jumpstart, but I believe it's something more like redhat kickstart,
no? I.e. a system for automated installs with a file containing answers to all the questions
that are usually asked when going through the interactive installer.
cobbler is more like a system for managing kickstarts and pxe boot infrastructure. It has 3
basic types of objects; distros (pxe boot images), profiles (associates a distro with a
specific kickstart file) and finally systems (associates a profile with a MAC and IP address).
So you create a bunch of systems with cobbler, and cobbler can generate a dhcp config (based
on a template you can modify e.g. to include non-cobblerized entries), sets up a tftp server
etc. Then you just PXE boot the systems and they are automatically installed with the profiles
you have specified.
Multi-system administration with Func
Multi-system administration with Func
No, this is not FUD. I really love all the ET stuff Red Hat is working on, but when I put on
my "busy sysadmin" hat, I realized I got really lost trying to figure out a) where to start b)
was the pain of setup worth the reward.
Multi-system administration with Func
virt-df
to get a 'df'-style listing
of the disks of all your virtual machines, or
virt-ps
to get process listings
from all of them.
Multi-system administration with Func
Actually one easy example is doing an update of some package on all your systems, but only
where that service is a version with a known security risk.
I can easily think of many tasks which I had to do fairly recently where a package like func
would have helped me. I can imagine though that if you don't have at least a few systems to
manage that setting up something like func is more a bother than a helper.
Ernest.
Multi-system administration with Func
My company has a few host management systems. So many, in fact, that we're trying to get rid
of most of them. The most effective one I've used had a simple PostgreSQL configuration
database that had a profile of sorts on all of the hosts it managed. Putting new hosts on the
network looked like this:
-Add the host to the system. (command line stanza that included host name, who operated the
host, and who managed the host)
-Create or clone an existing host's profile. The profile also held software that would be
pushed to the host. Think Apache plus configs, cron jobs, users, etc.
-Run a command to push software to the host. (stuff gets pushed over ssh)
That's it. You could go from a basic OS install to full production in minutes. Lose a host?
Put a new one there and push the profile. You're back up in minutes. Need to put some brand
new software on all your hosts? Simply make a package, assign it to your hosts, and then push
it out. Regexes are understood.
Now, Func appears to cover some of the picture. For example, it appears to have a bit of
understanding as to what it governs, somewhat like a profile. What Func can do for restarting
services, we do via something like "hostname_command | parrallel_version_of_ssh
/etc/init.d/httpd restart".
I'm happy to see software like Func out there. I have already talked to a few higher-ups in
my organization about making our host management system Free Software. Sadly those in charge
now don't see the value.
Multi-system administration with Func
You have to have quite large scale of requirements to decide to deploy
and use such a system. And you would not replace it with ease. There are
several already designed systems, like cfengine, bcfg2, automateit,
puppet (whihc is used by func), etc. But usually you pick such a system
only once and then cope with its limitations for a long time before you
decide to try something next.