Puppets, chefs, and community competition [LWN.net]

By Jonathan Corbet
March 10, 2009

There are many criticisms that one can make of the applications offered by the free software community, but lack of choice is generally not one of them. Our community thrives on competition while our licensing makes it hard to keep secrets from competitors. A recent episode in the Puppet community shows that, while this competition can sometimes take unwelcome forms, there is often little to do but to welcome it anyway.

Puppet is an automated configuration management system intended to make life easier for system administrators; it can be seen as a competitor to venerable tools like cfengine. Over time, Puppet has attracted an active community of users and developers; it would appear to be a tool which is growing in capability and popularity. Puppet is managed by Reductive Labs, which has a clear commercial interest in providing training and support services for Puppet users.

Recently (January, 2009), a project named Chef announced its existence. Chef's developers, who have previously worked with the Puppet code, set out to solve a similar problem. Chef is not a fork of Puppet, though; it's a new project developed from the beginning. Among other things, the Chef developers decided to use Ruby as the configuration language and they chose the Apache License (Puppet, instead, is distributed under the GPL). This project claims to be in active, production use, but its community, at this point, is clearly small. As of this writing, the chef-dev mailing list shows a total of four messages over its entire history.

Initially, the Puppet developers responded confidently to the Chef announcement:

Everything else in Chef seems pretty basic. They certainly have a smaller code base than Puppet does, but they're also brand new - Puppet didn't start this large, either, of course. To me, it's mostly a question of who has the best vision and who can execute. On those fronts, given my experiences (albeit tempered a bit these days by fatigue), I'm not afraid of competition.

More recently, though, Puppet developer Luke Kanies posted to the project's user list that Chef wasn't competing entirely fairly:

We've recently had some problems where one or two people are maintaining their presence in the Puppet community solely as a way to recruit people out of Puppet and into their community, at the expense of ours, and I think we need a straightforward community policy on this....

My take is that if your participation in our community is *solely* for purposes of shrinking it by drawing people into your community at the expense of ours, then you should be kicked from our community.

In particular, it is said that one developer from the Chef project has been sending private mail to Puppet users - especially those experiencing problems with Puppet - suggesting that they should switch to Chef. Luke, clearly, sees this activity as a threat to his livelihood; every Puppet user who deserts is one less potential customer. Even without that incentive, though; it can be hard to stand by and watch as others try to woo users away from your project. One need only think back to the days when "Ubuntu is better" posts were a semi-regular feature of the Fedora mailing lists to see how galling it can be.

In this case, a cooler perspective quickly won over and it became clear that there was little to be done. If nothing else, the objectionable messages were private email; there is little that the project could do to stop them even if it wanted to. Beyond that, though, certain things are inherent in the running of a free software project, including:

There will be competition, in some form or other. Somebody, somewhere, is sure to decide to scratch an itch, even if that itch is no more substantial than "I want to run my own project." This is both a strength and a weakness in our community. The ability for new and different ideas to develop into functioning projects is the source for much of the great software we now have, but it also leads to a certain amount of duplication of effort and confusion of users.
Some Puppet users expressed dissatisfaction that the Chef developers had clearly drawn a lot of inspiration and knowledge from the Puppet project. But, again, that's how our community works. Anybody who wants to hide the ideas that go into an application would be well advised to keep their software proprietary and closed. In the free software world we learn from each other - at least some of the time.
In a community which values freedom, attempts to silence or banish inconvenient characters will not get very far. When inappropriate or unethical behavior is seen (and spamming users of a competing project to urge them to switch is certainly pushing the boundary), shining light on that behavior is usually the best thing to do. In this case, the discussion made it clear that this email campaign did not inspire respect; it would not be surprising to learn that the pro-Chef emails have already stopped.

Andrew Shafer summed up the situation nicely:

Puppet is awesome, except when it isn't, and the best way to move things forward is to address those and get back to making more awesome. That's what we need to be worried about. Just more awesome, this is not a zero sum game.

Projects which are focused on "awesome" tend, over the long term, to be rather more successful than projects which worry about what others might be saying about them. They are also likely to be more successful than projects which put their effort into trying to poach another project's users. Puppet appears to have good code and an active and engaged user community. If it can stay focused on that code and that community, this project need not fear what its competitors are doing.

(Thanks to Friedrich Clausen for calling our attention to this discussion).

to post comments

Puppets, chefs, and community competition

Posted Mar 10, 2009 21:54 UTC (Tue) by dberkholz (guest, #23346) [Link] (8 responses)

In this case, the discussion made it clear that this email campaign did not inspire respect; it would not be surprising to learn that the pro-Chef emails have already stopped.

Cute idea. In my experience, people like that don't have the same morals as everyone else.

Puppets, chefs, and community competition

Posted Mar 10, 2009 22:31 UTC (Tue) by drag (guest, #31333) [Link] (1 responses)

SPAM, aka unsolicited solicitation, has always met with quick end in my inbox.

Puppets, chefs, and spam

Posted Mar 12, 2009 21:49 UTC (Thu) by giraffedata (guest, #1954) [Link]

The solicitations here aren't really spam, because spam isn't just unsolicited solicitations. The defining characteristic of spam is that it's blasted indiscriminantly to so many people that the chance of any particular recipient being interested in its message is miniscule. It is thus bad because it takes more from the recipients than it gives.

But sending an advertisement for product A to users of a similar product B is quite focused advertising, and there's a significant chance the user of B really will prefer A, and be glad he got the ad. It's not spam, and I'm all for it.

Puppets, chefs, and community competition

Posted Mar 10, 2009 22:37 UTC (Tue) by branden (guest, #7029) [Link] (5 responses)

To be fair, this sort of attempted poaching is precisely what (what was left of) the XFree86 Core Team accused Keith Packard of doing.

Keith denied undertaking such poaching (and since I am unaware of any contradictory information, I believe him), but I suspect most people can agree that everyone has benefitted from the realignment of loyalties that followed. Even the XFree86 diehards themselves are happy to all appearances, now that they have a nice, small, quiet sandbox to play in, untrammeled by the distractions of actual users.

Apart from Keith's legendary reputation and proven resume, what distinguishes these cases? I mean socially, not in terms of the technology used.

Was it simply that the XFree86 leadership didn't obviously care about "awesome" first and foremost?

Puppets, chefs, and community competition

Posted Mar 10, 2009 23:23 UTC (Tue) by rahulsundaram (subscriber, #21946) [Link] (3 responses)

The question can be boiled down to: whom do you trust?

Puppets, chefs, and community competition

Posted Mar 11, 2009 14:16 UTC (Wed) by njd27 (subscriber, #5770) [Link] (2 responses)

The lurkers [are trying to steal my users] in email

Is this free software or not?

Posted Mar 11, 2009 23:02 UTC (Wed) by xoddam (subscriber, #2322) [Link]

Is this free software or not? Free software is defined in terms of what its users are free to do. It doesn't make any sense at all to describe the users as belonging to the developers of the software.

Puppets, chefs, and community competition

Posted Mar 12, 2009 1:41 UTC (Thu) by Ze (guest, #54182) [Link]

>>The lurkers [are trying to steal my users] in email
You can't stop this unless you close up the support/development process.

Realistically the only way you can combat this is by providing a better option for your users than the alternative the lurkers are pushing. This includes guidance , support and user experience.

Puppets, chefs, and community competition

Posted Mar 11, 2009 19:04 UTC (Wed) by PO8 (guest, #41661) [Link]

XFree86 was open source code, but at the end not much of an open source project. That is, while the source was under an open license, the development wasn't done in a fully open way, and participation in the development of the codebase and guidance of the organization wasn't really possible for those outside of a group of long-time participants. Even Keith Packard got his commit access removed, not because of any social sins but because he was deemed not technically competent to commit to the code base by other members of the core team!

As someone who was peripherally involved with the transition to X.Org, I am quite confident that had the leaders of XFree86 agreed to open governance and open development there would have been no X.Org. This is quite a different situation than what is described in the article for Puppet / Chef.

Puppets, chefs, and community competition

Posted Mar 11, 2009 9:49 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (17 responses)

Ruby as a configuration language?

I hate this new trend of using Turing-complete languages for configurations. It makes automatic processing of configurations (using Augeas, for example) impossible.

Puppets, chefs, and community competition

Posted Mar 11, 2009 11:44 UTC (Wed) by nix (subscriber, #2304) [Link] (1 responses)

What new trend? Emacs is, what, 32 years old now...

Puppets, chefs, and community competition

Posted Mar 13, 2009 4:20 UTC (Fri) by njs (subscriber, #40338) [Link]

And on this issue, emacs has come full circle -- ";; custom-set-variables was added by Custom. If you edit it by hand, you could mess it up, so be careful..."

Puppets, chefs, and community competition

Posted Mar 11, 2009 11:50 UTC (Wed) by angdraug (subscriber, #7487) [Link]

As much as I like Ruby, I couldn't agree more. In my experience, YAML (or any other similar format with widely available parsing tools) is best for configuration. If you want your functionality to be controllable by a programming language, publish a plugins API.

Puppets, chefs, and community competition

Posted Mar 11, 2009 13:47 UTC (Wed) by alankila (guest, #47141) [Link] (7 responses)

My wild guess is that chef is written in ruby and they see no point to invent any kind of configuration language when you can just eval the config file, and it produces some datastructure. Being turing complete has the advantage of being also the most flexible way imaginable to do whatever it is doing.

Puppets, chefs, and community competition

Posted Mar 11, 2009 14:00 UTC (Wed) by clugstj (subscriber, #4020) [Link] (6 responses)

I don't know Ruby, but if there isn't a way of preventing the "configuration" from mucking with the internals of the implementation of the tool, then you can't ever change the implementation without risking breaking the "configuration".

Puppets, chefs, and community competition

Posted Mar 11, 2009 14:09 UTC (Wed) by alankila (guest, #47141) [Link] (5 responses)

Most people probably do not care about this sort of issue. The culture surrounding scripting languages tends to be one that favors flexibility, and responds to this kind of concerns with just "do not do that (unless you must)".

In general case the configuration often needs to be updated whenever the tool changes, too. New fields appear, old fields disappear, some are reinterpreted...

Puppets, chefs, and community competition

Posted Mar 11, 2009 16:12 UTC (Wed) by drag (guest, #31333) [Link] (3 responses)

Ya. This is one of the things that has irritated me about using python. A lot of the configuration stuff is actually scripts.

Personally I prefer to seperate data from code as much as possible. Configuration is data, not code, and should not be treated as such.

Also configuration files are user interfaces and should be treated consciously as such, too. So configuration files should be clean, concise, and well documented. This is how you have good usability for a program that is configured through text files.

Even XML can be good, even though it does make your life harder when it comes to good interfaces.

Puppets, chefs, and community competition

Posted Mar 12, 2009 2:54 UTC (Thu) by flewellyn (subscriber, #5047) [Link] (2 responses)

You can do both: sandbox the configuration language as a subset of whatever language you're working in or providing for scripting. The function or method that loads the config file can refuse to execute code which does not match a whitelist of allowed operations. Or, you can provide "sanitized" versions of those operations if you need to.

It's not as "easy" as simply evaluating a text file, granted, but as you said, config files are a form of user interface. Still, there's no reason that, with a bit of care, you can't have the flexibility of a full Turing-complete language for config, while maintaining security.

Puppets, chefs, and community competition

Posted Mar 12, 2009 22:42 UTC (Thu) by rwmj (subscriber, #5474) [Link] (1 responses)

Can do, but no one does.

/etc/xen configuration files are a good example: It's useful to be able to parse and manipulate
them from outside Xen itself, but because they are really Python code this isn't possible to do in the
general case.

Puppets, chefs, and community competition

Posted Mar 13, 2009 1:31 UTC (Fri) by flewellyn (subscriber, #5047) [Link]

Well, I have not seen the contents of an /etc/xen file, so I can't comment on the specifics. What about those files makes them problematic to parse with other tools, even Python tools? Do they make calls to xen-specific functions?

I think one thing about configuration-via-programming that should be adhered to, is that the configuration code should only consist of variable assignments. In a language which has first-class functions, closures, lambdas, and so forth, this still allows for a good bit of customizing, via binding functions to hooks. But, combined with the sandboxing approach, this could keep the application's config secure, while still allowing a good deal of flexibility.

Puppets, chefs, and community competition

Posted Mar 11, 2009 16:14 UTC (Wed) by man_ls (guest, #15091) [Link]

There is also the issue of security. If your config file can contain code then any user with write permissions can make you run arbitrary code. The config file is probably not executable and yet it is executed. The same is true for .bashrc though, so perhaps it's not that important.

Puppets, chefs, and community competition

Posted Mar 11, 2009 18:06 UTC (Wed) by bart (guest, #466) [Link]

Chef and Puppet are both tools to manage system configuration. They not only manage
configuration files, but also packages, users, cron jobs, etc.: everything you need to get a fully
configured server. By providing a language they allow you to do things like: only install this file on
the server if the operating system is Debian.

Augeas is something Puppet and Chef can use to actually manage the configuration files on the
system. (And in fact, Puppet can use Augeas for doing just that do that.)

Turing complete configs

Posted Mar 11, 2009 23:07 UTC (Wed) by prl (guest, #44893) [Link] (2 responses)

If you have to set up 1000 instances of something or other, you need an API and config via a proper programming language (i.e. not tcl, even if it was invented for just this purpose). mod_perl made Apache really usable on a large scale especially for multiple virtual servers. The lack of an API for this is the biggest limitations of BIND.

It's perfectly reasonable to want to have either a static config file or a scripting language or both. But what I REALLY object to is being forced to use ONLY the author's favourite language or config format - especially if the author has an interest in selling support for that format...

Turing complete configs

Posted Mar 12, 2009 1:17 UTC (Thu) by jimparis (guest, #38647) [Link] (1 responses)

I don't understand. How does mod_perl affect Apache configuration?
I use mod_perl and I still need to set up my virtual hosts in a text file (or generate that text file externally with a script).

Turing complete configs

Posted Mar 12, 2009 12:02 UTC (Thu) by Klavs (guest, #10563) [Link]

with an embedded perl in apache -you can config apache using <perl> sections :)

Puppets, chefs, and community competition

Posted Mar 13, 2009 3:41 UTC (Fri) by jamtur01 (guest, #48573) [Link]

Actually Puppet fully integrates with Augeas so that issue is moot.

Puppets, chefs, and community competition

Posted Mar 14, 2009 0:22 UTC (Sat) by dmag (guest, #17775) [Link]

Any configuration tool is going to have a "configuration language". Puppet defined it's own language. It was neat (even revolutionary). Instead of writing scripts to start/stop services and add/remove users, you just declared what Resources (users, services, files) you wanted, and Puppet "made it so". Your declarations are fairly simple, and the Puppet engine implements them in an idempotent way (so it only does what needs doing. By default it runs every 30 minutes to implement any new changes).

But the Puppet "config language" had to be extended in a number of different directions. People need to be able to express complex things like "All servers must have this program installed, except the staging/test servers". Worse, some configuration is repetitive (think vhosts for a webhost). Puppet has a limited way of dealing with that. You can quickly devolve into generating your Puppet config, which is the wrong answer.

Chef started copying the resource declaration style of Puppet. But they implemented the config files as Ruby with extensions. Unlike Python or Perl, you can write very simple to understand config files that are actually valid Ruby. But writing in Ruby allows you to DRY (Don't Repeat Yourself) your configuration. Not just simple variable substitutions, but loops anywhere you want them, calling out to external services (databases, parsing custom config files), etc. Like Rails, Chef gives you a pre-made directory structure (definitions, attributes, recipes, etc.). It looks complex, but it's nice because "everything has a place" and it keeps you organized.

Chef is very useful when building complex apps "in the cloud" (i.e. on Amazon EC2 where servers come and go).

Self-promotion

Posted Mar 12, 2009 14:03 UTC (Thu) by pboddie (guest, #50784) [Link]

You see this kind of thing happening in the Python community quite a bit. Someone will say that they're having specific problems with solution A, to which a developer of competing solution B will ask them why they aren't using solution B instead. Quite often the problems aren't anything to do with solution A, but are more general issues with programming or the language, and they'd probably arise when using solution B, too.

This kind of thing gets old very fast. It's like someone phoning in a problem with the CD player in a Honda half way (or further) through a journey, and then being asked why they don't sell their car, return home, buy a Toyota instead, and set off once more.

OT: What would be the Python equivalent to cfengine/puppet/chef?

Posted Mar 13, 2009 0:33 UTC (Fri) by debacle (subscriber, #7114) [Link] (2 responses)

func?

OT: What would be the Python equivalent to cfengine/puppet/chef?

Posted Feb 25, 2010 14:52 UTC (Thu) by keithlard (guest, #50464) [Link]

There is a Python-based clone of Chef available, called Kokki.

However, I'm not sure what it means to ask about a 'Python equivalent of Puppet'. Puppet is written in Ruby, but the configuration language that you use to describe the resources on your servers is a dedicated language designed especially for the job. So you don't need to know Ruby to use Puppet (though you do if you want to use Chef).

There is an interesting discussion about the merits, or otherwise, of dedicated configuration languages (cfengine, Puppet) as opposed to an embedded DSL (Chef, Kokki) here:

Puppet vs Chef

OT: What would be the Python equivalent to cfengine/puppet/chef?

Posted Jun 12, 2012 9:40 UTC (Tue) by baijum (guest, #85078) [Link]

Salt is a remote execution and configuration management tool written in Python: http://saltstack.org/

Puppet vs Chef: configuration management showdown

Posted Jan 26, 2010 14:30 UTC (Tue) by keithlard (guest, #50464) [Link]

Clearly 'Puppet vs Chef' is a debate which will run and run - I've addressed some of the divisions between the two projects in my article Puppet vs Chef: 10 reasons Puppet wins. It's clear from the article that many of Puppet's current advantages are directly due to its large user base - itself due to Puppet being the first mover in this market.

It seems that the Chef community know this as well, hence their efforts to recruit Puppet users, especially disillusioned ones. I don't think that's inherently bad - when Puppet and Chef have equal market share, then they will be able to compete directly on technical merit, which can only be good for both projects.