LWN: Comments on "Puppets, chefs, and community competition"

OT: What would be the Python equivalent to cfengine/puppet/chef?

keithlard — 2010-02-25T14:52:42+00:00

There is a Python-based clone of Chef available, called Kokki.

However, I'm not sure what it means to ask about a 'Python equivalent of Puppet'. Puppet is written in Ruby, but the configuration language that you use to describe the resources on your servers is a dedicated language designed especially for the job. So you don't need to know Ruby to use Puppet (though you do if you want to use Chef).

There is an interesting discussion about the merits, or otherwise, of dedicated configuration languages (cfengine, Puppet) as opposed to an embedded DSL (Chef, Kokki) here:

Puppet vs Chef

Puppet vs Chef: configuration management showdown

keithlard — 2010-01-26T14:30:14+00:00

Clearly 'Puppet vs Chef' is a debate which will run and run - I've addressed some of the divisions between the two projects in my article Puppet vs Chef: 10 reasons Puppet wins. It's clear from the article that many of Puppet's current advantages are directly due to its large user base - itself due to Puppet being the first mover in this market.

It seems that the Chef community know this as well, hence their efforts to recruit Puppet users, especially disillusioned ones. I don't think that's inherently bad - when Puppet and Chef have equal market share, then they will be able to compete directly on technical merit, which can only be good for both projects.

Puppets, chefs, and community competition

dmag — 2009-03-14T00:22:30+00:00

Any configuration tool is going to have a "configuration language". Puppet defined it's own language. It was neat (even revolutionary). Instead of writing scripts to start/stop services and add/remove users, you just declared what Resources (users, services, files) you wanted, and Puppet "made it so". Your declarations are fairly simple, and the Puppet engine implements them in an idempotent way (so it only does what needs doing. By default it runs every 30 minutes to implement any new changes).

But the Puppet "config language" had to be extended in a number of different directions. People need to be able to express complex things like "All servers must have this program installed, except the staging/test servers". Worse, some configuration is repetitive (think vhosts for a webhost). Puppet has a limited way of dealing with that. You can quickly devolve into generating your Puppet config, which is the wrong answer.

Chef started copying the resource declaration style of Puppet. But they implemented the config files as Ruby with extensions. Unlike Python or Perl, you can write very simple to understand config files that are actually valid Ruby. But writing in Ruby allows you to DRY (Don't Repeat Yourself) your configuration. Not just simple variable substitutions, but loops anywhere you want them, calling out to external services (databases, parsing custom config files), etc. Like Rails, Chef gives you a pre-made directory structure (definitions, attributes, recipes, etc.). It looks complex, but it's nice because "everything has a place" and it keeps you organized.

Chef is very useful when building complex apps "in the cloud" (i.e. on Amazon EC2 where servers come and go).

Puppets, chefs, and community competition

njs — 2009-03-13T04:20:48+00:00

And on this issue, emacs has come full circle -- ";; custom-set-variables was added by Custom. If you edit it by hand, you could mess it up, so be careful..."

Puppets, chefs, and community competition

jamtur01 — 2009-03-13T03:41:12+00:00

Actually Puppet fully integrates with Augeas so that issue is moot.

Puppets, chefs, and community competition

flewellyn — 2009-03-13T01:31:10+00:00

Well, I have not seen the contents of an /etc/xen file, so I can't comment on the specifics. What about those files makes them problematic to parse with other tools, even Python tools? Do they make calls to xen-specific functions?

I think one thing about configuration-via-programming that should be adhered to, is that the configuration code should only consist of variable assignments. In a language which has first-class functions, closures, lambdas, and so forth, this still allows for a good bit of customizing, via binding functions to hooks. But, combined with the sandboxing approach, this could keep the application's config secure, while still allowing a good deal of flexibility.

OT: What would be the Python equivalent to cfengine/puppet/chef?

debacle — 2009-03-13T00:33:35+00:00

func?

Puppets, chefs, and community competition

rwmj — 2009-03-12T22:42:01+00:00

Can do, but no one does.

/etc/xen configuration files are a good example: It's useful to be able to parse and manipulate
them from outside Xen itself, but because they are really Python code this isn't possible to do in the
general case.

Puppets, chefs, and spam

giraffedata — 2009-03-12T21:49:52+00:00

The solicitations here aren't really spam, because spam isn't just unsolicited solicitations. The defining characteristic of spam is that it's blasted indiscriminantly to so many people that the chance of any particular recipient being interested in its message is miniscule. It is thus bad because it takes more from the recipients than it gives.

But sending an advertisement for product A to users of a similar product B is quite focused advertising, and there's a significant chance the user of B really will prefer A, and be glad he got the ad. It's not spam, and I'm all for it.

Self-promotion

pboddie — 2009-03-12T14:03:04+00:00

You see this kind of thing happening in the Python community quite a bit. Someone will say that they're having specific problems with solution A, to which a developer of competing solution B will ask them why they aren't using solution B instead. Quite often the problems aren't anything to do with solution A, but are more general issues with programming or the language, and they'd probably arise when using solution B, too.

This kind of thing gets old very fast. It's like someone phoning in a problem with the CD player in a Honda half way (or further) through a journey, and then being asked why they don't sell their car, return home, buy a Toyota instead, and set off once more.

Turing complete configs

Klavs — 2009-03-12T12:02:08+00:00

with an embedded perl in apache -you can config apache using <perl> sections :)

Puppets, chefs, and community competition

flewellyn — 2009-03-12T02:54:45+00:00

You can do both: sandbox the configuration language as a subset of whatever language you're working in or providing for scripting. The function or method that loads the config file can refuse to execute code which does not match a whitelist of allowed operations. Or, you can provide "sanitized" versions of those operations if you need to.

It's not as "easy" as simply evaluating a text file, granted, but as you said, config files are a form of user interface. Still, there's no reason that, with a bit of care, you can't have the flexibility of a full Turing-complete language for config, while maintaining security.

Puppets, chefs, and community competition

Ze — 2009-03-12T01:41:48+00:00

>>The lurkers [are trying to steal my users] in email
You can't stop this unless you close up the support/development process.

Realistically the only way you can combat this is by providing a better option for your users than the alternative the lurkers are pushing. This includes guidance , support and user experience.

Turing complete configs

jimparis — 2009-03-12T01:17:07+00:00

I don't understand. How does mod_perl affect Apache configuration?
I use mod_perl and I still need to set up my virtual hosts in a text file (or generate that text file externally with a script).

Turing complete configs

prl — 2009-03-11T23:07:33+00:00

If you have to set up 1000 instances of something or other, you need an API and config via a proper programming language (i.e. not tcl, even if it was invented for just this purpose). mod_perl made Apache really usable on a large scale especially for multiple virtual servers. The lack of an API for this is the biggest limitations of BIND.

It's perfectly reasonable to want to have either a static config file or a scripting language or both. But what I REALLY object to is being forced to use ONLY the author's favourite language or config format - especially if the author has an interest in selling support for that format...

Is this free software or not?

xoddam — 2009-03-11T23:02:09+00:00

Is this free software or not? Free software is defined in terms of what its users are free to do. It doesn't make any sense at all to describe the users as belonging to the developers of the software.

Puppets, chefs, and community competition

PO8 — 2009-03-11T19:04:23+00:00

XFree86 was open source code, but at the end not much of an open source project. That is, while the source was under an open license, the development wasn't done in a fully open way, and participation in the development of the codebase and guidance of the organization wasn't really possible for those outside of a group of long-time participants. Even Keith Packard got his commit access removed, not because of any social sins but because he was deemed not technically competent to commit to the code base by other members of the core team!

As someone who was peripherally involved with the transition to X.Org, I am quite confident that had the leaders of XFree86 agreed to open governance and open development there would have been no X.Org. This is quite a different situation than what is described in the article for Puppet / Chef.

Puppets, chefs, and community competition

bart — 2009-03-11T18:06:08+00:00

Chef and Puppet are both tools to manage system configuration. They not only manage
configuration files, but also packages, users, cron jobs, etc.: everything you need to get a fully
configured server. By providing a language they allow you to do things like: only install this file on
the server if the operating system is Debian.

Augeas is something Puppet and Chef can use to actually manage the configuration files on the
system. (And in fact, Puppet can use Augeas for doing just that do that.)

Puppets, chefs, and community competition

man_ls — 2009-03-11T16:14:05+00:00

There is also the issue of security. If your config file can contain code then any user with write permissions can make you run arbitrary code. The config file is probably not executable and yet it is executed. The same is true for .bashrc though, so perhaps it's not that important.

Puppets, chefs, and community competition

drag — 2009-03-11T16:12:54+00:00

Ya. This is one of the things that has irritated me about using python. A lot of the configuration stuff is actually scripts.

Personally I prefer to seperate data from code as much as possible. Configuration is data, not code, and should not be treated as such.

Also configuration files are user interfaces and should be treated consciously as such, too. So configuration files should be clean, concise, and well documented. This is how you have good usability for a program that is configured through text files.

Even XML can be good, even though it does make your life harder when it comes to good interfaces.

Puppets, chefs, and community competition

njd27 — 2009-03-11T14:16:28+00:00

The lurkers [are trying to steal my users] in email

Puppets, chefs, and community competition

alankila — 2009-03-11T14:09:53+00:00

Most people probably do not care about this sort of issue. The culture surrounding scripting languages tends to be one that favors flexibility, and responds to this kind of concerns with just "do not do that (unless you must)".

In general case the configuration often needs to be updated whenever the tool changes, too. New fields appear, old fields disappear, some are reinterpreted...

Puppets, chefs, and community competition

clugstj — 2009-03-11T14:00:26+00:00

I don't know Ruby, but if there isn't a way of preventing the "configuration" from mucking with the internals of the implementation of the tool, then you can't ever change the implementation without risking breaking the "configuration".

Puppets, chefs, and community competition

alankila — 2009-03-11T13:47:11+00:00

My wild guess is that chef is written in ruby and they see no point to invent any kind of configuration language when you can just eval the config file, and it produces some datastructure. Being turing complete has the advantage of being also the most flexible way imaginable to do whatever it is doing.

Puppets, chefs, and community competition

angdraug — 2009-03-11T11:50:24+00:00

As much as I like Ruby, I couldn't agree more. In my experience, YAML (or any other similar format with widely available parsing tools) is best for configuration. If you want your functionality to be controllable by a programming language, publish a plugins API.

Puppets, chefs, and community competition

nix — 2009-03-11T11:44:45+00:00

What new trend? Emacs is, what, 32 years old now...

Puppets, chefs, and community competition

Cyberax — 2009-03-11T09:49:46+00:00

Ruby as a configuration language?

I hate this new trend of using Turing-complete languages for configurations. It makes automatic processing of configurations (using Augeas, for example) impossible.

Puppets, chefs, and community competition

rahulsundaram — 2009-03-10T23:23:16+00:00

The question can be boiled down to: whom do you trust?

Puppets, chefs, and community competition

branden — 2009-03-10T22:37:13+00:00

To be fair, this sort of attempted poaching is precisely what (what was left of) the XFree86 Core Team accused Keith Packard of doing.

Keith denied undertaking such poaching (and since I am unaware of any contradictory information, I believe him), but I suspect most people can agree that everyone has benefitted from the realignment of loyalties that followed. Even the XFree86 diehards themselves are happy to all appearances, now that they have a nice, small, quiet sandbox to play in, untrammeled by the distractions of actual users.

Apart from Keith's legendary reputation and proven resume, what distinguishes these cases? I mean socially, not in terms of the technology used.

Was it simply that the XFree86 leadership didn't obviously care about "awesome" first and foremost?

Puppets, chefs, and community competition

drag — 2009-03-10T22:31:54+00:00

SPAM, aka unsolicited solicitation, has always met with quick end in my inbox.

Puppets, chefs, and community competition

dberkholz — 2009-03-10T21:54:16+00:00

In this case, the discussion made it clear that this email campaign did not inspire respect; it would not be surprising to learn that the pro-Chef emails have already stopped.

Cute idea. In my experience, people like that don't have the same morals as everyone else.