Configuration Management Software Sucks

Yes.  Configuration Management Software Sucks.  Horribly.

The main problem is that n-th order tweakability is preferred over convention.  It’s just stupid.  There are a core set of things that just about everybody needs to do.  Those should be dead simple.  Ready to uncomment and run.  The set operating systems used in the enterprise is fairly small:  RHEL5, RHEL6, Debian 6, Ubuntu LTS.  A configuration system should be opinionated and have complete out of the box support for these platforms.  Simple rulesets for the basics that nearly everyone uses should be ready to go..  package management, process initialization, file management, ssh, sudo, DNS, Apache, PAM, PostgreSQL, MySQL, OpenLDAP, etc.  Keep it simple.  Keep it simple.  Keep it simple.  Resist all urges to add complexity.

That’s not the case.

You’d think after 30 years of Unix, BSD and Linux network deployments this would be pretty well trodden ground.  Wrong.  It’s a complete crapshoot and everybody does things differently.  Pick your poisons and reinvent the stack ad infinitum.

This is one of the few areas I’m green with envy of the Microsoft side of the fence.  Between Active Directory, Group Policy,  and maybe a third party tool or two for cloning and installs and such, Microsoft environments can easily be set up and managed well by complete morons (and often are).

Puppet

Puppet seems to have potential.  Of course, out of the box you’re pissing in the wind with a blank slate and most books and sites will have you following tutorials to rewrite rulesets that thousands of other people before you have similarly cobbled poorly together.  As a Ruby project, it unsurprisingly has vocal hipster fanboys.  Unfortunately, they forgot to parrot their DRY principle to each other.

It centers around a domain specific convention which isn’t so bad..  but in no time flat you’ll start seeing full blown Ruby programs intermingled.  Ugh.  But it’s not so bad if you stick to the basics.

If you look around you can find reasonably complete module sets, i.e. http://www.example42.com/.  It’s not all gravy as these are heavily interdependent and kludgy.  If you want a clean, simple solution you’re back to rolling your own with some healthy copy and paste.

Since it’s a Ruby project, aside from the annoying fanboys, you’re also going to run into scalability problems past a few hundred nodes.  There are mitigation strategies, but it’s a joke compared to something like Cfengine.

Due to hype, you’ll find decent versions in the Debian and Ubuntu backports repos.  RHEL 5 and 6 are covered by a Puppet Labs repo.  2.6 and 2.7 are therefore readily available and as long as your master is running the later version you shouldn’t have interop problems.

All things considered, Puppet is probably the best choice at the moment.  It sucks, but it’s got a lot of momentum behind it.  There are mountains of docs, books, and tutorials to get you going and nothing is too foreign or hard to grasp.

Cfengine 3

I really want to like Cfengine.  It’s incredibly light weight and hardcore ROFLscale.  It’s got serious theory behind it and older versions have been used in massive deployments.  But it’s not just a blank slate.  It’s even lower level and incomplete compared to the others.

You really need to add a promise library to get features that should be included by default.  These are all stagnate though, and still leave much to be desired.

There’s a company behind it doing something or another, but the open source version is raw.  If you have more than one Linux distribution, I’ll pretty much guarantee the packages are incompatible.

The repo choices aren’t great either.  Uncle Bob’s PPA on Ubuntu, out of luck on Debian.  RPMs in the EL repos look out of date.  You can of course get source and binaries from the Cfengine company, but it’s not my preferred way to install things and makes bootstrapping harder than it needs to be.

I haven’t tried the latest release, but quickly gave this one up when I found severe incompatibilities between point releases.  Madness.  You’d think people inventing something like promise theory could handle something as simple as version stability.

Ping me when a corporation backs Cfengine with a good promise library, some standard tasks, and repos for the common operating systems.

Bcfg2

Bcfg2 made the most sense to me out of the box.  XML is yucky and out of fashion these days, but Bcfg2 manages to use it acceptably.  Consequently, most things are declarative, easily read, and overall easy to mimic.  Beyond that, you can tap into some Python template and generator stuff.  But yes, these guys finally didn’t put n-th order above the common cases!  Installing packages and ensuring services are on is a snap.

They’ve got their own repos for many distros so installation isn’t bad.

The client and server are Python so you’ll have similar scaling problems to Puppet in large environments.

My biggest grievance with Bcfg2 is that the server needs intimate knowledge of each operating system version’s package repos.  You’ll fumble around writing a good bit of XML definitions for this in a heterogeneous environment.

The main thing Bcfg2 is lacking right now is community momentum.  Including repo definitions by default and some  more doc work.. I think this would be a great system for small to medium deployments.

Conclusions

The lot of this stuff is really terrible.  End to end system management under *nix is a major pain point.  On top of this, you’ll need a fairly free form monitoring framework (these also all suck) and directory service.  Mix and match an impossible array of projects and eventually you’ll find your own recipe that sort of works.  Except everyone does it differently so you’ll constantly be learning and redoing the same things over and over anyway.

It’s not fun.  What we need is end to end integrated thinking.  This area is still ripe for picking.  Oh RedHat, where art thou?

Be Sociable, Share!
  • email
  • Reddit
  • HackerNews

16 thoughts on “Configuration Management Software Sucks”

  1. We are coming, but it’s going to be a bit. Red Hat Cloud Engine will have a lot of the pain points removed, though it will be XML based too :-) so it will have some of the downsides of BCFG2.

    I have complained a million times about everything you have here and I agree. Hopefully, with pulp, delta-cloud, and application engine, we will eventually have a unified way to specify configuration.

  2. You might find Blueprint interesting. It reverse engineers an existing server to figure out what was installed and configured. It works its magic by inspecting existing package control: RPM, Apt, NPM, Python, etc. Then it produces shell scripts (or Puppet or Chef scripts) to set up another server like the old one. It’s a little literal minded, of course, but it worked surprisingly well. I think you’re right that having out of the box support for existing software is a good idea.

  3. @mina – That looks nice. I’ll keep an eye on it. Indeed, something close to UNIX style is what is needed to manage UNIX machines. The other solutions decidedly do not have that style.

    @Scott – cloud stuff has some overlap but I will be surprised if something in that market completely nails the problem.

    @Nelson – hmm, I will give Blueprint a try, especially to see how it dits in with Puppet

    @qzio – The sentiments I’ve seen are that Puppet is superior to chef. Is it worth evaluating vs these other systems?

  4. I have the same frustration with system config management. Starting with Puppet, I found Chef was much more straight forward and in fact enforced really sensible conventions (as far as recipe layout, configuration, etc.).

    But I think your underlying point is not the conventions of the CM system, but conventions of the target operating system. And I agree. There is just “too much shit” you have to do to configure things. I think this stems from the fact that operating systems, even server operating systems, were designed for persistent human interaction, not transient automation. My own feeling is that operating systems need to change entirely in the cloud so that they are stateless and stripped down thin wrappers around the actual services you want to run – and this may include radical changes such as not actually having filesystems or “file”-based configuration. There are some Linux distributions starting to do this (although I am hard pressed to actually find them now that I look…).

  5. There is “too much to configure” because Unix has flexible tools, and requirements may not be as common as they seem. I have 888 organizational requirements regarding the configuration of my hosts (it sounds made up but it is real). If I put out an “opinionated” tool that configures my 888 things, “ready to uncomment and run,” would anyone use it, or would most people deem it overkill? Who would write books and meet at conventions about my tool? But if I factored my opinion out of it, I’d have a tool that could work for lots of people, and a policy that works for me. And that’s how Puppet, cfengine and others do it. The “UNIX style” is to separate policy and mechanism.

    “End to end integrated thinking” doesn’t sound very Unixy to me either: see flexibility all the way down and the rule of diversity.

    The admin knows what the requirements on the ground are, in a way that developers usually don’t. When I do the final step of “end to end system integration,” it results in a system I can administer, my users can use, and my security folks can trust. That’s not a “pain point,” it’s a fulfilling job, and a result my Windows colleagues are “green with envy of.”

  6. You really should take a look at Chef and its community. I am one of the original developers. Three years ago we were all Puppet users who were pretty happy with it, ecstatic compared to using nothing at all, but we wanted to go in a different direction than the Puppet developers wanted to at the time.

    I think we “keep it simpler” than you describe. The use of primitives was a design goal from the start. Puppet wanted you to abstract everything you controlled into a core resource and when what you wanted to control lacked a resource, or if the existing resource lacked a feature you needed, it leaned on you more to fix the problem in Puppet rather than giving you the tools to solve the real problem. Thus Puppet has a historical list of resources for controlling Nagios, MTAs and zfs filesystems mixed in with primitives like package and service.

    For Chef, we absolutely wanted to control those things but we didn’t want to burden core Chef with all of the nuances of individual applications. Those can be built into Cookbooks using the basic DSL or full Ruby utilizing the support for reusable definitions and Lightweight Resource and Providers (LWRPs) [2], which are very similar to the primitives in Chef but can be created, shared, released and re-released quickly and easily. The use of the template resource (ERB) and leveraging our search capabilities to create configuration files dynamically is encouraged in the Chef community, while we were discouraged from doing so with Puppet.

    These primitives are supported on the platforms you listed, but from my experience that isn’t an accurate “enterprise” list. Supporting the applications on top of these platforms becomes a community “cookbook” project. Opscode maintains and stewards contributions to a large number of common applications [3], but other companies and users participate in the community of open source cookbooks.

    Visit our community site to check out many of these cookbooks: http://community.opscode.com/

    And of course, Opscode is actively working on improving Chef both for existing users and new users. For instance our Windows support [4] and installer [5] have been very well received and this is the tip of the Windows iceberg.

    [1] http://docs.puppetlabs.com/references/stable/type.html
    [2] http://wiki.opscode.com/display/chef/Lightweight+Resources+and+Providers+%28LWRP%29
    [3] https://github.com/opscode/cookbooks
    [4] http://www.opscode.com/blog/2011/05/24/chef-hearts-windows/
    [5] http://www.opscode.com/blog/2011/12/15/chef-client-installer-for-windows/

  7. If we look at where many of the linux utilities originated from, we’ll find that most of the system administration and packaging came from the 1980′s, targeted at standalone systems. What if a community of server administrators (system/network) went after the problem by creating a linux server distribution that is inherently manageable. Packaging for example is based on 1980′s AT&T packaging, makes no sense in 2012. System startup scripts are relics from the 1980′s too. Create a distro that takes into consideration the entire life cycle, which starts when a developer introduces a bug fix or new feature, all the way thru deployment, and onto escalations. DevOPs as I see it is a reaction based eco-system. Stop reacting, create a distro that fundamentally removes the symptoms that DevOPs systems are designed to automate. Anyone that has done a code review of the autotools and process to build packages, and same thing for the debian distros — it’s really brutal. I really think the DevOPs community can make a huge contribution, by taking the bull by the horns and designing a server distro from the ground up that is designed to be managed

  8. Hmmm, you are obviously very Linux-centric, what about AIX, Solaris, HP-UX,…(just to name a few)?
    Yes, Linux is becoming more Enterprise aware, but, still not nearly on top of the world like we’d like to think (I was a large S/W contributor to the FSF community starting back in 1985).

    Instead of trying to control all the rings, how about One Ring to rule them all (perhaps we need to re-investigate “United Linux” (w/o MS of course, just saying).

    But I digress.

  9. All CM systems suck because they’re typically employed by people who think pushing configuration files out over a network requires some huge software stack. You can use Make, SSH and a few scripts and templates and probably have a better tailored system in less lines of code than the equivalent Puppet config. If you think otherwise then you’re hopelessly inept at Unix.

    CM consultants are like “Agile” consultants or Java programmers. They’re only too glad to give you the most complex, ever-engineer system possible. Billing more hours and maximising your dependence on them can only be good for (their) business.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>