Category Archives: Redhat

Configuration Management Software Sucks

Yes.  Configuration Management Software Sucks.  Horribly.

The main problem is that n-th order tweakability is preferred over convention.  It’s just stupid.  There are a core set of things that just about everybody needs to do.  Those should be dead simple.  Ready to uncomment and run.  The set operating systems used in the enterprise is fairly small:  RHEL5, RHEL6, Debian 6, Ubuntu LTS.  A configuration system should be opinionated and have complete out of the box support for these platforms.  Simple rulesets for the basics that nearly everyone uses should be ready to go..  package management, process initialization, file management, ssh, sudo, DNS, Apache, PAM, PostgreSQL, MySQL, OpenLDAP, etc.  Keep it simple.  Keep it simple.  Keep it simple.  Resist all urges to add complexity.

That’s not the case.

You’d think after 30 years of Unix, BSD and Linux network deployments this would be pretty well trodden ground.  Wrong.  It’s a complete crapshoot and everybody does things differently.  Pick your poisons and reinvent the stack ad infinitum.

This is one of the few areas I’m green with envy of the Microsoft side of the fence.  Between Active Directory, Group Policy,  and maybe a third party tool or two for cloning and installs and such, Microsoft environments can easily be set up and managed well by complete morons (and often are).

Puppet

Puppet seems to have potential.  Of course, out of the box you’re pissing in the wind with a blank slate and most books and sites will have you following tutorials to rewrite rulesets that thousands of other people before you have similarly cobbled poorly together.  As a Ruby project, it unsurprisingly has vocal hipster fanboys.  Unfortunately, they forgot to parrot their DRY principle to each other.

It centers around a domain specific convention which isn’t so bad..  but in no time flat you’ll start seeing full blown Ruby programs intermingled.  Ugh.  But it’s not so bad if you stick to the basics.

If you look around you can find reasonably complete module sets, i.e. http://www.example42.com/.  It’s not all gravy as these are heavily interdependent and kludgy.  If you want a clean, simple solution you’re back to rolling your own with some healthy copy and paste.

Since it’s a Ruby project, aside from the annoying fanboys, you’re also going to run into scalability problems past a few hundred nodes.  There are mitigation strategies, but it’s a joke compared to something like Cfengine.

Due to hype, you’ll find decent versions in the Debian and Ubuntu backports repos.  RHEL 5 and 6 are covered by a Puppet Labs repo.  2.6 and 2.7 are therefore readily available and as long as your master is running the later version you shouldn’t have interop problems.

All things considered, Puppet is probably the best choice at the moment.  It sucks, but it’s got a lot of momentum behind it.  There are mountains of docs, books, and tutorials to get you going and nothing is too foreign or hard to grasp.

Cfengine 3

I really want to like Cfengine.  It’s incredibly light weight and hardcore ROFLscale.  It’s got serious theory behind it and older versions have been used in massive deployments.  But it’s not just a blank slate.  It’s even lower level and incomplete compared to the others.

You really need to add a promise library to get features that should be included by default.  These are all stagnate though, and still leave much to be desired.

There’s a company behind it doing something or another, but the open source version is raw.  If you have more than one Linux distribution, I’ll pretty much guarantee the packages are incompatible.

The repo choices aren’t great either.  Uncle Bob’s PPA on Ubuntu, out of luck on Debian.  RPMs in the EL repos look out of date.  You can of course get source and binaries from the Cfengine company, but it’s not my preferred way to install things and makes bootstrapping harder than it needs to be.

I haven’t tried the latest release, but quickly gave this one up when I found severe incompatibilities between point releases.  Madness.  You’d think people inventing something like promise theory could handle something as simple as version stability.

Ping me when a corporation backs Cfengine with a good promise library, some standard tasks, and repos for the common operating systems.

Bcfg2

Bcfg2 made the most sense to me out of the box.  XML is yucky and out of fashion these days, but Bcfg2 manages to use it acceptably.  Consequently, most things are declarative, easily read, and overall easy to mimic.  Beyond that, you can tap into some Python template and generator stuff.  But yes, these guys finally didn’t put n-th order above the common cases!  Installing packages and ensuring services are on is a snap.

They’ve got their own repos for many distros so installation isn’t bad.

The client and server are Python so you’ll have similar scaling problems to Puppet in large environments.

My biggest grievance with Bcfg2 is that the server needs intimate knowledge of each operating system version’s package repos.  You’ll fumble around writing a good bit of XML definitions for this in a heterogeneous environment.

The main thing Bcfg2 is lacking right now is community momentum.  Including repo definitions by default and some  more doc work.. I think this would be a great system for small to medium deployments.

Conclusions

The lot of this stuff is really terrible.  End to end system management under *nix is a major pain point.  On top of this, you’ll need a fairly free form monitoring framework (these also all suck) and directory service.  Mix and match an impossible array of projects and eventually you’ll find your own recipe that sort of works.  Except everyone does it differently so you’ll constantly be learning and redoing the same things over and over anyway.

It’s not fun.  What we need is end to end integrated thinking.  This area is still ripe for picking.  Oh RedHat, where art thou?

KDE SC 4.4 – Steady, Incremental Improvements

I haven’t noticed any killer features in KDE SC 4.4 and I’ve been running it since Beta 1. I’ve noticed a lot of subtle improvements.  Things like app stacking and selection in the task bar seem much more responsive.  All around, plasma looks subtly better and my favorite KDE apps seem to just keep getting better.

KSysGuard is really impressive and now has the ability to connect to remote hosts for monitoring.  However, the biggest change is in the greater ecosystem.  It seems all the external apps like Amarok, K3b, and digiKam are coming along to fruition.

Other than that, this is a smooth release and shows that the platform is starting to mature.  I think the Summer release distros will be able to do a good job delivering a nice desktop experience based on KDE 4.4.  I’ll end with my obligatory “try KDE 4.4 if you had previous bad KDE4 experiences”.

2010 SpamAssasin Public Service Announcement

If you run a public mail server, there is a good chance you run SpamAssassin.

There is a New Year’s SNAFU in which any dates 2010+ are marked “grossly in the future”.  That is a problem since it is now 2010.

Run sa-update and restart SpamAssassin ASAP, or install the following rule to local.cf:

score FH_DATE_PAST_20XX 0.0

Thanks to LWN for pointing this out to me before too much damage was done.

DRBD merged with kernel 2.6.33

DRBD has been a long standing external patch in many distribution kernels.  It has finally been merged in the 2.6.33 window.  Colloquially the “Distributed Redundant Block Device”, this piece of code allows you to mirror blocks of storage across multiple nodes.

drbd_overview

This is primarily useful in high availability setups.  By synchronously mirroring storage across two systems, you can run an active-passive cluster where the backup machine will take over if the primary fails.  Using a more advanced clustering file system such as GFS2 or OCFS, you can even do active-active setups although there are certain considerations there.

This is exciting because it alleviates the need for specialized hardware like SAN storage.  Standard Linux servers with direct attached storage may be used and indeed even give appreciable performance.  In practice, redundancy will be even greater than all but the highest end SAN equipment due to the lack of single point of failure.

DRBD also allows for asynchronous mirroring, that is, writes to the primary do not wait on completion to the secondary.  This is useful for cold site backups and perhaps meeting legal compliance in certain industries as well.

Take a look at the DRBD site for more information and use cases.

Xen 3.4.1 on RHEL/CentOS 5.4

I’m happy to report that the updated Gitco Xen 3.4.1 repo is working well on CentOS 5.4.

If you are doing link bonding and bridging in accordance with my previous post “Xen 3.3 in RHEL/CentOS 5 and more Link Aggregation Fun“, you no longer need to patch the network scripts as RedHat fixed the initscripts package in RHEL 5.4.

Upgrade procedure for CentOS 5.3 to 5.4:

yum clean all
yum update glibc\*
yum update yum\* rpm\* python\*
yum clean all
yum upgrade
reboot

Updated Xen Install Guide From My Previous Article:

Head over to http://www.gitco.de/repo/ and grab the repo for your arch.  (Most likely wget http://www.gitco.de/repo/CentOS5-GITCO_x86_64.repo in /etc/yum.repos.d/ for the uninitiated).

If you already have Xen installed, you may need to remove and readd it.

yum groupremove Virtualization
yum groupinstall Virtualization

You’ll also get some updated tools like Virtual Machine Monitor 0.7.0 that make it easier to install newer guests such as Fedora 11 or Ubuntu.  Sweet!

Double check /etc/sysconfig/kernel.  It should be set to kernel-xen.  Likewise, check /boot/grub.conf and make sure that the Xen kernel is the default if the aforementioned was not done beforehand.

Reboot!

Computer e-Recycling (an I.T. WTF Odyssey)

Story Time: Computer e-Recycling

an I.T. WTF Odyssey

I had the displeasure of working at an erecycler several years ago. Even watching stuff come off the trucks, it was very hard to get anything before it was utterly destroyed by the yard goons. Inserting a forklift blade into a 19″ rack cabinet was common practice and I witnessed on numerous occasions the dropping of them in this fashion. Once, they even rolled a forklift off the ramp. These events were always followed by a flurry of Spanish profanity and I usually had to check my pants for continence afterward from laughing so hard. It is a miracle nobody has ever been seriously maimed there to the best of my knowledge.

The highlight of this job experience was when the greedy goons resold a defective Siemens blood handling instrument of some sort that was sent specifically to them to be destroyed (as was EVERYTHING, in theory). The serial number was traced back to them and there was an all out shitstorm. A team of inspectors was flown out from Germany. The goons put on a particularly hilarious show buying hardhats, safety vests, warning signs, identification badges, and more. Somehow, they kept the contract (it was probably just “check you ass” on Siemen’s part — dumping obsolete X-Ray, MRI machines, medical waste, etc for free must be hard to pass up) and all of this change disappeared within a couple days.

What is remarkable is that the above business got multi-millions of dollars of inventory for FREE every year. I really can’t think of any other business model like it. The owner is completely incompetent and morally bankrupt. Very little was put back into the company’s facilities, employees (except maybe a couple at the top), or development. None of the employees had an IQ above room temperature, and everybody seemed content to keep it that way.

There is a happy ending though. A skid of IBM RS/6000 7012 systems from Intel came in at one time. Among the 20-30 machines was a lone -397 that I snagged. This is my favorite collectors box to date (it is the same POWER2 CPU type used in the famous Deep Blue super computer).

Mirroring Fedora

Introduction

This post details setting up your own private mirror of Fedora’s repos.  There are many ways to do this, but this method is by far the best for heavy usage.  By using MirrorManager, clients in your IP range need no custom configuration.  Roaming laptop users automagically hit your mirror while on the premises, yet use the public infrastructure elsewhere.  Setup isn’t exactly hard, but it isn’t well documented so I’ll write about my experience here.

Some background info.. we have at least 50 Linux desktops, laptops, servers and VMs running about half Fedora 10 and half Fedora 11 at work.  Due to the number of systems, breadth of packages used, and desire to quickly update when new releases are out, I decided on a full mirror setup.  If you only have a handful of systems, you may be better off simply using a general purpose caching proxy like Squid, perhaps telling MirrorManager to point to it.

This guide should be used in addition to http://fedoraproject.org/wiki/Infrastructure/Mirroring which has some background info.

Initial setup and mirror

First, get prepared by installing MirrorManager-client, which contains the report_mirror script you will need.  If your mirror isn’t running Fedora, you can clone the source of this app from their GIT repo.

yum install mirrormanager-client

You’ll be using rsync, a sysadmin’s best friend, for efficient mirroring.

Set up a shell script like mine below (d0mirror.sh) one level up from where your mirror will be accessible (http, ftp, rsync, nfs – covered later).  This one mirrors against kernel.org.  Choose a mirror close to you on the Internet.

rsync -vaH --exclude-from=fedora-excludes.txt --numeric-ids --delete --delete-delay \
 --delay-updates rsync://mirrors.kernel.org/fedora-enchilada fedora-mirror
report_mirror

And a text file (fedora-excludes.txt) excluding things you don’t want/need.  Take a look through a public mirror and decide if you want to eliminate anything else.  You may want to remove the *.iso line below if you want users to be able to pull disc images from this box.  Otherwise, this is probably a good list for most people.  You can exclude all of linux/updates/testing/ if you don’t enable the testing repo on any of your machines.

**/debug/**
**/alpha/**
**/source/**
**/SRPMS/**
**/*.iso
**/ppc/**
**/ppc64/**
linux/core/**
linux/development/**
linux/releases/7/**
linux/releases/8/**
linux/releases/9/**
linux/releases/test/**
linux/updates/8/**
linux/updates/9/**
linux/updates/testing/7/**
linux/updates/testing/8/**
linux/updates/testing/9/**

Run your shell script and sit back for up to a day or two depending on your connection speed.  My current mirror weighs in at about 80G.

Internal distribution

While you wait for sync, decide how you want to run the service internally.  HTTP is nice because it is easy for users to browse and decently quick with keep-alive.   Using NFS, rsync, or FTP may be a bit more efficient if you are worried about this.  You can list several URLs in MirrorManager for the best of all worlds.

Add the following to your Apache configuration if you decide to use HTTP:

Alias /fedora/ "/mnt/ar1/fedora-mirror/"

AddType application/octet-stream .rpm

<Directory "/mnt/ar1/fedora-mirror">
    Options Indexes FollowSymLinks
    Order allow,deny
    Allow from all
</Directory>

<LocationMatch "\.(xml|xml\.gz|xml\.asc|sqlite)">
    Header set Cache-Control "must-revalidate"
    ExpiresActive On
    ExpiresDefault "now"
</LocationMatch>

Set up any other services of you choice to push that directory out in addition.

Working with MirrorManager client and server

Next, open up /etc/mirrormanager-client/report_mirror.conf.  Take notice of the site name, password, and host name.  You will need to set these up in MirrorManager in a bit.  The paths here are all local and used by report_mirror to check what you have available.

# if enabled=0, no data is sent to the database
enabled=1
server=https://admin.fedoraproject.org/mirrormanager/xmlrpc

[site]
# if enabled=0, no data about this site is sent to the database
enabled=1
name=<yoursitename>
password=<yourhostpassword>

[host]
# if enabled=0, no data about this host is sent to the database
enabled=1
name=x345-a2.internal
# if user_active=0, no data about this category is given to the public
# This can be used to toggle between serving and not serving data,
# such enabled during the nighttime (when you have more idle bandwidth
# available) and disabled during the daytime.
# not passing it means leave it alone in the database.

[stats]
# Stats are only sent when run with the -s option
# and when this section is enabled.
enabled=0
apache=/var/log/httpd/access_log
vsftpd=/var/log/vsftpd.log
# remember to enable log file and transfer logging in rsyncd.conf
rsyncd=/var/log/rsyncd.log

[Fedora Linux]
enabled=1
path=/mnt/ar1/fedora-mirror/linux

[Fedora EPEL]
path=/var/www/html/pub/epel
enabled=0

# lesser used categories below

[Fedora Web]
enabled=0
path=/var/www/html/pub/fedora/web

[Fedora Secondary Arches]
enabled=0
path=/var/www/html/pub/fedora-secondary

[Fedora Other]
enabled=0
path=/var/www/html/pub/alt

# historical content

[Fedora Core]
# if enabled=0, no data about this host is sent to the database
enabled=0
path=/var/www/html/pub/fedora/linux/core

[Fedora Extras]
enabled=0
path=/var/www/html/pub/fedora/linux/extras

Log into https://admin.fedoraproject.org/mirrormanager, creating a new account if you need to.  Add a new site with the same name as the config file from above.  You’ll set the site password here, and make sure to check the ‘private’ box if this is only for internal users.  Now, add a host under this site.  The name here should probably be a FQDN of your actual mirror, even if it is internal only (i.e x345-a2.internal from my example above).  Once that is done, add a “site-local netblock”.  This is your public IP network/netmask or network in CIDR notation.  If you only have one public IP, it will be in the format nnn.nnn.nnn.nnn/32.

Almost done.  Now, click Add Category.  “Fedora Linux” is the only one you are concerned with if you followed all the values in this guide so far.  Add the others if needed.  Tell them your upstream source (rsync://mirrors.kernel.org/fedora-enchilada from above) and then your internal URL (http://x345-a2.internal/fedora/linux for my setup).

Conclusion

Once your rsync is complete and report_mirror is done, you should see clients start hitting your box.   Don’t forget to add your mirror script (domirror.sh from above — rsync and report_mirror) to cron!  You may wish to join the private ‘fedora-mirrors’ mail lists to be informed of new releases and changes.

The best thing is that it works across all package requests, including new machines, roaming users,  ‘preupgrade’, etc.   All in all, pretty nifty!  Your users will love you when their upgrades are almost instant!  The Fedora infrastructure is set up very well for mirroring, public and private, and this is how the project copes with the huge demand for new releases.  Comment away if you need clarification or help.

Kernel 2.6.30 is a Go

I initially thought this would be a rather uninteresting release, especially when we learned Xen dom0 didn’t make the cut. Following the changelog line-by-line, this one still didn’t seem very interesting to me. But analyzing the sum of parts, I have to consider 2.6.30 a ‘golden’ kernel — certainly the best in a while.

There is solid improvement top to bottom here.  A lot of the new KMS/DRM stuff from Fedora 11 has worked its way up stream.  File system work is too much to mention, but highlights include relatime, writeback by default for Extfs, NILFS2, Btrfs development and more. FSCache works as advertised.  Also some groundwork for NFS 4.1, which will eventually bring us pNFS.

Boot speed seems fast as ever, but I haven’t taken the time to do any empirical analysis.  Your results here will be hardware dependent but async initialization of certain subsystems is a welcome move in the right direction.

Basically, a solid release with a good balance of new stuff but mainly refinement of existing systems and merging of longstanding patches.

Kernel Newbies has, as usual, a great change summary: http://kernelnewbies.org/Linux_2_6_30

Kernel developers don’t get Xen

The recent bruhaha surrounding Xen on LKML (http://lkml.org/lkml/2009/6/2/475) is really disheartening.  Essentially, the Linux kernel devs are at a disconnect with users.  Some are proposing narrow-minded ideas such as DROPPING software paravirt or merging Xen as a whole into the kernel.

I use Xen for a few primary reasons:  it bar none has the best speed — full software paravirtualization pays dividends here;  it is mature;  it works on perfectly good machines that don’t happen to have the latest chips;  it does hardware passthrough on these same systems;  it has great live migration that actually works.

Ingo Molnar wants you to send all your perfectly good enterprise iron to the landfill even though these systems will last 10+ useful years without boneheaded software decisions such as this.

These same FUDsters want to strip the crossplatform nature of Xen dom0 out too.  Xen dom0 runs on NetBSD and Solaris.  It is a true hypervisor and will plug into exisiting architectures, and not force you to use Linux for everything.

I have to admire all the hoops Jeremy Fitzhardinge has jumped through to date, as I know my patience is wearing thin.

Xen powers huge sites such as Amazon and services like linode.com/slicehost.com.  By not having dom0 in the kernel where distros such as Ubuntu and Fedora can easily integrate it, kernel devs are doing a disservice to users.

I use KVM, VMWare, and Virtual Box at work in addition, but Xen is firmly entrenched in my toolbox.  The roadmap they have looks great, and I just don’t see a reason for decline in Xen popularity.  High availability in Xen 4.0 is what I’ve always been waiting for.

Jeremy has gone to great lengths to work with upstream but keeps getting shot down and asked to do something else when he meets one requirement.  The solution is to merge Jeremy’s conservative dom0 patch set and work on a technical solution to the patches that the FUDsters consider bad.  It’s what the users want!

To users that miss xorg.conf and complain about it

I get requests from users and see questions all the time for “where did my xorg.conf go in the latest Ubuntu or Fedora?”, though it is usually a bit more of a flame.

The quick answer… press Ctrl+Alt+F2 or similar to log into a TTY console, or type ‘init 3′ into a root X terminal.

If you haven’t already, log in as root and  kill X or type ‘init 3′ if you want to be heavy handed.  Then run:

X -configure
mv ~/xorg.conf.new /etc/X11/xorg.conf

xorg.config in two commands.  Run the ‘init 5′ command to get back to your GUI login (or kdm or gdm or startx, etc if you know what you are doing.  Worst case remove the .conf and restart.)

If you are advanced enough to edit an xorg.conf, the above should be a cakewalk and you shouldn’t complain about it.

Regardless, you should investigate ‘xrandr’ which makes it simple to do runtime adjustments.

If you are a newbie, look into a gui.  KDE has KRandRTray which makes controlling outputs and resolutions a breeze.  Don’t forget to toggle the output on with the Fn key if you are a laptop user.

Needless to say, Xorg is moving in the right direction.  Stop complaining about it.