Kernel developers don’t get Xen

The recent bruhaha surrounding Xen on LKML (http://lkml.org/lkml/2009/6/2/475) is really disheartening.  Essentially, the Linux kernel devs are at a disconnect with users.  Some are proposing narrow-minded ideas such as DROPPING software paravirt or merging Xen as a whole into the kernel.

I use Xen for a few primary reasons:  it bar none has the best speed — full software paravirtualization pays dividends here;  it is mature;  it works on perfectly good machines that don’t happen to have the latest chips;  it does hardware passthrough on these same systems;  it has great live migration that actually works.

Ingo Molnar wants you to send all your perfectly good enterprise iron to the landfill even though these systems will last 10+ useful years without boneheaded software decisions such as this.

These same FUDsters want to strip the crossplatform nature of Xen dom0 out too.  Xen dom0 runs on NetBSD and Solaris.  It is a true hypervisor and will plug into exisiting architectures, and not force you to use Linux for everything.

I have to admire all the hoops Jeremy Fitzhardinge has jumped through to date, as I know my patience is wearing thin.

Xen powers huge sites such as Amazon and services like linode.com/slicehost.com.  By not having dom0 in the kernel where distros such as Ubuntu and Fedora can easily integrate it, kernel devs are doing a disservice to users.

I use KVM, VMWare, and Virtual Box at work in addition, but Xen is firmly entrenched in my toolbox.  The roadmap they have looks great, and I just don’t see a reason for decline in Xen popularity.  High availability in Xen 4.0 is what I’ve always been waiting for.

Jeremy has gone to great lengths to work with upstream but keeps getting shot down and asked to do something else when he meets one requirement.  The solution is to merge Jeremy’s conservative dom0 patch set and work on a technical solution to the patches that the FUDsters consider bad.  It’s what the users want!

To users that miss xorg.conf and complain about it

I get requests from users and see questions all the time for “where did my xorg.conf go in the latest Ubuntu or Fedora?”, though it is usually a bit more of a flame.

The quick answer… press Ctrl+Alt+F2 or similar to log into a TTY console, or type ‘init 3′ into a root X terminal.

If you haven’t already, log in as root and  kill X or type ‘init 3′ if you want to be heavy handed.  Then run:

X -configure
mv ~/xorg.conf.new /etc/X11/xorg.conf

xorg.config in two commands.  Run the ‘init 5′ command to get back to your GUI login (or kdm or gdm or startx, etc if you know what you are doing.  Worst case remove the .conf and restart.)

If you are advanced enough to edit an xorg.conf, the above should be a cakewalk and you shouldn’t complain about it.

Regardless, you should investigate ‘xrandr’ which makes it simple to do runtime adjustments.

If you are a newbie, look into a gui.  KDE has KRandRTray which makes controlling outputs and resolutions a breeze.  Don’t forget to toggle the output on with the Fn key if you are a laptop user.

Needless to say, Xorg is moving in the right direction.  Stop complaining about it.

Arora, a refreshing new Qt/WebKit browser

The Gentoo Qt maintainers have been doing a fantastic job of getting cutting edge Qt software into shape with the qting-edge overlay.  I’ve been running Qt 4.5 since beta1 and am pleased with the direction it is going.  Recently the devs bumped the Arora ebuild to version 0.5.  Arora is a lightweight browser based on Qt and WebKit.

I must say, I am impressed.  This browser is lightning fast.  The interface is pleasing, yet minimalist.  You get all the benefits of recent WebKit, and the nice cross-platform nature of Qt.  It feels much faster than Konqueror4.2, and the recent WebKit engine means better compatibility.

It works with all the sites I’ve thrown at it so far and performs great on the Acid 3 test.

Arora 0.5 Acid3 Test - 100/100

Arora 0.5 Acid3 Test - 100/100

All in all it reminds me of Firefox’s early days.  Very mean and lean.  Firefox 3.1 beta2 has served me pretty well, so I don’t think I’ll abandon it yet, but WebKit browsers are becoming quite compelling.

The WebKit devs are also pushing some interesting new ideas out with CSS animation.  This example gives Flash-esque falling leaves.  Maybe we will see less reliance on Flash in the future with the new video/audio tags and work such as this.  It would be nice to move presentation back into structured [X]HTML and CSS, which is portable, easier to parse, and light weight.

Leaves CSS Animation

Leaves CSS Animation

If you are on Gentoo, bump to Qt 4.5rc1 and take a look easily (both are in portage):

emerge -av arora

I’ve seen it hit Fedora Rawhide recently as well.

Only one problem: icon is a bit conspicuous  :-P

Arora Icon

I hate Ubuntu

I hate Ubuntu.  I immediately lose respect for anyone who runs it, and especially those who advocate it.   Here’s why:

Name 20 features, release-for-release or year-for-year that have not come from Redhat.  Redhat basically runs the show when it comes to Linux.  This includes things like NetworkManager, Gnome, Xorg, GCC, glibc, LVM, KVM, kernel, file systems et al.  Redhat has developers making significant contributions to the entire FOSS software stack upstream.

Ubuntu on the other hand pulls most of the heavy weight packaging from Debian with each release.  They then perform minor patching and testing.  It generally lags behind Fedora by a release or two in parts of the software stack.  I never see @ubuntu or @cannocial email addresses in upstream changelogs.

So tell me again, how exactly does Ubuntu innovate?  They even struggle to release a new theme with each release, and artwork is about the only original thing in Ubuntu.

Yes, Ubuntu is stable because they are standing on the shoulders of giants.  Most of the hard work is hashed out before they ever import software into their repositories.  This is fine, and what FOSS is all about, but I prefer to be in with the leaders rather than the followers.

What really irks me and what has really brewed my hatred are Ubuntu users.  They seem to think Ubuntu is responsible for all that is good in the FOSS world.  I have just proven how false this is.  In my experience, Ubuntu support mechanisms (IRC, mailing lists, forums) are much less helpful than the alternative.

If you want a nice desktop distro, run Fedora or OpenSUSE.  If you like control, run Gentoo.  If you need stability, run RHEL/CentOS or Debian.  But please, don’t feed the idiot magnet that is Ubuntu!

KDE 4.2 on Gentoo

KDE 4.2 is out officially.  The ebuilds for Gentoo have been ready for a while.  This is a truly fantastic release.  If you’ve ever made an opinion about KDE in the past, I encourage you to give it another go.

My beta1 review back in December sums up most of my thoughts on the release.  There isn’t anything significantly changed since, just lots of polish and bug fixing.  Everything has been stable and functional since I’ve been using it in the RC phase.   This is a worthy opponent to KDE 3.5, GNOME, Windows and OS X.

Thanks again to the Gentoo KDE team.  The ebuilds are in great shape!

One Small Step for QT, One Giant Leap for Free Software

QT Software, under the graces of Nokia, has released the superb QT cross-platform toolkit under the LGPL.

This. is. HUGE.

For the libre software purist, this still benefits you, if indirectly.  Companies that make changes to the toolkit must still submit patches.  More influential, GPL incompatible software may now readily use QT for free.  This will likely foster more QT centric developers, boost adoption of the underlying stack (Linux, etc), and lower the barrier for vendors to release cross-platform tools.

From a Nokia business perspective, it makes perfect sense and makes the whole thing that much more beautiful.  “QT Everywhere” is really a possibility now.  And, it’s beneficial to Nokia as well as the ecosystem they are enriching.  The more QT developers, the bigger the talent pool for Nokia software.  The more contributors, the better the toolkit.  Win.  A small company like Trolltech could not afford to do this, but to a big dog like Nokia, the revenue from commercial licensing is insignificant and unimportant compared to device sales.

I know the company I work for, Analog Rails, will be able to take advantage of the license switch.  Being previous commercial QT customers, it was expensive to juggle around machines to maintain compliance.  For a companies like VMWare that deploy cross-platform software and maintain their own cross-platform extensions, this surely must be compelling.  I say, the more the merrier!

What a great day for free software, computing, and life in general :-) .

Ars Technica has outstanding coverage of the news: http://arstechnica.com/news.ars/post/20090114-nokia-qt-lgpl-switch-huge-win-for-cross-platform-development.html

Political correctness in open source doesn’t matter

See: http://www.itwire.com/content/view/22467/1090/.  To answer the sensationalist title, no.

I can’t believe people are trying to make this an issue.  I guess it was only a matter of time before the crap that we deal with in the rest of the world met up with open source.

I have seen several prominent developers on just the kernel that just happen to be women.  Many more on large projects like KDE.  Great, big deal.  They shouldn’t receive special privileges, recognition, or anything because of it.

“The strange thing about this episode is that it looks like the FOSS community seemingly doesn’t want to know about it.”
No shit.  This kind of crybaby attitude is why governments and large corporations can’t get anything done, too worried about offending people.  Most of us FOSS people are here for the goods, not to set up bureaucracy, politics, and political correctness.

This Gentoo developer is spot on: http://steveno.wordpress.com/2008/12/17/mad-gnu-women/.  The existence of women’s only groups like Debian women are wrong in the first place and harmful.  In the same category as Richard Stallman – well intentioned but counter-productive.

Isn’t the goal to write and use good software?  Gender has nothing to do with that.  Neither does race, color, or being the stereotypical guy that spends countless hours hacking away in the parent’s basement.  Yet we think nothing about laughing at the last.  The world would be a better place if people just grew thicker skin.

How to upgrade to ext4 in place

Here’s how you upgrade to ext4.  The process is pretty easy, but requires an fsck which means unmounting or rebooting if the file system is in use.

Make sure you are using at least e2fstools 1.41.3 and kernel 2.6.28 (or a vendor kernel with latest ext4 patches applied)!  Also, its probably a good idea to have proper backups (really!).  ext4 has just been declared stable, but what that really means is that the battle hardening has just begun.  I’ve done several heavily used systems without fault so far though, so its probably good enough for your desktop.

WARNING: DON’T CONVERT YOUR /boot PARTITION. Right now, there is no stable version of grub with ext3 support.  Even if there was, it really won’t gain you anything  :-) .

Run tune2fs, e.g.:

tune2fs -I 256 -O sparse_super,filetype,resize_inode,dir_index,ext_attr,has_journal,\
extents,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize /dev/sd[x][n]

Those are the default options for an ext4 file system if you were to create it with mkfs.ext4 (e2fsprogs 1.41.3 – see /etc/mke2fs.conf).  I’m getting pretty damn good performance with this!  The ‘-I 256′ option sets 256 bit inodes, which most recent ext3 FSs use already. If this is the case, and you get a message telling you so, remove this option.  Note that extents will make the FS backwards INCOMPATIBLE with ext3.

Next, edit /etc/fstab, e.g.:

/dev/vg/home /home ext4 defaults 0 0

Either unmount and mount or reboot your system.  tune2fs marks the fs as dirty and performs a fsck and conversion.
NOTICE: distros with initrds may need to be regenerated or you won’t be able to mount your root file system.  In Fedora (replace kernel version with your own):

cd /boot
mv initrd-2.6.27.7-134.fc10.i686.img initrd-2.6.27.7-134.fc10.i686.img.old
mkinitrd initrd-2.6.27.7-134.fc10.i686.img initrd-2.6.27.7-134.fc10.i686.

That’s all there is to it.  Stay tuned for future ext4 developments like online defragmentation.

Also, ext{2,3,4} reserve 5% of space for root in case the drive fills up.  On large modern drives, this can be excessive (e.g: 50GB on a 1TB disk).  Consider running ‘tune2fs -m 1 /dev/sd[x][n]‘ to reduce this to 1%.

For more information and tweaking:

  1. Documentation/filesystems/ext4.txt from the latest kernel sources
  2. http://ext4.wiki.kernel.org/index.php/Main_Page
  3. man tune2fs
  4. http://e2fsprogs.sourceforge.net/

Linux and the PC Mentaility

Here’s a minor thought dump of the moment: no hard numbers, just some general observations over the past year or two.

I’ve heard the comparison between Linux and other UNIX like OSes a few times in the past.  Of course it is a bit hard to compare “Linux” – which is a kernel – and the others which most often are complete operating systems.  Many times they even share a lot of GNU userland tools, so I will disregard userland.

The general feel I get from looking at some parts of Linux and its development process is that of the PC world.  Build cheap, throwaway solutions and rip and replace later.  Looking at things like the scheduler, FreeBSD was able to grow similarly advanced solution in a much more organized fashion without ripping and replacing this critical component several times and has far less developers.  *BSD has a more traditional UNIX process from the days of the minicomputer and professional workstation, while Linux is deeply rooted in PC culture.

Aside from the scheduler, there seem to be other places where development has tarnish before shine.  hrtimers, dyntick, CFS, and SLUB are some of the more controversial moves in the past year or so.  Indeed, kernels 2.6.23+ didn’t “feel” as polished as the few leading up to it and “felt” less responsive to me when playing audio or video concurrent with other tasks like compiling – something CFS was supposed to fix, but I never had a problem with before.  Going into 2.6.28, these big changes are starting to settle down.  Things once again “feel” better, more stable, and better performing – so I guess it is just a phase.

From a software engineering standpoint, it is interesting to ponder how to improve the process.  One thing Linux has that many other projects don’t is an enormous amount of contributors.  A lot of people and companies are trying to different things with the kernel.  It is important for them to get their code into the kernel as soon as possible, because it is such a fast moving process, but new features lead to bugs.  Linux-next appears to be helping with keeping in progress trees so that developers can explore ideas before pushing to mainline, and Adrian Bunk’s idea of the “long time support” kernels for those of us that don’t run large distro kernels seem like a step in the right direction.  I do wish he would forgo 2.6.27 and base it on 2.6.28 for the next LTS though.  2.6.27 has had quite a lot of major patching done since release and as I said earlier, 2.6.28 just looks better so far.

Another point: technically superior solutions don’t always get favored.  I’ve said before that I think there were better choices for file systems than ext3 for the past 8 or so years.  CFS vs. RDSL is another example.  The latest is SCST vs. SGST (http://lwn.net/Articles/311344/).  SCST is clearly the only solution for the enterprise where Fibre Channel and Infiniband are likely to be used.  And Ingo and Thomas are at it again reinventing the wheel with performance counters.

The counterargument is that Linux runs on most of the worlds supercomputers all the way down to small critical embedded systems.  Most of these are running obsolete kernels with heavy vendor backports though.

Running the latest kernel is fun; after all I am a Gentoo user so I don’t mind when things break, but for truly mission critical tasks I would not be afraid to look outside of Linux to BSD.  Ironically, Solaris, once the stalwart of enterprise, seems to be embracing the Linux mentality.

Phoronix Benchmarking.. Statistically Significant? and Other Performance Concerns

Phoronix has been cranking out a slew of benchmarks recently, pitting various different Linux distros against each other and even different operating systems with their own automated test suite.

  1. Ubuntu 7.04 to 8.10 Benchmarks
  2. Mac OS 10.5 vs. Ubuntu 8.10 Benchmarks
  3. Ubuntu vs. OpenSolaris vs. FreeBSD benchmarks
  4. Fedora 10 vs Ubuntu 8.10 Benchmarks
  5. “Real World” Benchmarks of the [sic x2...]EXT4 File-System
  6. OpenSolaris 2008.05 vs. 2008.11 Benchmarks

What I would like to know is… are they bullshit?  I’m no statistician, yet the proximity of the numbers and lack of error bars raise my own bullshit detection meter.  See this URL for some background on statistical significance and error bars: http://www.graphpad.com/articles/errorbars.htm.

I spot plenty of outlandish things, such as FPS benchmarks for file system tests and JDK version changes through product life cycles, also to mention the somewhat unfairness of crappy binary graphics drivers across versions.  JDK 1.6.10 is SUPPOSED to be faster; results from benchmarking against this are insignificant unless it is run across all versions.  Yes.. GCC, glibc, and the kernel change between releases as well but these are not typically components that a user can swap out as easily as a JVM which should probably be bumped for security reasons anyways.

Furthermore, I realize all benchmarking should be taken with a grain of salt – one particular set of hardware and software will never map correctly to another set of hardware or software, but it should be possible to set up tests to gain some useful intelligence.

Can this kind of macro/micro benchmarking (depending on how you look at it) help weed out regressions?  GCC 4.0 was noticeably slower on x86 than 3.x (See: http://www.coyotegulch.com/reviews/gcc4/index.html, http://people.redhat.com/bkoz/benchmarks/).  At the same time I think PowerPC saw significant improvement due to auto vectorization and use of Altivec/VMX.   But it also seems to be improving over time.  I’ve heard 4.4 is supposed to be much better with a new register allocator (IRA).  This probably the most important component of modern open source operating systems, so some of the blame might be placed here if the numbers have meaning.

All of this makes LLVM look more and more appealing.  LLVM is able to do not only compile time, but also link and run time optimization.  This is very appealing for commercial software where you are given a binary blob by the manufacturer that will likely that will not change through its lifetime.  It also reminds me of Java and speedup through JVM upgrades, except this should work on any language.

“LLVM is… designed to enable effective program optimization across the entire lifetime of a program. LLVM supports effective optimization at compile time, link-time (particularly interprocedural), run-time and offline (i.e., after software is installed), while remaining transparent to developers”

One thing the Phoronix numbers do show is that things seemed to go down hill coincident with CFS (Completely Fair Scheduler), dyntick, and SLUB merging as well.

Evgeniy Polyakov of POHMELFS fame raised the alarm with some fairly significant networking regressions – how financial crisis affects tbench performance – that seem to support a general slowdown between 2.6.22 and 2.6.27.  This resulted in noise on LKML and hopefully we will see improvements soon.

I guess what I am getting at is that compute power is so cheap that it seems stupid to not have automated tests against such things these days.  Diego Petteno of Gentoo fame has been doing such things recently with Gentoo’s excellent build system.  I have set up Hudson, a Java Continuous Integration system, before to track commit regressions and such a system seems ideal for all modern software testing.

Anyways, I am interested in hearing your thoughts on benchmarking, software testing, and automation and how it can be used to improve modern software.