Category Archives: Datacenter

Mirroring Fedora

Posted on by
3

Introduction

This post details setting up your own private mirror of Fedora’s repos.  There are many ways to do this, but this method is by far the best for heavy usage.  By using MirrorManager, clients in your IP range need no custom configuration.  Roaming laptop users automagically hit your mirror while on the premises, yet use the public infrastructure elsewhere.  Setup isn’t exactly hard, but it isn’t well documented so I’ll write about my experience here.

Some background info.. we have at least 50 Linux desktops, laptops, servers and VMs running about half Fedora 10 and half Fedora 11 at work.  Due to the number of systems, breadth of packages used, and desire to quickly update when new releases are out, I decided on a full mirror setup.  If you only have a handful of systems, you may be better off simply using a general purpose caching proxy like Squid, perhaps telling MirrorManager to point to it.

This guide should be used in addition to http://fedoraproject.org/wiki/Infrastructure/Mirroring which has some background info.

Initial setup and mirror

First, get prepared by installing MirrorManager-client, which contains the report_mirror script you will need.  If your mirror isn’t running Fedora, you can clone the source of this app from their GIT repo.

yum install mirrormanager-client

You’ll be using rsync, a sysadmin’s best friend, for efficient mirroring.

Set up a shell script like mine below (d0mirror.sh) one level up from where your mirror will be accessible (http, ftp, rsync, nfs – covered later).  This one mirrors against kernel.org.  Choose a mirror close to you on the Internet.

rsync -vaH --exclude-from=fedora-excludes.txt --numeric-ids --delete --delete-delay \
 --delay-updates rsync://mirrors.kernel.org/fedora-enchilada fedora-mirror
report_mirror

And a text file (fedora-excludes.txt) excluding things you don’t want/need.  Take a look through a public mirror and decide if you want to eliminate anything else.  You may want to remove the *.iso line below if you want users to be able to pull disc images from this box.  Otherwise, this is probably a good list for most people.  You can exclude all of linux/updates/testing/ if you don’t enable the testing repo on any of your machines.

**/debug/**
**/alpha/**
**/source/**
**/SRPMS/**
**/*.iso
**/ppc/**
**/ppc64/**
linux/core/**
linux/development/**
linux/releases/7/**
linux/releases/8/**
linux/releases/9/**
linux/releases/test/**
linux/updates/8/**
linux/updates/9/**
linux/updates/testing/7/**
linux/updates/testing/8/**
linux/updates/testing/9/**

Run your shell script and sit back for up to a day or two depending on your connection speed.  My current mirror weighs in at about 80G.

Internal distribution

While you wait for sync, decide how you want to run the service internally.  HTTP is nice because it is easy for users to browse and decently quick with keep-alive.   Using NFS, rsync, or FTP may be a bit more efficient if you are worried about this.  You can list several URLs in MirrorManager for the best of all worlds.

Add the following to your Apache configuration if you decide to use HTTP:

Alias /fedora/ "/mnt/ar1/fedora-mirror/"

AddType application/octet-stream .rpm

<Directory "/mnt/ar1/fedora-mirror">
    Options Indexes FollowSymLinks
    Order allow,deny
    Allow from all
</Directory>

<LocationMatch "\.(xml|xml\.gz|xml\.asc|sqlite)">
    Header set Cache-Control "must-revalidate"
    ExpiresActive On
    ExpiresDefault "now"
</LocationMatch>

Set up any other services of you choice to push that directory out in addition.

Working with MirrorManager client and server

Next, open up /etc/mirrormanager-client/report_mirror.conf.  Take notice of the site name, password, and host name.  You will need to set these up in MirrorManager in a bit.  The paths here are all local and used by report_mirror to check what you have available.

# if enabled=0, no data is sent to the database
enabled=1
server=https://admin.fedoraproject.org/mirrormanager/xmlrpc

[site]
# if enabled=0, no data about this site is sent to the database
enabled=1
name=<yoursitename>
password=<yourhostpassword>

[host]
# if enabled=0, no data about this host is sent to the database
enabled=1
name=x345-a2.internal
# if user_active=0, no data about this category is given to the public
# This can be used to toggle between serving and not serving data,
# such enabled during the nighttime (when you have more idle bandwidth
# available) and disabled during the daytime.
# not passing it means leave it alone in the database.

[stats]
# Stats are only sent when run with the -s option
# and when this section is enabled.
enabled=0
apache=/var/log/httpd/access_log
vsftpd=/var/log/vsftpd.log
# remember to enable log file and transfer logging in rsyncd.conf
rsyncd=/var/log/rsyncd.log

[Fedora Linux]
enabled=1
path=/mnt/ar1/fedora-mirror/linux

[Fedora EPEL]
path=/var/www/html/pub/epel
enabled=0

# lesser used categories below

[Fedora Web]
enabled=0
path=/var/www/html/pub/fedora/web

[Fedora Secondary Arches]
enabled=0
path=/var/www/html/pub/fedora-secondary

[Fedora Other]
enabled=0
path=/var/www/html/pub/alt

# historical content

[Fedora Core]
# if enabled=0, no data about this host is sent to the database
enabled=0
path=/var/www/html/pub/fedora/linux/core

[Fedora Extras]
enabled=0
path=/var/www/html/pub/fedora/linux/extras

Log into https://admin.fedoraproject.org/mirrormanager, creating a new account if you need to.  Add a new site with the same name as the config file from above.  You’ll set the site password here, and make sure to check the ‘private’ box if this is only for internal users.  Now, add a host under this site.  The name here should probably be a FQDN of your actual mirror, even if it is internal only (i.e x345-a2.internal from my example above).  Once that is done, add a “site-local netblock”.  This is your public IP network/netmask or network in CIDR notation.  If you only have one public IP, it will be in the format nnn.nnn.nnn.nnn/32.

Almost done.  Now, click Add Category.  “Fedora Linux” is the only one you are concerned with if you followed all the values in this guide so far.  Add the others if needed.  Tell them your upstream source (rsync://mirrors.kernel.org/fedora-enchilada from above) and then your internal URL (http://x345-a2.internal/fedora/linux for my setup).

Conclusion

Once your rsync is complete and report_mirror is done, you should see clients start hitting your box.   Don’t forget to add your mirror script (domirror.sh from above — rsync and report_mirror) to cron!  You may wish to join the private ‘fedora-mirrors’ mail lists to be informed of new releases and changes.

The best thing is that it works across all package requests, including new machines, roaming users,  ‘preupgrade’, etc.   All in all, pretty nifty!  Your users will love you when their upgrades are almost instant!  The Fedora infrastructure is set up very well for mirroring, public and private, and this is how the project copes with the huge demand for new releases.  Comment away if you need clarification or help.

Kernel 2.6.30 is a Go

Posted on by
Reply

I initially thought this would be a rather uninteresting release, especially when we learned Xen dom0 didn’t make the cut. Following the changelog line-by-line, this one still didn’t seem very interesting to me. But analyzing the sum of parts, I have to consider 2.6.30 a ‘golden’ kernel — certainly the best in a while.

There is solid improvement top to bottom here.  A lot of the new KMS/DRM stuff from Fedora 11 has worked its way up stream.  File system work is too much to mention, but highlights include relatime, writeback by default for Extfs, NILFS2, Btrfs development and more. FSCache works as advertised.  Also some groundwork for NFS 4.1, which will eventually bring us pNFS.

Boot speed seems fast as ever, but I haven’t taken the time to do any empirical analysis.  Your results here will be hardware dependent but async initialization of certain subsystems is a welcome move in the right direction.

Basically, a solid release with a good balance of new stuff but mainly refinement of existing systems and merging of longstanding patches.

Kernel Newbies has, as usual, a great change summary: http://kernelnewbies.org/Linux_2_6_30

Kernel developers don’t get Xen

Posted on by
7

The recent bruhaha surrounding Xen on LKML (http://lkml.org/lkml/2009/6/2/475) is really disheartening.  Essentially, the Linux kernel devs are at a disconnect with users.  Some are proposing narrow-minded ideas such as DROPPING software paravirt or merging Xen as a whole into the kernel.

I use Xen for a few primary reasons:  it bar none has the best speed — full software paravirtualization pays dividends here;  it is mature;  it works on perfectly good machines that don’t happen to have the latest chips;  it does hardware passthrough on these same systems;  it has great live migration that actually works.

Ingo Molnar wants you to send all your perfectly good enterprise iron to the landfill even though these systems will last 10+ useful years without boneheaded software decisions such as this.

These same FUDsters want to strip the crossplatform nature of Xen dom0 out too.  Xen dom0 runs on NetBSD and Solaris.  It is a true hypervisor and will plug into exisiting architectures, and not force you to use Linux for everything.

I have to admire all the hoops Jeremy Fitzhardinge has jumped through to date, as I know my patience is wearing thin.

Xen powers huge sites such as Amazon and services like linode.com/slicehost.com.  By not having dom0 in the kernel where distros such as Ubuntu and Fedora can easily integrate it, kernel devs are doing a disservice to users.

I use KVM, VMWare, and Virtual Box at work in addition, but Xen is firmly entrenched in my toolbox.  The roadmap they have looks great, and I just don’t see a reason for decline in Xen popularity.  High availability in Xen 4.0 is what I’ve always been waiting for.

Jeremy has gone to great lengths to work with upstream but keeps getting shot down and asked to do something else when he meets one requirement.  The solution is to merge Jeremy’s conservative dom0 patch set and work on a technical solution to the patches that the FUDsters consider bad.  It’s what the users want!

El Reg Humor and Java in free software

Posted on by
1

The Register has a good article on Sphinx search with some entertaining pop-shots at Java and “enterprise software” that got a rise out of me:

Solr is popular with the enterprise crowd, who love its Java. Being a Java program, Solr includes no shortage of technology whose acronyms contain the letters J and X.

This tickles the enterprise pink, because these sorts of developers love nothing more than hanging out around a whiteboard drawing boxes and arrows and, from time to time, writing XML to make it look like they’re doing real work. Solr thrives in this environment, being an Apache Foundation project, the Apache Foundation, of course, widely known as a cruel experiment to see what happens when bureaucrats do open source.

Having a bit of experience with Java from academia and a few open source projects I make use of, I can’t help but laugh at how comically and concisely the editor summed it up.

By and large, successful open source projects tend to be written in languages other than Java. The entire GNU/Linux OS stack is primarily C, with some components using C++ like KDE, OpenOffice and Firefox.  On the ever popular web front, PHP, Ruby, and Python lead the pack.

I think it turned out this way for a multitude of reasons.  When working on the OS stack, the power and control of C and C++ are hard to beat.  The plethora of libraries and raw speed of these compiled languages set the bar high for any newcomers.  Java exists as a kludge, mildly useful for desktop apps and mildly useful for web apps while historically having a lot of problems.  Native look and feel have long been the layman’s complaint, though SWT has done a pretty good job there.  Of course, omnipresent Java in the Linux world is relatively new.  I think Java would have been the darling language of client apps had it been open sourced sooner, but this came about 7 years too late to have a large impact on shaping the common FOSS userland.

It is interesting how the open source projects built with Java tend to be highly bureaucratic and abstract.  I think the bottom line is that FOSS programmers do what they do because it is fun and demand pragmatism.  The “enterprise software” attitude/baggage that many Java apps and libraries carry are a big turn off to pragmatism and the hacking culture.  The barrier to entry for Java web programming is also much higher than its “scripting language” competitors, which carry light and simple frameworks that focus on results, not procedure.

Java itself isn’t that of a bad language.  I actually enjoy working with it in school (…though I think it really isn’t appropriate as an introductory teaching language, shielding important concepts from students.  Maybe a future post?..).  When it comes time for real work though, I consider Python, C,  C++ more pragmatic depending on the job at hand.  That, and the fact that most of the common scripting languages are gaining JIT compilers may accelerate Java toward status as a legacy language.

Your thoughts?

FS-Cache merged in Kernel 2.6.30

Posted on by
2

FS-Cache has been merged into the upcoming kernel 2.6.30.  This allows for a generic caching interface in the kernel for other file systems.  For example, you can use local hard disks to cache data accessed via NFS, AFS, or CD-Rom.  Since these tend to be high-latency while the disks are low latency, it should provide for a nice speedup.

Of particular interest to me, I contacted maintainer David Howells who is a Redhat employee.  I asked whether this infrastructure would help with large disk image files stored on NFS — a common though not particularly efficient case for VMWare, Xen, KVM, etc.  His exact response was “Quite feasible.  As long as you have a local disk on which to cache the files.”

I am quite happy as I run this setup at work for some production VMs since it allows for easy migration and backup without the complexity and cost of a SAN or cluster FS.  I look forward to testing when 2.6.30 hits the stable tree.

Good Linux File System Developments

Posted on by
Reply

ext4 has sparked good controversy on the LKML. Aside from the recent delayed alloc and fsync issues, the whole FS stack is getting some much needed attention.  Indeed, Linux file systems are starting to feel like first class citizens again with ext4 and Btrfs (merged in 2.6.29 for testing!) and the surrounding infrastructure being worked on.  A lot of long overdue problems are being mitigated.  Jens Axboe claims 8% single drive and 25% array speedup with some recent pdflush patches.  This is very good news for all users since disk I/O has had a fast growing gap with CPU and main memory bandwidth, even with SSDs.  The fruits of this labor are quite visible with recent boot speedups in distros like the upcoming Fedora 11.

Mandatory reading:

I dream of pervasive virtualization…

Posted on by
2

I dream of a day where virtualization is pervasive.

Instead of thinking about services in terms of servers, CPUs or directly mapped resources, I should be able to to add virtual machines in terms of guaranteed throughput rate over a whole grid.  Scaling out should be as easy as adding a blade or racking another server.

At the low level, I should have the option of running N+N redundancy.  That is, the VM should run in lockstep across multiple machines – so if it is running on 2 vcpus, 4 in total would be used.  This would allow for any node to fail.  And the VM should be an aggregate of the low level hardware – e.g. a VM grid across 4 8-core servers should scale near-linearly when a single OS instance is running 32 processes.

Current solutions only attempt to do some of the tasks above, and most fail miserably.  IBM mainframes have been doing it for ages.

If I had the time, I know I could build software to do this better than anyone else.  All the puzzle pieces are there, especially the tough ones like hypervisors and Infiniband.  This could have been done at least 3 years ago.  I bet it will take the industry 3-4 years yet to get anywhere close.

This is a real virtual datacenter.

Xen 3.3 in RHEL/CentOS 5 and more Link Aggregation Fun

Posted on by
2

RHEL 5 includes the now ancient Xen 3.0 hypervisior.  A lot has been improved since then, especially in the current 3.3 release.  Additionally, RedHat now owns the company behind KVM, so it is unlikely they will spend much time backporting Xen stuff for RHEL 5.3 or the likes.

Why Xen?

Xen is a proven hypervisor.  It works well on lots of hardware, including servers without hardware virtualization and older 64-bit Opterons that wont run 64-bit guests in the likes of VMWare.  Since the OS is usually paravirtualized, performance is top notch.  By making an OS aware of the environment it is running in, you can optimize it for virtualization.  KVM is playing catchup here, realizing that paravirtualization is still ideal for many things.

How..

Okay, so we are using or want to use Xen. Others have already built the packages we need, thankfully!

Head over to http://www.gitco.de/repo/ and grab the repo for your arch.  (Most likely wget http://www.gitco.de/repo/CentOS5-GITCO_x86_64.repo in /etc/yum.repos.d/ for the uninitiated).

If you already have Xen installed, you may need to remove and readd it.

yum groupremove Virtualization
yum groupinstall Virtualization

You’ll also get some updated tools like Virtual Machine Monitor 0.6.0 that make it easier to install newer guests such as Fedora 10 or Ubuntu.  Sweet!

Double check /etc/sysconfig/kernel.  It should be set to kernel-xen.  Likewise, check /boot/grub.conf and make sure that the Xen kernel is the default if the aforementioned was not done beforehand.

Reboot!

Xen 3.3 and Link Bonding

See my previous post for general information, but it gets harder.

This one is a nightmare.  In my previous post, I detailed how to get Xen to work with link aggregation with Xen 3.0.  Well, it doesn’t work in 3.3.  Xen decides that it still owns eth0 and completely destroys your bond0 setup.

Like these people, I’ve come to the conclusion that the integrated network scripts suck.  This is alarming since you’d think link bonded setups would be the norm for Xen setups.

The quick fix is to let the OS handle networking.  We do that like so: add a br0 interface and tell the bond to bridge with it.

File /etc/sysconfig/network-scripts/ifcfg-br0

DEVICE=br0
ONBOOT=yes
BOOTPROTO=none
IPADDR=10.0.6.201
NETMASK=255.255.255.0
GATEWAY=10.0.6.1
NO_ALIASROUTING=yes
TYPE=Bridge

Then, edit your /etc/sysconfig/network-scripts/ifcfg-bond0 and add “BRIDGE=br0″ and comment out any IP related information (since you are now defining that in the bridge.  Head over to /etc/sysctl.conf and add:

net.ipv4.ip_forward = 1

Now, edit your Xen VMs in /etc/xen/ or /etc/xen/auto and change xenbr0 to br0:

vif = [ ‘mac=ee:cc:aa:88:66:44, bridge=br0′, ]

Okay, now disable the Xen networking garbage.  Open /etc/xen/xend-config.sxp and comment out anything  that looks like (network-script ….).

Almost done, but wait!  RHEL 5.2 has a bug that prevents the bridge coming up on a bonded interface.  Hopefully this will make the 5.3 cut or be pushed to 5.2, but until then go here.  Download the new patch into /etc/sysconfig/network-scripts/ and run patch -p0 < ifup-eth.patch for instance.

Finish

Reboot.  You now have Xen 3.3 goodness on a big Ethernet channel!  Post a comment if you have any trouble or questions.