• Software Freedom Day 2010 – Gentoo Linux Presentation – .odp / Open Office Impress
• Software Freedom Day 2010 – Gentoo Linux PDF – .pdf / Portable Document Format

Here is the presentation.  Feel free to use or modify this under the terms of CC-SA3.0.

No related posts.

Unfortunately, IBM’s “transmogrifier” utility doesn’t work very well in Wine, so you should preprocess older books in Windows before running them through the boo2pdf web service (download is on that page).

Source code is now available from boo2pdf gitweb.

No related posts.

Motivation

I have a large collection of old IBM machines and documentation. I want this documentation indexed by my own search facilities and Google for easy retrieval. PDF is widely read, while BookManager requires proprietary software and no search engines I know of parse it. This will probably be useful to Mainframers as well.

Take the web service for a spin here:

http://ps-2.kev009.com:8081/boo2pdf/.

No related posts.

I aim to do the same by showing “Java: The Good Parts” here at a high level.

When I was younger, I used to despise Java for political reasons and bad memories of early applets and applications.  I suspect many users and developers (especially Libre software devs!) are on the same boat.  By the end of this article, I hope I swayed your opinion or at least caused you to reevaluate your bias.  I also wish to encourage further discussion about these points and ways we can improve any deficiencies.

Rough and Tumble Upbringing

When Java first started gaining popularity, it was loudly hyped as the end all language.  It was expected that Java would take the “rich client” by storm, and applets would be the go to solution for enhancing web pages.  What happened was a bit different.  Java floundered and struggled to find a niche.  On the client side, AWT apps looked horrendous despite using native widgets.  Then Swing came about and despite easing development, it looked equally bad on all platforms (by default).  Applets were basically a stillbirth.  The ugly gray box, loadtime sometime measuring in minutes, and no coordination with the DOM and web browser made the average user hate Java.

One area Java was able to develop and secure a foundation, however, was the back end of large web applications.  The Virtual Machine approach provided a marked advantage over the CGI and interpreted scripts of the day.  Java’s rich networking libraries, clean Object Oriented design, and safety made this the language de jour for large web applications.

Open Source Matters

In my opinion, the open sourcing of Java during its early infancy would have had little impact on most of the teething issues.  The Virtual Machine, JIT, and Garbage Collection required many years of tuning to get acceptable performance and Sun did an acceptable job keeping it under wing.  The relatively limited CPU and RAM of the mid ’90s also made these concepts a bit ahead of their time.  Somewhere in the 1999-2002 time frame, though, Sun really dropped the ball.  An Open Source Java would have led to ubiquity on the booming Linux platform and a chance for all sorts of cross-platform software.

Open Source matters, and not just for the source code.  Open Source projects naturally bring about very pragmatic and intelligent developers.  These are the folks that thoroughly enjoy their hobby, work, and tools.  The marketing guys and pointed haired bosses have much less pull here.  On one hand, a vibrant community built itself with the many Apache Software Foundation projects.  However, most of these were squarely focused on web applications or low level things such as build tools, testing frameworks, and message buses.

Due to the void, interpreted languages such as Python rose to the challenge while C and C++ remained the mainstay for applications programming.  Microsoft started dominating Windows development with their .NET CLR languages.  The glib/Gtk+ and Qt toolkits brought about a renaissance in cross-platform development with C and C++ respectively [though not limited].

It wasn’t until the open-sourcing of SWT that GUI development in Java became attractive.  The obvious killer apps here were the Eclipse IDE and the Azureus (now Vuze) bittorrent client.

Sun’s closed grip of Java really stagnated any chance of abundant expansion in these middle years (2001-2006).  Microsoft leveraged this weakness to create the excellent .NET platform and associated languages to maintain their closed platform and market dominance.  The counterbalance that would have been Java was thus left playing catchup.

We are just beginning to see the fruits of this labor from 2006 through today.  The OpenJDK project is now distributed with popular Linux distributions such as Ubuntu and Fedora.  We finally have decent browser plugins and Java Web Start applications across 32 and 64-bit machines.  The Java deployment problem will slowly fade from memory.

Application Development

SWT made Java apps beautiful.  OpenJDK should make them ubiquitous.  We finally have an Open Source platform that is widely deployed.  The strong built in standard library and clean OO design patterns of Java make it a very pleasant host for developing rich client apps.  Obvious areas for improvement here include better layout/form design tools and closer integration with upstream Linux distributors.

Somewhere along the line, Trolltech/QT Software (now owned by Nokia) released Jambi — the complete Qt bindings, GUI framework, and incredibly rich library — for Java.  Oddly, this bombshell received little of the community and fanfare I thought it would or deserves.  Indeed, QT Software demoted Jambi from their teir-1 platforms and hopes the community will pick it up.  I hope this project isn’t allowed to stagnate as there is a lot of potential here.

Web Apps

Along the “Enterprise Web Application” lineage of Java, we wound up with some disgustingly overcomplicated and bloated frameworks for building web apps.  Ruby on Rails and Python Django came about and put a new spin on the development of rapid and robust web apps.  The learning curve of these frameworks is much less than Java EE and I will go as far as saying they are more capable because of it.

By using Java, JSP, and Servlets directly on top of a light Model-View-Controller, I believe Java is just as compelling as some of the more popular scripting languages.  Developers need to know they can trim the fat and that there are many advantages to developing in Java, namely because of the next topic…

Dynamic Languages, its all about the VM

It’s all about the JVM stupid!  One of the best features of Java and .NET are the underlying Virtual Machines.  By using JIT compiled VMs, Java code has a distinct advantage over the common interpreted languages such as Perl, Python, and PHP.  In the case of Java, the resultant is even naturally crossplatform.

The really interesting developments here focus on extending the JVM to syntax and paradigms other than the statically typed C++ lookalike.  Clojure and Scala deliver innovative new techniques while Jython and JRuby bring these excellent languages to the Java software platform and virtual machine.

In short, Java provides everyone with a counter to Microsoft’s .NET CLR.  The Java VM has been around the block and tuned by giants such as Sun, IBM, Oracle, SAP and more.

I call on the community to discuss how we can encourage use of the JVM for languages other than Java and build this into a defacto runtime.  Continued tuning and integration with Windows, Mac OS X, and Gnome/KDE *NIX systems is paramount.  Research for easy multi-core development is also worthwhile.  Meanwhile, distributions need to continue packaging the JRE and make it a default.  Individual developers need to be made aware of “Java: The Good Parts” and myths debunked.

Applets, Rich Media, Native Code!?

With the release of Java FX, widespread deployment of the JRE, and better browser integration, Java has set the stage for a comeback to its roots.

During the late ’90s and 2000s, Adobe Flash became the tool of choice for web animation and interactive pages.  It really exploded with the advent of Youtube and other internet streaming sites making use of the Flash video format.  Unfortunately, Flash player is notoriously insecure, resource intensive, and crash-prone.  It is also not widely available for the millions of smartphones that have become more accessible than computers.

Luckily, there seems to be a shift back to the browser with new developments in AJAX, JavaScript, and HTML.  The <video> tag will hopefully make video as easy and portable as graphics are today in the browser.  Clean JavaScript libraries and fast JIT JS engines make it practical to use this paradigm for many domains.

Yet one must acknowledge that somewhere along the line, manipulating a DOM/markup language with a scripting language isn’t the most effective development platform for everything.  Google even thinks it poignant to run x86 machine code in a sandboxed environment in your browser.  I personally fail to see the logic behind this.  Java provides a well evolved, cross-platform solution.   Java can run on your ARM powered Android.  Requiring an x86 CPU just seems like the wrong track in this modern age.  Hopefully JavaFX will pick up the slack and return Java to its roots.  I would love to see the demise of the terrible Flash plugin.

Future and Conclusion

It’s time we considered Java for The Good Parts.

I hope some of my points caused you to reevaluate any bad preconceptions or past experiences you may have had with Java.  Java has undergone great change since its birth and I think it is capable of becoming the premier development platform for applications programming of all types.  Particularly interesting are some of the new languages such as Scala and Clojure.  Java has long been a staple in web development, but has traditionally scared away amateur coders.  If you cut the fat, Servlets and JSP are not much harder to set up than common place scripting languages.  Frameworks such as Grails bring it to parity with Rails or Django.

Java underwent a sea change in 2006 with the releasing of the source code and opening of the development process.  Java and the JVM should be championed by Libre software developers and users alike!

No related posts.

I use Xen for a few primary reasons:  it bar none has the best speed — full software paravirtualization pays dividends here;  it is mature;  it works on perfectly good machines that don’t happen to have the latest chips;  it does hardware passthrough on these same systems;  it has great live migration that actually works.

Ingo Molnar wants you to send all your perfectly good enterprise iron to the landfill even though these systems will last 10+ useful years without boneheaded software decisions such as this.

These same FUDsters want to strip the crossplatform nature of Xen dom0 out too.  Xen dom0 runs on NetBSD and Solaris.  It is a true hypervisor and will plug into exisiting architectures, and not force you to use Linux for everything.

I have to admire all the hoops Jeremy Fitzhardinge has jumped through to date, as I know my patience is wearing thin.

Xen powers huge sites such as Amazon and services like linode.com/slicehost.com.  By not having dom0 in the kernel where distros such as Ubuntu and Fedora can easily integrate it, kernel devs are doing a disservice to users.

I use KVM, VMWare, and Virtual Box at work in addition, but Xen is firmly entrenched in my toolbox.  The roadmap they have looks great, and I just don’t see a reason for decline in Xen popularity.  High availability in Xen 4.0 is what I’ve always been waiting for.

Jeremy has gone to great lengths to work with upstream but keeps getting shot down and asked to do something else when he meets one requirement.  The solution is to merge Jeremy’s conservative dom0 patch set and work on a technical solution to the patches that the FUDsters consider bad.  It’s what the users want!

No related posts.

This. is. HUGE.

For the libre software purist, this still benefits you, if indirectly.  Companies that make changes to the toolkit must still submit patches.  More influential, GPL incompatible software may now readily use QT for free.  This will likely foster more QT centric developers, boost adoption of the underlying stack (Linux, etc), and lower the barrier for vendors to release cross-platform tools.

From a Nokia business perspective, it makes perfect sense and makes the whole thing that much more beautiful.  “QT Everywhere” is really a possibility now.  And, it’s beneficial to Nokia as well as the ecosystem they are enriching.  The more QT developers, the bigger the talent pool for Nokia software.  The more contributors, the better the toolkit.  Win.  A small company like Trolltech could not afford to do this, but to a big dog like Nokia, the revenue from commercial licensing is insignificant and unimportant compared to device sales.

I know the company I work for, Analog Rails, will be able to take advantage of the license switch.  Being previous commercial QT customers, it was expensive to juggle around machines to maintain compliance.  For a companies like VMWare that deploy cross-platform software and maintain their own cross-platform extensions, this surely must be compelling.  I say, the more the merrier!

What a great day for free software, computing, and life in general :-) .

Ars Technica has outstanding coverage of the news: http://arstechnica.com/news.ars/post/20090114-nokia-qt-lgpl-switch-huge-win-for-cross-platform-development.html

No related posts.

Instead of thinking about services in terms of servers, CPUs or directly mapped resources, I should be able to to add virtual machines in terms of guaranteed throughput rate over a whole grid.  Scaling out should be as easy as adding a blade or racking another server.

At the low level, I should have the option of running N+N redundancy.  That is, the VM should run in lockstep across multiple machines – so if it is running on 2 vcpus, 4 in total would be used.  This would allow for any node to fail.  And the VM should be an aggregate of the low level hardware – e.g. a VM grid across 4 8-core servers should scale near-linearly when a single OS instance is running 32 processes.

Current solutions only attempt to do some of the tasks above, and most fail miserably.  IBM mainframes have been doing it for ages.

If I had the time, I know I could build software to do this better than anyone else.  All the puzzle pieces are there, especially the tough ones like hypervisors and Infiniband.  This could have been done at least 3 years ago.  I bet it will take the industry 3-4 years yet to get anywhere close.

This is a real virtual datacenter.

No related posts.

Make sure you are using at least e2fstools 1.41.3 and kernel 2.6.28 (or a vendor kernel with latest ext4 patches applied)!  Also, its probably a good idea to have proper backups (really!).  ext4 has just been declared stable, but what that really means is that the battle hardening has just begun.  I’ve done several heavily used systems without fault so far though, so its probably good enough for your desktop.

WARNING: DON’T CONVERT YOUR /boot PARTITION. Right now, there is no stable version of grub with ext3 support.  Even if there was, it really won’t gain you anything  :-) .

Run tune2fs, e.g.:

tune2fs -I 256 -O sparse_super,filetype,resize_inode,dir_index,ext_attr,has_journal,\
extents,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize /dev/sd[x][n]

Those are the default options for an ext4 file system if you were to create it with mkfs.ext4 (e2fsprogs 1.41.3 – see /etc/mke2fs.conf).  I’m getting pretty damn good performance with this!  The ‘-I 256′ option sets 256 bit inodes, which most recent ext3 FSs use already. If this is the case, and you get a message telling you so, remove this option.  Note that extents will make the FS backwards INCOMPATIBLE with ext3.

Next, edit /etc/fstab, e.g.:

/dev/vg/home /home ext4 defaults 0 0

Either unmount and mount or reboot your system.  tune2fs marks the fs as dirty and performs a fsck and conversion.
NOTICE: distros with initrds may need to be regenerated or you won’t be able to mount your root file system.  In Fedora (replace kernel version with your own):

cd /boot
mv initrd-2.6.27.7-134.fc10.i686.img initrd-2.6.27.7-134.fc10.i686.img.old
mkinitrd initrd-2.6.27.7-134.fc10.i686.img initrd-2.6.27.7-134.fc10.i686.

That’s all there is to it.  Stay tuned for future ext4 developments like online defragmentation.

Also, ext{2,3,4} reserve 5% of space for root in case the drive fills up.  On large modern drives, this can be excessive (e.g: 50GB on a 1TB disk).  Consider running ‘tune2fs -m 1 /dev/sd[x][n]‘ to reduce this to 1%.

For more information and tweaking:

1. Documentation/filesystems/ext4.txt from the latest kernel sources
2. http://ext4.wiki.kernel.org/index.php/Main_Page
3. man tune2fs
4. http://e2fsprogs.sourceforge.net/

No related posts.

What is retrocomputing?

I define retrocomputing [wikipedia] as the collecting and use of old computers.  Why might one do this?  Well, for one, enterprises cycle out machines fairly frequently.  2,3,4 and 5 year old systems are often sent out to scrappers in droves despite still being plenty useful.  Top of the line systems for large companies often have more than enough power for small and medium sized ones at pennies on the dollar compared to new hardware.  These machines are likely complete overkill for home use, but none the less are very useful for fun and learning.

IBM mainframe ops in the 1980s

Why?!

A lot of what I know about computers has been learned on old machines.  Hooking up a couple of servers and desktops and trying to make something useful out of them is a great exercise for the aspiring system administrator.  With open source software, it can all be done freely and easily.

Yes, you can run Linux, BSD, and Solaris from the comfort of your Windows desktop in a virtual machine (weak sauce…).  Yet there is something much different when you cluster several high technology servers together, tethered to a Fibre Channel storage array and have them share a single distributed file system.  The knowledge of setup, installation, and troubleshooting I’ve gained from mock scenarios like this I cannot compare to anyone else I’ve ever met.  Breaking things here usually means digging deep and fixing it.  If you were to screw something up at work like some of the things I’ve gotten into, it would probably cost you your job.

BENCHNET - where I rip into computers that cost as much as a house and my "production" rack

Retrocomputing is also fun.  I am personally into old IBM hardware, though old UNIX workstations of all sorts are interesting to me.  You can see my collection of IBM PS/2 and RS/6000 knowledge here: http://ps-2.kev009.com:8081/.  There is a particular thrill to booting up a machine that cost between $20,000 and $50,000 10 years ago.  Knowing that these same machine models were used to design the Boeing 777, composed the famous Deep Blue machine, and were used in the largest automotive and shipbuilding firms not to mention some of the most important space craft to date also brings a sense of power and nostalgia.  In some ways its similar to having a classic car, but different.  Maybe if that classic car was a big ass bulldozer, tank, jet or some other well engineered piece of equipment :-P.

IBM PS/2e, RS/6000 43p, PowerMac 7100, RS/6000 x2, HP Visualize c360

Nostalgia is one of the biggest things I get out of using particularly old hardware.  I missed the mainframe days, the minicomputer days, the PC and DOS days, the Apple II days (well, actually I used these a bit at a very young age), and to a degree the early Windows days.  Just like a history class, studying these old machines gives me insight as to why things are done the way they are today.  It gives me appreciation for modern systems and makes me write clean and well optimized code.  The old computer games that captivated me as a child (Sim City, Sim Tower, Sim Ant, Sim Farm, Gizmos and Gadgets, The Incredible Machine, Oregon Tail etc.) implanted a high degree of logic and understanding at a young age and it is heartwarming to revisit these.  I grew up a Mac user as well, so seeing what I was(or: was not :>) missing on PCs is also interesting.

Old MIPS UNIX server booting and logging in

Some of the benefits of retrocomputing:

• Enterprise class hardware
• Cheap, possibly even free
• Different design philosophies – not everything is x86 – a lot of this gear is quite different.  For example, UNIX workstations integrated most of what we enjoy on our PCs years before it became available to consumers.  SGI machines were doing A/V and 3D in the early 90s.  IBM midrange AS/400s have an advanced integrated database, programming languages, and environment that make PCs look like a joke for business programming.  WinFS, Object Storage Devices, etc are just now being talked about for PCs.  The channel philosophy from mainframes is still pretty new to PC servers (fibre channel), not to mention virtualization.
• If you break it, you can fix it and learn from it or toss it
• The engineering and craftsmanship in some of these systems is downright astonishing
• Old computers are works of art: they give you a window into the technology and culture of times past
• You should never trust a computer you can’t lift

It is interesting that we as humans produce such elaborate machines, only to discard them as scarp a few years later.  It is humbling and shows you the incredible progress we are making.

How?

eBay is your friend, but also look for local scrapyards or businesses doing overhauls.

If you are faint of heart, plenty of good abandonware sites exist for games and operating systems that can be run on emulators or VMs.  Check out this IBM mainframe emulator, Hercules.  Some of the original IBM OSes are public domain.

If you don’t want old PCs and big iron overtaking your house, there is plenty of good material on YouTube as well.  The Computer Museum is a good start.  Some of the consoles, offices, and outfits are hilarious.

Old SGI tech demo – pretty impressive!

No related posts.

In an earlier post, I discussed what was necessary to cable 1U servers up with power to run SATA drives. Next, you’ll need some drives and a controller if you have not already purchased them.

Controllers

The SATA controller you select is one of the most important pieces of the NAS box. It is my recommendation you get a high quality card, even if you are just doing RAID-1 (Mirroring). There are basically three types of controllers: standard, fakeraid, and hardware RAID. A standard controller is just a host-bus adapter. Any advanced features like striping and mirroring will need to be performed by your OS. Fakeraid includes a RAID BIOS, but all processing is left to a software driver. Hardware RAID is a complete subsystem with a dedicated CPU.

Standard controllers and fakeraid cards are generally cheap and maybe even integrated on your motherboard. However, these are rarely a good choice for servers. Aside from requiring the host to perform a lot more work, these can also cause a lot of trouble down the road. For instance, if a hard disk fails with software RAID, the system may not be able to reboot without intervention. Fakeraid may present the disks to the machine as one logical disk to avoid this, but these cards are the most evil. They tend to use closed drivers and proprietary disk formats which basically means avoid these like the plague. The cards themselves are fine, just use them like a standard controller and use the software RAID of your operating system. That leaves us with hardware RAID controllers.

Hardware RAID controllers are the only logical choice for a server. Hardware RAID cards offload all RAID processing to a subsystem on the card. These cards have an integrated CPU to perform parity calculations for RAID-5, and usually a large amount of RAM to act as cache. The RAM can even be used as write cache, but it is important to have a battery ON THE CARD in case of a power failure or crash to avoid potentially much greater data loss. Hardware RAID controllers present their RAID volumes as logical disks to the computer and operating system, so faults are transparent to the host. They also tend to allow hot addition of storage. Finally, since the card is handling all the I/O, bandwidth requirements are heavily reduced. If I am writing to a hardware RAID-1 device, data only needs to be sent across the PCI bus once as opposed to software RAID which will need to write the data to each disk. This is a very important consideration for older machines that have limited bus bandwidth and large arrays.

All things considered, I went with a 3Ware 9500S. Cost: $100 on eBay.

3Ware has good drivers in the Linux kernel, and has been manufacturing SATA RAID controllers for some time. Other decent manufactures are Adaptec and LSI Logic. Be sure to check for OS compatibility before purchasing a card.

Hard Drives

Just because you are getting the price point of SATA, you should not disregard the quality of drives you are purchasing. Although most standard consumer drives will work fine, there is a much more attractive option. Seagate and others offer what they call “nearline” drives which are basically the consumer drives with a RAID friendly firmware and continuous duty cycle. What’s best is that these usually only have a $10 or so premium over their consumer counterparts. When it comes to selecting a manufacturer, take a look at the warranty and technology integrated in the drives. Seagate and Hitachi are both good choices. Take a look at reviews too, especially StorageReview.

When most people purchase a hard disk, they make their choice simply on size. For a server (and even desktop!), you should also consider the spindle speed and size of the drive cache. 7,200RPM is enough for most servers, but 10,000RPM will deliver far greater performance under concurrent and random access. Of course, these usually come at a a heavy price premium and lesser capacity.

Here, I went with a pair of nearline Seagate Barracuda ES drives. The 320GB model was ample for my need, but these go all the way up to 750GB. They use perpendicular magnetic recording which increases density and speed, and feature a large 16MB cache. My initial testing shows nearly 80MB/s throughput! That level of speed has traditionally only been available on expensive SCSI disks. Cost: $100/drive.

Networking

Equally important to a NAS server is ample networking bandwidth.  This is largely dependent on the scale of services, but a decent gigabit NIC should get you near disk performance.  I picked up an Intel Pro/1000 MT Server Adapter, which features various offloading schemes to free the host’s CPUs from networking tasks.  If your needs are greater, consider aggregating several gigabit ports together.  Cost: $20.

Final Thoughts

The total cost for storage hardware comes to about $320.

Cabling the drives up and configuring the array is a pretty straight forward task. If you chose software RAID, there are plenty of guides on the internet to assist you in setting up md and device-mapper. Try http://linas.org/linux/raid.html.

When it comes to RAID levels, the choice depends on the number of drives you have and the level of protection you need. RAID-1 is a great choice for most applications in that you get true redundancy and greater read performance, but at the cost of half the physical capacity. RAID-5 is also commonly used in which you lose the capacity of one drive, with a three drive minimum. It allows for the failure of a single drive. With the size of modern hard disks, RAID-5 is less attractive than it once was and also suffers from heavy write performance loss due to parity calculations. Try http://www.acnc.com/04_01_00.html for the lowdown on various RAID levels.

My next post will cover file system selection.

No related posts.