Xen 3.4.1 on RHEL/CentOS 5.4

I’m happy to report that the updated Gitco Xen 3.4.1 repo is working well on CentOS 5.4.

If you are doing link bonding and bridging in accordance with my previous post “Xen 3.3 in RHEL/CentOS 5 and more Link Aggregation Fun“, you no longer need to patch the network scripts as RedHat fixed the initscripts package in RHEL 5.4.

Upgrade procedure for CentOS 5.3 to 5.4:

yum clean all
yum update glibc\*
yum update yum\* rpm\* python\*
yum clean all
yum upgrade
reboot

Updated Xen Install Guide From My Previous Article:

Head over to http://www.gitco.de/repo/ and grab the repo for your arch.  (Most likely wget http://www.gitco.de/repo/CentOS5-GITCO_x86_64.repo in /etc/yum.repos.d/ for the uninitiated).

If you already have Xen installed, you may need to remove and readd it.

yum groupremove Virtualization
yum groupinstall Virtualization

You’ll also get some updated tools like Virtual Machine Monitor 0.7.0 that make it easier to install newer guests such as Fedora 11 or Ubuntu.  Sweet!

Double check /etc/sysconfig/kernel.  It should be set to kernel-xen.  Likewise, check /boot/grub.conf and make sure that the Xen kernel is the default if the aforementioned was not done beforehand.

Reboot!

Announcing boo2pdf

I’ve just uploaded a beta of boo2pdf, an IBM BookManager to PDF conversion app & web service. I’m currently experimenting with the HTML to PDF backends and would like feedback with book files I haven’t tried. Once the code is cleaned up, I will dump it on my site.

Motivation

I have a large collection of old IBM machines and documentation. I want this documentation indexed by my own search facilities and Google for easy retrieval. PDF is widely read, while BookManager requires proprietary software and no search engines I know of parse it. This will probably be useful to Mainframers as well.

Take the web service for a spin here:

http://ps-2.kev009.com:8081/boo2pdf/.

Java: The Good Parts

javascript-the-good-partsA while back, a book entitled JavaScript: The Good Parts made waves on the internet, especially social networking sites.  This book purported to show the inner beauty of a language that was long considered second or third rate, coming of age.  With the advent of toolkits like JQuery, Javascript/AJAX development has become easy and even fun.

I aim to do the same by showing “Java: The Good Parts” here at a high level.java_powered_logo_rgb

When I was younger, I used to despise Java for political reasons and bad memories of early applets and applications.  I suspect many users and developers (especially Libre software devs!) are on the same boat.  By the end of this article, I hope I swayed your opinion or at least caused you to reevaluate your bias.  I also wish to encourage further discussion about these points and ways we can improve any deficiencies.

Rough and Tumble Upbringing

When Java first started gaining popularity, it was loudly hyped as the end all language.  It was expected that Java would take the “rich client” by storm, and applets would be the go to solution for enhancing web pages.  What happened was a bit different.  Java floundered and struggled to find a niche.  On the client side, AWT apps looked horrendous despite using native widgets.  Then Swing came about and despite easing development, it looked equally bad on all platforms (by default).  Applets were basically a stillbirth.  The ugly gray box, loadtime sometime measuring in minutes, and no coordination with the DOM and web browser made the average user hate Java.

One area Java was able to develop and secure a foundation, however, was the back end of large web applications.  The Virtual Machine approach provided a marked advantage over the CGI and interpreted scripts of the day.  Java’s rich networking libraries, clean Object Oriented design, and safety made this the language de jour for large web applications.

Open Source Matters

In my opinion, the open sourcing of Java during its early infancy would have had little impact on most of the teething issues.  The Virtual Machine, JIT, and Garbage Collection required many years of tuning to get acceptable performance and Sun did an acceptable job keeping it under wing.  The relatively limited CPU and RAM of the mid ’90s also made these concepts a bit ahead of their time.  Somewhere in the 1999-2002 time frame, though, Sun really dropped the ball.  An Open Source Java would have led to ubiquity on the booming Linux platform and a chance for all sorts of cross-platform software.

Open Source matters, and not just for the source code.  Open Source projects naturally bring about very pragmatic and intelligent developers.  These are the folks that thoroughly enjoy their hobby, work, and tools.  The marketing guys and pointed haired bosses have much less pull here.  On one hand, a vibrant community built itself with the many Apache Software Foundation projects.  However, most of these were squarely focused on web applications or low level things such as build tools, testing frameworks, and message buses.

ThumbsUpDue to the void, interpreted languages such as Python rose to the challenge while C and C++ remained the mainstay for applications programming.  Microsoft started dominating Windows development with their .NET CLR languages.  The glib/Gtk+ and Qt toolkits brought about a renaissance in cross-platform development with C and C++ respectively [though not limited].

It wasn’t until the open-sourcing of SWT that GUI development in Java became attractive.  The obvious killer apps here were the Eclipse IDE and the Azureus (now Vuze) bittorrent client.

Sun’s closed grip of Java really stagnated any chance of abundant expansion in these middle years (2001-2006).  Microsoft leveraged this weakness to create the excellent .NET platform and associated languages to maintain their closed platform and market dominance.  The counterbalance that would have been Java was thus left playing catchup.

We are just beginning to see the fruits of this labor from 2006 through today.  The OpenJDK project is now distributed with popular Linux distributions such as Ubuntu and Fedora.  We finally have decent browser plugins and Java Web Start applications across 32 and 64-bit machines.  The Java deployment problem will slowly fade from memory.

Application Development

SWT made Java apps beautiful.  OpenJDK should make them ubiquitous.  We finally have an Open Source platform that is widely deployed.  The strong built in standard library and clean OO design patterns of Java make it a very pleasant host for developing rich client apps.  Obvious areas for improvement here include better layout/form design tools and closer integration with upstream Linux distributors.

qt-logoSomewhere along the line, Trolltech/QT Software (now owned by Nokia) released Jambi — the complete Qt bindings, GUI framework, and incredibly rich library — for Java.  Oddly, this bombshell received little of the community and fanfare I thought it would or deserves.  Indeed, QT Software demoted Jambi from their teir-1 platforms and hopes the community will pick it up.  I hope this project isn’t allowed to stagnate as there is a lot of potential here.

Web Apps

Along the “Enterprise Web Application” lineage of Java, we wound up with some disgustingly overcomplicated and bloated frameworks for building web apps.  Ruby on Rails and Python Django came about and put a new spin on the development of rapid and robust web apps.  The learning curve of these frameworks is much less than Java EE and I will go as far as saying they are more capable because of it.

By using Java, JSP, and Servlets directly on top of a light Model-View-Controller, I believe Java is just as compelling as some of the more popular scripting languages.  Developers need to know they can trim the fat and that there are many advantages to developing in Java, namely because of the next topic…

Dynamic Languages, its all about the VM

It’s all about the JVM stupid!  One of the best features of Java and .NET are the underlying Virtual Machines.  By using JIT compiled VMs, Java code has a distinct advantage over the common interpreted languages such as Perl, Python, and PHP.  In the case of Java, the resultant is even naturally crossplatform.

The really interesting developments here focus on extending the JVM to syntax and paradigms other than the statically typed C++ lookalike.  Clojure and Scala deliver innovative new techniques while Jython and JRuby bring these excellent languages to the Java software platform and virtual machine.

In short, Java provides everyone with a counter to Microsoft’s .NET CLR.  The Java VM has been around the block and tuned by giants such as Sun, IBM, Oracle, SAP and more.

I call on the community to discuss how we can encourage use of the JVM for languages other than Java and build this into a defacto runtime.  Continued tuning and integration with Windows, Mac OS X, and Gnome/KDE *NIX systems is paramount.  Research for easy multi-core development is also worthwhile.  Meanwhile, distributions need to continue packaging the JRE and make it a default.  Individual developers need to be made aware of “Java: The Good Parts” and myths debunked.

Applets, Rich Media, Native Code!?

javafx_logoWith the release of Java FX, widespread deployment of the JRE, and better browser integration, Java has set the stage for a comeback to its roots.

During the late ’90s and 2000s, Adobe Flash became the tool of choice for web animation and interactive pages.  It really exploded with the advent of Youtube and other internet streaming sites making use of the Flash video format.  Unfortunately, Flash player is notoriously insecure, resource intensive, and crash-prone.  It is also not widely available for the millions of smartphones that have become more accessible than computers.

Luckily, there seems to be a shift back to the browser with new developments in AJAX, JavaScript, and HTML.  The <video> tag will hopefully make video as easy and portable as graphics are today in the browser.  Clean JavaScript libraries and fast JIT JS engines make it practical to use this paradigm for many domains.

Yet one must acknowledge that somewhere along the line, manipulating a DOM/markup language with a scripting language isn’t the most effective development platform for everything.  Google even thinks it poignant to run x86 machine code in a sandboxed environment in your browser.  I personally fail to see the logic behind this.  Java provides a well evolved, cross-platform solution.   Java can run on your ARM powered Android.  Requiring an x86 CPU just seems like the wrong track in this modern age.  Hopefully JavaFX will pick up the slack and return Java to its roots.  I would love to see the demise of the terrible Flash plugin.

Future and Conclusion

It’s time we considered Java for The Good Parts.

I hope some of my points caused you to reevaluate any bad preconceptions or past experiences you may have had with Java.  Java has undergone great change since its birth and I think it is capable of becoming the premier development platform for applications programming of all types.  Particularly interesting are some of the new languages such as Scala and Clojure.  Java has long been a staple in web development, but has traditionally scared away amateur coders.  If you cut the fat, Servlets and JSP are not much harder to set up than common place scripting languages.  Frameworks such as Grails bring it to parity with Rails or Django.

Java underwent a sea change in 2006 with the releasing of the source code and opening of the development process.  Java and the JVM should be championed by Libre software developers and users alike!

javascript-the-good-parts

Mobile phones spur cross-platform applications. Open Source is mainstream.

I had an interesting day today.  At school, we had a social event with an industry governing board and several local software companies in the Charleston, South Carolina region.  Aside from meeting a lot of new people, I was able to ask some of the industry leaders present about the platform and languages they used.

Perhaps the most interesting was that many Microsoft shops are moving away from fat client apps to web apps, and not simply because it is a buzzworthy thing to do.  The primary driver is the proliferation of advanced mobile devices, namely iPhone, Palm Pre, and Blackberry.  Two of the companies I talked to were VB.NET or C# shops and used to do traditional fat client software.  Due partially to the smartphone craze, they are moving SaaS.  They also mentioned the Mac as a rising popularity, but no Linux or netbooks or anything like that.   It’s funny how venerable HTML has become.. now the medium of choice for displaying cross-platform applicaitons.  I doubt anybody every imagined just how important HTML and JavaScript would become during their infancy.

Surprising and delightful to me was the talk of Open Source in the enterprise.  I spoke with two gentlemen from different large defense contractors and they were spot on with there assertions that Open Source software is superior in many ways.  Both were large Java EE shops and mentioned how they could check and verify FOSS for security much better than any proprietary software would allow.  They mentioned that the US Government is one of the largest purchasers of software but even then working with COTS vendors is difficult and FOSS solves many of these problems by allowing them to commit back changes.  The thing that made me the most happy was when one rep said that active participation in an Open Source project was a surefire way to boost a resume to the top of a stack.

Another interesting tidbit was that virtualization is synonymous with VMWare here (everyone specified this by name) even today.  I’d go as far as to say that I’d probably have received strange looks if I had mentioned KVM or even a heavyweight like Xen or MS Virtual Server.  Aside from Windows, the defense guys talked a lot about Solaris.  I didn’t get much reaction when mentioning Linux to anybody, sadly.  (somewhat comically) the non-profits were largely the 100% Microsoft shops.

Computer e-Recycling (an I.T. WTF Odyssey)

Story Time: Computer e-Recycling

an I.T. WTF Odyssey

I had the displeasure of working at an erecycler several years ago. Even watching stuff come off the trucks, it was very hard to get anything before it was utterly destroyed by the yard goons. Inserting a forklift blade into a 19″ rack cabinet was common practice and I witnessed on numerous occasions the dropping of them in this fashion. Once, they even rolled a forklift off the ramp. These events were always followed by a flurry of Spanish profanity and I usually had to check my pants for continence afterward from laughing so hard. It is a miracle nobody has ever been seriously maimed there to the best of my knowledge.

The highlight of this job experience was when the greedy goons resold a defective Siemens blood handling instrument of some sort that was sent specifically to them to be destroyed (as was EVERYTHING, in theory). The serial number was traced back to them and there was an all out shitstorm. A team of inspectors was flown out from Germany. The goons put on a particularly hilarious show buying hardhats, safety vests, warning signs, identification badges, and more. Somehow, they kept the contract (it was probably just “check you ass” on Siemen’s part — dumping obsolete X-Ray, MRI machines, medical waste, etc for free must be hard to pass up) and all of this change disappeared within a couple days.

What is remarkable is that the above business got multi-millions of dollars of inventory for FREE every year. I really can’t think of any other business model like it. The owner is completely incompetent and morally bankrupt. Very little was put back into the company’s facilities, employees (except maybe a couple at the top), or development. None of the employees had an IQ above room temperature, and everybody seemed content to keep it that way.

There is a happy ending though. A skid of IBM RS/6000 7012 systems from Intel came in at one time. Among the 20-30 machines was a lone -397 that I snagged. This is my favorite collectors box to date (it is the same POWER2 CPU type used in the famous Deep Blue super computer).

Mirroring Fedora

Introduction

This post details setting up your own private mirror of Fedora’s repos.  There are many ways to do this, but this method is by far the best for heavy usage.  By using MirrorManager, clients in your IP range need no custom configuration.  Roaming laptop users automagically hit your mirror while on the premises, yet use the public infrastructure elsewhere.  Setup isn’t exactly hard, but it isn’t well documented so I’ll write about my experience here.

Some background info.. we have at least 50 Linux desktops, laptops, servers and VMs running about half Fedora 10 and half Fedora 11 at work.  Due to the number of systems, breadth of packages used, and desire to quickly update when new releases are out, I decided on a full mirror setup.  If you only have a handful of systems, you may be better off simply using a general purpose caching proxy like Squid, perhaps telling MirrorManager to point to it.

This guide should be used in addition to http://fedoraproject.org/wiki/Infrastructure/Mirroring which has some background info.

Initial setup and mirror

First, get prepared by installing MirrorManager-client, which contains the report_mirror script you will need.  If your mirror isn’t running Fedora, you can clone the source of this app from their GIT repo.

yum install mirrormanager-client

You’ll be using rsync, a sysadmin’s best friend, for efficient mirroring.

Set up a shell script like mine below (d0mirror.sh) one level up from where your mirror will be accessible (http, ftp, rsync, nfs – covered later).  This one mirrors against kernel.org.  Choose a mirror close to you on the Internet.

rsync -vaH --exclude-from=fedora-excludes.txt --numeric-ids --delete --delete-delay \
 --delay-updates rsync://mirrors.kernel.org/fedora-enchilada fedora-mirror
report_mirror

And a text file (fedora-excludes.txt) excluding things you don’t want/need.  Take a look through a public mirror and decide if you want to eliminate anything else.  You may want to remove the *.iso line below if you want users to be able to pull disc images from this box.  Otherwise, this is probably a good list for most people.  You can exclude all of linux/updates/testing/ if you don’t enable the testing repo on any of your machines.

**/debug/**
**/alpha/**
**/source/**
**/SRPMS/**
**/*.iso
**/ppc/**
**/ppc64/**
linux/core/**
linux/development/**
linux/releases/7/**
linux/releases/8/**
linux/releases/9/**
linux/releases/test/**
linux/updates/8/**
linux/updates/9/**
linux/updates/testing/7/**
linux/updates/testing/8/**
linux/updates/testing/9/**

Run your shell script and sit back for up to a day or two depending on your connection speed.  My current mirror weighs in at about 80G.

Internal distribution

While you wait for sync, decide how you want to run the service internally.  HTTP is nice because it is easy for users to browse and decently quick with keep-alive.   Using NFS, rsync, or FTP may be a bit more efficient if you are worried about this.  You can list several URLs in MirrorManager for the best of all worlds.

Add the following to your Apache configuration if you decide to use HTTP:

Alias /fedora/ "/mnt/ar1/fedora-mirror/"

AddType application/octet-stream .rpm

<Directory "/mnt/ar1/fedora-mirror">
    Options Indexes FollowSymLinks
    Order allow,deny
    Allow from all
</Directory>

<LocationMatch "\.(xml|xml\.gz|xml\.asc|sqlite)">
    Header set Cache-Control "must-revalidate"
    ExpiresActive On
    ExpiresDefault "now"
</LocationMatch>

Set up any other services of you choice to push that directory out in addition.

Working with MirrorManager client and server

Next, open up /etc/mirrormanager-client/report_mirror.conf.  Take notice of the site name, password, and host name.  You will need to set these up in MirrorManager in a bit.  The paths here are all local and used by report_mirror to check what you have available.

# if enabled=0, no data is sent to the database
enabled=1
server=https://admin.fedoraproject.org/mirrormanager/xmlrpc

[site]
# if enabled=0, no data about this site is sent to the database
enabled=1
name=<yoursitename>
password=<yourhostpassword>

[host]
# if enabled=0, no data about this host is sent to the database
enabled=1
name=x345-a2.internal
# if user_active=0, no data about this category is given to the public
# This can be used to toggle between serving and not serving data,
# such enabled during the nighttime (when you have more idle bandwidth
# available) and disabled during the daytime.
# not passing it means leave it alone in the database.

[stats]
# Stats are only sent when run with the -s option
# and when this section is enabled.
enabled=0
apache=/var/log/httpd/access_log
vsftpd=/var/log/vsftpd.log
# remember to enable log file and transfer logging in rsyncd.conf
rsyncd=/var/log/rsyncd.log

[Fedora Linux]
enabled=1
path=/mnt/ar1/fedora-mirror/linux

[Fedora EPEL]
path=/var/www/html/pub/epel
enabled=0

# lesser used categories below

[Fedora Web]
enabled=0
path=/var/www/html/pub/fedora/web

[Fedora Secondary Arches]
enabled=0
path=/var/www/html/pub/fedora-secondary

[Fedora Other]
enabled=0
path=/var/www/html/pub/alt

# historical content

[Fedora Core]
# if enabled=0, no data about this host is sent to the database
enabled=0
path=/var/www/html/pub/fedora/linux/core

[Fedora Extras]
enabled=0
path=/var/www/html/pub/fedora/linux/extras

Log into https://admin.fedoraproject.org/mirrormanager, creating a new account if you need to.  Add a new site with the same name as the config file from above.  You’ll set the site password here, and make sure to check the ‘private’ box if this is only for internal users.  Now, add a host under this site.  The name here should probably be a FQDN of your actual mirror, even if it is internal only (i.e x345-a2.internal from my example above).  Once that is done, add a “site-local netblock”.  This is your public IP network/netmask or network in CIDR notation.  If you only have one public IP, it will be in the format nnn.nnn.nnn.nnn/32.

Almost done.  Now, click Add Category.  “Fedora Linux” is the only one you are concerned with if you followed all the values in this guide so far.  Add the others if needed.  Tell them your upstream source (rsync://mirrors.kernel.org/fedora-enchilada from above) and then your internal URL (http://x345-a2.internal/fedora/linux for my setup).

Conclusion

Once your rsync is complete and report_mirror is done, you should see clients start hitting your box.   Don’t forget to add your mirror script (domirror.sh from above — rsync and report_mirror) to cron!  You may wish to join the private ‘fedora-mirrors’ mail lists to be informed of new releases and changes.

The best thing is that it works across all package requests, including new machines, roaming users,  ‘preupgrade’, etc.   All in all, pretty nifty!  Your users will love you when their upgrades are almost instant!  The Fedora infrastructure is set up very well for mirroring, public and private, and this is how the project copes with the huge demand for new releases.  Comment away if you need clarification or help.

Kernel 2.6.30 is a Go

I initially thought this would be a rather uninteresting release, especially when we learned Xen dom0 didn’t make the cut. Following the changelog line-by-line, this one still didn’t seem very interesting to me. But analyzing the sum of parts, I have to consider 2.6.30 a ‘golden’ kernel — certainly the best in a while.

There is solid improvement top to bottom here.  A lot of the new KMS/DRM stuff from Fedora 11 has worked its way up stream.  File system work is too much to mention, but highlights include relatime, writeback by default for Extfs, NILFS2, Btrfs development and more. FSCache works as advertised.  Also some groundwork for NFS 4.1, which will eventually bring us pNFS.

Boot speed seems fast as ever, but I haven’t taken the time to do any empirical analysis.  Your results here will be hardware dependent but async initialization of certain subsystems is a welcome move in the right direction.

Basically, a solid release with a good balance of new stuff but mainly refinement of existing systems and merging of longstanding patches.

Kernel Newbies has, as usual, a great change summary: http://kernelnewbies.org/Linux_2_6_30

Kernel developers don’t get Xen

The recent bruhaha surrounding Xen on LKML (http://lkml.org/lkml/2009/6/2/475) is really disheartening.  Essentially, the Linux kernel devs are at a disconnect with users.  Some are proposing narrow-minded ideas such as DROPPING software paravirt or merging Xen as a whole into the kernel.

I use Xen for a few primary reasons:  it bar none has the best speed — full software paravirtualization pays dividends here;  it is mature;  it works on perfectly good machines that don’t happen to have the latest chips;  it does hardware passthrough on these same systems;  it has great live migration that actually works.

Ingo Molnar wants you to send all your perfectly good enterprise iron to the landfill even though these systems will last 10+ useful years without boneheaded software decisions such as this.

These same FUDsters want to strip the crossplatform nature of Xen dom0 out too.  Xen dom0 runs on NetBSD and Solaris.  It is a true hypervisor and will plug into exisiting architectures, and not force you to use Linux for everything.

I have to admire all the hoops Jeremy Fitzhardinge has jumped through to date, as I know my patience is wearing thin.

Xen powers huge sites such as Amazon and services like linode.com/slicehost.com.  By not having dom0 in the kernel where distros such as Ubuntu and Fedora can easily integrate it, kernel devs are doing a disservice to users.

I use KVM, VMWare, and Virtual Box at work in addition, but Xen is firmly entrenched in my toolbox.  The roadmap they have looks great, and I just don’t see a reason for decline in Xen popularity.  High availability in Xen 4.0 is what I’ve always been waiting for.

Jeremy has gone to great lengths to work with upstream but keeps getting shot down and asked to do something else when he meets one requirement.  The solution is to merge Jeremy’s conservative dom0 patch set and work on a technical solution to the patches that the FUDsters consider bad.  It’s what the users want!

El Reg Humor and Java in free software

The Register has a good article on Sphinx search with some entertaining pop-shots at Java and “enterprise software” that got a rise out of me:

Solr is popular with the enterprise crowd, who love its Java. Being a Java program, Solr includes no shortage of technology whose acronyms contain the letters J and X.

This tickles the enterprise pink, because these sorts of developers love nothing more than hanging out around a whiteboard drawing boxes and arrows and, from time to time, writing XML to make it look like they’re doing real work. Solr thrives in this environment, being an Apache Foundation project, the Apache Foundation, of course, widely known as a cruel experiment to see what happens when bureaucrats do open source.

Having a bit of experience with Java from academia and a few open source projects I make use of, I can’t help but laugh at how comically and concisely the editor summed it up.

By and large, successful open source projects tend to be written in languages other than Java. The entire GNU/Linux OS stack is primarily C, with some components using C++ like KDE, OpenOffice and Firefox.  On the ever popular web front, PHP, Ruby, and Python lead the pack.

I think it turned out this way for a multitude of reasons.  When working on the OS stack, the power and control of C and C++ are hard to beat.  The plethora of libraries and raw speed of these compiled languages set the bar high for any newcomers.  Java exists as a kludge, mildly useful for desktop apps and mildly useful for web apps while historically having a lot of problems.  Native look and feel have long been the layman’s complaint, though SWT has done a pretty good job there.  Of course, omnipresent Java in the Linux world is relatively new.  I think Java would have been the darling language of client apps had it been open sourced sooner, but this came about 7 years too late to have a large impact on shaping the common FOSS userland.

It is interesting how the open source projects built with Java tend to be highly bureaucratic and abstract.  I think the bottom line is that FOSS programmers do what they do because it is fun and demand pragmatism.  The “enterprise software” attitude/baggage that many Java apps and libraries carry are a big turn off to pragmatism and the hacking culture.  The barrier to entry for Java web programming is also much higher than its “scripting language” competitors, which carry light and simple frameworks that focus on results, not procedure.

Java itself isn’t that of a bad language.  I actually enjoy working with it in school (…though I think it really isn’t appropriate as an introductory teaching language, shielding important concepts from students.  Maybe a future post?..).  When it comes time for real work though, I consider Python, C,  C++ more pragmatic depending on the job at hand.  That, and the fact that most of the common scripting languages are gaining JIT compilers may accelerate Java toward status as a legacy language.

Your thoughts?