What ultimately drove Con Kolivas to exit the Linux developer community forever.
 Kernel developers: Con at the Kernel Summit in Ottawa, Canada, 2005 |
APC: So what did you do next to try to convince the Linux kernel devs of the need for more focus on end-users?
I set out to invent some benchmark to try and quantify performance problems in Linux on the desktop PC. The first benchmark I created called 'Contest' (pun intended). It was horribly complicated to use and set up and the results were difficult to interpret but at least I tried. It did help. Some changes did come about as a result of these benchmarks, but mostly on the disk I/O front. So I was still left looking at this stuttering CPU scheduling performance.
I had some experience at merging patches from previous kernels and ironically most of them were code around the CPU scheduler. Although I'd never learnt how to program, looking at the code it eventually started making sense.
I was left pointing out to people what I thought the problem was from looking at that particular code. After ranting and raving and saying what I thought the problem was, I figured I'd just start tinkering myself and try and tune the thing myself.
After a few failed experiments I started writing some code which helped... a lot. As it turns out people did pay attention and eventually my code got incorporated. I was never very happy with how the CPU scheduler tackled interactivity but at least it was now usable on the desktop.
Not being happy with how the actual underlying mechanism worked I set out to redesign that myself from scratch, and the second generation of the -ck patchset was born. This time it was mostly my own code. So this is the story of how I started writing my own code for the linux kernel.
In brief, following this I found myself writing code which many many desktop users found helped, but I had no way to quantify these changes. There is always a placebo element which makes the end user experience difficult to clarify as to whether there is an improvement or not.
However I have tried very hard to make myself relatively resistant to this placebo effect myself from years of testing, and I guess the fact that my website has close to 1 million hits suggests there are people who agree it makes a difference. This inability to prove quantitatively the advantage that -ck offered, though, was basically what would eventually spell out the death of it.
APC: What code did get incorporated into the mainline kernel?
Looking back, while the patchset has been well known, little of the actual code itself ended up in the mainline kernel, even if it spurned interest and development from other people in that time. Most of what did end up going in were changes to the CPU scheduler to improve interactivity, fairness, SMP user fairness, making 'nice' behave itself, hyperthread fairness and so on. There were lots of other minor contributions in other areas, such as the virtual memory subsystem, software suspend, disk i/o scheduling and random bugfixes.
The emphasis of what I did get involved in is still the area that there are many -ck patches in the last release I ever made (2.6.22-ck1). The remaining patches reflect where I still believe massive problems exist for the desktop, and ironically, despite my efforts to the contrary, the same problems seem to get magnified with each passing year.
It seems that the emerging challenges for the linux kernel on the desktop never seem to get whole-heartedly tackled by any full time developer, and only get a sideways glance when the problems are so obvious that even those on the linux kernel mailing list are willing to complain about them.
APC: Was there a toll on you for this voluntary involvement?
Yes. There is no doubt that whenever I was heavily involved, I would stay awake at night thinking up code solutions, I would be sleep deprived, and it had the possibility to impact on my work and family life. I tried very hard to never compromise those two for the sake of kernel development, but perhaps on occasion I did push it to far. I make an issue of never regretting anything I have done so I'm still pleased with my involvement till now.
 In black and white: Con Kolivas' CK patch homepage |
APC: What was the driving force that caused you to hangup your kernel coding keyboard and the -ck patchset?
That's one of those 'I wish I had a dime for ever time I was asked that' sort of questions.
As most people are well aware, my involvement in kernel development was motivated on three fronts, and it has zero to do with my full time career and life.
First, it was lots of fun. I've always been a computer geek and enjoyed spending hours in front of the computer doing... well whatever really. So spending it on something that had become a passion for me was an obvious source of great enjoyment for me.
Second, it was an intellectual challenge. There seemed to be things that I always wish were done in the Linux kernel, and there were issues people didn't really care to tackle and so on. Trying to confront them head-on I was doing something I really hadn't done before in terms of high level programming. I learnt a heck of a lot about operating system design and all other really basic computer science things that most CS students probably are bored with.
However it was all new to me. There is precious little (and some would argue zero) new research in operating system design. Looking at linux there is innovation in the approach to tackling things, but it's not like there's anything new about the problems being fixed. The same issues have always been there in hardware vs software designs, and all the research has been done. I've seen many people accuse me of claiming I invented fair scheduling. Let me set the record straight. I make no such ridiculous claim.
What I did was take what was known and try and apply it to the constraints of current hardware designs and the Linux kernel framework. While the 'global picture' of the problems and solutions have been well known for many years, actually narrowing down on where the acute differences in hardware vs software have evolved has not. Innovation only lies in the application of those ideas. Academic approaches to solutions tend not to be useful in the real world. On the flip side, pure hackery also tend to be long term disasters even if initially they're a quick fix. Finding the right balance between hackery and not ignoring the many years of academic research is what is needed.
APC: Did you get that balance right?
Heck no. But that's what I strived for. I think there are precious few developers who do. If there is one failing in the human driven decision making in choosing what code goes where it is that it is impossible to recognise that (again I'm not saying it was my code).
Third, it was an ego trip. If I improved something that I cared about, I found that usually lots of other people cared about it. The reason for that obviously is that I was actually an ordinary desktop PC user that pretended to be a kernel hacker. So whenever I made improvements that affected desktop users, lo and behold lots of people cared about them. I recall a kernel hacker that I respected very much joked about my 'fans' and asking how he could attract a crowd. He has contributed possibly 1000 times more lines of code to linux kernel than myself, yet I had a 'following'.
I only attracted a following because I hacked on things I cared about. And the users told me so. The one thing that drove my development over the many years were the 'thank you's that I got from the users of the -ck patchset.
So I still haven't answered your question about what made me stop kernel development have I? I guess explaining my motivations helps me explain why I stopped.
The user response was still there. If anything, the users got more vocal than ever as I was announcing quitting kernel development.
The intellectual challenge? Well that still existed of course.
The fun? Yes that's what was killed. It stopped being fun. In my now quite public email announcing that I was quitting I explained it briefly. The -ck patchset was for quite a while, a meaningless, out of mainline's spotlight playground for my experiments. As the scope of changes got larger, the improvements became more drastic and were more acutely noticeable. This led to more and more people asking me when the changes would me merged into mainline. As the number of requests grew, my resolve to not get mainline involved diminished. So I'd occasionally post patches as examples to the linux kernel mailing list. This generated more interest and requests to get mainline involved. So I tried.
You asked before what patches from -ck got into mainline and I listed a whole lot of random minor patches. The magnitude of the changes in the patches that did _not_ get involved stands out far more than those that did get in.
APC: was there something that was the 'final straw' for you?
My first major rejection was the original staircase CPU scheduler. It stood out as being far better in interactivity than the mainline CPU scheduler, but ultimately, just as the mainline CPU scheduler, it had corner cases that meant it was not perfect. While I was still developing it, the attention moved away from the CPU scheduler at that time. The reason given by Andrew Morton (the maintainer and second last gateway into the mainline kernel) at the time was that the kernel had more burning issues and bugs to address.
Of course it did. There were so many subsystems being repeatedly rewritten that there was never-ending breakage. And rewriting working subsystems and breaking them is far more important than something that might improve the desktop right? Oops, some of my bitterness crept in there. I'll try and keep emotion out and just tell the rest of the story as objectively as I can.
The main problem was that there simply was not a convincing way to prove that staircase was better on the desktop. User reports were not enough. There was no benchmark. There was no way to prove it was better, and the user reports if anything just angered the kernel maintainers further for their lack of objectivity. I even tried writing a benchmark called Interbench. Note that interactivity is not responsiveness (I have a little summary of the difference as I see it with the Interbench code). This was much better code than Contest but even though I could demonstrate advantage with my scheduler on Interbench, the magnitude of the differences was difficult if not impossible to elicit based on the raw numbers generated by Interbench.
With a truckload of help from William Lee Irwin III (who wrote the main architecture) I posted a pluggable CPU scheduler framework that would allow you to build into the kernel as many of multiple CPU schedulers as you like and choose at boot time which one to run. I planned to extend that to runtime selection as well. This is much like the modular pluggable I/O scheduler framework that Linux kernel currently has. It was flat out refused by both Linus and Ingo (who is the CPU scheduler maintainer) as leading to specialisation of CPU schedulers and they both preferred there to be one CPU scheduler that was good at everything. I guess you can say the CPU scheduler is a steamroller that we as desktop users use to crack nuts with, and they didn't want us to build a nutcracker into the kernel.
The purpose of Plugsched was simply to provide a seamless way for the staircase CPU scheduler to be integrated into the mainline kernel. I didn't feel strongly about the presence of pluggable CPU schedulers, but many people still do and the code is still maintained today mainly by Peter Williams.
Then along came swap prefetch. I spent a long time maintaining and improving it. It was merged into the -mm kernel 18 months ago and I've been supporting it since. Andrew to this day remains unconvinced it helps and that it 'might' have negative consequences elsewhere. No bug report or performance complaint has been forthcoming in the last 9 months. I even wrote a benchmark that showed how it worked, which managed to quantify it! In a hilarious turnaround Linus asked me offlist 'yeah but does it really help'. Well, user reports and benchmarks weren't enough... It's still in limbo now but they'll have no choice but to drop it without anyone maintaining it.
Finally the nail in the coffin. The Staircase Deadline CPU scheduler. Initially started as a side project from the Staircase CPU scheduler I soon realised that it was possible to have excellent interactivity while fixing the horrible fairness issues that an unfair design had. Furthermore, it actually improved interactivity issues elsewhere that ended up being fairness problems, and fairness is of course paramount to servers and multiuser environments.
A lot of users and even kernel developers found that many long lasting complaints with the mainline and other schedulers were fixed by this code and practically begged me to push it into mainline, and one user
demanded Linus merge it as soon as possible as a bugfix. So I supported the code and fixed it as problems arose and did many bugfixes and improvements along the way.
Then I hit an impasse. One very vocal user found that the unfair behaviour in the mainline scheduler was something he came to expect. A flamewar of sorts erupted at the time, because to fix 100% of the problems with the CPU scheduler we had to sacrifice interactivity on some workloads. It wasn't a dramatic loss of interactivity, but it was definitely there. Rather than use 'nice' to proportion CPU according to where the user told the operating system it should be, the user believed it was the kernel's responsibility to guess. As it turns out, it is the fact that guessing means that no matter how hard and how smart you make the CPU scheduler, it will get it wrong some of the time. The more it tries to guess, the worse will be the corner cases of misbehaving.
The option is to throttle the guessing, or not guess at all. The former option means you have a CPU scheduler which is difficult to model, and the behaviour is right 95% of the time and ebbs and flows in its metering out of CPU and latency. The latter option means there is no guessing and the behaviour is correct 100% of the time... it only gives what you tell it to give. It seemed so absurdly clear to me, given that interactivity mostly was better anyway with the fair approach, yet the maintainers demanded I address this as a problem with the new design. I refused. I insisted that we had to compromise a small amount to gain a heck of a great deal more. A scheduler that was deterministic and predictable and still interactive is a much better option long term than the hack after hack approach we were maintaining.
Then I became very unwell. A disc prolapse in my neck basically meant I need to lie flat on my back for about 6 weeks. Yes, kernel development did contribute to this problem. So I was drugged to the eyeballs and spent most of the day and night lying down... having nothing to think about but stew over the kernel. Whenever I could I snuck back on the PC to try and support the code I had thrown out there. I refused to budge on the fairness aspect, and I kept getting that thrown back in my face as an unfixable failure in the design.
Then one day presumably Ingo decided it was a good idea and the way forward and... wrote his own fair scheduling interactive design with a modular almost pluggable CPU scheduling framework... and had help with the code from the person who refused to accept fair behaviour in my flamewar.
So I had plenty of time lying on my back to reflect on what I was doing and why, and whether I was going to regret it from that point on. I decided to complete the work on Staircase Deadline to make sure it was the reference for comparison, instead of having the CPU scheduler maintainer's new code comparing to the old clunky scheduler. Then I quit forever.
APC: Are developer egos a problem in the open source development model in general?
I think any problem with any development model has multiple factors, and ultimately, it is humans that make decisions. I won't comment on the humans themselves.
If there is any one big problem with kernel development and Linux it is the complete disconnection of the development process from normal users. You know, the ones who constitute 99.9% of the Linux user base.
The Linux kernel mailing list is the way to communicate with the kernel developers. To put it mildly, the Linux kernel mailing list (lkml) is about as scary a communication forum as they come. Most people are absolutely terrified of mailing the list lest they get flamed for their inexperience, an inappropriate bug report, being stupid or whatever. And for the most part they're absolutely right. There is no friendly way to communicate normal users' issues that are kernel related. Yes of course the kernel developers are fun loving, happy-go-lucky friendly people. Just look at any interview with Linus and see how he views himself.
I think the kernel developers at large haven't got the faintest idea just how big the problems in userspace are. It is a very small brave minority that are happy to post to lkml, and I keep getting users telling me on IRC, in person, and via my own mailing list, what their problems are. And they've even become fearful of me, even though I've never viewed myself as a real kernel developer.
Just trawl the normal support forums (which I did for Gentoo users as a way of finding bug reports often because the users were afraid to tell me) and see how many obvious kernel related issues there are. I'd love to tell them all to suddenly flood lkml with their reports of failed boots with various kernels, hardware disappearing, stopping working suddenly, memory disappearing, trying to use software suspend and having your balls blown off by your laptop, and so on.
And there are all the obvious bug reports. They're afraid to mention these. How scary do you think it is to say 'my Firefox tabs open slowly since the last CPU scheduler upgrade'? To top it all off, the enterprise users are the opposite. Just watch each kernel release and see how quickly some $bullshit_benchmark degraded by .1% with patch $Y gets reported. See also how quickly it gets attended to.
APC: What are you doing in your spare time now, and what future projects do you have planned?
I had some small userspace toys that I was experimenting with (compression tools, video encoding/conversion tools and a few others) that I thought I'd get back to. However just looking at any code gives me a bad taste in the mouth so that's not happening.
My current passion is learning Japanese. It's fun, it's a huge intellectual challenge, will allow me to eventually understand lots of interesting Japanese media in its native language (short stories, movies, manga and anime) and comes with no strings attached. I never really know what hobby I'll end up taking up but I usually get completely engrossed in whatever it was.
Thankyou for your time Con.
Page 3/3
 Con's pride and joy: Future coders? |