Linux: Reflecting on the New Development Model

Submitted by Jeremy
on April 22, 2005 - 4:40am

At the July 2004 kernel summit, it was decided that there was no need to fork a 2.7 kernel [forum] to introduce new functionality into the Linux kernel. Instead, the decision was made that it was possible for Andrew Morton [interview] and Linus Torvalds to continue working together to first merge things into Andrew's -mm tree, and then after testing the changes to merge them into Linus' mainline tree [story]. This of course led to discussion, with some confusion as to how the 2.6 kernel [forum] could be considered stable while new features were still being merged in [story]. During another short discussion nine months after this decision, Rik van Riel [interview] offered some insight into why the new development model works:

"Things get merged one change at a time, and stabilised one change at a time. This is a big change from the even/odd numbered kernel series, where sometimes a bug crops up without anybody knowing exactly what change introduced it. The current development model seems to go much smoother than anything I've seen before."


From: Chuck Wolber [email blocked]
To:  linux-kernel
Subject: Development Model
Date: 	Mon, 18 Apr 2005 22:31:43 -0700 (PDT)


Greetings,

For months I have been reading as much as I can about the current 
stable/unstable development model, but still have a question.

Has the Linux Kernel reached a point where the majority of developers feel 
that (at least for now) no *MAJOR* "rip it out, stomp on it, burn it and 
start over" parts of the kernel exist any longer? In other words, do you 
feel that we're in a good place at this point and that incremental 
improvements will rule the day until $COOL_IDEA comes along and requires a 
refactoring of 2.x (where x is an odd number) porportions?


Thank you,

..Chuck..


P.S. Please CC me on replies, as I am not subscribed at this time.

-- 
http://www.quantumlinux.com 
 Quantum Linux Laboratories, LLC.
 ACCELERATING Business with Open Technology

 "The measure of the restoration lies in the extent to which we apply 
  social values more noble than mere monetary profit." - FDR


From: Arjan van de Ven [email blocked] Subject: Re: Development Model Date: Tue, 19 Apr 2005 16:50:48 +0200 On Mon, 2005-04-18 at 22:31 -0700, Chuck Wolber wrote: > Greetings, > > For months I have been reading as much as I can about the current > stable/unstable development model, but still have a question. > > Has the Linux Kernel reached a point where the majority of developers feel > that (at least for now) no *MAJOR* "rip it out, stomp on it, burn it and > start over" parts of the kernel exist any longer? In other words, do you > feel that we're in a good place at this point and that incremental > improvements will rule the day until $COOL_IDEA comes along and requires a > refactoring of 2.x (where x is an odd number) porportions? actually we have shown (and I like the model very much, it's a great way to get many features production ready and in the hand of users/customers really fast) that it doesn't take an odd number release branch to get major changes in. Instead it takes careful design and sufficient testing and review and most of the changes can go in anyway. A good example is the 4 level page tables... we entirely changed how pagetables worked (eg added another level) midway the 2.6 series and we could because of the factors I wrote above. It even didn't majorly destabilize the 2.6 tree to become unusable (although the change was obviously not pain free for kernel architecture maintainers). The VM got replaced as well on the fly. The tty layer got half fixed and seems to be in progress to be fixed even more. The scsi layer has gone through an overhaul with the creation of transport classes etc etc. All on the fly by doing careful design (eg stepwise), testing and review. If we can do all that... what WOULD warrant a odd numbered release which would be out of the hands of users for a long time (resulting in features not being available for the users for a likewise long time, which in turn means very little testing etc etc)
From: Rik van Riel [email blocked] Subject: Re: Development Model Date: Tue, 19 Apr 2005 15:24:00 -0400 (EDT) On Tue, 19 Apr 2005, Arjan van de Ven wrote: > actually we have shown (and I like the model very much, it's a great way > to get many features production ready and in the hand of users/customers > really fast) that it doesn't take an odd number release branch to get > major changes in. Instead it takes careful design and sufficient testing > and review and most of the changes can go in anyway. Perhaps even more importantly, things get merged one change at a time, and stabilised one change at a time. This is a big change from the even/odd numbered kernel series, where sometimes a bug crops up without anybody knowing exactly what change introduced it. The current development model seems to go much smoother than anything I've seen before. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan

Related Links:

i like the model as it truly

Anonymous (not verified)
on
April 22, 2005 - 9:21am

i like the model as it truly speeds up development: new features get included faster and get more testing, while people how want a stable kernel can either use the 2.6.y.z releases or recur to the quite stable 2.4 tree.

Yep, you say that things get

Stefano (not verified)
on
April 22, 2005 - 9:44am

Yep, you say that things get into the kernel faster, but if I want stability, I have to rely on 2.4. So, if there will be no "unstable" 2.7, when I'll begin to use new feature?

Even if I'm not into kernel develpement I don't like this approach very much, e.g., when we will have a major change (really major) like the one needed by reiser, if we're going to see it? I don't think they can make only smooth changes to introduce it, sometimes changes have to be like little revolution.

Stefano

Stable Kernel == Vendor Support

Anonymous (not verified)
on
April 22, 2005 - 11:05am

If you need a stable a kernel, you should use kernel provided by vendor, NOT vanilla kernels.

Then, if we should rely on ve

Federico (not verified)
on
April 22, 2005 - 12:08pm

Then, if we should rely on vendor's kernel, in which way we take advantages of having a faster kernel developement ?

By saying that, you are denying the advantages of the 2.6 developement model.

Federico.

You just can't win, can you.

Anonymous (not verified)
on
April 22, 2005 - 12:54pm

You just can't win, can you.

Sure you can win

Anonymous (not verified)
on
April 22, 2005 - 3:11pm

Want upgrades hot off the griddle? Run 2.6.x. Want something a little more stable? Run 2.6.x.x. Oh, you needed something guaranteed to be tested in a more systematic way by a third party, at say, 6 month intervals? Choose a distro that releases twice a year. Then again, Mandrake is getting ready to start releasing yearly. (This is not an endorsement of the quality of any particular distro's kernel, BTW.) Want to upgrade kernels every 12-18 months? CentOS, RHEL, Whitebox, etc. are for you. Are you hesitant to upgrade kernels more often than every 5 years? Check out Debian.

The current kernel development model protects the rest of us from Linus' inablility to say "No" during a development cycle which is expected to be relatively longer than the current one. Remember how development cycles started out with an off-hand estimate of 6 months or a year? And ended up being 1.5 years, 2.5 years, more? Remember how distros like RedHat were forced to maintain expensive trees with major pieces of the development kernel backported?

All that is gone now.

Everyone has much more choice and the distros have less burden.

The change does necessitate a corresponding change of mindset for people who are still thinking that they can blindly grab the latest vanilla x.x.x kernel and assume that it is "tried and true".

Redhat

Anonymous (not verified)
on
April 23, 2005 - 12:10am

Redhat still patches the linux 2.6.x kernels despite the new development model.

All distros do

Anonymous (not verified)
on
April 23, 2005 - 8:28am

All distros patch their kernels, mostly with bug fixes and a few features that their target audience needs.

*But* the patches are a lot smaller than they used to be. For all intents and purposes, a RedHat kernel isn't that different from a SUSE kernel or a Debian kernel.

Fedora kernels are actually very lightly patched

Anonymous (not verified)
on
April 27, 2005 - 10:32am

Actually they patch the Fedora kernels very lightly compared to the old RHL 2.4 kernels. The RHEL kernels are a "stable branch" so they do get bugfix backports, driver additions, etc, and stay at the basic version number that they had when the distro was released, so of course they are more heavily patched after a while. That's the point of a stable vendor branch!

Oh, of course they patch some things in Fedora as well. I certainly want them to stay on top of security holes, preferably at least as fast as the 2.6.x.y series if not faster...

There are some cases where Red Hat have had long lived patches that don't end up in the mainline kernel but they pretty much have to have because of customer demand; one example is the 4G/4G patch (which, by the way, has been dropped in the Fedora kernels now). Another example is Exec-shield, bits of which are finally getting merged now, and they are experimenting with Xen which is very likely mainline-bound (but basically doesn't touch anything if you don't use the Xen kernels so it should not introduce any real changes for non-Xen users).

The Red Hat 2.4 kernels (including the FC1 kernels) had loads of patches, to a large extent feature backports from 2.6 development. That's basically gone now. And pretty much whenever anyone complains about stuff not being patched into the Fedora kernels (new drivers etc) the answer they get is "get it included in mainline, and it will end up in Fedora soon". (Soon often being before the next Fedora core release - the Fedora kernels are actually rebased to new upstream version within the distro lifetime, e.g. FC3 started on 2.6.9 but is now on 2.6.11.) So trying to use Red Hat as an example of a company that patches the h*ck out of the kernel is thoroughly misleading nowadays.

2.6.x.y???

ravster (not verified)
on
April 22, 2005 - 5:11pm

Isn't that what the 2.6.x.y releases are for?
Also, isn't there a ck or co or something tree that is meant only for security fixes and 'obvious bug-fixes' for the 2.6 kernel? Came on kerneltrap one or two months ago.

Surely if you want the 2.6 AND you want it stable, then you should be willing a little bit to find out about these TWO alternatives. And I'm pretty sure that there are other alternatives too.

You are talking about -as. Bu

Anonymous (not verified)
on
April 25, 2005 - 1:29am

You are talking about -as. But there is no need for that anymore now when there is 2.6.x.y releases.

How about Slackware?

Anonymous (not verified)
on
April 22, 2005 - 7:50pm

My Slackware kernel is a vanilla kernel... Oops, it's a 2.4.29...

Don't agree with you

Peteris Krisjanis (not verified)
on
April 22, 2005 - 12:19pm

2.6 series kernel are very stable almost for everyone. I personally use 2.6.11 on my Gentoo desktop and it is a snap - altough I have very heavy use of HAL/DBUS, ALSA supported multichannel sound card, also USB storage works out of box...it is not conntected with kernel, but I put audio CD in and is started playing it - with digital extraction - out of box. I was very surprised. :)
For server I have used it for some five boxes and had only one problem with broken Realtek cards which I should avoid anyway. Not other problems or strange expierences, things went only better.

Of coarse, if you have some isolated problem, it should be dealed with accordingly. But in general, 2.6 is much faster and stable for anything you would like to do.

Not everyone

Anonymous (not verified)
on
April 23, 2005 - 5:33pm

See, that's exactly the problem. 2.6.11 may be perfect for you, but for me, it broke my touchpad and my soundcard (before that, 2.6.10 broke my USB disk and some USB flash keys). That kind of thing is bound to happen when you merge stuff faster than you can really test it.

Linux 2.6.x

Anonymous (not verified)
on
April 24, 2005 - 2:28am

This highlights one of the biggest problem that is lack of testing or release management failures.

Everyone seems to have varied mileage with linux 2.6.x.

Wrong kind of stability

cout
on
April 24, 2005 - 3:57am

The kind of stability people are looking for isn't just that their computer doesn't crash or doesn't exhibit strange bugs. Stability of API and feature set is also important from update to update; when a security hole is found, upgrading the kernel should not cause an unrelated feature to break. This is exactly what can happen if you are always upgrading to the latest development kernel, though, which is why the system of vendor patches was introduced.

You are confused

Cabal
on
April 24, 2005 - 10:06am

The plural of anecdote is not "data."

Realtek cards

Andreas Mohr (not verified)
on
April 25, 2005 - 8:01am

Realtek cards aren't broken, in fact they've been reported to work more reliably than some other cards for ages, probably because they're ubiquitous and thus the Linux driver is now extremely stable.
So if you think that Realtek cards are broken then this might well be another problem of a driver stability regression in the kernel you're using.
Unless you meant that your cards *are* broken for real, either due to hardware damage or due to a cheap card design...

bugs do occur

Anthony (not verified)
on
April 22, 2005 - 6:44pm

I've perhaps been unluckier than most, but I've had showstopper bugs pretty regularly in new versions of the 2.6 kernel.

Suse 9.2 was affected by the CD burner memory leak, even though distros are supposed to stabalize the kernel and they are one of the biggest, and on my workstation I currently can't connect an SATA hard drive and a PATA burner to my motherboard at the same time, I have to connect the burner to a Promise card. I also haven't been able to get DMA on the burner since 2.6.4, with Suse, Fedora, or Debian-testing. I know the hardware works because it works on 2.4 kernels and with FreeBSD.

It's not that I've never seen 2.6 work, it's just that it breaks so frequently that I've pretty much given up on it. I can deal with it on my workstation because of the performance advantages and because nothing critical is on that machine, but 2.6 is too much of a risk anywhere else.

FC3 is pretty decent

ac (not verified)
on
April 22, 2005 - 7:58pm

The Fedora Core 770 errata kernel is basically an -ac kernel. It's been pretty decent. I've been having trouble with the ICH5(?) Intel SATA chip. Also it doesn't have the bcm5700 driver. And the 3ware 8506 driver has a bug with over 4G of RAM so you need to install the 3ware driver from 3ware.com.

But other than that's it's been pretty decent.

To be honest Fedora Core is not as good as RHEL. The installer is a bit crap. GRUB is broken in the FC3 installer.

I haven't tried RHEL4 except to see if it fixed the problems with the ICH5 chip I mentioned before but it didn't.

Suse 9.2 is pretty horrible. Especially if you have over 4G of RAM.

The suse 9.1 installer is a bit crap because it has a race condition where it random deletes /dev/hda2 or whatever and you have to run mknod.

To be honest RHEL3u3 is the only distro that impresses me at all. It seems like Redhat really tested which is remarkable and rare and should be honoured.

Some FUD correction

Warren Togami (not verified)
on
April 24, 2005 - 2:36am

bcm5700 is not even shipped in the upstream kernel. IIRC it is an alternative implementation of the tg3 driver supplied by Broadcom. Their tech support suggests that folks use bcm5700 instead of tg3, despite tg3 being the upstream driver. Fedora ships ONLY upstream kernel technology (with the exception of some stuff that we add that makes sense, in 2.6 that list of added stuff is very small like exec-shield and xen in FC4).

You have it backwards...

Matthew Berg (not verified)
on
April 24, 2005 - 6:38am

Broadcom's supplied driver is bcm5700, which is why they suggest it. The tg3 driver was independently developed because the broadcom driver was, to put it bluntly, utter crap. I haven't investigated the claim yet, but the author of tg3 has indicated that Broadcom has been assisting with development lately and plan on later dropping their code base entirely.

ReiserFS4?

Anonymous (not verified)
on
April 24, 2005 - 11:46am

[Rant-o-phobiac, skip this]

Is ReiserFS4 is a taboo subject now because it can't be introduced with the current 2.6 model? Kernel developers complained the patch was introducing ideas that should belong in the VFS layer; playing with the VFS layer in the middle of a "stable" tree is not a great idea isn't it?

I had a couple of problems in the 2.6.8-2.6.10 times, when suddenly after an upgrade a DVD-ROM drive didn't show up at all. Is that what you'd expect from a stable branch?

Though this method "helps because more people test" (unsufficient testing was one of the reasons for leaving the 2.ODD model), it breaks things for everybody, so where's the win? you can't expect SuSE or RH or whoelse to be able to test _EVERY_ single hardware configuration. Though these two giants "might", what does happen to minor distributions? Fixes should go _in_the_main_tree_, not in vendor-specific-customized-kernel-packages.

The 2.4 kernel was pretty stable in the 2.4.11 days, it remained so (and is still). Less people were able (less Linux users!) to test the 2.3 branch than there would be for a 2.7. Now that it's becoming mainstream, linux HAS to become a nightmarish kernel?

I really don't like this "broken-by-design" scheme, "fix our errors, distributors".

so you'd preferred the 2.6.8

Anonymous (not verified)
on
April 24, 2005 - 2:09pm

so you'd preferred the 2.6.8 kernels, where it was _really_ easy to trash your data as user. great way to rant.
thanks to alan cox for having pointed that out.

I don't follow. A link to AC'

Anonymous (not verified)
on
April 24, 2005 - 9:37pm

I don't follow. A link to AC's explanation would be enlightening.

SG_IO and security - as remin

Anonymous (not verified)
on
April 25, 2005 - 12:53am

SG_IO and security - as reminder this hole issue settled with 2.6.10. 2.6.9 was already safer, but denied to many commands in the read-ok and write-ok SG_IO table.

so you like ranting, but aren't reading changelogs nor following lkml dev - questionable method!?

my request for clarification but not my rant

Anonymous (not verified)
on
April 26, 2005 - 1:23am

You have me confused with the other Anonymous.

If the Reiser4 devs (Namesys)

Ano Nymous
on
April 25, 2005 - 7:27am

If the Reiser4 devs (Namesys) would push a patch which doesn't make drastic changes outside its own module and with the controversial part removed (the reiser4 sycall, files as dirs), then the core could be long ago in the main kernel. To me it seems more a lack of development/different priorities from Namesys than the willingness to merge Reiser4 in mainline from the kernel devs.

I really would prefer to have

Anonymous (not verified)
on
April 25, 2005 - 10:15pm

I really would prefer to have a 2.7 tree..
2.6 works pretty well.. as long as I reley on the vendor.. but say right now.. Ububtu is running a 386 opt. kernel. I feel it would be a shame to waste the mmx, sse, and sse2 on my box. Any version outside the stock kernel usally has mixed results. the 2.6.11-1 and 2.6.11.7 both are unstable and hang the system at the strangest moments for me. (the -1 is from Ubuntu's tree.. the .7 I compiled myself, vanilla from kernel.org)

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.