login
Search
Jesper Juhl summarized a recent experiment of compiling the latest Linux kernel 100 times with various configurations, resulting in 82 failed builds and thousands of warnings. Most of the builds utilized make randconfig which generates a random .config, and further inspection revealed that a significant percentage of these builds failed due to known configuration issues. Regarding warnings, Adrian Bunk pointed out that for most normal configurations things are much better, "not that our current situation [is] perfect, but the number of warnings in .config's people usually use isn't that bad."
Jesper acknowledged that things aren't as bad as they first look, but went on to explain that he's trying to motivate more people into helping track down warnings and build errors, "there's a lot of focus on implementing new features - and that's great - but there's little emphasis on fixing the problems we have and already know about - I'd like to see that change, and my post was mainly an attempt at making that happen :)" Adrian agreed that this is a good goal, then pointed to the kernel bug tracker noting that most build errors are known, "and in these cases, the bugs in unmaintained areas of the kernel like APM or the floppy driver are the worst ones."
From: Jesper Juhl [email blocked] To: linux-kernel Subject: Building 100 kernels; we suck at dependencies and drown in warnings Date: Sun, 26 Feb 2006 17:21:17 +0100 Hi everyone, I just sat down and build 100 kernels (2.6.16-rc4-mm2 kernels to be exact) 95 kernels were build with 'make randconfig'. 1 kernel was build with the config I normally use for my own box. 1 kernel was build from 'make defconfig'. 1 kernel was build from 'make allmodconfig'. 1 kernel was build from 'make allnoconfig'. 1 kernel was build from 'make allyesconfig'. That was an interresting experience. First of all not very many of the kernels actually build correctly and secondly, if I grep the build logs for warnings I'm swamped. Out of 100 kernels 82 failed to build - that's an 18% success rate people, not very impressive. Some of the failed builds are due to things like CONFIG_STANDALONE that will break the build if not set to Y (unless you have the firmware available ofcourse), but looking at the config files I find that only 26 kernels have CONFIG_STANDALONE unset, so that only accounts for a quarter of the kernels. A lot of failed builds are due to invalid combinations of some stuff being build-in and some stuff being build as modules. This, as far as I'm concerned, is something that the dependencies in Kconfig should make impossible - hence my conclusion that we suck at deps. From 100 kernel builds there was a total of 16152 warnings and 645 of those are unique warnings, the rest are duplicates. We are drowning in warnings people. Sure, many of the warnings are due to gcc getting something wrong and shouldn't really be emitted, but a lot of them point to actual problems or deficiencies (I obviously haven't looked at them all in detail yet, so take that with a grain of salt please). In any case, it looks to me like we have some serious clean-up work to do. Unfortunately I don't have anywhere to put all the configs and logs online, but I can send them on request, or if someone can point at a space to upload them to I'll gladly make them available. That's it for now, I'll get to work trying to clean up some of the breakage I've seen, if anyone wants to join in feel free :) -- Jesper Juhl [email blocked] Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html
From: Adrian Bunk [email blocked] Subject: Re: Building 100 kernels; we suck at dependencies and drown in warnings Date: Sun, 26 Feb 2006 18:00:38 +0100 On Sun, Feb 26, 2006 at 05:21:17PM +0100, Jesper Juhl wrote: > > Hi everyone, Hi Jesper, > I just sat down and build 100 kernels (2.6.16-rc4-mm2 kernels to be exact) > > 95 kernels were build with 'make randconfig'. > 1 kernel was build with the config I normally use for my own box. > 1 kernel was build from 'make defconfig'. > 1 kernel was build from 'make allmodconfig'. > 1 kernel was build from 'make allnoconfig'. > 1 kernel was build from 'make allyesconfig'. > > That was an interresting experience. > > First of all not very many of the kernels actually build correctly and > secondly, if I grep the build logs for warnings I'm swamped. > > Out of 100 kernels 82 failed to build - that's an 18% success rate people, > not very impressive. > > Some of the failed builds are due to things like CONFIG_STANDALONE that > will break the build if not set to Y (unless you have the firmware > available ofcourse), but looking at the config files I find that only 26 > kernels have CONFIG_STANDALONE unset, so that only accounts for a quarter > of the kernels. > > A lot of failed builds are due to invalid combinations of some stuff > being build-in and some stuff being build as modules. > This, as far as I'm concerned, is something that the dependencies in > Kconfig should make impossible - hence my conclusion that we suck at deps. Yes, it should be fixed. Our dependencies are usually relatively good in all the normal cases. I'd expect e.g. half of your randconfig builds to have EMBEDDED=y set, and this often exposes problems. They should be fixed, but it is far from the .config's most users use. And then there's the usual problem with numbers, e.g. each of CONFIG_STANDALONE=n or breakage of the OSS sonicvibes driver will account for a two digit number of build failures. I'd guess fixing two or three problems will bring your 18% number > 50%. > >From 100 kernel builds there was a total of 16152 warnings and 645 of those > are unique warnings, the rest are duplicates. > > We are drowning in warnings people. Sure, many of the warnings are due to > gcc getting something wrong and shouldn't really be emitted, but a lot of > them point to actual problems or deficiencies (I obviously haven't looked > at them all in detail yet, so take that with a grain of salt please). >... It's well-known that BROKEN_ON_SMP drivers often spit 50 warnings in one warning. If you remove the dozen worst drivers the numbers should look much better. Not that our current situation was perfect, but the number of warnings in .config's people usually use isn't that bad. > Jesper Juhl [email blocked] cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed
From: "Jesper Juhl" [email blocked] Subject: Re: Building 100 kernels; we suck at dependencies and drown in warnings Date: Sun, 26 Feb 2006 18:29:54 +0100 On 2/26/06, Adrian Bunk [email blocked] wrote: > On Sun, Feb 26, 2006 at 05:21:17PM +0100, Jesper Juhl wrote: > > > > Hi everyone, > > Hi Jesper, > > > I just sat down and build 100 kernels (2.6.16-rc4-mm2 kernels to be exact) > > > > 95 kernels were build with 'make randconfig'. > > 1 kernel was build with the config I normally use for my own box. > > 1 kernel was build from 'make defconfig'. > > 1 kernel was build from 'make allmodconfig'. > > 1 kernel was build from 'make allnoconfig'. > > 1 kernel was build from 'make allyesconfig'. > > > > That was an interresting experience. > > > > First of all not very many of the kernels actually build correctly and > > secondly, if I grep the build logs for warnings I'm swamped. > > > > Out of 100 kernels 82 failed to build - that's an 18% success rate people, > > not very impressive. > > > > Some of the failed builds are due to things like CONFIG_STANDALONE that > > will break the build if not set to Y (unless you have the firmware > > available ofcourse), but looking at the config files I find that only 26 > > kernels have CONFIG_STANDALONE unset, so that only accounts for a quarter > > of the kernels. > > > > A lot of failed builds are due to invalid combinations of some stuff > > being build-in and some stuff being build as modules. > > This, as far as I'm concerned, is something that the dependencies in > > Kconfig should make impossible - hence my conclusion that we suck at deps. > > Yes, it should be fixed. > > Our dependencies are usually relatively good in all the normal cases. Agreed, most common configs don't spew too many warnings, but that doesn't mean the warnings for the uncommon cases shouldn't be fixed. > I'd expect e.g. half of your randconfig builds to have EMBEDDED=y set, you are dead on : $ grep "EMBEDDED=y" *.config | wc -l 50 > and this often exposes problems. They should be fixed, but it is far > from the .config's most users use. > > And then there's the usual problem with numbers, e.g. each of > CONFIG_STANDALONE=n or breakage of the OSS sonicvibes driver will > account for a two digit number of build failures. I'd guess fixing two > or three problems will bring your 18% number > 50%. > Let's get to work then :) Actually I already send a few patches for a few errors and I plan to send many more. > > >From 100 kernel builds there was a total of 16152 warnings and 645 of those > > are unique warnings, the rest are duplicates. > > > > We are drowning in warnings people. Sure, many of the warnings are due to > > gcc getting something wrong and shouldn't really be emitted, but a lot of > > them point to actual problems or deficiencies (I obviously haven't looked > > at them all in detail yet, so take that with a grain of salt please). > >... > > It's well-known that BROKEN_ON_SMP drivers often spit 50 warnings in one > warning. If you remove the dozen worst drivers the numbers should look > much better. > Better yet, let's fix the warnings. > Not that our current situation was perfect, but the number of warnings > in .config's people usually use isn't that bad. > I agree it's not too bad for most people. The point of my post was mostly a "call to arms" trying to get people interrested in fixing all the warnings and build errors we have. There's a lot of focus on implementing new features - and that's great - but there's little emphasis on fixing the problems we have and already know about - I'd like to see that change, and my post was mainly an attempt at making that happen :) -- Jesper Juhl [email blocked] Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html
From: Adrian Bunk [email blocked] Subject: Re: Building 100 kernels; we suck at dependencies and drown in warnings Date: Sun, 26 Feb 2006 18:41:23 +0100 On Sun, Feb 26, 2006 at 06:29:54PM +0100, Jesper Juhl wrote: > On 2/26/06, Adrian Bunk [email blocked] wrote: >... > > > >From 100 kernel builds there was a total of 16152 warnings and 645 of those > > > are unique warnings, the rest are duplicates. > > > > > > We are drowning in warnings people. Sure, many of the warnings are due to > > > gcc getting something wrong and shouldn't really be emitted, but a lot of > > > them point to actual problems or deficiencies (I obviously haven't looked > > > at them all in detail yet, so take that with a grain of salt please). > > >... > > > > It's well-known that BROKEN_ON_SMP drivers often spit 50 warnings in one > > warning. If you remove the dozen worst drivers the numbers should look > > much better. > > > Better yet, let's fix the warnings. It's never bad if someone converts drivers still using cli/sti. Unfortunately, this is non-trivial... > > Not that our current situation was perfect, but the number of warnings > > in .config's people usually use isn't that bad. > > I agree it's not too bad for most people. The point of my post was > mostly a "call to arms" trying to get people interrested in fixing all > the warnings and build errors we have. I'm agreeing that this is a good idea, I simply disagreed with your "we suck at..." point, since although we can become better we aren't that bad. > There's a lot of focus on implementing new features - and that's great > - but there's little emphasis on fixing the problems we have and > already know about - I'd like to see that change, and my post was > mainly an attempt at making that happen :) IMHO, the real problems we already know about are not warnings, they are the ones listed at http://bugzilla.kernel.org/ And in these cases, the bugs in unmaintained areas of the kernel like APM or the floppy driver are the worst ones. > Jesper Juhl [email blocked] cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed
Submitted
Yes I've summited two build errors, REISERFS_FS_XATTR build error (fixed in 2.6.16-rc3) and MEMORY_HOTPLUG build error. Apparently MEMORY_HOTPLUG builds fine with highmem enabled, but since the memory splits are in the kernel now it errors when enabling 1Gb.
Also lot of warnings coming from the kernel headers about being deprecated. Would it be worth sending these?
Who cares? 2.6 is development
Who cares? 2.6 is development branch. Not anywhere near stable. Or is not that correct? So why bother with bugs that will come and go with relatively large chunks of code.
Only if the development branch got broken more obiously and often as it happened with 2.3 and 2.5 ...
*beats you with a clue-stick
*beats you with a clue-stick
Go to another site, this is not your place.
some people would agree with
some people would agree with you that 2.6 is unstable. I'm one of them.
Who cares? I'll tell you who
Who cares? I'll tell you who cares: everyone who wants to help make 2.6 more stable than it already is.
There's never any harm in fixing bugs, even small ones.
I use 2.6 on all my Linux box
I use 2.6 on all my Linux boxes; it's stable enough for my purposes. Haven't had any real problems. Gentoo systems, one of them an old HP laptop.
The gcc warnings generated during compile do worry me somewhat, but I haven't had problems yet. I'd like to see all of that fixed, but I'm not a programmer, and while I might have a clue as to where to look, the kernel is daunting to me. So I don't bother. And it's not like most of the rest of the distro doesn't generate the same kinds of warnings.
But it would be nice to see that all fixed.
Try 2.4
Of course 2.4 kernels don't even compile with gcc4.1 but if you were to fix the obvious errors enough for it to compile you would get so many warnings that you'd have no idea where to begin.
I use 2.6 compiled with gcc
I use 2.6 compiled with gcc 4.2.0-branch (prerelease) and it is quite stable.
I have never had the kernel crash on it's own. I have only had problems when I tried messing with the .config file and adding experimental features.
The whole idea is to get programs to compile without errors OR warnings. You can't accept a compile with errors since you'd end up with nothing but you accept a compile with errors and sluff it off.
That attitude is exactly why most programs seldom terminate the build with an unfixable compiler error (sometimes a tiny change is neccesary to accomidate a different method not anticipated by the program writer - since they only have one OS to play with).
It is also why we have all these warnings that go unchecked.
We don't need to hear "2.6 is an unstable development branch - don't worry about fixing any warnings" (or errors). People ought to submit code that works as well as it possibly can with as few warning as possible (IE: NO WARNINGS on _the_authors's_ computer, and very few warnings for the rest of us).
People can then submit patches for the few warnings that occur and then the next release will have nearly no warnings and also no problems (and certainly no errors).
But it's people who don't understand computers / software development / correctness
that are complacent with an unacceptable mess.
I can't hit you with my smartening-up stick - it's got a warning on it about being broken!
I come across broken configurations fairly often
i come across broken configurations fairly often, and usually assume that I've done something stupid. But really - it shouldn't be possible to pick a combination that breaks - so this series of emails is my wake-up call to report them.