With the recent release of 2.6.0-test10 [story], Linus Torvalds acknowledged a problem that seems to be triggered by enabling CONFIG_PREEMPT. Bradley Chapman posted to the lkml asking, "what exactly are the issues with 2.6.0-test10 preempt?" During the following discussion, Linus explained:

"Basically, there's something strange going on, which _seems_ to be memory corruption, and seems to correlate [reasonably] well (but not 100%) with CONFIG_PREEMPT."

"It's actually unlikely to be preemption itself that is broken: it's much more likely that some driver or other subsystem is broken, and preempt is just better at triggering it by making some race conditions much easier to see due to bigger windows for them to happen."

He goes on to add, "The problem is finding enough of a pattern to the reports to make sense of what seems to be the common thread. A lot of people use preemption without any trouble." Read on for the continued discussion.

From: Bradley Chapman [email blocked]
To:  linux-kernel
Subject: What exactly are the issues with 2.6.0-test10 preempt?
Date: Mon, 24 Nov 2003 11:14:59 -0800 (PST)

I saw in Linus' 2.6.0-test10 announcement that preempt is suffering from some
problems and should not be used. However, I am currently running 2.6.0-test10
with CONFIG_PREEMPT=y and nothing has appeared yet. To see if the problem appeared
under stress, I started an A/V trailer playback with mplayer and then ran the
find command on both my home directory and the 2.6.0-test10 kernel source directory,
with the expected result - mplayer did not skip, neither find invocation broke,
and there were no nasty errors in dmesg.

So what exactly is the problem?

I can provide more system info on request.

TIA

Brad

=====



From: Guennadi Liakhovetski [email blocked]
Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?
Date: Mon, 24 Nov 2003 22:08:27 +0100 (CET)

On Mon, 24 Nov 2003, Bradley Chapman wrote:

> I saw in Linus' 2.6.0-test10 announcement that preempt is suffering from some
> problems and should not be used. However, I am currently running 2.6.0-test10
> with CONFIG_PREEMPT=y and nothing has appeared yet. To see if the problem appeared
> under stress, I started an A/V trailer playback with mplayer and then ran the
> find command on both my home directory and the 2.6.0-test10 kernel source directory,
> with the expected result - mplayer did not skip, neither find invocation broke,
> and there were no nasty errors in dmesg.
>
> So what exactly is the problem?

Well, FWIW, I'm getting 100% reproducible Oopses on __boot__ by enabling
preemption AND (almost) all kernel-hacking CONFIG_DEBUG_* options - see my
post of 21.11.2003 with subject "[OOPS] 2.6.0-test7 + preempt + hacking".
If required, could try to narrow it down to 1 CONFIG option. However, the
Oops itself happens somewhere in NFS code (see backtrace in above email
for details).

Guennadi
---
Guennadi Liakhovetski



From: Linus Torvalds [email blocked]
Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?
Date: Mon, 24 Nov 2003 13:59:24 -0800 (PST)

On Mon, 24 Nov 2003, Guennadi Liakhovetski wrote:
>
> Well, FWIW, I'm getting 100% reproducible Oopses on __boot__ by enabling
> preemption AND (almost) all kernel-hacking CONFIG_DEBUG_* options - see my
> post of 21.11.2003 with subject "[OOPS] 2.6.0-test7 + preempt + hacking".
> If required, could try to narrow it down to 1 CONFIG option.

I'd love to have more info - I actually looked at your original report,
and it's one of those "impossible" things as far as I can tell. The low
bit of the work "pending" flag should acts as a lock on workqueues, and
serialize access to one workqueue totally - so having it show up with a
pending timer is "strange" to say the least. The only two ways to clear
the "pending" timer is by running the work-queue - either for the timer to
have gone off (for the delayed case) _or_ the timer not to have evern been
set in the first place (for the immediate case).

So more information would be wonderful.

		Linus



From: Bradley Chapman [email blocked]
Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?
Date: Mon, 24 Nov 2003 14:26:52 -0800 (PST)

Mr. Torvalds,

--- Linus Torvalds [email blocked] wrote:
> 
> So more information would be wonderful.

What sort of information would you like me to provide, sir? The bug you're
discussing here isn't affecting me; CONFIG_PREEMPT has been solid on 2.6.0-test10.
This is on a Gateway 600S laptop with a P4-M 2Ghz processor and an i845 Brookdale
chipset.

> 
> 		Linus

Brad



From: Linus Torvalds [email blocked]
Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?
Date: Mon, 24 Nov 2003 14:32:31 -0800 (PST)

On Mon, 24 Nov 2003, Bradley Chapman wrote:
>
> What sort of information would you like me to provide, sir? The bug you're
> discussing here isn't affecting me; CONFIG_PREEMPT has been solid on 2.6.0-test10.
> This is on a Gateway 600S laptop with a P4-M 2Ghz processor and an i845 Brookdale
> chipset.

Basically, there's something strange going on, which _seems_ to be memory
corruption, and seems to correlate reasonable well (but not 100%) with
CONFIG_PREEMPT.

It's actually unlikely to be preemption itself that is broken: it's much
more likely that some driver or other subsystem is broken, and preempt is
just better at triggering it by making some race conditions much easier to
see due to bigger windows for them to happen.

The problem is finding enough of a pattern to the reports to make sense of
what seems to be the common thread. A lot of people use preemption without
any trouble.

		Linus



From: Bradley Chapman [email blocked]
Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?
Date: Mon, 24 Nov 2003 14:45:14 -0800 (PST)

Mr. Torvalds,

--- Linus Torvalds [email blocked] wrote:
> 
> Basically, there's something strange going on, which _seems_ to be memory
> corruption, and seems to correlate reasonable well (but not 100%) with
> CONFIG_PREEMPT.

Ah, I see. I thought there was a definite issue with a certain subsystem that
just hadn't been fixed yet when CONFIG_PREEMPT=y.

> 
> It's actually unlikely to be preemption itself that is broken: it's much
> more likely that some driver or other subsystem is broken, and preempt is
> just better at triggering it by making some race conditions much easier to
> see due to bigger windows for them to happen.
> 
> The problem is finding enough of a pattern to the reports to make sense of
> what seems to be the common thread. A lot of people use preemption without
> any trouble.

Indeed. Do the same subsystems usually show the memory corruption issue with
preempt active, or does it just pop up all over the place, unpredictably?

> 
> 		Linus

Brad



From: Linus Torvalds [email blocked]
Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?
Date: Mon, 24 Nov 2003 15:00:40 -0800 (PST)


On Mon, 24 Nov 2003, Bradley Chapman wrote:
>
> Indeed. Do the same subsystems usually show the memory corruption issue with
> preempt active, or does it just pop up all over the place, unpredictably?

There are a few reports of "predictable" memory corruption, in the sense
that the same people tend to see the same kinds of oopses _without_ any
other signs of memory corruption (ie no random SIGSEGV's in user space
etc).

There's the magic slab corruption thing, there's a strange thread data
corruption (one person), and there's the sunrpc timer bug. All are
"impossible" bugs that would indicate a small amount of data corruption in
some core data structure.

They are hard to trigger, which makes me personally suspect some
user-after-free thing, where the bug happens only when somebody else
allocates (and uses) the entry immediately afterwards (so that the old
user overwrites stuff that just got initialized for the new user).

It's not likely to be a wild pointer: those tend to corrupt random memory,
and that in turn is a lot more likely to result in _user_ corruptions
(causing SIGSEGV's, corrupted files that magically become ok again when
re-read, etc), since 99% of all memory tends to be non-kernel data
structures.

The PAGEFREE debug option works well for page allocations, but the slab
cache is not very amenable to it. For slab debugging, it would be
wonderful if somebody made a _truly_ debugging slab allocator that didn't
use the slab cache at all, but used the page allocator (and screw the fact
that you use too much memory ;) instead.

(Sadly, some slab users actually use that stupid "initialize" crap. We
should rip it out: it's a disaster from a data cache standpoint too, since
it tends to do all the wrong things there, even though it's literally
meant to help).

		Linus


		
From: Linus Torvalds [email blocked]
Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?
Date: Mon, 24 Nov 2003 15:45:24 -0800 (PST)


On Mon, 24 Nov 2003, Linus Torvalds wrote:
>
> The PAGEFREE debug option works well for page allocations, but the slab
> cache is not very amenable to it. For slab debugging, it would be
> wonderful if somebody made a _truly_ debugging slab allocator that didn't
> use the slab cache at all, but used the page allocator (and screw the fact
> that you use too much memory ;) instead.

Ok, this is quite possibly the ugliest patch I've ever made, but it might
catch some of the "use-after-free" issues with slab.

IT WILL NOT WORK IN GENERAL! In particular, you'll need to have tons of
memory free for this patch to work, since it basically scrapes off all the
regular slab code, and replaces it with some really nasty crud. I was
lazy. Whatever. It means that the slab memory balancing won't work etc,
but maybe somebody else can integrate my quick hack better.

It boots for me, but I won't say anything beyond that (oh, do NOT enable
slab debugging if you want to test this out - I pretty much guarantee it
won't work with this hack. You should enable DEBUG_PAGEALLOC instead,
and see if you get anything interesting out of that..).

		Linus

[patch]

• Archive of above thread