login
Search
"Applications with dynamic input and dynamic memory usage have some issues with the current overcommitting kernel," Daniel Spång explained looking for ideas on how to best manage out of memory (OOM) situations on embedded systems with little memory and without swap. He noted, "some kind of notification to the application that the available memory is scarce and let the application free up some memory (e.g., by flushing caches), could be used to improve the situation and avoid the OOM killer." Daniel then briefly described four possible solutions, looking for other ideas:
"1) Turn off overcommit. Results in a waste of memory. 2) Nokia uses a lowmem security module to signal on predetermined thresholds. Currently available in the -omap tree. But this requires manual tuning of the thresholds. 3) Using madvise() with MADV_FREE to get the kernel to free mmaped memory, typically application caches, when the kernel needs the memory. 4) A OOM handler that the application registers with the kernel, and that the kernel executes before the OOM-killer steps in."
From: Daniel Spång Subject: Out of memory management in embedded systems Date: Sep 28, 5:55 am 2007 Applications with dynamic input and dynamic memory usage have some issues with the current overcommitting kernel. A high memory usage situation eventually results in that a process is killed by the OOM killer. This is especially evident in swapless embedded systems with limited memory and no swap available. Some kind of notification to the application that the available memory is scarce and let the application free up some memory (e.g., by flushing caches), could be used to improve the situation and avoid the OOM killer. I am currently not aware of any general solution to this problem, but I have found some approaches that might (or might not) work: o Turn off overcommit. Results in a waste of memory. o Nokia uses a lowmem security module to signal on predetermined thresholds. Currently available in the -omap tree. But this requires manual tuning of the thresholds. http://www.linuxjournal.com/article/8502 o Using madvise() with MADV_FREE to get the kernel to free mmaped memory, typically application caches, when the kernel needs the memory. o A OOM handler that the application registers with the kernel, and that the kernel executes before the OOM-killer steps in. Does it exist any other solutions to this problem? Daniel -
From: linux-os (Dick Johnson) Subject: Re: Out of memory management in embedded systems Date: Sep 28, 6:09 am 2007 On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote: > Applications with dynamic input and dynamic memory usage have some > issues with the current overcommitting kernel. A high memory usage > situation eventually results in that a process is killed by the OOM > killer. This is especially evident in swapless embedded systems with > limited memory and no swap available. > > Some kind of notification to the application that the available memory > is scarce and let the application free up some memory (e.g., by > flushing caches), could be used to improve the situation and avoid the > OOM killer. I am currently not aware of any general solution to this > problem, but I have found some approaches that might (or might not) > work: > > o Turn off overcommit. Results in a waste of memory. > > o Nokia uses a lowmem security module to signal on predetermined > thresholds. Currently available in the -omap tree. But this requires > manual tuning of the thresholds. > http://www.linuxjournal.com/article/8502 > > o Using madvise() with MADV_FREE to get the kernel to free mmaped > memory, typically application caches, when the kernel needs the > memory. > > o A OOM handler that the application registers with the kernel, and > that the kernel executes before the OOM-killer steps in. > > Does it exist any other solutions to this problem? > > Daniel > - But an embedded system contains all the software that will ever be executed on that system! If it is properly designed, it can never run out of memory because everything it will ever do is known at design time. This should never be an issue with an embedded system. If you have such a system issue, then you have application(s) that have memory leaks because of improper design or coding. For instance, there is a common open-source web-server that is used in some embedded systems. It has memory leaks. The solution, if the server can't be fixed, is to execute a supervisor process which periodically shuts it down and restarts it --ugly, but effective if the developers refuse to accept patches. You shouldn't expect a kernel to be modified to "fix" broken application code. Cheers, Dick Johnson Penguin : Linux version 2.6.22.1 on an i686 machine (5588.29 BogoMips). My book : http://www.AbominableFirebug.com/ _ **************************************************************** The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them. Thank you. -
From: Daniel Spång Subject: Re: Out of memory management in embedded systems Date: Sep 28, 6:30 am 2007 On 9/28/07, linux-os (Dick Johnson) <linux-os@analogic.com> wrote: > > On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote: > > > Applications with dynamic input and dynamic memory usage have some > > issues with the current overcommitting kernel. A high memory usage > > situation eventually results in that a process is killed by the OOM > > killer. This is especially evident in swapless embedded systems with > > limited memory and no swap available. > > > > Some kind of notification to the application that the available memory > > is scarce and let the application free up some memory (e.g., by > > flushing caches), could be used to improve the situation and avoid the > > OOM killer. I am currently not aware of any general solution to this > > problem, but I have found some approaches that might (or might not) > > work: > > > > o Turn off overcommit. Results in a waste of memory. > > > > o Nokia uses a lowmem security module to signal on predetermined > > thresholds. Currently available in the -omap tree. But this requires > > manual tuning of the thresholds. > > http://www.linuxjournal.com/article/8502 > > > > o Using madvise() with MADV_FREE to get the kernel to free mmaped > > memory, typically application caches, when the kernel needs the > > memory. > > > > o A OOM handler that the application registers with the kernel, and > > that the kernel executes before the OOM-killer steps in. > > > > Does it exist any other solutions to this problem? > > > > Daniel > > - > > But an embedded system contains all the software that will > ever be executed on that system! If it is properly designed, > it can never run out of memory because everything it will > ever do is known at design time. Not if its input is not known beforehand. Take a browser in a mobile phone as an example, it does not know at design time how big the web pages are. On the other hand we want to use as much memory as possible, for cache etc., a method that involves the kernel would simplify this and avoids setting manual limits. Daniel -
From: linux-os (Dick Johnson) Subject: Re: Out of memory management in embedded systems Date: Sep 28, 7:04 am 2007 On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote: > On 9/28/07, linux-os (Dick Johnson) <linux-os@analogic.com> wrote: >> >> On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote: >> >>> Applications with dynamic input and dynamic memory usage have some >>> issues with the current overcommitting kernel. A high memory usage >>> situation eventually results in that a process is killed by the OOM >>> killer. This is especially evident in swapless embedded systems with >>> limited memory and no swap available. >>> >>> Some kind of notification to the application that the available memory >>> is scarce and let the application free up some memory (e.g., by >>> flushing caches), could be used to improve the situation and avoid the >>> OOM killer. I am currently not aware of any general solution to this >>> problem, but I have found some approaches that might (or might not) >>> work: >>> >>> o Turn off overcommit. Results in a waste of memory. >>> >>> o Nokia uses a lowmem security module to signal on predetermined >>> thresholds. Currently available in the -omap tree. But this requires >>> manual tuning of the thresholds. >>> http://www.linuxjournal.com/article/8502 >>> >>> o Using madvise() with MADV_FREE to get the kernel to free mmaped >>> memory, typically application caches, when the kernel needs the >>> memory. >>> >>> o A OOM handler that the application registers with the kernel, and >>> that the kernel executes before the OOM-killer steps in. >>> >>> Does it exist any other solutions to this problem? >>> >>> Daniel >>> - >> >> But an embedded system contains all the software that will >> ever be executed on that system! If it is properly designed, >> it can never run out of memory because everything it will >> ever do is known at design time. > > Not if its input is not known beforehand. Take a browser in a mobile > phone as an example, it does not know at design time how big the web > pages are. On the other hand we want to use as much memory as > possible, for cache etc., a method that involves the kernel would > simplify this and avoids setting manual limits. > > Daniel > Any networked appliance can (will) throw data away if there are no resources available. The length of a web-page is not relevent, nor is the length of any external data. Your example will buffer whatever it can and not read anything more from the external source until it has resources available unless it is broken. Cheers, Dick Johnson Penguin : Linux version 2.6.22.1 on an i686 machine (5588.29 BogoMips). My book : http://www.AbominableFirebug.com/ _ **************************************************************** The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them. Thank you. -
From: Rik van Riel Subject: Re: Out of memory management in embedded systems Date: Sep 28, 7:17 am 2007 On Fri, 28 Sep 2007 10:04:23 -0400 "linux-os \(Dick Johnson\)" <linux-os@analogic.com> wrote: > On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote: > > > On 9/28/07, linux-os (Dick Johnson) <linux-os@analogic.com> wrote: > >> > >> On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote: > >>> Some kind of notification to the application that the available memory > >>> is scarce and let the application free up some memory (e.g., by > >>> flushing caches), could be used to improve the situation > Any networked appliance can (will) throw data away if there are > no resources available. That is exactly what Daniel proposed in his first email. I think his idea makes sense. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan -
From: Daniel Spång Subject: Re: Out of memory management in embedded systems Date: Sep 28, 7:14 am 2007 On 9/28/07, linux-os (Dick Johnson) <linux-os@analogic.com> wrote: > > On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote: > > > On 9/28/07, linux-os (Dick Johnson) <linux-os@analogic.com> wrote: > >> > >> But an embedded system contains all the software that will > >> ever be executed on that system! If it is properly designed, > >> it can never run out of memory because everything it will > >> ever do is known at design time. > > > > Not if its input is not known beforehand. Take a browser in a mobile > > phone as an example, it does not know at design time how big the web > > pages are. On the other hand we want to use as much memory as > > possible, for cache etc., a method that involves the kernel would > > simplify this and avoids setting manual limits. > > > > Daniel > > > > Any networked appliance can (will) throw data away if there are > no resources available. > > The length of a web-page is not relevent, nor is the length > of any external data. Your example will buffer whatever it > can and not read anything more from the external source until > it has resources available unless it is broken. And how do you determine when no resources are availabe? We are using overcommit here so malloc() will always return non null. -
From: linux-os (Dick Johnson) Subject: Re: Out of memory management in embedded systems Date: Sep 28, 8:16 am 2007 On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote: > On 9/28/07, linux-os (Dick Johnson) <linux-os@analogic.com> wrote: >> >> On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote: >> >>> On 9/28/07, linux-os (Dick Johnson) <linux-os@analogic.com> wrote: >>>> >>>> But an embedded system contains all the software that will >>>> ever be executed on that system! If it is properly designed, >>>> it can never run out of memory because everything it will >>>> ever do is known at design time. >>> >>> Not if its input is not known beforehand. Take a browser in a mobile >>> phone as an example, it does not know at design time how big the web >>> pages are. On the other hand we want to use as much memory as >>> possible, for cache etc., a method that involves the kernel would >>> simplify this and avoids setting manual limits. >>> >>> Daniel >>> >> >> Any networked appliance can (will) throw data away if there are >> no resources available. >> >> The length of a web-page is not relevent, nor is the length >> of any external data. Your example will buffer whatever it >> can and not read anything more from the external source until >> it has resources available unless it is broken. > > And how do you determine when no resources are availabe? We are using > overcommit here so malloc() will always return non null. > A networked appliance using embedded software is not your daddy's Chevrolet. Any task that is permanent needs to allocate all its resources when it starts. That's how it knows how much there are, and incidentally, it doesn't do it blindly. The system designer must know how much memory is available in the system and how much is allocated to the kernel. The fact that you can give a fictitious value to malloc() is not relevant. If you don't provide resources for malloc(), like (ultimately) a swap file, then you can't assume that it can do any design work for you. An embedded system is NOT an ordinary system that happens to boot from flash. An embedded system requires intelligent design. It is important to understand how a virtual memory system operates. The basics are that the kernel only "knows" that a new page needs to be allocated when it encounters a trap called a "page fault." If you don't have any memory resources to free up (read no swap file to write a seldom-used task's working set), then you are screwed --pure and simple. So, if you don't provide any resources to actually use virtual memory, then you need to make certain that virtual memory and physical memory are, for all practical purposes, the same. With embedded servers, it's usually very easy to limit the number of connections allowed, therefore the amount of dynamic resources that must be provided. With clients it should be equally easy, but generic software won't work because, for instance, Mozilla doesn't keep track of the number of "windows" you have up and the number of connections you have. HOWEVER, remember that malloc() is a library call. You can substitute your own using LD_PRELOAD, they keeps track of everything if you must use generic software. Cheers, Dick Johnson Penguin : Linux version 2.6.22.1 on an i686 machine (5588.29 BogoMips). My book : http://www.AbominableFirebug.com/ _ **************************************************************** The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them. Thank you. -
Come On!
This is not an embedded problem only, desktop systems ca go out of memory as well, and i think its better to send a (global) signal to Firefox (and friends) to free some old-pages-cache, then swapping out other data/processes to disk if you don't really need to. I think its to easy for the kernel to swap out data today, swapping out data shuould really be the last option.
point there. hitting swap
point there. hitting swap should be a last resort, if at all available.
if one can have the kernel go "sorry, unless you can do some house cleaning of you own, your out of luck" to a program, it would be helpful.
maybe with a more polite "do some cleaning. and of that dont help, ill see what i can do" for when there is swap available.
WTF kind of "embedded" is he talking about?
That's more of a "closed" real-time system than a generic "embedded" one. Lots of embedded platforms have non-permanent tasks, and dynamically varying resource availability.
The argument from Dick Johnson is a bunch of hooey. At least, as a general argument against this feature. There are systems for which it applies -- the canonical 8-bit microcontroller managing a saw, which may well be *managed* by a Linux box, for one -- but it's completely wrong to say that applies to all embedded systems.
Completely agree on this.
Completely agree on this. The demagogy by Dick Johnson applied only to extremely small and simple embedded systems.
Actually from what he tells, one can conclude that Linux (with overcommitting switched ON) is not suitable for embedded system at all, as according to his argumentation, applications always know how much memory they will ever use, and so there is no need for overcommitting - just reserve everything from the beginning.
This is just a way for him to disregard the problem instead of thinking of solution.
Implementation
An implementation in "another operating system" pops up a message box that warns the user that the virtual memory is low.
message box
Implementation, yeah... How a message box (leaving aside there's often no display to show such a message box on) enables applications to voluntarily free some memory they do not need so badly in order to reduce memory pressure?
Showing message boxes means admitting you really screwed up and begging the user to try some random actions (which is exactly what she will do) in the hope some of them will mitigate the situation.
So the problem is that you
So the problem is that you have an application that can be written to use differing amounts of memory to make different speed/space tradeoffs. You would like that application to always make the best use of memory.
As far as I can tell, the only solution is to have the memory/space tradeoff managed by the part of the system that knows how much memory is available globally - the kernel. Specifically, there would have to be a new alloc call, that allocates a block of memory of a given size *along with a function to regenerate it if it is freed*.
void *cache_alloc(size_t size, void regenerate(void *mem, void *user_data), void *regenerate_data);
When the cache_alloc call is made, the kernel will pretend to allocate memory (it could even do it, if it was available). The idea is, if an application page faults into a cache area, the kernel has the choice of either allocating a new page, or possibly just letting the regenerate function do its thing and then returning the value it needs.
The regenerate function is basically an unfold. For a browser, regenerate would probably be a page-loading function, and the user_data would be the URL of a page. A browser that used this to cache pages would always use the optimal amount of memory (or at least, decently close).
Dont forget.
Dont forget to add a call to move an early malloc() object to an cache-object, for example in a web-browser, the content on the page you are looking for is malloc(), and when the user goes to next page, he will change state on the malloced() data, to a cached data that may be freed. I wonder how symbian solution works, symbian have a system-call "Reclaim Memory) that tells all applications to leave back unnessesary used ram, and to rezise over-sized allocations to the size of the actual content.
A fifth possibility for user mode/kernel OOM interaction
One other possibility that occurs to me for handling out of memory interactions between kernel and user space (note, this is a very rough first sketch): Allow registration of sections of virtual memory with the kernel and associate them with a flagn, very possibly implemented as a a word in memory. This flag can have one of three states:
The application can set the flag to Locked or Unlocked, if the kernel finds flag has value Unlocked, it will set it to Gone before it grabs the memory.
This could be implemented, in part, by adding a flag to mmap() to indicate that the memory has the special characteristic of being grabbable, but we might need to add a system call to register the status location associated with the memory. Once the memory has been grabbed, you'd use mmap() to get it mapped again.
There can be multiple grabbable pieces of memory.
One nice thing about this, if the status location is in memory, it would be very quick if the memory was still available.
--
Note: this is my opinion and may not represent that of my employer
SIGNOMEM
Wouldn't it be easier to just invent a new signal- SIGNOMEM, and send that to processes. The default handler can do nothing, and if the application wants to play along, it can handle the signal by freeing up some memory. Having an application register a callback with the kernel seems too messy, given that we already have a mechanism for this type of notification.
Using signals to notify apps about OOM conditions
I'm a little leery of using signals when in OOM conditions. One concern is that signal delivery will itself require allocating memory if the stack needs to grow. Additionally, you have to get scheduled before memory can be reclaimed, which means that there is an opportunity for your memory condition to worsen before you can do anything. I'd prefer a solution where the kernel can grab the memory immediately, and then the application can respond appropriately after the fact.
A modification of my previous half-baked suggestion where you could use a "Grabbable" flag to mmap() to identify memory that the kernel can grab, coupled with a signal to the process to indicate that memory has been grabbed might be a reasonable thing to do. I'm a little concerned that you might have issues should you chose to defer the signal for a while, but that might just be an application programming issue.
--
This opinion is mine, and may not represent that of my employer...
AmigaOS had a similar system
Back in the old Amiga days it was possible to install a 'low memory handler' into the system whereby if the kernel realised it had no memory when it was attempted to be allocated, it would call a set of chained, prioritised handlers installed by applications which would free up memory from the apps, by the apps callbacks/handlers.
For instance, if a web browser is running and a user loads a game - and the game really needs memory for loading textures, it might AllocVec() a bunch of buffers. At the point AllocVec() has not got enough memory to spare, it would go through the list of low memory handlers - the web browser would have one installed, and when called, might free cached pages, images or close network sockets in order to free resources. As soon as a handler finished it would check to see if it had enough memory to fulfil the allocation - if not, it moved to the next handler, and so on. If it got to the end it would simply give up :)
It's possible for a task to install MULTIPLE handlers at different priorities depending on the importance of it's memory for speed or effective operation.
This is not so different to Firefox's "Trim on Minimize" on Windows. If an application does not have focus and other work is being done, it's sometimes a good idea to preemptively kill off some of it's cached data and buffers until the point that it IS in focus and in use.
It really is a useful system, and I'd be really excited to see this in Linux.. for X.org to dump it's pixmap caches, for Firefox to cull certain parts of it's caches or whatever. However, I am concerned that a single signal callback is not enough - and how do you make sure only the least important data is freed first? Relying on UNIX signals is probably a bad idea, it needs a different system. Could it be done through sysfs or some syscall callback registration?
Relying on UNIX signals is
signal() is a syscall. The passed signal handler is a callback. :)
The alternative Unix-ish way would be explicitly opening a character device (e.g. /dev/oomevent) and reading notifications from there.
Signals are portable, using
Signals are portable, using sysfs for this purpose would make this feature permanently Linux-only.
zojOFpeyQdVIDF
paxil cuurk discount elidel 756237 buy flomax syv
BWVlXRyuCTKkQJ
accutane aypbyx