6

I've been having this severe performance issue, both in Ubuntu 11.10, and Debian Wheezy:

If one process uses up a great deal of RAM (say, 3700 MB out of 3900 MB), the system quickly becomes unusable, with constant disk thrashing. It gets to the point where the window manager doesn't respond, and killing the offending process takes several minutes. (Actually getting to a terminal, issuing pkill, waiting a while longer until the signal actually gets through)

This problem occurs even if the swap partition is completely disabled.

Investigating with iotop, it looks like most of the disk activity is disk read, going to processes such as google chrome, and the X window renderer. However, this is only limited information, as iotop no longer updates while the system is in full-thrash mode.

My question is:

1) what is causing the massive disk read? 2) how can I prevent this from making the system unusable?

Improve this question
2
  • Don't disable swap. That will force the system to keep all dirty pages in memory, making the problem much worse. Tell us more about your outer problem. Why is a process using so much memory? Is it because it's broken? Is it malicious?
    – David Schwartz
    Commented Mar 19, 2012 at 19:14
  • 1
    I've run into the problem in a few different ways. sometimes opening up too many chrome tabs will trigger it, sometimes trying to cache 1M+ data structures in a python script will cause it. Running make with -j of 6 or 7 on something like the Boost libs will eat up memory pretty reliably, and then leave me thrashing.
    – YenTheFirst
    Commented Mar 19, 2012 at 19:45

3 Answers 3

Reset to default
3

The system is moving data in RAM in and out of the hard drive to make way for other data in RAM. Disk operations are far slower than RAM operations, causing the system to slow down significantly. When your system is running out of physical memory, the system does this to expand the amount of "virtual" memory available, at the cost of performance. This condition is called thrashing.

Unfortunately, the only full solution to this problem is to add more RAM. Memory modules are relatively inexpensive; if you are consistently reaching the limits of available physical memory, you should add memory to your computer.

Edit: Because your system is low on physical memory, programs need to read from the hard drive more often, because free space in RAM is normally used to store files frequently or recently accessed to reduce disk reading. With low free RAM, fewer files can be cached, reducing performance. This is especially the case with applications like Chrome, which is quite memory- and I/O-intensive. Once again, adding RAM will increase performance. For more information on caching, see the Wikipedia article.

Improve this answer
11
  • 1
    The problem occurs even if I completely disable a swap partition. Also, I don't think simply more RAM will help, since the problem tends to occur with degenerate programs (some of which are written by me) using up arbitrary amounts of memory.
    – YenTheFirst
    Commented Mar 19, 2012 at 18:47
  • Some system process may be blocking normal I/O operations in applications such as Chrome. In this case, it appears the OOM killer is attempting to kill processes using abnormal amounts of memory.
    – bwDraco
    Commented Mar 19, 2012 at 18:51
  • 2
    I'm sorry, I don't quite understand this comment. The I/O isn't blocked - Chrome, and X, and other random processes, are doing a lot of disk reading. No processes are killed by the Out of Memory killer, since the memory's not actually full, just nearly full. If the OOM would at least kill the offending process, it wouldn't be so bad.
    – YenTheFirst
    Commented Mar 19, 2012 at 18:59
  • 1
    This answer most clearly describes what's going on, so I'm accepting it. Any ideas on 2)? Simply adding more RAM isn't always an option (especially if you've maxed out the RAM on a laptop). Low memory situations happen in life, how can Linux be made to not become unusable in those situations? i.e., if I've only got 200MB free, I don't care about rendering chrome, I just want to switch to a terminal and kill some processes.
    – YenTheFirst
    Commented Mar 19, 2012 at 22:52
  • 1
    You could use a USB flash drive and dedicate it to swap space. Buy a high-speed flash drive (at least 4 GB), create a swap partition on it, and activate it. The effect of this is similar to using Windows ReadyBoost on a Windows Vista or Windows 7 system.
    – bwDraco
    Commented Mar 20, 2012 at 0:01
3

If what causes your problems are known processes, you can restrict them using ulimit For example, script the launcher of your app like this:

#!/bin/bash
ulimit -v 1048576
/usr/bin/greedyapp

Your app will be able to get only 1 GiB of virtual memory before getting out-of-memory errors. I've tried it with browsers and it will effectively protect your RAM without killing the browser straight away when reaches its limit. But the app may then behave oddly (like stop responding user input), so you may try it first with a very strict limit to see what it does in an out-of-memory. Still, way better than those hd trashes.

Note that ulimit only affects the calling process and its childs, setting the limit for all the processes of that branch. I wouldn't recommend to put it in your session script.

Improve this answer
1

DragonLord is correct, the issue is the fact that the OS does not like that your using an excessive amount of RAM. The closer you are to maxing out your RAM. The system will fall into a state where it starts using virtual memory. The faster your harddrive the faster your computer will be when in this state, however there is only so much performance you can gain from that. More RAM is needed in order to stop this from happening.

Clearly you already realize the issue is only when using certain programs, some in which you have created. Due to the information that has been given it is only logical to assume your issue is RAM.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .