4

I have an ARM based server with just under 2GB of addressable memory and 4GB of swap activated:

root@bang:~> free -m
              total        used        free      shared  buff/cache   available
Mem:           1976         388          48          15        1539        1487
Swap:          4095           1        4094

Once the system has been up for a day or so, the OOM killer starts getting a bit aggressive and starts killing things:

Aug  3 12:59:01 bang kernel: [51585.822794] dump1090 invoked oom-killer: gfp_mask=0x24040c0(GFP_KERNEL|__GFP_COMP), order=2, oom_score_adj=0
Aug  3 12:59:01 bang kernel: [51585.822851] dump1090 cpuset=/ mems_allowed=0
Aug  3 12:59:01 bang kernel: [51585.822963] CPU: 6 PID: 25989 Comm: dump1090 Tainted: G         C      4.7.0-41238-g206dbde-dirty #16
Aug  3 12:59:01 bang kernel: [51585.823010] Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
Aug  3 12:59:01 bang kernel: [51585.823120] [<c010e4ec>] (unwind_backtrace) from [<c010b234>] (show_stack+0x10/0x14)
Aug  3 12:59:01 bang kernel: [51585.823203] [<c010b234>] (show_stack) from [<c04eff84>] (dump_stack+0x88/0x9c)
Aug  3 12:59:01 bang kernel: [51585.823283] [<c04eff84>] (dump_stack) from [<c0227830>] (dump_header+0x5c/0x1b0)
Aug  3 12:59:01 bang kernel: [51585.823357] [<c0227830>] (dump_header) from [<c01d1aec>] (oom_kill_process+0x328/0x494)
Aug  3 12:59:01 bang kernel: [51585.823420] [<c01d1aec>] (oom_kill_process) from [<c01d1fa0>] (out_of_memory+0x2e0/0x338)
Aug  3 12:59:01 bang kernel: [51585.823487] [<c01d1fa0>] (out_of_memory) from [<c01d6724>] (__alloc_pages_nodemask+0xd80/0xda0)
Aug  3 12:59:01 bang kernel: [51585.823555] [<c01d6724>] (__alloc_pages_nodemask) from [<c01d6a28>] (alloc_kmem_pages+0x18/0xb0)
Aug  3 12:59:01 bang kernel: [51585.823620] [<c01d6a28>] (alloc_kmem_pages) from [<c01ee7a4>] (kmalloc_order+0x10/0x20)
Aug  3 12:59:01 bang kernel: [51585.823688] [<c01ee7a4>] (kmalloc_order) from [<c06435b4>] (proc_submiturb+0x60c/0xe88)
Aug  3 12:59:01 bang kernel: [51585.823749] [<c06435b4>] (proc_submiturb) from [<c06446e4>] (usbdev_do_ioctl+0x8b4/0x1bfc)
Aug  3 12:59:01 bang kernel: [51585.823816] [<c06446e4>] (usbdev_do_ioctl) from [<c023c74c>] (do_vfs_ioctl+0x98/0x8e4)
Aug  3 12:59:01 bang kernel: [51585.823879] [<c023c74c>] (do_vfs_ioctl) from [<c023d004>] (SyS_ioctl+0x6c/0x7c)
Aug  3 12:59:01 bang kernel: [51585.823948] [<c023d004>] (SyS_ioctl) from [<c0107740>] (ret_fast_syscall+0x0/0x3c)
Aug  3 12:59:01 bang kernel: [51585.823987] Mem-Info:
Aug  3 12:59:01 bang kernel: [51585.824073] active_anon:43846 inactive_anon:46454 isolated_anon:0
Aug  3 12:59:01 bang kernel: [51585.824073]  active_file:132799 inactive_file:109909 isolated_file:19
Aug  3 12:59:01 bang kernel: [51585.824073]  unevictable:1408 dirty:56 writeback:0 unstable:0
Aug  3 12:59:01 bang kernel: [51585.824073]  slab_reclaimable:17104 slab_unreclaimable:6387
Aug  3 12:59:01 bang kernel: [51585.824073]  mapped:13368 shmem:3582 pagetables:971 bounce:0
Aug  3 12:59:01 bang kernel: [51585.824073]  free:92967 free_pcp:31 free_cma:32601
Aug  3 12:59:01 bang kernel: [51585.824216] Normal free:13240kB min:3420kB low:4272kB high:5124kB active_anon:26652kB inactive_anon:26692kB active_file:360240kB inactive_file:194904kB unevictable:1336kB isolated(anon):0kB isolated(file):76kB present:770048kB managed:736192kB mlocked:1336kB dirty:16kB writeback:0kB mapped:11600kB shmem:900kB slab_reclaimable:68416kB slab_unreclaimable:25548kB kernel_stack:3384kB pagetables:3884kB unstable:0kB bounce:0kB free_pcp:124kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Aug  3 12:59:01 bang kernel: [51585.824259] lowmem_reserve[]: 0 9040 9040
Aug  3 12:59:01 bang kernel: [51585.824442] HighMem free:358664kB min:512kB low:1864kB high:3216kB active_anon:148732kB inactive_anon:159124kB active_file:170956kB inactive_file:244732kB unevictable:4296kB isolated(anon):0kB isolated(file):0kB present:1288192kB managed:1288192kB mlocked:4296kB dirty:208kB writeback:0kB mapped:41872kB shmem:13428kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:130404kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Aug  3 12:59:01 bang kernel: [51585.824483] lowmem_reserve[]: 0 0 0
Aug  3 12:59:01 bang kernel: [51585.824592] Normal: 1300*4kB (UMEH) 525*8kB (UMEH) 11*16kB (H) 9*32kB (H) 8*64kB (H) 5*128kB (H) 3*256kB (H) 1*512kB (H) 1*1024kB (H) 0*2048kB 0*4096kB = 13320kB
Aug  3 12:59:01 bang kernel: [51585.825061] HighMem: 1212*4kB (UMC) 538*8kB (UM) 160*16kB (UM) 140*32kB (UMC) 108*64kB (UMC) 34*128kB (UM) 19*256kB (UMC) 10*512kB (UM) 8*1024kB (UMC) 7*2048kB (UMC) 73*4096kB (UMC) = 358976kB
Aug  3 12:59:01 bang kernel: [51585.825558] 247387 total pagecache pages
Aug  3 12:59:01 bang kernel: [51585.825596] 18 pages in swap cache
Aug  3 12:59:01 bang kernel: [51585.825636] Swap cache stats: add 1360, delete 1342, find 33/71
Aug  3 12:59:01 bang kernel: [51585.825672] Free swap  = 4190368kB
Aug  3 12:59:01 bang kernel: [51585.825705] Total swap = 4194300kB
Aug  3 12:59:01 bang kernel: [51585.825739] 514560 pages RAM
Aug  3 12:59:01 bang kernel: [51585.825772] 322048 pages HighMem/MovableOnly
Aug  3 12:59:01 bang kernel: [51585.825804] 8464 pages reserved
Aug  3 12:59:01 bang kernel: [51585.825836] 32768 pages cma reserved
Aug  3 12:59:01 bang kernel: [51585.825869] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Aug  3 12:59:01 bang kernel: [51585.825958] [ 2363]     0  2363     2724      664       8       0       13         -1000 systemd-udevd
Aug  3 12:59:01 bang kernel: [51585.826019] [ 4035]     0  4035     1736      445       7       0       16             0 syslog-ng
Aug  3 12:59:01 bang kernel: [51585.826073] [ 4036]     0  4036    11306     1067      15       0       38             0 syslog-ng
Aug  3 12:59:01 bang kernel: [51585.826123] [ 4037]     0  4037     1149      639       7       0        0             0 log_to_sql.sh
Aug  3 12:59:01 bang kernel: [51585.826173] [ 4235]    60  4235    57365    13082      62       0      881             0 mysqld
Aug  3 12:59:01 bang kernel: [51585.826222] [ 4283]   107  4283     2557     1006       9       0        0             0 ulogd
Aug  3 12:59:01 bang kernel: [51585.826268] [ 4698]     0  4698      899      404       5       0        0             0 pppd
Aug  3 12:59:01 bang kernel: [51585.826316] [ 4762]   105  4762     1183      472       6       0        0             0 dnsmasq
Aug  3 12:59:01 bang kernel: [51585.826363] [ 4970]     0  4970     1292      542       7       0        0         -1000 sshd
Aug  3 12:59:01 bang kernel: [51585.826410] [ 5079]     0  5079    32467     4668      25       0        0             0 apache2
Aug  3 12:59:01 bang kernel: [51585.826457] [ 5081]    81  5081   168576    28259     140       0        0             0 apache2
Aug  3 12:59:01 bang kernel: [51585.826504] [ 5082]    81  5082   173465    34888     154       0        0             0 apache2
Aug  3 12:59:01 bang kernel: [51585.826550] [ 5211]     0  5211      594       29       5       0        0             0 atd
Aug  3 12:59:01 bang kernel: [51585.826597] [ 5239]   102  5239      777      430       5       0        0             0 dbus-daemon
Aug  3 12:59:01 bang kernel: [51585.826644] [ 5299]   103  5299     2665     2156      11       0        0             0 dhcpd
Aug  3 12:59:01 bang kernel: [51585.826691] [ 5365]   240  5365      601      209       5       0        0             0 distccd
Aug  3 12:59:01 bang kernel: [51585.826738] [ 5366]   240  5366      601       25       4       0        0             0 distccd
Aug  3 12:59:01 bang kernel: [51585.826784] [ 5399]   123  5399     1874     1411      10       0        0             0 ntpd
Aug  3 12:59:01 bang kernel: [51585.826830] [ 5428]   240  5428      601       25       4       0        0             0 distccd
Aug  3 12:59:01 bang kernel: [51585.826876] [ 5433]     0  5433      929      617       7       0        0             0 dovecot
Aug  3 12:59:01 bang kernel: [51585.826922] [ 5443]    97  5443      700      512       6       0        0             0 anvil
Aug  3 12:59:01 bang kernel: [51585.826968] [ 5444]     0  5444      733      561       5       0        0             0 log
Aug  3 12:59:01 bang kernel: [51585.827015] [ 5470]     8  5470    10720     1045      14       0        0             0 exim
Aug  3 12:59:01 bang kernel: [51585.827061] [ 5477]   240  5477      601       25       4       0        0             0 distccd
Aug  3 12:59:01 bang kernel: [51585.827107] [ 5497]   240  5497      601       25       4       0        0             0 distccd
Aug  3 12:59:01 bang kernel: [51585.827153] [ 5500]     0  5500    20882     2674      21       0        0             0 fail2ban-server
Aug  3 12:59:01 bang kernel: [51585.827199] [ 5502]     0  5502     1677     1007       7       0        0             0 screen
Aug  3 12:59:01 bang kernel: [51585.827246] [ 5503]   240  5503      601       25       4       0        0             0 distccd
Aug  3 12:59:01 bang kernel: [51585.827291] [ 5504]     0  5504     1295      804       8       0        0             0 bash
Aug  3 12:59:01 bang kernel: [51585.827339] [ 5505]     0  5505     1347      704       6       0        0             0 top
Aug  3 12:59:01 bang kernel: [51585.827385] [ 5506]     0  5506      842      102       5       0        0             0 tail
Aug  3 12:59:01 bang kernel: [51585.827431] [ 5507]     0  5507      842      100       6       0        0             0 tail
Aug  3 12:59:01 bang kernel: [51585.827477] [ 5510]     0  5510     1150      584       7       0        0             0 multitail.sh
Aug  3 12:59:01 bang kernel: [51585.827524] [ 5519]     0  5519     2466     1794       9       0        0             0 multitail
Aug  3 12:59:01 bang kernel: [51585.827572] [ 5526]     0  5526      941      651       6       0        0             0 gam_server
Aug  3 12:59:01 bang kernel: [51585.827618] [ 5527]     0  5527      842      108       6       0        0             0 tail
Aug  3 12:59:01 bang kernel: [51585.827664] [ 5528]     0  5528      842      105       5       0        0             0 tail
Aug  3 12:59:01 bang kernel: [51585.827710] [ 5529]     0  5529      842      100       5       0        0             0 tail
Aug  3 12:59:01 bang kernel: [51585.827756] [ 5530]     0  5530      842      355       6       0        0             0 tail
Aug  3 12:59:01 bang kernel: [51585.827802] [ 5531]     0  5531      843      386       6       0        0             0 tail
Aug  3 12:59:01 bang kernel: [51585.827848] [ 5532]   240  5532      601       25       4       0        0             0 distccd
Aug  3 12:59:01 bang kernel: [51585.827894] [ 5550]   240  5550      601       25       4       0        0             0 distccd
Aug  3 12:59:01 bang kernel: [51585.827940] [ 5622]     0  5622      615      442       5       0        0             0 rpcbind
Aug  3 12:59:01 bang kernel: [51585.827986] [ 5634]   240  5634      601       25       4       0        0             0 distccd
Aug  3 12:59:01 bang kernel: [51585.828032] [ 5652]     0  5652      787      572       5       0        0             0 rpc.statd
Aug  3 12:59:01 bang kernel: [51585.828078] [ 5707]     0  5707      789       46       5       0        0             0 rpc.idmapd
Aug  3 12:59:01 bang kernel: [51585.828124] [ 5733]   240  5733      601       25       4       0        0             0 distccd
Aug  3 12:59:01 bang kernel: [51585.828170] [ 5747]     0  5747      856      497       5       0        0             0 rpc.mountd
Aug  3 12:59:01 bang kernel: [51585.828220] [ 5804]   101  5804      562      367       6       0        0             0 radvd
Aug  3 12:59:01 bang kernel: [51585.828266] [ 5805]     0  5805      562      239       6       0        0             0 radvd
Aug  3 12:59:01 bang kernel: [51585.828313] [ 5839]   240  5839      601       25       4       0        0             0 distccd
Aug  3 12:59:01 bang kernel: [51585.828359] [ 5860]     0  5860     1150      618       5       0        0             0 heating.sh
Aug  3 12:59:01 bang kernel: [51585.828405] [ 5898]     0  5898     1007      451       6       0        0             0 agetty
Aug  3 12:59:01 bang kernel: [51585.828451] [ 5899]     0  5899     1007      436       7       0        0             0 agetty
Aug  3 12:59:01 bang kernel: [51585.828497] [ 5900]     0  5900     1007      419       6       0        0             0 agetty
Aug  3 12:59:01 bang kernel: [51585.828543] [ 5901]     0  5901     1007      435       5       0        0             0 agetty
Aug  3 12:59:01 bang kernel: [51585.828589] [ 5902]     0  5902     1007      436       6       0        0             0 agetty
Aug  3 12:59:01 bang kernel: [51585.828779] [ 5903]     0  5903     1007      449       7       0        0             0 agetty
Aug  3 12:59:01 bang kernel: [51585.828827] [ 5904]     0  5904      609      420       5       0        0             0 agetty
Aug  3 12:59:01 bang kernel: [51585.828875] [ 6004]     0  6004     1455      921       7       0        0             0 bluetoothd
Aug  3 12:59:01 bang kernel: [51585.828926] [ 6010]     0  6010    39540     7714      43       0        0             0 python2
Aug  3 12:59:01 bang kernel: [51585.828974] [ 3224]     0  3224     2247     1027      10       0        0             0 sshd
Aug  3 12:59:01 bang kernel: [51585.829021] [ 3227]  1000  3227     2247      945       9       0        0             0 sshd
Aug  3 12:59:01 bang kernel: [51585.829066] [ 3228]  1000  3228     1298      774       8       0        0             0 bash
Aug  3 12:59:01 bang kernel: [51585.829111] [ 3236]  1000  3236     1347      645       6       0        0             0 su
Aug  3 12:59:01 bang kernel: [51585.829155] [ 3238]     0  3238     1298      799       7       0        0             0 bash
Aug  3 12:59:01 bang kernel: [51585.829202] [  880]     0   880     1082      759       7       0        0             0 config
Aug  3 12:59:01 bang kernel: [51585.829247] [ 1099]   106  1099     1327     1093       7       0        0             0 imap-login
Aug  3 12:59:01 bang kernel: [51585.829334] [ 1111]     8  1111     1046      872       6       0        0             0 imap
Aug  3 12:59:01 bang kernel: [51585.829449] [10717]     0 10717     1299      765       7       0        0             0 bash
Aug  3 12:59:01 bang kernel: [51585.829564] [10784]     0 10784     2885     1232       9       0        0             0 mysql
Aug  3 12:59:01 bang kernel: [51585.829701] [16321]    40 16321    32298     9969      39       0        0             0 named
Aug  3 12:59:01 bang kernel: [51585.829900] [24379]     0 24379      996      411       6       0        0             0 cron
Aug  3 12:59:01 bang kernel: [51585.830042] [25814]     0 25814     2270     1056      10       0        0             0 sshd
Aug  3 12:59:01 bang kernel: [51585.830162] [25818]  1000 25818     2304      943       8       0        0             0 sshd
Aug  3 12:59:01 bang kernel: [51585.830290] [25819]  1000 25819     1298      769       6       0        0             0 bash
Aug  3 12:59:01 bang kernel: [51585.830405] [25827]  1000 25827     1347      642       7       0        0             0 su
Aug  3 12:59:01 bang kernel: [51585.830505] [25828]     0 25828     1298      760       8       0        0             0 bash
Aug  3 12:59:01 bang kernel: [51585.830620] [25834]     0 25834     1242      565       7       0        0             0 screen
Aug  3 12:59:01 bang kernel: [51585.830753] [12903]     0 12903     1299      788       7       0        0             0 bash
Aug  3 12:59:01 bang kernel: [51585.830872] [25975]     0 25975     6895      579      11       0        0             0 dump1090
Aug  3 12:59:01 bang kernel: [51585.831006] Out of memory: Kill process 5082 (apache2) score 22 or sacrifice child
Aug  3 12:59:01 bang kernel: [51585.832683] Killed process 5082 (apache2) total-vm:693860kB, anon-rss:118856kB, file-rss:13300kB, shmem-rss:7396kB

The thing is, swap is hardly being used at all. Why hasn't anything been swapped out rather than invoking the OOM killer?

Here's the VM details:

root@bang:~> grep ''     /proc/sys/vm/*
/proc/sys/vm/admin_reserve_kbytes:8192
/proc/sys/vm/block_dump:0
grep:     /proc/sys/vm/compact_memory: Permission denied
/proc/sys/vm/compact_unevictable_allowed:1
/proc/sys/vm/dirty_background_bytes:0
/proc/sys/vm/dirty_background_ratio:10
/proc/sys/vm/dirty_bytes:0
/proc/sys/vm/dirty_expire_centisecs:3000
/proc/sys/vm/dirty_ratio:20
/proc/sys/vm/dirtytime_expire_seconds:43200
/proc/sys/vm/dirty_writeback_centisecs:500
/proc/sys/vm/drop_caches:0
/proc/sys/vm/extfrag_threshold:500
/proc/sys/vm/highmem_is_dirtyable:0
/proc/sys/vm/laptop_mode:0
/proc/sys/vm/legacy_va_layout:0
/proc/sys/vm/lowmem_reserve_ratio:32    32
/proc/sys/vm/max_map_count:65530
/proc/sys/vm/min_free_kbytes:3420
/proc/sys/vm/mmap_min_addr:4096
/proc/sys/vm/mmap_rnd_bits:8
/proc/sys/vm/nr_pdflush_threads:0
/proc/sys/vm/oom_dump_tasks:1
/proc/sys/vm/oom_kill_allocating_task:0
/proc/sys/vm/overcommit_kbytes:0
/proc/sys/vm/overcommit_memory:0
/proc/sys/vm/overcommit_ratio:50
/proc/sys/vm/page-cluster:3
/proc/sys/vm/panic_on_oom:0
/proc/sys/vm/percpu_pagelist_fraction:0
/proc/sys/vm/stat_interval:1
/proc/sys/vm/swappiness:50
/proc/sys/vm/user_reserve_kbytes:62869
/proc/sys/vm/vfs_cache_pressure:100
/proc/sys/vm/watermark_scale_factor:10

Kernel is mainline 4.7 with some Exynos patches:

Linux bang 4.7.0-41238-g206dbde-dirty #16 SMP PREEMPT Tue Aug 2 22:35:38 BST 2016 armv7l SAMSUNG EXYNOS (Flattened Device Tree) GNU/Linux

Now, since I built the kernel myself, it's entirely possible that I got an option wrong somewhere. Any help would be appreciated.

[EDIT1]: This seems to happen when there's high I/O usage, but I haven't determined whether that's down to the cache filling or something else.

[EDIT2]: It seems there's an ongoing (at this time) discussion on the kernel mailing lists about what appears to be an identical problem. I'll monitor it and report back on the outcome.

Improve this question
9
  • What does the server do, why the memory is all used up, why it kills up things...the post is rather vague as it is.
    – Rui F Ribeiro
    Commented Aug 3, 2016 at 19:02
  • 3
    because linux malloc() is still broken, making the OOM killer a requirement in the first place? Sorry.. couldn't resist. Carry on.
    – alzee
    Commented Aug 3, 2016 at 21:15
  • @RuiFRibeiro the server isn't specialised; it runs Exim, Dovecot, mysql, apache2.4 and some other things - none of which have especially high throughput or memory requirements. As you can see from the free -m in the post, there's only 381MB of 2GB+4GB swap memory used, the rest being cache.
    – Steve
    Commented Aug 4, 2016 at 11:46
  • 2GB RAM is rather on the low side for all the software you are running.
    – Rui F Ribeiro
    Commented Aug 4, 2016 at 12:28
  • I have a very similar system here, sized it to 6GB, and I am waiting it to go in production build a baseline to have a better idea wether to size it between 4GB-8GB.
    – Rui F Ribeiro
    Commented Aug 4, 2016 at 14:56

2 Answers 2

Reset to default
3

This was caused by a kernel bug present in Linux kernels 4.7.0 to 4.7.4 (it's fixed by this commit in 4.7.5 and this commit in 4.8.0).

Improve this answer
2

I am getting this error with FC25's kernel 4.8.8 Same usage as above, apache and dovecot. According to https://marius.bloggt-in-braunschweig.de/category/fedora/ it is fragmentation of the filecache, and there is a workaround to clear that on a regular basis via cron until the fix goes into 4.9:

sync && echo 1 > /proc/sys/vm/drop_caches

Improve this answer
1
  • This Linux OOM thing has been bothering me for months. Linux is VERY dumb when it comes to caches, it never clears it up, even after simple operations like a file copy, the cache to the copied file just sits there, very annoying having to manually clear caches almost daily to avoid a system freeze. On top of that, disk swapping is also obsolete and I have been stumbling into slowness problems when dealing with disk swapping, the code to it was never really updated over the years, very upsetting how something crucial would be left behind like this.
    – Winampah
    Commented May 19, 2021 at 10:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .