[Durham] Swap fills up then out of memory errors
Dougie Nisbet
dougie at highmoor.co.uk
Fri Feb 17 22:40:58 UTC 2012
Can anyone help me with a puzzle? The system is Debian (testing). The
problem is occasional out of memory errors caused, I think, by a memory
leak somewhere in an as yet, unidentified program. My suspicions
(shortlist) are exaile (music player) or flickr_upload (perl based
command line uploaded to flickr).
What appears to be happening is that the amount of swap being allocated
gradually creeps up, usually over several days, as observed using
something like gnome-system-monitor. I think I understand the principle
that it's not necessarily a bad thing that all swap becomes allocated,
while memory used remains low, as idle programs can be swapped out even
when memory is available. But something is clearly not right.
Occasionally, and I suspect coinciding with the amount of swap used
hitting 100%, the system grinds to a halt, and eventually recovers some
speed or reboots.
My suspicion is that it's flickr_upload (although some old exaile
related posts mention a memory leak - but the symptoms don't match
mine). I suppose the next logical step would be to run
gnome-system-monitor, then repeatedly run test flickr_upload instances
on the command line, and see if it corresponds to an increase in swap
allocation.
But for the moment here's a snaphot from my kern.log file. This happens
periodically, and presumably on many occasions I'm unaware that it's
happened as I'm not present, and the program that it fails on changes
too. So I don't think too much can be read into the crashed program
name. Interested in any thoughts.
Feb 17 22:11:41 phoenix kernel: [793305.048319] bash invoked oom-killer:
gfp_mask=0xd0, order=1, oom_adj=0, oom_score_adj=0
Feb 17 22:11:41 phoenix kernel: [793305.048324] bash cpuset=/ mems_allowed=0
Feb 17 22:11:41 phoenix kernel: [793305.048328] Pid: 13446, comm: bash
Not tainted 3.1.0-1-amd64 #1
Feb 17 22:11:41 phoenix kernel: [793305.048331] Call Trace:
Feb 17 22:11:41 phoenix kernel: [793305.048340] [<ffffffff810b25c0>] ?
dump_header+0x78/0x1bd
Feb 17 22:11:41 phoenix kernel: [793305.048346] [<ffffffff810948fe>] ?
delayacct_end+0x72/0x7d
Feb 17 22:11:41 phoenix kernel: [793305.048351] [<ffffffff8115b7e0>] ?
security_real_capable_noaudit+0x40/0x4f
Feb 17 22:11:41 phoenix kernel: [793305.048355] [<ffffffff810b2965>] ?
oom_kill_process+0x49/0x24e
Feb 17 22:11:41 phoenix kernel: [793305.048360] [<ffffffff810b304b>] ?
out_of_memory+0x2f8/0x341
Feb 17 22:11:41 phoenix kernel: [793305.048365] [<ffffffff810b6da9>] ?
__alloc_pages_nodemask+0x600/0x748
Feb 17 22:11:41 phoenix kernel: [793305.048370] [<ffffffff81043d0d>] ?
copy_process+0x109/0x112e
Feb 17 22:11:41 phoenix kernel: [793305.048376] [<ffffffff81330dea>] ?
do_page_fault+0x2fc/0x337
Feb 17 22:11:41 phoenix kernel: [793305.048380] [<ffffffff81044e3c>] ?
do_fork+0xe1/0x219
Feb 17 22:11:41 phoenix kernel: [793305.048385] [<ffffffff810f2cd8>] ?
fd_install+0x25/0x4a
Feb 17 22:11:41 phoenix kernel: [793305.048390] [<ffffffff81333133>] ?
stub_clone+0x13/0x20
Feb 17 22:11:41 phoenix kernel: [793305.048394] [<ffffffff81332e12>] ?
system_call_fastpath+0x16/0x1b
Feb 17 22:11:41 phoenix kernel: [793305.048397] Mem-Info:
Feb 17 22:11:41 phoenix kernel: [793305.048399] Node 0 DMA per-cpu:
Feb 17 22:11:41 phoenix kernel: [793305.048402] CPU 0: hi: 0,
btch: 1 usd: 0
Feb 17 22:11:41 phoenix kernel: [793305.048405] CPU 1: hi: 0,
btch: 1 usd: 0
Feb 17 22:11:41 phoenix kernel: [793305.048407] Node 0 DMA32 per-cpu:
Feb 17 22:11:41 phoenix kernel: [793305.048411] CPU 0: hi: 186,
btch: 31 usd: 0
Feb 17 22:11:41 phoenix kernel: [793305.048413] CPU 1: hi: 186,
btch: 31 usd: 0
Feb 17 22:11:41 phoenix kernel: [793305.048420] active_anon:651377
inactive_anon:130497 isolated_anon:0
Feb 17 22:11:41 phoenix kernel: [793305.048422] active_file:137
inactive_file:215 isolated_file:23
Feb 17 22:11:41 phoenix kernel: [793305.048423] unevictable:0 dirty:0
writeback:0 unstable:0
Feb 17 22:11:41 phoenix kernel: [793305.048424] free:19181
slab_reclaimable:9702 slab_unreclaimable:6841
Feb 17 22:11:41 phoenix kernel: [793305.048425] mapped:1835
shmem:609966 pagetables:5268 bounce:0
Feb 17 22:11:41 phoenix kernel: [793305.048429] Node 0 DMA free:13244kB
min:208kB low:260kB high:312kB active_anon:704kB inactive_anon:924kB
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:15688kB mlocked:0kB dirty:0kB writeback:0kB
mapped:0kB shmem:0kB slab_reclaimable:280kB slab_unreclaimable:432kB
kernel_stack:248kB pagetables:60kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Feb 17 22:11:41 phoenix kernel: [793305.048443] lowmem_reserve[]: 0 3256
3256 3256
Feb 17 22:11:41 phoenix kernel: [793305.048448] Node 0 DMA32
free:63480kB min:44844kB low:56052kB high:67264kB active_anon:2604804kB
inactive_anon:521064kB active_file:548kB inactive_file:860kB
unevictable:0kB isolated(anon):0kB isolated(file):92kB present:3334576kB
mlocked:0kB dirty:0kB writeback:0kB mapped:7340kB shmem:2439864kB
slab_reclaimable:38528kB slab_unreclaimable:26932kB kernel_stack:2400kB
pagetables:21012kB unstable:0kB bounce:0kB writeback_tmp:0kB
pages_scanned:11218 all_unreclaimable? yes
Feb 17 22:11:41 phoenix kernel: [793305.048463] lowmem_reserve[]: 0 0 0 0
Feb 17 22:11:41 phoenix kernel: [793305.048467] Node 0 DMA: 27*4kB
42*8kB 48*16kB 30*32kB 9*64kB 4*128kB 3*256kB 2*512kB 2*1024kB 1*2048kB
1*4096kB = 13244kB
Feb 17 22:11:41 phoenix kernel: [793305.048479] Node 0 DMA32: 14395*4kB
182*8kB 9*16kB 6*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB
1*4096kB = 63532kB
Feb 17 22:11:41 phoenix kernel: [793305.048491] 619720 total pagecache pages
Feb 17 22:11:41 phoenix kernel: [793305.048493] 9359 pages in swap cache
Feb 17 22:11:41 phoenix kernel: [793305.048496] Swap cache stats: add
2221727, delete 2212368, find 8440555/8485693
Feb 17 22:11:41 phoenix kernel: [793305.048499] Free swap = 0kB
Feb 17 22:11:41 phoenix kernel: [793305.048501] Total swap = 6792188kB
Feb 17 22:11:41 phoenix kernel: [793305.062610] 849280 pages RAM
Feb 17 22:11:41 phoenix kernel: [793305.062613] 13597 pages reserved
Feb 17 22:11:41 phoenix kernel: [793305.062615] 4174 pages shared
Feb 17 22:11:41 phoenix kernel: [793305.062618] 811387 pages non-shared
Feb 17 22:11:41 phoenix kernel: [793305.062621] [ pid ] uid tgid
total_vm rss cpu oom_adj oom_score_adj name
Feb 17 22:11:41 phoenix kernel: [793305.062632] [ 239] 0 239
5449 16 0 -17 -1000 udevd
Feb 17 22:11:41 phoenix kernel: [793305.062639] [ 860] 0 860
4738 16 1 0 0 rpcbind
Feb 17 22:11:41 phoenix kernel: [793305.062645] [ 876] 105 876
6349 2 1 0 0 rpc.statd
Feb 17 22:11:41 phoenix kernel: [793305.062651] [ 892] 0 892
7393 0 1 0 0 rpc.idmapd
Feb 17 22:11:41 phoenix kernel: [793305.062657] [ 1092] 0 1092
29720 387 1 0 0 rsyslogd
Feb 17 22:11:41 phoenix kernel: [793305.062665] [ 1127] 0 1127
6457 1 0 0 0 rpc.mountd
Feb 17 22:11:41 phoenix kernel: [793305.062671] [ 1133] 101 1133
6158 249 0 0 0 dbus-daemon
Feb 17 22:11:41 phoenix kernel: [793305.062677] [ 1143] 0 1143
1032 24 1 0 0 acpid
Feb 17 22:11:41 phoenix kernel: [793305.062684] [ 1153] 0 1153
38499 360 0 0 0 NetworkManager
Feb 17 22:11:41 phoenix kernel: [793305.062690] [ 1163] 0 1163
33021 380 0 0 0 polkitd
Feb 17 22:11:41 phoenix kernel: [793305.062695] [ 1169] 0 1169
20185 61 1 0 0 modem-manager
Feb 17 22:11:41 phoenix kernel: [793305.062701] [ 1190] 0 1190
20926 49 1 0 0 gdm3
Feb 17 22:11:41 phoenix kernel: [793305.062707] [ 1198] 0 1198
25667 2 1 0 0 gdm-simple-slav
Feb 17 22:11:41 phoenix kernel: [793305.062713] [ 1202] 0 1202
62753 3561 1 0 0 Xorg
Feb 17 22:11:41 phoenix kernel: [793305.062718] [ 1216] 0 1216
4162 3 0 0 0 atd
Feb 17 22:11:41 phoenix kernel: [793305.062724] [ 1228] 106 1228
8532 43 0 0 0 avahi-daemon
Feb 17 22:11:41 phoenix kernel: [793305.062730] [ 1229] 106 1229
8501 1 1 0 0 avahi-daemon
Feb 17 22:11:41 phoenix kernel: [793305.062736] [ 1332] 0 1332
32383 161 0 0 0 console-kit-dae
Feb 17 22:11:41 phoenix kernel: [793305.062742] [ 1401] 0 1401
20009 84 1 0 0 cupsd
Feb 17 22:11:41 phoenix kernel: [793305.062748] [ 1671] 104 1671
12717 16 0 0 0 exim4
Feb 17 22:11:41 phoenix kernel: [793305.062754] [ 1684] 0 1684
19913 31 1 0 0 kerneloops
Feb 17 22:11:41 phoenix kernel: [793305.062760] [ 1757] 0 1757
38927 110 1 0 0 upowerd
Feb 17 22:11:41 phoenix kernel: [793305.062766] [ 1940] 0 1940
12424 31 0 -17 -1000 sshd
Feb 17 22:11:41 phoenix kernel: [793305.062772] [ 1952] 0 1952
1016 1 1 0 0 minissdpd
Feb 17 22:11:41 phoenix kernel: [793305.062777] [ 1964] 109 1964
26285 9 1 0 0 rtkit-daemon
Feb 17 22:11:41 phoenix kernel: [793305.062783] [ 1974] 0 1974
17771 119 0 0 0 login
Feb 17 22:11:41 phoenix kernel: [793305.062789] [ 1975] 0 1975
4058 2 1 0 0 getty
Feb 17 22:11:41 phoenix kernel: [793305.062794] [ 1976] 0 1976
4058 2 1 0 0 getty
Feb 17 22:11:41 phoenix kernel: [793305.062800] [ 1977] 0 1977
4058 2 1 0 0 getty
Feb 17 22:11:41 phoenix kernel: [793305.062805] [ 1978] 0 1978
4058 2 1 0 0 getty
Feb 17 22:11:41 phoenix kernel: [793305.062811] [ 1979] 0 1979
4058 2 1 0 0 getty
Feb 17 22:11:41 phoenix kernel: [793305.062817] [ 1985] 102 1985
92643 0 1 0 0 colord
Feb 17 22:11:41 phoenix kernel: [793305.062823] [ 2085] 1000 2085
2889 244 1 0 0 upload_camera_f
Feb 17 22:11:41 phoenix kernel: [793305.062828] [ 2088] 1000 2088
48167 5682 1 0 0 motion
Feb 17 22:11:41 phoenix kernel: [793305.062835] [17100] 0 17100
27140 0 0 0 0 gdm-session-wor
Feb 17 22:11:41 phoenix kernel: [793305.062841] [17120] 1000 17120
1040 2 0 0 0 sh
Feb 17 22:11:41 phoenix kernel: [793305.062846] [17155] 1000 17155
3062 17 1 0 0 ssh-agent
Feb 17 22:11:41 phoenix kernel: [793305.062852] [17158] 1000 17158
6587 0 0 0 0 dbus-launch
Feb 17 22:11:41 phoenix kernel: [793305.062858] [17159] 1000 17159
6471 316 0 0 0 dbus-daemon
Feb 17 22:11:41 phoenix kernel: [793305.062864] [17167] 1000 17167
12348 78 1 0 0 xfconfd
Feb 17 22:11:41 phoenix kernel: [793305.062869] [17173] 1000 17173
14530 63 0 0 0 xscreensaver
Feb 17 22:11:41 phoenix kernel: [793305.062876] [17175] 1000 17175
39961 531 1 0 0 xfce4-session
Feb 17 22:11:41 phoenix kernel: [793305.062883] [17180] 1000 17180
78298 832 1 0 0 xfwm4
Feb 17 22:11:41 phoenix kernel: [793305.062889] [17182] 1000 17182
31463 74 1 0 0 xfsettingsd
Feb 17 22:11:41 phoenix kernel: [793305.062896] [17183] 1000 17183
41128 60 0 0 0 Thunar
Feb 17 22:11:41 phoenix kernel: [793305.062902] [17185] 1000 17185
15491 44 1 0 0 gvfsd
Feb 17 22:11:41 phoenix kernel: [793305.062908] [17188] 1000 17188
92393 1362 1 0 0 xfce4-panel
Feb 17 22:11:41 phoenix kernel: [793305.062914] [17189] 1000 17189
88285 738 1 0 0 xfdesktop
Feb 17 22:11:41 phoenix kernel: [793305.062921] [17190] 1000 17190
104503 2202 1 0 0 gnome-terminal
Feb 17 22:11:41 phoenix kernel: [793305.062927] [17193] 1000 17193
37647 136 0 0 0 xfce4-settings-
Feb 17 22:11:41 phoenix kernel: [793305.062933] [17200] 1000 17200
14444 400 0 0 0 gconfd-2
Feb 17 22:11:41 phoenix kernel: [793305.062940] [17210] 1000 17210
31499 43 1 0 0 kerneloops-appl
Feb 17 22:11:41 phoenix kernel: [793305.062947] [17212] 1000 17212
165774 396 0 0 0 xfce4-mixer-plu
Feb 17 22:11:41 phoenix kernel: [793305.062954] [17215] 1000 17215
40339 133 1 0 0 gvfs-gdu-volume
Feb 17 22:11:41 phoenix kernel: [793305.062960] [17217] 1000 17217
57884 224 1 0 0 system-config-p
Feb 17 22:11:41 phoenix kernel: [793305.062967] [17219] 1000 17219
36841 194 0 0 0 panel-2-systray
Feb 17 22:11:41 phoenix kernel: [793305.062973] [17221] 0 17221
32428 169 1 0 0 udisks-daemon
Feb 17 22:11:41 phoenix kernel: [793305.062980] [17242] 0 17242
11835 20 1 0 0 udisks-daemon
Feb 17 22:11:41 phoenix kernel: [793305.062986] [17249] 1000 17249
76712 440 1 0 0 pulseaudio
Feb 17 22:11:41 phoenix kernel: [793305.062993] [17255] 1000 17255
91840 212 0 0 0 evolution-alarm
Feb 17 22:11:41 phoenix kernel: [793305.062999] [17257] 1000 17257
100471 354 0 0 0 nm-applet
Feb 17 22:11:41 phoenix kernel: [793305.063005] [17262] 1000 17262
46652 140 0 0 0 polkit-gnome-au
Feb 17 22:11:41 phoenix kernel: [793305.063012] [17270] 1000 17270
141558 141 1 0 0 xfce4-volumed
Feb 17 22:11:41 phoenix kernel: [793305.063018] [17285] 1000 17285
3632 0 1 0 0 gnome-pty-helpe
Feb 17 22:11:41 phoenix kernel: [793305.063025] [17287] 1000 17287
5096 1 1 0 0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063031] [17289] 1000 17289
5110 14 0 0 0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063038] [17290] 1000 17290
5128 319 0 0 0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063044] [17293] 1000 17293
5121 120 0 0 0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063050] [17294] 1000 17294
5096 1 1 0 0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063056] [17717] 1000 17717
15106 40 1 0 0 gvfs-gphoto2-vo
Feb 17 22:11:41 phoenix kernel: [793305.063063] [17719] 1000 17719
35540 36 0 0 0 gvfs-afc-volume
Feb 17 22:11:41 phoenix kernel: [793305.063070] [17722] 1000 17722
17869 1173 1 0 0 gvfsd-trash
Feb 17 22:11:41 phoenix kernel: [793305.063076] [ 1528] 1000 1528
87917 2826 1 0 0 gnome-terminal
Feb 17 22:11:41 phoenix kernel: [793305.063083] [ 1533] 1000 1533
3632 21 0 0 0 gnome-pty-helpe
Feb 17 22:11:41 phoenix kernel: [793305.063089] [ 1534] 1000 1534
5096 0 1 0 0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063095] [ 1618] 1000 1618
1402 18 1 0 0 tail
Feb 17 22:11:41 phoenix kernel: [793305.063102] [ 1620] 1000 1620
33197 107 0 0 0 dconf-service
Feb 17 22:11:41 phoenix kernel: [793305.063108] [ 2541] 1000 2541
5132 164 0 0 0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063115] [ 8213] 1000 8213
13042 1 0 0 0 sudo
Feb 17 22:11:41 phoenix kernel: [793305.063121] [ 8214] 0 8214
14020 2 1 0 0 su
Feb 17 22:11:41 phoenix kernel: [793305.063125] [ 8222] 0 8222
5095 160 0 0 0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063130] [ 8297] 1000 8297
123486 4655 1 0 0 gnome-system-mo
Feb 17 22:11:41 phoenix kernel: [793305.063135] [ 8759] 0 8759
3632 1 0 0 0 gnome-pty-helpe
Feb 17 22:11:41 phoenix kernel: [793305.063140] [10677] 0 10677
1467 1 0 0 0 hddtemp
Feb 17 22:11:41 phoenix kernel: [793305.063145] [ 5744] 1000 5744
54931 141 1 0 0 gnome-settings-
Feb 17 22:11:41 phoenix kernel: [793305.063151] [17709] 1000 17709
5097 27 0 0 0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063156] [17710] 1000 17710
5099 279 1 0 0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063160] [17711] 1000 17711
5096 30 0 0 0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063165] [22787] 1000 22787
5333 81 0 0 0 top
Feb 17 22:11:41 phoenix kernel: [793305.063170] [28979] 0 28979
5097 22 0 0 0 cron
Feb 17 22:11:41 phoenix kernel: [793305.063175] [29177] 0 29177
17290 42 0 0 0 nmbd
Feb 17 22:11:41 phoenix kernel: [793305.063180] [29180] 0 29180
23819 58 0 0 0 smbd
Feb 17 22:11:41 phoenix kernel: [793305.063184] [29187] 0 29187
23845 43 0 0 0 smbd
Feb 17 22:11:41 phoenix kernel: [793305.063191] [ 3626] 0 3626
31227 93 0 0 0 accounts-daemon
Feb 17 22:11:41 phoenix kernel: [793305.063196] [ 3694] 0 3694
5239 1 0 0 0 bluetoothd
Feb 17 22:11:41 phoenix kernel: [793305.063200] [ 7949] 1000 7949
5105 21 1 0 0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063205] [11357] 1000 11357
5102 2 1 0 0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063210] [11418] 1000 11418
13042 1 1 0 0 sudo
Feb 17 22:11:41 phoenix kernel: [793305.063214] [11419] 0 11419
14020 0 0 0 0 su
Feb 17 22:11:41 phoenix kernel: [793305.063219] [11427] 0 11427
5101 2 1 0 0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063223] [11503] 0 11503
5448 15 0 -17 -1000 udevd
Feb 17 22:11:41 phoenix kernel: [793305.063228] [26994] 1000 26994
1402 16 1 0 0 tail
Feb 17 22:11:41 phoenix kernel: [793305.063233] [31542] 1000 31542
11738 147 0 0 0 gvfsd-metadata
Feb 17 22:11:41 phoenix kernel: [793305.063237] [10803] 1000 10803
5110 272 0 0 0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063242] [19791] 1000 19791
15524 110 1 0 0 gvfsd-burn
Feb 17 22:11:41 phoenix kernel: [793305.063247] [19830] 1000 19830
49434 902 1 0 0 tracker-store
Feb 17 22:11:41 phoenix kernel: [793305.063251] [12901] 1000 12901
135507 3618 1 0 0 nautilus
Feb 17 22:11:41 phoenix kernel: [793305.063256] [18926] 1000 18926
11373 162 1 0 0 ssh
Feb 17 22:11:41 phoenix kernel: [793305.063261] [ 8721] 1000 8721
230865 35018 1 0 0 x-www-browser
Feb 17 22:11:41 phoenix kernel: [793305.063266] [13247] 1000 13247
13042 96 1 0 0 sudo
Feb 17 22:11:41 phoenix kernel: [793305.063271] [13257] 0 13257
14020 94 0 0 0 su
Feb 17 22:11:41 phoenix kernel: [793305.063275] [13265] 0 13265
5101 373 1 0 0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063280] [13322] 0 13322
74838 7219 0 0 0 synaptic
Feb 17 22:11:41 phoenix kernel: [793305.063285] [13343] 0 13343
96166 84274 1 0 0 update-apt-xapi
Feb 17 22:11:41 phoenix kernel: [793305.063289] [13446] 0 13446
5106 385 0 0 0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063293] [13529] 1000 13529
1393 22 1 0 0 sleep
Feb 17 22:11:41 phoenix kernel: [793305.063297] Out of memory: Kill
process 8721 (x-www-browser) score 14 or sacrifice child
Feb 17 22:11:41 phoenix kernel: [793305.063370] Killed process 8721
(x-www-browser) total-vm:923460kB, anon-rss:140072kB, file-rss:0kB
More information about the Durham
mailing list