[Durham] Swap fills up then out of memory errors

Dougie Nisbet dougie at highmoor.co.uk
Fri Feb 17 22:40:58 UTC 2012


Can anyone help me with a puzzle? The system is Debian (testing). The 
problem is occasional out of memory errors caused, I think, by a memory 
leak somewhere in an as yet, unidentified program. My suspicions 
(shortlist) are exaile (music player) or flickr_upload  (perl based 
command line uploaded to flickr).

What appears to be happening is that the amount of swap being allocated 
gradually creeps up, usually over several days, as observed using 
something like gnome-system-monitor. I think I understand the principle 
that it's not necessarily a bad thing that all swap becomes allocated, 
while memory used remains low, as idle programs can be swapped out even 
when memory is available. But something is clearly not right. 
Occasionally, and I suspect coinciding with the amount of swap used 
hitting 100%, the system grinds to a halt, and eventually recovers some 
speed or reboots.

My suspicion is that it's flickr_upload (although some old exaile 
related posts mention a memory leak - but the symptoms don't match 
mine). I suppose the next logical step would be to run 
gnome-system-monitor, then repeatedly run test flickr_upload instances 
on the command line, and see if it corresponds to an increase in swap 
allocation.

But for the moment here's a snaphot from my kern.log file. This happens 
periodically, and presumably on many occasions I'm unaware that it's 
happened as I'm not present, and the program that it fails on changes 
too. So I don't think too much can be read into the crashed program 
name. Interested in any thoughts.


Feb 17 22:11:41 phoenix kernel: [793305.048319] bash invoked oom-killer: 
gfp_mask=0xd0, order=1, oom_adj=0, oom_score_adj=0
Feb 17 22:11:41 phoenix kernel: [793305.048324] bash cpuset=/ mems_allowed=0
Feb 17 22:11:41 phoenix kernel: [793305.048328] Pid: 13446, comm: bash 
Not tainted 3.1.0-1-amd64 #1
Feb 17 22:11:41 phoenix kernel: [793305.048331] Call Trace:
Feb 17 22:11:41 phoenix kernel: [793305.048340]  [<ffffffff810b25c0>] ? 
dump_header+0x78/0x1bd
Feb 17 22:11:41 phoenix kernel: [793305.048346]  [<ffffffff810948fe>] ? 
delayacct_end+0x72/0x7d
Feb 17 22:11:41 phoenix kernel: [793305.048351]  [<ffffffff8115b7e0>] ? 
security_real_capable_noaudit+0x40/0x4f
Feb 17 22:11:41 phoenix kernel: [793305.048355]  [<ffffffff810b2965>] ? 
oom_kill_process+0x49/0x24e
Feb 17 22:11:41 phoenix kernel: [793305.048360]  [<ffffffff810b304b>] ? 
out_of_memory+0x2f8/0x341
Feb 17 22:11:41 phoenix kernel: [793305.048365]  [<ffffffff810b6da9>] ? 
__alloc_pages_nodemask+0x600/0x748
Feb 17 22:11:41 phoenix kernel: [793305.048370]  [<ffffffff81043d0d>] ? 
copy_process+0x109/0x112e
Feb 17 22:11:41 phoenix kernel: [793305.048376]  [<ffffffff81330dea>] ? 
do_page_fault+0x2fc/0x337
Feb 17 22:11:41 phoenix kernel: [793305.048380]  [<ffffffff81044e3c>] ? 
do_fork+0xe1/0x219
Feb 17 22:11:41 phoenix kernel: [793305.048385]  [<ffffffff810f2cd8>] ? 
fd_install+0x25/0x4a
Feb 17 22:11:41 phoenix kernel: [793305.048390]  [<ffffffff81333133>] ? 
stub_clone+0x13/0x20
Feb 17 22:11:41 phoenix kernel: [793305.048394]  [<ffffffff81332e12>] ? 
system_call_fastpath+0x16/0x1b
Feb 17 22:11:41 phoenix kernel: [793305.048397] Mem-Info:
Feb 17 22:11:41 phoenix kernel: [793305.048399] Node 0 DMA per-cpu:
Feb 17 22:11:41 phoenix kernel: [793305.048402] CPU    0: hi:    0, 
btch:   1 usd:   0
Feb 17 22:11:41 phoenix kernel: [793305.048405] CPU    1: hi:    0, 
btch:   1 usd:   0
Feb 17 22:11:41 phoenix kernel: [793305.048407] Node 0 DMA32 per-cpu:
Feb 17 22:11:41 phoenix kernel: [793305.048411] CPU    0: hi:  186, 
btch:  31 usd:   0
Feb 17 22:11:41 phoenix kernel: [793305.048413] CPU    1: hi:  186, 
btch:  31 usd:   0
Feb 17 22:11:41 phoenix kernel: [793305.048420] active_anon:651377 
inactive_anon:130497 isolated_anon:0
Feb 17 22:11:41 phoenix kernel: [793305.048422]  active_file:137 
inactive_file:215 isolated_file:23
Feb 17 22:11:41 phoenix kernel: [793305.048423]  unevictable:0 dirty:0 
writeback:0 unstable:0
Feb 17 22:11:41 phoenix kernel: [793305.048424]  free:19181 
slab_reclaimable:9702 slab_unreclaimable:6841
Feb 17 22:11:41 phoenix kernel: [793305.048425]  mapped:1835 
shmem:609966 pagetables:5268 bounce:0
Feb 17 22:11:41 phoenix kernel: [793305.048429] Node 0 DMA free:13244kB 
min:208kB low:260kB high:312kB active_anon:704kB inactive_anon:924kB 
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB 
isolated(file):0kB present:15688kB mlocked:0kB dirty:0kB writeback:0kB 
mapped:0kB shmem:0kB slab_reclaimable:280kB slab_unreclaimable:432kB 
kernel_stack:248kB pagetables:60kB unstable:0kB bounce:0kB 
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Feb 17 22:11:41 phoenix kernel: [793305.048443] lowmem_reserve[]: 0 3256 
3256 3256
Feb 17 22:11:41 phoenix kernel: [793305.048448] Node 0 DMA32 
free:63480kB min:44844kB low:56052kB high:67264kB active_anon:2604804kB 
inactive_anon:521064kB active_file:548kB inactive_file:860kB 
unevictable:0kB isolated(anon):0kB isolated(file):92kB present:3334576kB 
mlocked:0kB dirty:0kB writeback:0kB mapped:7340kB shmem:2439864kB 
slab_reclaimable:38528kB slab_unreclaimable:26932kB kernel_stack:2400kB 
pagetables:21012kB unstable:0kB bounce:0kB writeback_tmp:0kB 
pages_scanned:11218 all_unreclaimable? yes
Feb 17 22:11:41 phoenix kernel: [793305.048463] lowmem_reserve[]: 0 0 0 0
Feb 17 22:11:41 phoenix kernel: [793305.048467] Node 0 DMA: 27*4kB 
42*8kB 48*16kB 30*32kB 9*64kB 4*128kB 3*256kB 2*512kB 2*1024kB 1*2048kB 
1*4096kB = 13244kB
Feb 17 22:11:41 phoenix kernel: [793305.048479] Node 0 DMA32: 14395*4kB 
182*8kB 9*16kB 6*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 
1*4096kB = 63532kB
Feb 17 22:11:41 phoenix kernel: [793305.048491] 619720 total pagecache pages
Feb 17 22:11:41 phoenix kernel: [793305.048493] 9359 pages in swap cache
Feb 17 22:11:41 phoenix kernel: [793305.048496] Swap cache stats: add 
2221727, delete 2212368, find 8440555/8485693
Feb 17 22:11:41 phoenix kernel: [793305.048499] Free swap  = 0kB
Feb 17 22:11:41 phoenix kernel: [793305.048501] Total swap = 6792188kB
Feb 17 22:11:41 phoenix kernel: [793305.062610] 849280 pages RAM
Feb 17 22:11:41 phoenix kernel: [793305.062613] 13597 pages reserved
Feb 17 22:11:41 phoenix kernel: [793305.062615] 4174 pages shared
Feb 17 22:11:41 phoenix kernel: [793305.062618] 811387 pages non-shared
Feb 17 22:11:41 phoenix kernel: [793305.062621] [ pid ]   uid  tgid 
total_vm      rss cpu oom_adj oom_score_adj name
Feb 17 22:11:41 phoenix kernel: [793305.062632] [  239]     0   239     
5449       16   0     -17         -1000 udevd
Feb 17 22:11:41 phoenix kernel: [793305.062639] [  860]     0   860     
4738       16   1       0             0 rpcbind
Feb 17 22:11:41 phoenix kernel: [793305.062645] [  876]   105   876     
6349        2   1       0             0 rpc.statd
Feb 17 22:11:41 phoenix kernel: [793305.062651] [  892]     0   892     
7393        0   1       0             0 rpc.idmapd
Feb 17 22:11:41 phoenix kernel: [793305.062657] [ 1092]     0  1092    
29720      387   1       0             0 rsyslogd
Feb 17 22:11:41 phoenix kernel: [793305.062665] [ 1127]     0  1127     
6457        1   0       0             0 rpc.mountd
Feb 17 22:11:41 phoenix kernel: [793305.062671] [ 1133]   101  1133     
6158      249   0       0             0 dbus-daemon
Feb 17 22:11:41 phoenix kernel: [793305.062677] [ 1143]     0  1143     
1032       24   1       0             0 acpid
Feb 17 22:11:41 phoenix kernel: [793305.062684] [ 1153]     0  1153    
38499      360   0       0             0 NetworkManager
Feb 17 22:11:41 phoenix kernel: [793305.062690] [ 1163]     0  1163    
33021      380   0       0             0 polkitd
Feb 17 22:11:41 phoenix kernel: [793305.062695] [ 1169]     0  1169    
20185       61   1       0             0 modem-manager
Feb 17 22:11:41 phoenix kernel: [793305.062701] [ 1190]     0  1190    
20926       49   1       0             0 gdm3
Feb 17 22:11:41 phoenix kernel: [793305.062707] [ 1198]     0  1198    
25667        2   1       0             0 gdm-simple-slav
Feb 17 22:11:41 phoenix kernel: [793305.062713] [ 1202]     0  1202    
62753     3561   1       0             0 Xorg
Feb 17 22:11:41 phoenix kernel: [793305.062718] [ 1216]     0  1216     
4162        3   0       0             0 atd
Feb 17 22:11:41 phoenix kernel: [793305.062724] [ 1228]   106  1228     
8532       43   0       0             0 avahi-daemon
Feb 17 22:11:41 phoenix kernel: [793305.062730] [ 1229]   106  1229     
8501        1   1       0             0 avahi-daemon
Feb 17 22:11:41 phoenix kernel: [793305.062736] [ 1332]     0  1332    
32383      161   0       0             0 console-kit-dae
Feb 17 22:11:41 phoenix kernel: [793305.062742] [ 1401]     0  1401    
20009       84   1       0             0 cupsd
Feb 17 22:11:41 phoenix kernel: [793305.062748] [ 1671]   104  1671    
12717       16   0       0             0 exim4
Feb 17 22:11:41 phoenix kernel: [793305.062754] [ 1684]     0  1684    
19913       31   1       0             0 kerneloops
Feb 17 22:11:41 phoenix kernel: [793305.062760] [ 1757]     0  1757    
38927      110   1       0             0 upowerd
Feb 17 22:11:41 phoenix kernel: [793305.062766] [ 1940]     0  1940    
12424       31   0     -17         -1000 sshd
Feb 17 22:11:41 phoenix kernel: [793305.062772] [ 1952]     0  1952     
1016        1   1       0             0 minissdpd
Feb 17 22:11:41 phoenix kernel: [793305.062777] [ 1964]   109  1964    
26285        9   1       0             0 rtkit-daemon
Feb 17 22:11:41 phoenix kernel: [793305.062783] [ 1974]     0  1974    
17771      119   0       0             0 login
Feb 17 22:11:41 phoenix kernel: [793305.062789] [ 1975]     0  1975     
4058        2   1       0             0 getty
Feb 17 22:11:41 phoenix kernel: [793305.062794] [ 1976]     0  1976     
4058        2   1       0             0 getty
Feb 17 22:11:41 phoenix kernel: [793305.062800] [ 1977]     0  1977     
4058        2   1       0             0 getty
Feb 17 22:11:41 phoenix kernel: [793305.062805] [ 1978]     0  1978     
4058        2   1       0             0 getty
Feb 17 22:11:41 phoenix kernel: [793305.062811] [ 1979]     0  1979     
4058        2   1       0             0 getty
Feb 17 22:11:41 phoenix kernel: [793305.062817] [ 1985]   102  1985    
92643        0   1       0             0 colord
Feb 17 22:11:41 phoenix kernel: [793305.062823] [ 2085]  1000  2085     
2889      244   1       0             0 upload_camera_f
Feb 17 22:11:41 phoenix kernel: [793305.062828] [ 2088]  1000  2088    
48167     5682   1       0             0 motion
Feb 17 22:11:41 phoenix kernel: [793305.062835] [17100]     0 17100    
27140        0   0       0             0 gdm-session-wor
Feb 17 22:11:41 phoenix kernel: [793305.062841] [17120]  1000 17120     
1040        2   0       0             0 sh
Feb 17 22:11:41 phoenix kernel: [793305.062846] [17155]  1000 17155     
3062       17   1       0             0 ssh-agent
Feb 17 22:11:41 phoenix kernel: [793305.062852] [17158]  1000 17158     
6587        0   0       0             0 dbus-launch
Feb 17 22:11:41 phoenix kernel: [793305.062858] [17159]  1000 17159     
6471      316   0       0             0 dbus-daemon
Feb 17 22:11:41 phoenix kernel: [793305.062864] [17167]  1000 17167    
12348       78   1       0             0 xfconfd
Feb 17 22:11:41 phoenix kernel: [793305.062869] [17173]  1000 17173    
14530       63   0       0             0 xscreensaver
Feb 17 22:11:41 phoenix kernel: [793305.062876] [17175]  1000 17175    
39961      531   1       0             0 xfce4-session
Feb 17 22:11:41 phoenix kernel: [793305.062883] [17180]  1000 17180    
78298      832   1       0             0 xfwm4
Feb 17 22:11:41 phoenix kernel: [793305.062889] [17182]  1000 17182    
31463       74   1       0             0 xfsettingsd
Feb 17 22:11:41 phoenix kernel: [793305.062896] [17183]  1000 17183    
41128       60   0       0             0 Thunar
Feb 17 22:11:41 phoenix kernel: [793305.062902] [17185]  1000 17185    
15491       44   1       0             0 gvfsd
Feb 17 22:11:41 phoenix kernel: [793305.062908] [17188]  1000 17188    
92393     1362   1       0             0 xfce4-panel
Feb 17 22:11:41 phoenix kernel: [793305.062914] [17189]  1000 17189    
88285      738   1       0             0 xfdesktop
Feb 17 22:11:41 phoenix kernel: [793305.062921] [17190]  1000 17190   
104503     2202   1       0             0 gnome-terminal
Feb 17 22:11:41 phoenix kernel: [793305.062927] [17193]  1000 17193    
37647      136   0       0             0 xfce4-settings-
Feb 17 22:11:41 phoenix kernel: [793305.062933] [17200]  1000 17200    
14444      400   0       0             0 gconfd-2
Feb 17 22:11:41 phoenix kernel: [793305.062940] [17210]  1000 17210    
31499       43   1       0             0 kerneloops-appl
Feb 17 22:11:41 phoenix kernel: [793305.062947] [17212]  1000 17212   
165774      396   0       0             0 xfce4-mixer-plu
Feb 17 22:11:41 phoenix kernel: [793305.062954] [17215]  1000 17215    
40339      133   1       0             0 gvfs-gdu-volume
Feb 17 22:11:41 phoenix kernel: [793305.062960] [17217]  1000 17217    
57884      224   1       0             0 system-config-p
Feb 17 22:11:41 phoenix kernel: [793305.062967] [17219]  1000 17219    
36841      194   0       0             0 panel-2-systray
Feb 17 22:11:41 phoenix kernel: [793305.062973] [17221]     0 17221    
32428      169   1       0             0 udisks-daemon
Feb 17 22:11:41 phoenix kernel: [793305.062980] [17242]     0 17242    
11835       20   1       0             0 udisks-daemon
Feb 17 22:11:41 phoenix kernel: [793305.062986] [17249]  1000 17249    
76712      440   1       0             0 pulseaudio
Feb 17 22:11:41 phoenix kernel: [793305.062993] [17255]  1000 17255    
91840      212   0       0             0 evolution-alarm
Feb 17 22:11:41 phoenix kernel: [793305.062999] [17257]  1000 17257   
100471      354   0       0             0 nm-applet
Feb 17 22:11:41 phoenix kernel: [793305.063005] [17262]  1000 17262    
46652      140   0       0             0 polkit-gnome-au
Feb 17 22:11:41 phoenix kernel: [793305.063012] [17270]  1000 17270   
141558      141   1       0             0 xfce4-volumed
Feb 17 22:11:41 phoenix kernel: [793305.063018] [17285]  1000 17285     
3632        0   1       0             0 gnome-pty-helpe
Feb 17 22:11:41 phoenix kernel: [793305.063025] [17287]  1000 17287     
5096        1   1       0             0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063031] [17289]  1000 17289     
5110       14   0       0             0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063038] [17290]  1000 17290     
5128      319   0       0             0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063044] [17293]  1000 17293     
5121      120   0       0             0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063050] [17294]  1000 17294     
5096        1   1       0             0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063056] [17717]  1000 17717    
15106       40   1       0             0 gvfs-gphoto2-vo
Feb 17 22:11:41 phoenix kernel: [793305.063063] [17719]  1000 17719    
35540       36   0       0             0 gvfs-afc-volume
Feb 17 22:11:41 phoenix kernel: [793305.063070] [17722]  1000 17722    
17869     1173   1       0             0 gvfsd-trash
Feb 17 22:11:41 phoenix kernel: [793305.063076] [ 1528]  1000  1528    
87917     2826   1       0             0 gnome-terminal
Feb 17 22:11:41 phoenix kernel: [793305.063083] [ 1533]  1000  1533     
3632       21   0       0             0 gnome-pty-helpe
Feb 17 22:11:41 phoenix kernel: [793305.063089] [ 1534]  1000  1534     
5096        0   1       0             0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063095] [ 1618]  1000  1618     
1402       18   1       0             0 tail
Feb 17 22:11:41 phoenix kernel: [793305.063102] [ 1620]  1000  1620    
33197      107   0       0             0 dconf-service
Feb 17 22:11:41 phoenix kernel: [793305.063108] [ 2541]  1000  2541     
5132      164   0       0             0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063115] [ 8213]  1000  8213    
13042        1   0       0             0 sudo
Feb 17 22:11:41 phoenix kernel: [793305.063121] [ 8214]     0  8214    
14020        2   1       0             0 su
Feb 17 22:11:41 phoenix kernel: [793305.063125] [ 8222]     0  8222     
5095      160   0       0             0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063130] [ 8297]  1000  8297   
123486     4655   1       0             0 gnome-system-mo
Feb 17 22:11:41 phoenix kernel: [793305.063135] [ 8759]     0  8759     
3632        1   0       0             0 gnome-pty-helpe
Feb 17 22:11:41 phoenix kernel: [793305.063140] [10677]     0 10677     
1467        1   0       0             0 hddtemp
Feb 17 22:11:41 phoenix kernel: [793305.063145] [ 5744]  1000  5744    
54931      141   1       0             0 gnome-settings-
Feb 17 22:11:41 phoenix kernel: [793305.063151] [17709]  1000 17709     
5097       27   0       0             0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063156] [17710]  1000 17710     
5099      279   1       0             0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063160] [17711]  1000 17711     
5096       30   0       0             0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063165] [22787]  1000 22787     
5333       81   0       0             0 top
Feb 17 22:11:41 phoenix kernel: [793305.063170] [28979]     0 28979     
5097       22   0       0             0 cron
Feb 17 22:11:41 phoenix kernel: [793305.063175] [29177]     0 29177    
17290       42   0       0             0 nmbd
Feb 17 22:11:41 phoenix kernel: [793305.063180] [29180]     0 29180    
23819       58   0       0             0 smbd
Feb 17 22:11:41 phoenix kernel: [793305.063184] [29187]     0 29187    
23845       43   0       0             0 smbd
Feb 17 22:11:41 phoenix kernel: [793305.063191] [ 3626]     0  3626    
31227       93   0       0             0 accounts-daemon
Feb 17 22:11:41 phoenix kernel: [793305.063196] [ 3694]     0  3694     
5239        1   0       0             0 bluetoothd
Feb 17 22:11:41 phoenix kernel: [793305.063200] [ 7949]  1000  7949     
5105       21   1       0             0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063205] [11357]  1000 11357     
5102        2   1       0             0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063210] [11418]  1000 11418    
13042        1   1       0             0 sudo
Feb 17 22:11:41 phoenix kernel: [793305.063214] [11419]     0 11419    
14020        0   0       0             0 su
Feb 17 22:11:41 phoenix kernel: [793305.063219] [11427]     0 11427     
5101        2   1       0             0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063223] [11503]     0 11503     
5448       15   0     -17         -1000 udevd
Feb 17 22:11:41 phoenix kernel: [793305.063228] [26994]  1000 26994     
1402       16   1       0             0 tail
Feb 17 22:11:41 phoenix kernel: [793305.063233] [31542]  1000 31542    
11738      147   0       0             0 gvfsd-metadata
Feb 17 22:11:41 phoenix kernel: [793305.063237] [10803]  1000 10803     
5110      272   0       0             0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063242] [19791]  1000 19791    
15524      110   1       0             0 gvfsd-burn
Feb 17 22:11:41 phoenix kernel: [793305.063247] [19830]  1000 19830    
49434      902   1       0             0 tracker-store
Feb 17 22:11:41 phoenix kernel: [793305.063251] [12901]  1000 12901   
135507     3618   1       0             0 nautilus
Feb 17 22:11:41 phoenix kernel: [793305.063256] [18926]  1000 18926    
11373      162   1       0             0 ssh
Feb 17 22:11:41 phoenix kernel: [793305.063261] [ 8721]  1000  8721   
230865    35018   1       0             0 x-www-browser
Feb 17 22:11:41 phoenix kernel: [793305.063266] [13247]  1000 13247    
13042       96   1       0             0 sudo
Feb 17 22:11:41 phoenix kernel: [793305.063271] [13257]     0 13257    
14020       94   0       0             0 su
Feb 17 22:11:41 phoenix kernel: [793305.063275] [13265]     0 13265     
5101      373   1       0             0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063280] [13322]     0 13322    
74838     7219   0       0             0 synaptic
Feb 17 22:11:41 phoenix kernel: [793305.063285] [13343]     0 13343    
96166    84274   1       0             0 update-apt-xapi
Feb 17 22:11:41 phoenix kernel: [793305.063289] [13446]     0 13446     
5106      385   0       0             0 bash
Feb 17 22:11:41 phoenix kernel: [793305.063293] [13529]  1000 13529     
1393       22   1       0             0 sleep
Feb 17 22:11:41 phoenix kernel: [793305.063297] Out of memory: Kill 
process 8721 (x-www-browser) score 14 or sacrifice child
Feb 17 22:11:41 phoenix kernel: [793305.063370] Killed process 8721 
(x-www-browser) total-vm:923460kB, anon-rss:140072kB, file-rss:0kB




More information about the Durham mailing list