[7934037.688261] sys.fcgi invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [7934037.693085] CPU: 1 PID: 1459496 Comm: sys.fcgi Not tainted 5.10.0-8-amd64 #1 Debian 5.10.46-4 [7934037.695050] Hardware name: Red Hat Linux KVM, BIOS 0.5.1 01/01/2011 [7934037.696350] Call Trace: [7934037.697070] dump_stack+0x6b/0x83 [7934037.697806] dump_header+0x4a/0x1f0 [7934037.698585] oom_kill_process.cold+0xb/0x10 [7934037.699481] out_of_memory+0x1bd/0x500 [7934037.700318] __alloc_pages_slowpath.constprop.0+0xb8c/0xc60 [7934037.701460] __alloc_pages_nodemask+0x2da/0x310 [7934037.702411] pagecache_get_page+0x16d/0x380 [7934037.703346] filemap_fault+0x69e/0x900 [7934037.704378] ext4_filemap_fault+0x2d/0x40 [ext4] [7934037.705338] __do_fault+0x36/0x120 [7934037.706086] handle_mm_fault+0x118e/0x1b80 [7934037.706999] do_user_addr_fault+0x1b8/0x3f0 [7934037.707891] exc_page_fault+0x78/0x160 [7934037.708729] ? asm_exc_page_fault+0x8/0x30 [7934037.709581] asm_exc_page_fault+0x1e/0x30 [7934037.710426] RIP: 0033:0x55b5bc05aef4 [7934037.711229] Code: Unable to access opcode bytes at RIP 0x55b5bc05aeca. [7934037.712648] RSP: 002b:00007ffcc2250080 EFLAGS: 00010206 [7934037.713739] RAX: 000055b5bcb3fbb8 RBX: 000055b5bcb3fbb8 RCX: 0000000000000002 [7934037.715164] RDX: 000055b5bcebcc70 RSI: 000055b5bcf3bbc8 RDI: 000055b5bcdf3cf0 [7934037.716618] RBP: 000055b5bcdf3cf0 R08: 0000000000000000 R09: 000055b5bcb3fbb8 [7934037.718107] R10: 000055b5bc1ea6c0 R11: 0000000000000000 R12: 000055b5bc7ad2a0 [7934037.719573] R13: 000055b5bcb3fba8 R14: 000055b5bcf116c2 R15: 000055b5bcb3fb98 [7934037.721162] Mem-Info: [7934037.721946] active_anon:99623 inactive_anon:115585 isolated_anon:0 active_file:63 inactive_file:135 isolated_file:0 unevictable:0 dirty:0 writeback:0 slab_reclaimable:5788 slab_unreclaimable:5805 mapped:6827 shmem:14204 pagetables:3604 bounce:0 free:12683 free_pcp:386 free_cma:0 [7934037.729930] Node 0 active_anon:398492kB inactive_anon:462344kB active_file:368kB inactive_file:540kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:27308kB dirty:0kB writeback:0kB shmem:56816kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 71680kB writeback_tmp:0kB kernel_stack:2544kB all_unreclaimable? yes [7934037.738154] Node 0 DMA free:4408kB min:740kB low:924kB high:1108kB reserved_highatomic:0KB active_anon:4488kB inactive_anon:6520kB active_file:0kB inactive_file:68kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB pagetables:156kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [7934037.746022] lowmem_reserve[]: 0 928 928 928 928 [7934037.747834] Node 0 DMA32 free:45316kB min:44312kB low:55388kB high:66464kB reserved_highatomic:0KB active_anon:394004kB inactive_anon:455836kB active_file:2192kB inactive_file:748kB unevictable:0kB writepending:0kB present:1032168kB managed:983816kB mlocked:0kB pagetables:14260kB bounce:0kB free_pcp:976kB local_pcp:912kB free_cma:0kB [7934037.756261] lowmem_reserve[]: 0 0 0 0 0 [7934037.757829] Node 0 DMA: 43*4kB (UME) 14*8kB (UME) 12*16kB (UE) 13*32kB (UME) 2*64kB (UE) 7*128kB (UME) 4*256kB (UME) 3*512kB (UME) 0*1024kB 0*2048kB 0*4096kB = 4476kB [7934037.762116] Node 0 DMA32: 258*4kB (UME) 252*8kB (UME) 576*16kB (UME) 325*32kB (UME) 105*64kB (UME) 20*128kB (UME) 8*256kB (UE) 3*512kB (UME) 5*1024kB (UME) 0*2048kB 1*4096kB (M) = 44744kB [7934037.767004] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [7934037.769732] 14893 total pagecache pages [7934037.771684] 515 pages in swap cache [7934037.773398] Swap cache stats: add 4651488, delete 4650924, find 245358574/245989276 [7934037.775954] Free swap = 0kB [7934037.777541] Total swap = 3906556kB [7934037.779209] 262040 pages RAM [7934037.780856] 0 pages HighMem/MovableOnly [7934037.782851] 12109 pages reserved [7934037.784496] 0 pages hwpoisoned [7934037.786116] Tasks state (memory values in pages): [7934037.788113] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [7934037.791023] [ 216] 0 216 32545 4 253952 250 -250 systemd-journal [7934037.794638] [ 235] 0 235 5382 0 69632 297 -1000 systemd-udevd [7934037.798299] [ 628] 101 628 22094 0 81920 205 0 systemd-timesyn [7934037.801818] [ 636] 0 636 1214 24 45056 41 0 cron [7934037.804555] [ 637] 104 637 2089 0 53248 166 -900 dbus-daemon [7934037.807338] [ 644] 0 644 55185 0 73728 495 0 rsyslogd [7934037.810107] [ 646] 0 646 3489 0 69632 236 0 systemd-logind [7934037.813550] [ 655] 0 655 989 0 45056 33 0 agetty [7934037.816101] [ 657] 0 657 897 0 53248 34 0 agetty [7934037.818712] [ 683] 1001 683 2921 55 61440 673 0 ws_server.pl [7934037.821953] [ 738] 106 738 53064 271 151552 437 -900 postgres [7934037.824293] [ 852] 0 852 9997 2 65536 162 0 master [7934037.826790] [ 882] 0 882 3323 1 69632 238 -1000 sshd [7934037.829394] [ 944] 106 944 53147 225 405504 511 0 postgres [7934037.831900] [ 945] 106 945 53099 212 405504 462 0 postgres [7934037.834318] [ 946] 106 946 53064 44 135168 469 0 postgres [7934037.836516] [ 947] 106 947 53236 148 143360 514 0 postgres [7934037.838665] [ 948] 106 948 16728 81 118784 451 0 postgres [7934037.841267] [ 949] 106 949 53204 43 135168 561 0 postgres [7934037.843773] [ 1006] 107 1006 10108 0 69632 216 0 qmgr [7934037.846316] [ 1074] 107 1074 11011 0 73728 274 0 tlsmgr [7934037.848831] [ 567764] 0 567764 3245 19 65536 505 0 apache2 [7934037.851365] [1445160] 33 1445160 2881 0 65536 543 0 apache2 [7934037.854048] [1445161] 33 1445161 3177 14 65536 512 0 apache2 [7934037.856488] [1445162] 33 1445162 305005 29 249856 3044 0 apache2 [7934037.859162] [1445163] 33 1445163 304979 0 245760 3144 0 apache2 [7934037.861523] [1445480] 33 1445480 274283 36020 2207744 228490 0 sys.fcgi [7934037.864023] [1448748] 33 1448748 255885 21403 2068480 224659 0 sys.fcgi [7934037.866498] [1458940] 107 1458940 10060 0 65536 172 0 pickup [7934037.869087] [1459488] 106 1459488 53754 2346 409600 834 0 postgres [7934037.871577] [1459492] 106 1459492 53754 3277 409600 838 0 postgres [7934037.874035] [1459496] 33 1459496 229925 23244 1880064 201714 0 sys.fcgi [7934037.876546] [1459499] 106 1459499 53754 2694 409600 775 0 postgres [7934037.878983] [1459503] 33 1459503 216295 23028 1769472 188295 0 sys.fcgi [7934037.881472] [1459506] 106 1459506 53754 3050 409600 862 0 postgres [7934037.883875] [1459511] 33 1459511 198559 95132 1634304 98432 0 sys.fcgi [7934037.886279] [1459520] 106 1459520 53754 2589 405504 668 0 postgres [7934037.888678] [1459528] 106 1459528 53064 132 126976 377 0 postgres [7934037.891021] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/apache2.service,task=sys.fcgi,pid=1445480,uid=33 [7934037.895202] Out of memory: Killed process 1445480 (sys.fcgi) total-vm:1097132kB, anon-rss:144080kB, file-rss:0kB, shmem-rss:0kB, UID:33 pgtables:2156kB oom_score_adj:0 [7934038.318068] oom_reaper: reaped process 1445480 (sys.fcgi), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB