v4l2h264dec invoked oom-killer : Hardware name: BCM2711 - application crashed

Locked

Print view

3 posts • Page 1 of 1

abypaulvarghese: Posts: 8; Joined: Mon May 06, 2019 2:54 pm

v4l2h264dec invoked oom-killer : Hardware name: BCM2711 - application crashed

Sat Jun 12, 2021 11:12 am

Hello All,

Context :
I have a PI4 with 2 GB RAM. (SWAP memory 100MB, GPU 512MB ) .I am developing an application that can capture H264 stream from IP cameras and put them into a ZMQ. I am using GStreamer and the pipeline looks like below :
pipelinestring = "rtspsrc location={0} name=m_rtspsrc ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 ! video/x-raw ! v4l2convert output-io-mode=5 capture-io-mode=4 ! video/x-raw, format=(string)BGR, width=(int)640, height=(int)480 ! videorate ! video/x-raw,framerate=1/1 ! appsink name=m_appsink sync=false".format(self.streamLink)

I am getting the frame from 4 different IP cameras, a separate process is started to fetch the frame from each cameras. ie I have 4 child process and a parent process. I have two other python programs to perform the analysis . As you might have noted, I plan to grab only one frame in a second from a camera. which means, I am expecting 4 frames in a second. Once I get the frame, I push it to a ZMQ to perform analysis on it.
Problem
The application works okay. Today I try to do an endurance testing of it. The application works okay for sometime (may be around 45 minutes?). The CPU utilization was okay, somewhere around 30-40 % (this is maximum usage, otherwise it used to be between 10-20). The temperature was also around 50-60 as per RPI. However I noticed the desktop flashes immediately and restarted (only the desktop GUI ) and prompted the Login ( this is not the system restart- system is configured not to prompt for password). I tried to launch my program and found all of them working wired, ie it was not connecting to cameras. I remember there was a message 'killed' if I remember correctly.

Analysis so far

I am a beginner in GStreamer all together. I started looked looking at the syslog. And I found symptoms of OOM. The following the portion from the syslog.
Incident 1 : Crash pointing out in v4l2h264dec .

Code: Select all

Jun 12 06:17:55 raspberrypi dhcpcd[436]: wlan0: deleting route to 192.168.1.0/24
Jun 12 06:17:55 raspberrypi dhcpcd[436]: wlan0: deleting default route via 192.168.1.1
Jun 12 06:18:00 raspberrypi systemd[1]: systemd-rfkill.service: Succeeded.
Jun 12 06:20:27 raspberrypi kernel: [ 244.140910] usb 1-1.1: USB disconnect, device number 3
Jun 12 06:20:28 raspberrypi kernel: [ 244.278554] usb 2-1: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Jun 12 06:20:28 raspberrypi kernel: [ 244.309785] usb 2-1: New USB device found, idVendor=03e7, idProduct=f63b, bcdDevice= 1.00
Jun 12 06:20:28 raspberrypi kernel: [ 244.309795] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jun 12 06:20:28 raspberrypi kernel: [ 244.309803] usb 2-1: Product: VSC Loopback Device
Jun 12 06:20:28 raspberrypi kernel: [ 244.309810] usb 2-1: Manufacturer: Intel Corporation
Jun 12 06:20:28 raspberrypi kernel: [ 244.309817] usb 2-1: SerialNumber: DE2856CFB41C00
Jun 12 06:20:28 raspberrypi mtp-probe: checking bus 2, device 2: "/sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1"
Jun 12 06:20:28 raspberrypi mtp-probe: bus: 2, device: 2 was not an MTP device
Jun 12 06:20:28 raspberrypi mtp-probe: checking bus 2, device 2: "/sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1"
Jun 12 06:20:28 raspberrypi mtp-probe: bus: 2, device: 2 was not an MTP device
Jun 12 06:20:31 raspberrypi kernel: [ 247.516048] v3d fec00000.v3d: MMU error from client L2T (0) at 0x38a1000, pte invalid
Jun 12 06:25:01 raspberrypi CRON[2465]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Jun 12 06:31:33 raspberrypi systemd[1]: Starting Cleanup of Temporary Directories...
Jun 12 06:31:33 raspberrypi systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Jun 12 06:31:33 raspberrypi systemd[1]: Started Cleanup of Temporary Directories.
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205093] v4l2h264dec0:sr invoked oom-killer: gfp_mask=0x400dc2(GFP_KERNEL_ACCOUNT|__GFP_HIGHMEM|__GFP_ZERO), order=0, oom_score_adj=0
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205106] CPU: 2 PID: 1857 Comm: v4l2h264dec0:sr Tainted: G C 5.10.17-v7l+ #1414
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205111] Hardware name: BCM2711
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205115] Backtrace: 
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205137] [<c0b63050>] (dump_backtrace) from [<c0b633e4>] (show_stack+0x20/0x24)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205144] r7:ffffffff r6:00000000 r5:60000113 r4:c12e69fc
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205152] [<c0b633c4>] (show_stack) from [<c0b67774>] (dump_stack+0xcc/0xf8)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205159] [<c0b676a8>] (dump_stack) from [<c0b65a04>] (dump_header+0x64/0x208)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205165] r10:c120518c r9:00400dc2 r8:00000000 r7:c0e3709c r6:c6605800 r5:c55dbe00
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205170] r4:c46d1de8 r3:8f05e476
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205180] [<c0b659a0>] (dump_header) from [<c03b6cd0>] (oom_kill_process+0x1b4/0x1c0)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205185] r7:c0e3709c r6:c46d1de8 r5:c55dc380 r4:c55dbe00
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205192] [<c03b6b1c>] (oom_kill_process) from [<c03b7848>] (out_of_memory+0x2b8/0x390)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205197] r7:c1208580 r6:c1205048 r5:c55dbe00 r4:c46d1de8
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205205] [<c03b7590>] (out_of_memory) from [<c040a8bc>] (__alloc_pages_nodemask+0x7ec/0x1184)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205210] r7:c1336790 r6:00001400 r5:0000174f r4:00000000
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205220] [<c040a0d0>] (__alloc_pages_nodemask) from [<c03ec934>] (__pte_alloc+0x34/0x168)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205226] r10:c1205048 r9:00000040 r8:00000255 r7:00000000 r6:c70b98d8 r5:c3a4b000
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205230] r4:a371e000
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205238] [<c03ec900>] (__pte_alloc) from [<c03f0068>] (handle_mm_fault+0x100c/0x1124)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205243] r7:00000000 r6:070b9003 r5:00000000 r4:a371e000
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205251] [<c03ef05c>] (handle_mm_fault) from [<c0b739d4>] (do_page_fault+0x144/0x314)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205256] r10:00000255 r9:00000800 r8:c3a4b048 r7:c3a4b000 r6:00000a06 r5:a371e000
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205260] r4:c46d1fb0
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205266] [<c0b73890>] (do_page_fault) from [<c0b73c84>] (do_translation_fault+0xe0/0x10c)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205272] r10:00000000 r9:b6dcb258 r8:00000a06 r7:a371e000 r6:c0b73ba4 r5:00000a06
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205276] r4:c120aef8
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205283] [<c0b73ba4>] (do_translation_fault) from [<c0214c6c>] (do_DataAbort+0x40/0xc0)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205289] r9:b6dcb258 r8:c46d1fb0 r7:a371e000 r6:c0b73ba4 r5:00000a06 r4:c120aef8
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205296] [<c0214c2c>] (do_DataAbort) from [<c0200da4>] (__dabt_usr+0x44/0x60)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205300] Exception stack(0xc46d1fb0 to 0xc46d1ff8)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205305] 1fa0: a371e000 000e2002 00000000 a371e008
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205311] 1fc0: a9200010 000e2000 0001c201 000e1018 b6dcb000 b6dcb258 00000000 fffff000
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205316] 1fe0: b6dcb26c a49747b0 b6dcb258 b6cf18d8 40000010 ffffffff
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205322] r8:30c5387d r7:30c5383d r6:ffffffff r5:40000010 r4:b6cf18d8
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205326] Mem-Info:
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205338] active_anon:42279 inactive_anon:244363 isolated_anon:0
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205338] active_file:619 inactive_file:5129 isolated_file:0
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205338] unevictable:6795 dirty:0 writeback:0
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205338] slab_reclaimable:3594 slab_unreclaimable:11751
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205338] mapped:6721 shmem:8017 pagetables:2109 bounce:0
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205338] free:5252 free_pcp:0 free_cma:2
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205349] Node 0 active_anon:169116kB inactive_anon:977452kB active_file:2476kB inactive_file:20516kB unevictable:27180kB isolated(anon):0kB isolated(file):0kB mapped:26884kB dirty:0kB writeback:0kB shmem:32068kB writeback_tmp:0kB kernel_stack:2544kB all_unreclaimable? no
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205360] DMA free:18136kB min:20480kB low:24576kB high:28672kB reserved_highatomic:0KB active_anon:0kB inactive_anon:158976kB active_file:932kB inactive_file:4600kB unevictable:0kB writepending:0kB present:524288kB managed:421900kB mlocked:0kB pagetables:848kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:8kB
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205366] lowmem_reserve[]: 0 0 1024 0
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205402] HighMem free:2872kB min:31820kB low:42000kB high:52180kB reserved_highatomic:0KB active_anon:169116kB inactive_anon:818476kB active_file:1448kB inactive_file:15944kB unevictable:27180kB writepending:0kB present:1048576kB managed:1048576kB mlocked:16kB pagetables:7588kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205408] lowmem_reserve[]: 0 0 0 0
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205439] DMA: 604*4kB (UMEC) 272*8kB (UMEC) 206*16kB (UME) 103*32kB (UME) 82*64kB (UME) 14*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 18224kB
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205543] HighMem: 180*4kB (UM) 67*8kB (UM) 37*16kB (UM) 30*32kB (UM) 7*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3256kB
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205640] 13826 total pagecache pages
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205647] 23 pages in swap cache
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205654] Swap cache stats: add 25651, delete 25628, find 1089/1106
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205660] Free swap = 0kB
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205666] Total swap = 102396kB
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205672] 393216 pages RAM
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205678] 262144 pages HighMem/MovableOnly
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205684] 25597 pages reserved
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205690] 65536 pages cma reserved
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205696] Tasks state (memory values in pages):
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205702] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205718] [ 113] 0 113 5352 19 167936 224 0 systemd-journal
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205726] [ 148] 0 148 4651 49 49152 195 -1000 systemd-udevd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205736] [ 322] 100 322 5604 17 61440 146 0 systemd-timesyn
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205744] [ 366] 108 366 1442 34 40960 43 0 avahi-daemon
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205752] [ 370] 0 370 1987 15 45056 30 0 cron
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205759] [ 371] 0 371 16305 71 90112 397 0 udisksd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205767] [ 382] 108 382 1442 14 32768 49 0 avahi-daemon
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205775] [ 383] 0 383 6914 8 40960 15 0 rngd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205783] [ 387] 0 387 2942 14 49152 137 0 alsactl
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205791] [ 405] 104 405 1710 71 40960 96 -900 dbus-daemon
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205798] [ 413] 0 413 2686 13 45056 88 0 wpa_supplicant
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205806] [ 414] 0 414 6378 0 57344 219 0 rsyslogd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205814] [ 419] 0 419 3261 18 53248 151 0 systemd-logind
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205821] [ 426] 65534 426 1080 6 36864 38 0 thd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205829] [ 436] 0 436 726 8 28672 93 0 dhcpcd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205836] [ 487] 0 487 2157 0 45056 56 0 vncserver-x11-s
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205844] [ 496] 0 496 9827 78 90112 1344 0 vncserver-x11-c
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205851] [ 497] 0 497 10358 1 77824 427 0 polkitd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205859] [ 503] 0 503 10025 1 65536 198 0 lightdm
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205867] [ 513] 0 513 2682 0 49152 143 -1000 sshd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205874] [ 524] 0 524 2751 23 45056 121 0 wpa_supplicant
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205882] [ 542] 0 542 42377 3436 290816 3468 0 Xorg
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205890] [ 543] 0 543 1405 2 40960 111 0 login
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205898] [ 553] 0 553 2870 1 45056 123 0 vncagent
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205905] [ 561] 0 561 8125 0 69632 214 0 lightdm
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205913] [ 566] 1000 566 3682 2 53248 255 0 systemd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205920] [ 567] 1000 567 4185 14 57344 401 0 (sd-pam)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205928] [ 577] 1000 577 14052 2 98304 317 0 lxsession
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205936] [ 585] 1000 585 1637 0 40960 110 0 dbus-daemon
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205943] [ 609] 1000 609 1130 10 32768 62 0 ssh-agent
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205951] [ 620] 1000 620 10852 1 69632 166 0 gvfsd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205958] [ 625] 1000 625 14189 0 77824 188 0 gvfsd-fuse
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205966] [ 632] 1000 632 15583 2 94208 493 0 openbox
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205974] [ 634] 1000 634 11782 1 94208 275 0 lxpolkit
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205981] [ 643] 1000 643 105838 1025 217088 1035 0 lxpanel
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205989] [ 644] 1000 644 26595 1020 143360 975 0 pcmanfm
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205996] [ 652] 1000 652 1130 1 32768 71 0 ssh-agent
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206004] [ 654] 1000 654 1218 0 36864 67 0 xcompmgr
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206012] [ 668] 1000 668 12973 0 126976 3302 0 applet.py
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206019] [ 671] 1000 671 91936 0 86016 316 0 pulseaudio
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206027] [ 686] 110 686 5797 1 49152 50 0 rtkit-daemon
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206035] [ 693] 1000 693 2189 1 45056 303 0 bash
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206042] [ 706] 1000 706 19865 0 94208 344 0 gvfs-udisks2-vo
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206050] [ 712] 1000 712 8006 1 86016 541 0 vncserverui
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206058] [ 732] 1000 732 14007 1 94208 202 0 gvfs-afc-volume
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206065] [ 734] 1000 734 7174 0 57344 150 0 menu-cached
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206072] [ 742] 1000 742 10032 0 69632 133 0 gvfs-mtp-volume
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206080] [ 746] 1000 746 7934 30 90112 378 0 vncserverui
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206088] [ 747] 1000 747 10424 0 65536 161 0 gvfs-gphoto2-vo
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206095] [ 751] 1000 751 10026 0 61440 122 0 gvfs-goa-volume
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206103] [ 761] 0 761 535 1 28672 31 0 hciattach
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206111] [ 767] 0 767 2384 0 45056 94 0 bluetoothd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206118] [ 771] 1000 771 14239 0 90112 700 0 gvfsd-trash
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206126] [ 1046] 0 1046 6910 0 69632 231 0 cupsd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206134] [ 1048] 0 1048 10124 1 77824 272 0 cups-browsed
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206142] [ 1295] 1000 1295 42312 1756 290816 2000 0 lxterminal
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206149] [ 1304] 1000 1304 2226 2 40960 336 0 bash
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206157] [ 1328] 1000 1328 2226 55 36864 278 0 bash
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206173] [ 1347] 1000 1347 2226 59 40960 279 0 bash
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206210] [ 1376] 1000 1376 2193 14 40960 284 0 bash
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206218] [ 1544] 1000 1544 47592 3750 233472 282 0 python
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206225] [ 1670] 1000 1670 77117 10044 344064 0 0 python3
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206233] [ 1802] 1000 1802 44780 4787 225280 0 0 python3
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206241] [ 1810] 1000 1810 89999 13550 385024 7 0 python3
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206248] [ 1839] 1000 1839 84168 7402 331776 5 0 python3
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206256] [ 1859] 1000 1859 325845 249318 2273280 5 0 python3
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206263] [ 1887] 1000 1887 83785 8281 331776 5 0 python3
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206271] [ 1926] 1000 1926 2022 132 40960 0 0 htop
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206282] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=python3,pid=1859,uid=1000
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206371] Out of memory: Killed process 1859 (python3) total-vm:1303380kB, anon-rss:997260kB, file-rss:0kB, shmem-rss:12kB, UID:1000 pgtables:2220kB oom_score_adj:0
Jun 12 07:09:03 raspberrypi kernel: [ 3159.403435] oom_reaper: reaped process 1859 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:8kB
Jun 12 07:16:31 raspberrypi rngd[383]: stats: bits received from HRNG source: 80064
Jun 12 07:16:31 raspberrypi rngd[383]: stats: bits sent to kernel pool: 33824
Jun 12 07:16:31 raspberrypi rngd[383]: stats: entropy added to kernel pool: 33824

My observations :
The Free Swap memory is noted as 0 KB.

Questions : What is causing the crash ? is it because the Swap memory becoming 0 causing the crash? Could someone help me to understand how should I handle it ? is it something that I need to take care in the pipeline or in the python program?

Kindly note: Happy to share the complete source code on Github if required.

Thank You
Aby

6by9: Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator; Posts: 18476; Joined: Wed Dec 04, 2013 11:27 am

Re: v4l2h264dec invoked oom-killer : Hardware name: BCM2711 - application crashed

Sat Jun 12, 2021 1:50 pm

v4l2h264dec wanted some memory and there was none left. That triggers the Out Of Memory killer to try and free up some memory.
It's not the fault of v4l2h264dec, but probably indicative that something in the system has a memory leak.

gpu_mem of 512MB is generally excessive, although it does depend on what level your H264 streams are. 256MB should be sufficient.
You may wish to check how much CMA heap is available as V4L2 image buffers are allocated from there, as are all desktop graphics resources. "dmesg | grep -i cma" is probably the quickest check if done soon after boot.

Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

abypaulvarghese: Posts: 8; Joined: Mon May 06, 2019 2:54 pm

Re: v4l2h264dec invoked oom-killer : Hardware name: BCM2711 - application crashed

Sat Jun 12, 2021 4:35 pm

Thank you very much 6by9.

The overall process is given below:
App 1: Grab a frame from each of the cameras (4 cameras in total) in every second ->Push it to ZMQ (PUB - publisher mode) . Total 4 Publishers.
App 2: Grab the frame from ZMQ (SUB) by connecting to 4 publishers. Perform an AI inference using Intel NCS which is connected to one of the USB ports. The result is pushed into another ZMQ.
App3 : Grabs the data from ZMQ of App2 . Log / communicate the result over web.

I ran the command and the output is below.

Code: Select all

pi@raspberrypi:~ $ dmesg | grep -i cma
[ 0.000000] Reserved memory: created CMA memory pool at 0x000000000fc00000, size 256 MiB
[ 0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[ 0.000000] Memory: 1206284K/1572864K available (10240K kernel code, 1358K rwdata, 3160K rodata, 2048K init, 890K bss, 104436K reserved, 262144K cma-reserved, 1048576K highmem)
[ 4.917746] vc_sm_cma: module is from the staging directory, the quality is unknown, you have been warned.
[ 4.923118] bcm2835_vc_sm_cma_probe: Videocore shared memory driver

I observed similar issue once again, however this time it was clearly from one of the python application as per the syslog. The call stack was pointing to tcp_sendmsg() as per syslog.
Hence I my hypothesis is:
Hypothesis 1: The App2 (which is supposed to process the frames from the Queue ) is crashed due to some reason(may be an exception from NCS or other reasons). The 4 PUBs continued to push data into the Queue and since the SUB was down the memory kept on growing. It failed at one stage where there was no more memory left to add new item to the Queue.
I am planning to test the above hypothesis and planning to change the GPU memory to 256 MB. thoughts?

Thank you
Aby

Locked

Print view

3 posts • Page 1 of 1

Return to "OpenMAX"

Jump to