Customize Consent Preferences

We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.

The cookies that are categorized as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... 

Always Active

Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

No cookies to display.

Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

No cookies to display.

Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

No cookies to display.

Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.

No cookies to display.

Advertisement cookies are used to provide visitors with customized advertisements based on the pages you visited previously and to analyze the effectiveness of the ad campaigns.

No cookies to display.

Finding kernel memory leaks using kmemleak

Created on April 4, 2023

Note:
These details are with respect to RPI 3 and it is not validated in RPI 4 yet.

Introduction

  • Kmemleak is a Linux kernel feature designed to detect and report memory leaks in the kernel code. Memory leaks occur when a program allocates memory dynamically but fails to release it when it is no longer needed. This can result in a gradual loss of memory over time, eventually leading to system instability or even crashes.
  • Kmemleak works by scanning the kernel memory periodically to detect any memory that has been allocated but not freed. When it detects a potential memory leak, it reports it to the system log along with a stack trace that shows where the memory was allocated. This information can help developers to identify and fix the source of the memory leak.
  • Kmemleak is a useful tool for kernel developers and system administrators who want to ensure that their systems are stable and free of memory leaks. It can be enabled in the kernel configuration, and once enabled, it will run automatically in the background, periodically scanning the kernel memory for leaks.

Required kernel configs need to enable

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
CONFIG_HAVE_DEBUG_KMEMLEAK=y<br>CONFIG_DEBUG_KMEMLEAK=y<br>CONFIG_DEBUG_KMEMLEAK_TEST=m
CONFIG_HAVE_DEBUG_KMEMLEAK=y<br>CONFIG_DEBUG_KMEMLEAK=y<br>CONFIG_DEBUG_KMEMLEAK_TEST=m
CONFIG_HAVE_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK_TEST=m

Integrating Kmemleak tool  for RPI-3

1) Create kmemleak.cfg file under meta-cmf-raspberrypi/recipes-kernel/linux/files with below kernel configs.
Note: Need to add this change in corresponding platform OEM layer.

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
CONFIG_HAVE_DEBUG_KMEMLEAK=y<br>CONFIG_DEBUG_KMEMLEAK=y<br>CONFIG_DEBUG_KMEMLEAK_TEST=m
CONFIG_HAVE_DEBUG_KMEMLEAK=y<br>CONFIG_DEBUG_KMEMLEAK=y<br>CONFIG_DEBUG_KMEMLEAK_TEST=m
CONFIG_HAVE_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK_TEST=m

2) Add kmemleak.cfg file  in meta-cmf-raspberrypi/recipes-kernel/linux/linux-raspberrypi_5.10.52.bb 
Note
: Need to add this change in corresponding platform OEM layer.

Please find below diff file.
Ex:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diff --git a/recipes-kernel/linux/linux-raspberrypi_5.10.52.bb b/recipes-kernel/linux/linux-raspberrypi_5.10.52.bb<br>index 2c7f19a..224a64a 100644<br>--- a/recipes-kernel/linux/linux-raspberrypi_5.10.52.bb<br>+++ b/recipes-kernel/linux/linux-raspberrypi_5.10.52.bb<br>@@ -11,6 +11,7 @@ FILESEXTRAPATHS_prepend := "${THISDIR}/files:"<br><br> SRC_URI += " file://powersave.cfg <br> file://android-drivers.cfg <br>+ file://kmemleak.cfg <br> "
diff --git a/recipes-kernel/linux/linux-raspberrypi_5.10.52.bb b/recipes-kernel/linux/linux-raspberrypi_5.10.52.bb<br>index 2c7f19a..224a64a 100644<br>--- a/recipes-kernel/linux/linux-raspberrypi_5.10.52.bb<br>+++ b/recipes-kernel/linux/linux-raspberrypi_5.10.52.bb<br>@@ -11,6 +11,7 @@ FILESEXTRAPATHS_prepend := "${THISDIR}/files:"<br><br> SRC_URI += " file://powersave.cfg <br> file://android-drivers.cfg <br>+ file://kmemleak.cfg <br> "
diff --git a/recipes-kernel/linux/linux-raspberrypi_5.10.52.bb b/recipes-kernel/linux/linux-raspberrypi_5.10.52.bb
index 2c7f19a..224a64a 100644
--- a/recipes-kernel/linux/linux-raspberrypi_5.10.52.bb
+++ b/recipes-kernel/linux/linux-raspberrypi_5.10.52.bb
@@ -11,6 +11,7 @@ FILESEXTRAPATHS_prepend := "${THISDIR}/files:"

 SRC_URI += " file://powersave.cfg
              file://android-drivers.cfg
+             file://kmemleak.cfg
 "

3) Once build done with above changes, verify the  kernel configs are enabled in final kernel boot config file.
Path of kernel boot config file:  build-raspberrypi-rdk-mc/tmp/work/raspberrypi_rdk_mc-rdk-linux-gnueabi/linux-raspberrypi/1_5.10.52+gitAUTOINC+dbe03fa900_2697f74031-r0/image/boot/config-5.10.52-v7
Note:
Need to verify in corresponding platform specific final kernel boot config file
Ex:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
csures569@dvm-yocto3-docker-csures569:~/rpi-stack/build-raspberrypi-rdk-mc/tmp/work/raspberrypi_rdk_mc-rdk-linux-gnueabi/linux-raspberrypi/1_5.10.52+gitAUTOINC+dbe03fa900_2697f74031-r0/image/boot$ cat config-5.10.52-v7 | grep "KMEMLEAK"<br>CONFIG_HAVE_DEBUG_KMEMLEAK=y<br>CONFIG_DEBUG_KMEMLEAK=y<br>CONFIG_DEBUG_KMEMLEAK_MEM_POOL_SIZE=16000<br>CONFIG_DEBUG_KMEMLEAK_TEST=m<br># CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF is not set<br>CONFIG_DEBUG_KMEMLEAK_AUTO_SCAN=y
csures569@dvm-yocto3-docker-csures569:~/rpi-stack/build-raspberrypi-rdk-mc/tmp/work/raspberrypi_rdk_mc-rdk-linux-gnueabi/linux-raspberrypi/1_5.10.52+gitAUTOINC+dbe03fa900_2697f74031-r0/image/boot$ cat config-5.10.52-v7 | grep "KMEMLEAK"<br>CONFIG_HAVE_DEBUG_KMEMLEAK=y<br>CONFIG_DEBUG_KMEMLEAK=y<br>CONFIG_DEBUG_KMEMLEAK_MEM_POOL_SIZE=16000<br>CONFIG_DEBUG_KMEMLEAK_TEST=m<br># CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF is not set<br>CONFIG_DEBUG_KMEMLEAK_AUTO_SCAN=y
csures569@dvm-yocto3-docker-csures569:~/rpi-stack/build-raspberrypi-rdk-mc/tmp/work/raspberrypi_rdk_mc-rdk-linux-gnueabi/linux-raspberrypi/1_5.10.52+gitAUTOINC+dbe03fa900_2697f74031-r0/image/boot$ cat config-5.10.52-v7  | grep "KMEMLEAK"
CONFIG_HAVE_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK_MEM_POOL_SIZE=16000
CONFIG_DEBUG_KMEMLEAK_TEST=m
# CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF is not set
CONFIG_DEBUG_KMEMLEAK_AUTO_SCAN=y

4) On flashing the image with above changes , verify the kmemleak enabled in the device, it will create a kmemleak folder under /sys/modules/ and kmemleak file under /sys/kernel/debug/
Ex:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
root@raspberrypi-rdk-mc:/sys/module# ls -l | grep "kmemleak"<br>drwxr-xr-x 3 root root 0 Mar 5 09:49 kmemleak<br><br>root@raspberrypi-rdk-mc:~# ls /sys/kernel/debug/kmemleak<br>/sys/kernel/debug/kmemleak<br><br>
root@raspberrypi-rdk-mc:/sys/module# ls -l | grep "kmemleak"<br>drwxr-xr-x 3 root root 0 Mar 5 09:49 kmemleak<br><br>root@raspberrypi-rdk-mc:~# ls /sys/kernel/debug/kmemleak<br>/sys/kernel/debug/kmemleak<br><br>
root@raspberrypi-rdk-mc:/sys/module# ls -l | grep "kmemleak"
drwxr-xr-x    3 root     root             0 Mar  5 09:49 kmemleak

root@raspberrypi-rdk-mc:~# ls /sys/kernel/debug/kmemleak
/sys/kernel/debug/kmemleak

5) Once enabled kmemleak we will get kernel memory leaks in /sys/kernel/debug/kmemleak file if leaks are reported.

Testing kmemleak tool with kmemleak-test module in RPI-3

kmemleak-test is a test module available as part of kmemleak. By default, this module is not compiled in RPI3 and so there is no kmemleak-test.ko module. If needed, we need to enable to build as a module by following the above mentioned kernel configurations.
Path of kmemleak-test module source : build-raspberrypi-rdk-mc/tmp/work-shared/raspberrypi-rdk-mc/kernel-source/samples/kmemleak/kmemleak-test.c
kmemleak-test.ko

Note: Need to generate kmemleak-test.ko module file from corresponding platform specific kernel source code.

Step 1 : Copy kmemleak-test.ko module to this path /lib/modules/5.10.52-v7/  in box .
Step 2 : load the module to kernel space with insmod command.
Ex:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
root@raspberrypi-rdk-mc:~# insmod /lib/modules/5.10.52-v7/kmemleak-test.ko<br>root@raspberrypi-rdk-mc:~#
root@raspberrypi-rdk-mc:~# insmod /lib/modules/5.10.52-v7/kmemleak-test.ko<br>root@raspberrypi-rdk-mc:~#
root@raspberrypi-rdk-mc:~# insmod /lib/modules/5.10.52-v7/kmemleak-test.ko
root@raspberrypi-rdk-mc:~#

Step 3 : verify whether module is loaded or not
Ex: lsmod

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
root@raspberrypi-rdk-mc:~# lsmod<br><br>Module Size Used by<br><br>kmemleak_test 16384 0 // kmemleak test module loaded.<br>ip6t_REJECT 16384 2<br>nf_reject_ipv6 16384 1 ip6t_REJECT<br>ip6table_nat 16384 1<br>br_netfilter 32768 0<br>xt_state 16384 0
root@raspberrypi-rdk-mc:~# lsmod<br><br>Module Size Used by<br><br>kmemleak_test 16384 0 // kmemleak test module loaded.<br>ip6t_REJECT 16384 2<br>nf_reject_ipv6 16384 1 ip6t_REJECT<br>ip6table_nat 16384 1<br>br_netfilter 32768 0<br>xt_state 16384 0
root@raspberrypi-rdk-mc:~# lsmod

Module                Size      Used by

kmemleak_test    16384        0 // kmemleak test module loaded.
ip6t_REJECT         16384       2
nf_reject_ipv6      16384        1 ip6t_REJECT
ip6table_nat        16384        1
br_netfilter           32768       0
xt_state                16384       0

Step 4: Perform a scan
Ex: echo scan > /sys/kernel/debug/kmemleak

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
root@raspberrypi-rdk-mc:~# echo scan > /sys/kernel/debug/kmemleak<br>root@raspberrypi-rdk-mc:~#
root@raspberrypi-rdk-mc:~# echo scan > /sys/kernel/debug/kmemleak<br>root@raspberrypi-rdk-mc:~#
root@raspberrypi-rdk-mc:~# echo scan > /sys/kernel/debug/kmemleak
root@raspberrypi-rdk-mc:~#

Step 5: Collect memory leak data

Ex: cat /sys/kernel/debug/kmemleak



Note :
In  kmemleak-test module ( kmemleak-test.c) the  kernel memory leaks functionality present in the  kmemleak_test_init function and kmemleak_test_exit function, so memory leaks will be detect when module loaded into kernel space and removed from kernel space. 

Testing memory leaks from your kernel code

Suppose that you have a kernel module code which you want to test for kernel memory leaks. Make sure that kmemleak is enabled as mentioned above. Then, run the functionality of your kernel module by invoking it. Any memory leaks present in the kernel driver code will be reported in /sys/kernel/debug/kmemleak.

Memory leaks from module_init and module_exit functions get reported when the driver is inserted or removed. Inorder to test  the memory leaks present in other functions , we need to manually invoke those functions. 

Limitations and Drawbacks

While kmemleak can be a useful tool for detecting and reporting memory leaks in the Linux kernel, there are several limitations and drawbacks to consider:
The main drawback is the reduced performance of memory allocation and freeing. To avoid other penalties, the memory scanning is only performed when the /sys/kernel/debug/kmemleak file is read. Anyway, this tool is intended for debugging purposes where the performance might not be the most important requirement.

  1. False positives: Kmemleak may report potential memory leaks that are actually false positives, leading to unnecessary investigation and debugging.

  2. Overhead: Kmemleak can increase the overhead of the kernel, particularly during the initial scanning phase. This can impact system performance and resource utilization.

  3. Incomplete coverage: Kmemleak may not be able to detect all memory leaks, particularly those that occur in modules or code that are not regularly scanned.

  4. Debugging complexity: Debugging memory leaks can be a complex and time-consuming process, particularly in large and complex kernel codebases.

Ref: https://static.lwn.net/kerneldoc/dev-tools/kmemleak.html