KASAN based on MTE on Android Pixel 8 does not work

Question

I am working on a Pixel 8 device running CalyxOS. uname -a: Linux localhost 5.15.137-android14-11-gbc062a78e195-ab12057991 #1 SMP PREEMPT Mon Jul 8 12:34:46 UTC 2024 aarch64 Toybox.

The device supports MTE tagging, both in userspace and in the kernel (It has ArmV9 at all the cores, and the feature is present unser /proc/cpuinfo).

I am trying to enable it in the kernel.

I followed the following guide: https://outflux.net/blog/archives/2023/10/26/enable-mte-on-pixel-8/ Now I am trying to trigger MTE exception by accessing memory cell using wrongly tagged pointer. I've noticed that I get KASAN report in dmesg, BUT:

1. I only get it once, that is, if I rerun my kernel module which does the same access error, I don't get the report again.
2. The kernel module continues to run even after the report was produced (and at all the other runs afterwards). I want the access to raise an exception and crush (that is, I want it to run in MTE-sync mode).

How can I make those changes? According to https://source.android.com/docs/security/test/memory-safety/mte-configuration, the default is sync mode.

The mismatch code is the following:

static long __nocfi tester_ioctl(struct file* file, unsigned int cmd, unsigned long arg) {
 int err = 13056;
 unsigned long flags = 0;
 raw_local_irq_save(flags);
 for(int i = 0; i < 20; ++i) {
 module_err("Loaded with errorcode: %d", err);
 module_err("measurement code at: %px", measurement_code);
 if(0 <= err) {
 config_pfc();
 void* p = &sandbox; // Here is the start of interesting part
 const void* tagged_p = __tag_set(p, 7);
 stg(tagged_p);
 tagged_p = __tag_set(p, 5);
 *(char*)tagged_p = 1; // Here should trigger an MTE m
 }
 }
 raw_local_irq_restore(flags);
 module_err("finished");
 return 0;
}

The report I get is:

[10708.040449] testing_trace: Loaded with errorcode: 13056
[10708.040451] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.040477] ==================================================================
[10708.040485] BUG: KASAN: invalid-access in tester_ioctl+0x134/0x18c [testing_trace]
[10708.040501] Write at addr f5ffffe4dae450a0 by task executor_userla/8710
[10708.040506] Pointer tag: [f5], memory tag: [f7]
[10708.040510]
[10708.040517] CPU: 5 PID: 8710 Comm: executor_userla Tainted: G S B W OE 5.15.137-android14-11-gbc062a78e195-ab12057991 #1
[10708.040524] Hardware name: ZUMA SHIBA MP based on ZUMA (DT)
[10708.040528] Call trace:
[10708.040532] dump_backtrace+0xf8/0x1e8
[10708.040565] dump_stack_lvl+0x74/0xa4
[10708.040594] print_report+0x344/0x958
[10708.040624] kasan_report+0x90/0xe4
[10708.040631] __do_kernel_fault+0xc4/0x2ac
[10708.040642] do_bad_area+0x3c/0x154
[10708.040647] do_tag_check_fault+0x18/0x24
[10708.040651] do_mem_abort+0x60/0x134
[10708.040656] el1_abort+0x38/0x54
[10708.040687] el1h_64_sync_handler+0x58/0x9c
[10708.040693] el1h_64_sync+0x78/0x7c
[10708.040700] tester_ioctl+0x134/0x18c [testing_trace]
[10708.040707] __arm64_sys_ioctl+0x178/0x1f8
[10708.040733] invoke_syscall+0x58/0x138
[10708.040746] el0_svc_common+0xb0/0xe8
[10708.040752] do_el0_svc+0x20/0x7c
[10708.040756] el0_svc+0x28/0x9c
[10708.040764] el0t_64_sync_handler+0x7c/0xe4
[10708.040770] el0t_64_sync+0x1b4/0x1b8
[10708.040774]
[10708.040777] The buggy address belongs to the variable:
[10708.040781] sandbox+0x0/0xffffffffffff5f60 [testing_trace]
[10708.040789]
[10708.040811] The buggy address belongs to the virtual mapping at
 [f8ffffe4dae36000, f8ffffe4dae69000) created by:
 load_module+0x130c/0x18fc
[10708.040836]
[10708.040842] The buggy address belongs to the physical page:
[10708.040847] page:0000000086eff88f refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x90a04b
[10708.040855] flags: 0x41c0000000000000(zone=1|kasantag=0x7)
[10708.040864] raw: 41c0000000000000 0000000000000000 dead000000000122 0000000000000000
[10708.040870] raw: 0000000000000000 0000000000000000 00000001ffffffff 0000000000000000
[10708.040873] page dumped because: kasan: bad access detected
[10708.040878]
[10708.040881] Memory state around the buggy address:
[10708.040888] ffffffe4dae44e00: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8
[10708.040892] ffffffe4dae44f00: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8
[10708.040896] >ffffffe4dae45000: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f7 f6 f6 f6 f6 f6
[10708.040899] ^
[10708.040905] ffffffe4dae45100: f6 f6 f6 f6 f6 f6 f6 f6 f6 f6 f6 f6 f6 f6 f6 f6
[10708.040909] ffffffe4dae45200: f6 f6 f6 f6 f6 f6 f6 f6 f6 f6 f6 f6 f6 f6 f6 f6
[10708.040928] ==================================================================
[10708.040934] testing_trace: Loaded with errorcode: 13056
[10708.040935] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.040939] testing_trace: Loaded with errorcode: 13056
[10708.040943] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.040946] testing_trace: Loaded with errorcode: 13056
[10708.040950] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.040953] testing_trace: Loaded with errorcode: 13056
[10708.040957] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.040960] testing_trace: Loaded with errorcode: 13056
[10708.040963] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.040967] testing_trace: Loaded with errorcode: 13056
[10708.040970] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.040974] testing_trace: Loaded with errorcode: 13056
[10708.040977] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.040980] testing_trace: Loaded with errorcode: 13056
[10708.040984] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.040987] testing_trace: Loaded with errorcode: 13056
[10708.040990] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.040994] testing_trace: Loaded with errorcode: 13056
[10708.040997] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.041001] testing_trace: Loaded with errorcode: 13056
[10708.041004] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.041008] testing_trace: Loaded with errorcode: 13056
[10708.041011] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.041015] testing_trace: Loaded with errorcode: 13056
[10708.041018] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.041021] testing_trace: Loaded with errorcode: 13056
[10708.041025] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.041028] testing_trace: Loaded with errorcode: 13056
[10708.041031] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.041035] testing_trace: Loaded with errorcode: 13056
[10708.041038] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.041042] testing_trace: Loaded with errorcode: 13056
[10708.041045] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.041049] testing_trace: Loaded with errorcode: 13056
[10708.041052] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.041056] testing_trace: Loaded with errorcode: 13056
[10708.041059] testing_trace: measurement code at: ffffffe4dae3d0a0
[10708.041101] testing_trace: finished

Kernel command-line:

shiba:/data/local/tmp # cat /proc/cmdline
console=ttynull stack_depot_disable=on cgroup_disable=pressure kasan.page_alloc.sample=10 kasan.stacktrace=off bootconfig ioremap_guard kvm-arm.mode=protected root=/dev/ram0 rw clocksource=arch_sys_counter clk_ignore_unused loop.max_part=7 loop.hw_queue_depth=31 coherent_pool=4M firmware_class.path=/vendor/firmware irqaffinity=0 swiotlb=noforce sysrq_always_enabled no_console_suspend softlockup_panic=1 kasan_multi_shot kvm-arm.protected_modules=exynos-pd,pkvm_s2mpu-v9 exynos_drm.load_sequential=1 g2d.load_sequential=1 samsung_iommu_v9.load_sequential=1 swiotlb=noforce disable_dma32=on printk.devkmsg=on cma_sysfs.experimental=Y cgroup_disable=memory rcupdate.rcu_expedited=1 rcu_nocbs=all swiotlb=1024 cgroup.memory=nokmem sysctl.kernel.sched_pelt_multiplier=4 kasan=off at24.write_timeout=100 log_buf_len=1024K bootconfig console=null exynos_drm.panel_name=google-bigsurf.04a050 tcpci_max77759.conf_sbu=0 kasan=on ufs_pixel_fips140.fips_first_lba=151942 ufs_pixel_fips140.fips_last_lba=152453 ufs_pixel_fips140.fips_lu=0 id_aa64mmfr1.vh=0 bootconfig

Thank you!

• k1r1t0: Can you provide your module that generates tag mismatch fault and the report?
• k1r1t0: Note what docs says for Hardware Tag-Based KASAN on docs.kernel.org/dev-tools/kasan.html: Hardware Tag-Based KASAN only reports the first found bug. After that, MTE tag checking gets disabled.. Probably the way to get KASAN to work properly is to make it panic once a bug is found, i.e. kasan.fault=panic or something similar in command line should work
• k1r1t0: Also, there's a discussion groups.google.com/g/kasan-dev/c/BTh8IZcBxRU/m/… with the author of the patch that describes the reason
• Gal Kaptsenel: Thank you! I've read them, and indeed it explains some questions I had, but I TBH I don't know how can I change this on my Pixel 8 device as I can't recompile the kernel, as it is all signed by secure boot etc. Additionally, I've read here: source.android.com/docs/security/test/memory-safety/… that the default is "sync" mode, which means I should get a panic on the first invalid access, which I don't get. I'd add that as part of my module I track the cache lines (using Prime and Probe technique), and I can't see any cache line related to the tags being loaded.
• Gal Kaptsenel: I think that it is in Software Tag-Based KASAN mode and not in Hardware Tag-Based KASAN mode. The current configuration is: CONFIG_HAVE_ARCH_KASAN=y CONFIG_HAVE_ARCH_KASAN_SW_TAGS=y CONFIG_HAVE_ARCH_KASAN_HW_TAGS=y CONFIG_HAVE_ARCH_KASAN_VMALLOC=y CONFIG_CC_HAS_KASAN_GENERIC=y CONFIG_CC_HAS_KASAN_SW_TAGS=y CONFIG_KASAN=y CONFIG_KASAN_HW_TAGS=y CONFIG_KASAN_VMALLOC=y # CONFIG_KASAN_GENERIC is not set # CONFIG_KASAN_SW_TAGS is not set