Kernel crash: Fatal trap 12: page fault while in kernel mode

[melroy89]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

Kernel crash.

To Reproduce

I have no idea. I don't do anything special to be honest. OPNsense just keep crashing at random, this one was around 4am.

Expected behavior

No random kernel crashes.

Describe alternatives you considered

Move away from FreeBSD.

Relevant log files

dmesg.boot:

---<<BOOT>>---
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.1-RELEASE-p6 stable/24.7-n267939-fd5bc7f34e1 SMP amd64
FreeBSD clang version 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15a9eac9)
VT(vga): resolution 640x480
CPU: AMD EPYC 3201 8-Core Processor                  (1497.26-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x800f12  Family=0x17  Model=0x1  Stepping=2
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x7ed8320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
  AMD Features2=0x35c233ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,SKINIT,WDT,TCE,Topology,PCXC,PNXC,DBE,PL2I,MWAITX>
  Structured Extended Features=0x209c01a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  AMD Extended Feature Extensions ID EBX=0x1007<CLZERO,IRPerf,XSaveErPtr,IBPB>
  SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16452407296 (15690 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INSYDE WALLABY>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 2 cache groups x 4 core(s)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0: MADT APIC ID 128 != hw id 0
ioapic1: MADT APIC ID 129 != hw id 0
ioapic0 <Version 2.1> irqs 0-23
ioapic1 <Version 2.1> irqs 24-55
Launching APs: 6 4 7 1 2 3 5
random: entropy device external interface
wlan: mac acl policy registered
kbd0 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 15.0.
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
vtvga0: <VT VGA driver>
smbios0: <System Management BIOS> at iomem 0x793d2000-0x793d201e
smbios0: Version: 3.0, BCD Revision: 3.0
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256>
acpi0: <INSYDE WALLABY>
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 350
Event timer "HPET2" frequency 14318180 Hz quality 350
atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
apei0: <ACPI Platform Error Interface> on acpi0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <base peripheral, IOMMU> at device 0.2 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 1.3 on pci0
pci1: <ACPI PCI bus> on pcib1
nvme0: <Generic NVMe Device> mem 0x80d00000-0x80d03fff at device 0.0 on pci1
pcib2: <ACPI PCI-PCI bridge> at device 1.4 on pci0
pci2: <ACPI PCI bus> on pcib2
igc0: <Intel(R) Ethernet Controller I226-V> mem 0x80b00000-0x80bfffff,0x80c00000-0x80c03fff at device 0.0 on pci2
igc0: EEPROM V2.25-0 eTrack 0x800003b1
igc0: Using 1024 TX descriptors and 1024 RX descriptors
igc0: Using 4 RX queues 4 TX queues
igc0: Using MSI-X interrupts with 5 vectors
igc0: Ethernet address: f4:90:ea:01:2b:e0
igc0: netmap queues/slots: TX 4/1024, RX 4/1024
pcib3: <ACPI PCI-PCI bridge> at device 1.5 on pci0
pci3: <ACPI PCI bus> on pcib3
igc1: <Intel(R) Ethernet Controller I226-V> mem 0x80900000-0x809fffff,0x80a00000-0x80a03fff at device 0.0 on pci3
igc1: EEPROM V2.25-0 eTrack 0x800003b1
igc1: Using 1024 TX descriptors and 1024 RX descriptors
igc1: Using 4 RX queues 4 TX queues
igc1: Using MSI-X interrupts with 5 vectors
igc1: Ethernet address: f4:90:ea:01:2b:e1
igc1: netmap queues/slots: TX 4/1024, RX 4/1024
pcib4: <ACPI PCI-PCI bridge> at device 1.6 on pci0
pci4: <ACPI PCI bus> on pcib4
igc2: <Intel(R) Ethernet Controller I226-V> mem 0x80700000-0x807fffff,0x80800000-0x80803fff at device 0.0 on pci4
igc2: EEPROM V2.25-0 eTrack 0x800003b1
igc2: Using 1024 TX descriptors and 1024 RX descriptors
igc2: Using 4 RX queues 4 TX queues
igc2: Using MSI-X interrupts with 5 vectors
igc2: Ethernet address: f4:90:ea:01:2b:e2
igc2: netmap queues/slots: TX 4/1024, RX 4/1024
pcib5: <ACPI PCI-PCI bridge> at device 1.7 on pci0
pci5: <ACPI PCI bus> on pcib5
igc3: <Intel(R) Ethernet Controller I226-V> mem 0x80500000-0x805fffff,0x80600000-0x80603fff at device 0.0 on pci5
igc3: EEPROM V2.25-0 eTrack 0x800003b1
igc3: Using 1024 TX descriptors and 1024 RX descriptors
igc3: Using 4 RX queues 4 TX queues
igc3: Using MSI-X interrupts with 5 vectors
igc3: Ethernet address: f4:90:ea:01:2b:e3
igc3: netmap queues/slots: TX 4/1024, RX 4/1024
pcib6: <ACPI PCI-PCI bridge> at device 7.1 on pci0
pci6: <ACPI PCI bus> on pcib6
pci6: <encrypt/decrypt> at device 0.2 (no driver attached)
xhci0: <XHCI (generic) USB 3.0 controller> mem 0x80200000-0x802fffff at device 0.3 on pci6
xhci0: 64 bytes context size, 64-bit DMA
usbus0: waiting for BIOS to give up control
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pcib7: <ACPI PCI-PCI bridge> at device 8.1 on pci0
pci7: <ACPI PCI bus> on pcib7
pci7: <encrypt/decrypt> at device 0.1 (no driver attached)
ax0: <AMD 10 Gigabit Ethernet Driver> mem 0x80160000-0x8017ffff,0x80140000-0x8015ffff,0x80180000-0x80181fff at device 0.4 on pci7
ax0: Using 512 TX descriptors and 512 RX descriptors
ax0: Using 8 RX queues 8 TX queues
ax0: Using MSI-X interrupts with 12 vectors
ax0: Ethernet address: f4:90:ea:01:2b:e4
ax0: xgbe_config_sph_mode: SPH disabled in channel 0
ax0: xgbe_config_sph_mode: SPH disabled in channel 1
ax0: xgbe_config_sph_mode: SPH disabled in channel 2
ax0: xgbe_config_sph_mode: SPH disabled in channel 3
ax0: xgbe_config_sph_mode: SPH disabled in channel 4
ax0: xgbe_config_sph_mode: SPH disabled in channel 5
ax0: xgbe_config_sph_mode: SPH disabled in channel 6
ax0: xgbe_config_sph_mode: SPH disabled in channel 7
ax0: RSS Enabled
ax0: Receive checksum offload Enabled
ax0: VLAN filtering Enabled
ax0: VLAN Stripping Enabled
ax0: Checking GPIO expander validity
ax0: GPIO configuration valid
ax0: xgbe_phy_sfp_signals: port_sfp_inputs: 0x4
ax0: SFP detected:
ax0:   vendor:	 Zaram           
ax0:   part number:    ZXOS11NPI       
ax0:   revision level: 1B  
ax0:   serial number:  ZRMT23060637    
ax0: netmap queues/slots: TX 8/512, RX 8/512
ax1: <AMD 10 Gigabit Ethernet Driver> mem 0x80120000-0x8013ffff,0x80100000-0x8011ffff,0x80182000-0x80183fff at device 0.5 on pci7
ax1: Using 512 TX descriptors and 512 RX descriptors
ax1: Using 8 RX queues 8 TX queues
ax1: Using MSI-X interrupts with 12 vectors
ax1: Ethernet address: f4:90:ea:01:2b:e5
ax1: xgbe_config_sph_mode: SPH disabled in channel 0
ax1: xgbe_config_sph_mode: SPH disabled in channel 1
ax1: xgbe_config_sph_mode: SPH disabled in channel 2
ax1: xgbe_config_sph_mode: SPH disabled in channel 3
ax1: xgbe_config_sph_mode: SPH disabled in channel 4
ax1: xgbe_config_sph_mode: SPH disabled in channel 5
ax1: xgbe_config_sph_mode: SPH disabled in channel 6
ax1: xgbe_config_sph_mode: SPH disabled in channel 7
ax1: RSS Enabled
ax1: Receive checksum offload Enabled
ax1: VLAN filtering Enabled
ax1: VLAN Stripping Enabled
ax1: Checking GPIO expander validity
ax1: GPIO configuration valid
ax1: SFP detected:
ax1:   vendor:	 FS              
ax1:   part number:    SFPP-PC005      
ax1:   revision level: A   
ax1:   serial number:  G2340637796-2   
ax1: netmap queues/slots: TX 8/512, RX 8/512
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
uart2: <16x50 with 256 byte FIFO> iomem 0xfedc9000-0xfedc9fff,0xfedc7000-0xfedc7fff irq 3 on acpi0
ns8250: UART FCR is broken
uart2: console (115384,n,8,1)
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Timecounter "TSC" frequency 1497186964 Hz quality 1000
Timecounters tick every 1.000 msec
ugen0.1: <AMD XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <AMD XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ax1: Link is UP - 10Gbps/Full - flow control off
ax1: link state changed to UP
ax0: Link is UP - 10Gbps/Full - flow control off
ax0: link state changed to UP
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
nda0 at nvme0 bus 0 scbus0 target 0 lun 1
nda0: <TS256GMTE712A 82B2W2AA I689870120>
nda0: Serial Number I689870120
nda0: nvme version 1.4
nda0: 244198MB (500118192 512 byte sectors)
Trying to mount root from zfs:zroot/ROOT/default []...
uhub0: 8 ports with 8 removable, self powered
pid 30 (zpool) is attempting to use unsafe AIO requests - not logging anymore

/var/crash/info.0:

Dump header from device: /dev/gpt/swapfs
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 155648
  Blocksize: 512
  Compression: none
  Dumptime: 2025-01-27 04:05:57 +0100
  Hostname: OPNsense.home
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 14.1-RELEASE-p6 stable/24.7-n267939-fd5bc7f34e1 SMP
  Panic String: page fault
  Dump Parity: 1435364609
  Bounds: 0
  Dump Status: good

/var/crash/textdump.tar.0:

ddb.txt06000014000014745574025  7107 ustarrootwheeldb:0:kdb.enter.default>  run lockinfo
db:1:lockinfo> show locks
No such command; use "help" to list available commands
db:1:lockinfo>  show alllocks
No such command; use "help" to list available commands
db:1:lockinfo>  show lockedvnods
Locked vnodes
db:0:kdb.enter.default>  show pcpu
cpuid        = 0
dynamic pcpu = 0x12fc080
curthread    = 0xfffff80001959740: pid 11 tid 100003 critnest 3 "idle: cpu0"
curpcb       = 0xfffff80001959c60
fpcurthread  = none
idlethread   = 0xfffff80001959740: tid 100003 "idle: cpu0"
self         = 0xffffffff82e10000
curpmap      = 0xffffffff81b81670
tssp         = 0xffffffff82e10384
rsp0         = 0xfffffe001d71b000
kcr3         = 0xffffffffffffffff
ucr3         = 0xffffffffffffffff
scr3         = 0x0
gs32p        = 0xffffffff82e10404
ldt          = 0xffffffff82e10444
tss          = 0xffffffff82e10434
curvnet      = 0
db:0:kdb.enter.default>  bt
Tracing pid 11 tid 100003 td 0xfffff80001959740
kdb_enter() at kdb_enter+0x33/frame 0xfffffe001d71a8c0
panic() at panic+0x43/frame 0xfffffe001d71a920
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe001d71a980
trap_pfault() at trap_pfault+0x46/frame 0xfffffe001d71a9d0
calltrap() at calltrap+0x8/frame 0xfffffe001d71a9d0
--- trap 0xc, rip = 0xffffffff80c13be7, rsp = 0xfffffe001d71aaa0, rbp = 0xfffffe001d71aac0 ---
sleepq_remove_thread() at sleepq_remove_thread+0x67/frame 0xfffffe001d71aac0
sleepq_resume_thread() at sleepq_resume_thread+0x45/frame 0xfffffe001d71ab00
sleepq_timeout() at sleepq_timeout+0x94/frame 0xfffffe001d71ab20
softclock_call_cc() at softclock_call_cc+0x12c/frame 0xfffffe001d71abd0
callout_process() at callout_process+0x1e7/frame 0xfffffe001d71ac40
handleevents() at handleevents+0x180/frame 0xfffffe001d71ac80
timercb() at timercb+0x24c/frame 0xfffffe001d71acd0
lapic_handle_timer() at lapic_handle_timer+0xab/frame 0xfffffe001d71acf0
Xtimerint() at Xtimerint+0xb1/frame 0xfffffe001d71acf0
--- interrupt, rip = 0xffffffff81065b02, rsp = 0xfffffe001d71adc0, rbp = 0xfffffe001d71add0 ---
cpu_idle_hlt() at cpu_idle_hlt+0x32/frame 0xfffffe001d71add0
cpu_idle() at cpu_idle+0x9d/frame 0xfffffe001d71adf0
sched_idletd() at sched_idletd+0x576/frame 0xfffffe001d71aef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe001d71af30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001d71af30
--- trap 0x93d3a881, rip = 0x1b94a12917d4a12d, rsp = 0xbca33a65b0e33a61, rbp = 0x9c164b0cdc164f ---
db:0:kdb.enter.default>  ps
  pid  ppid  pgrp   uid  state   wmesg   wchan               cmd
 8190  6512  6512     0  S       accept  0xfffff8003b4821d0  php-cgi
98723     1 98723     0  Ss      (threaded)                  suricata
101832                   S       nanslp  0xffffffff81bc48e0  suricata
302928                   S       select  0xfffff803015a6bc0  W#01-ax1
302963                   S       select  0xfffff802d5149840  W#02-ax1
302964                   S       select  0xfffff802dc7f5d40  W#03-ax1
302965                   S       select  0xfffff8042d073dc0  W#04-ax1
302966                   S       select  0xfffff800142e75c0  W#05-ax1
302967                   S       select  0xfffff8023a1e6540  W#06-ax1
302968                   S       select  0xfffff803011201c0  W#07-ax1
302969                   S       select  0xfffff803e633cc40  W#08-ax1

[....]

<7>cannot forward src fe80:6::fccd:ef81:dd7b:6ece, dst 2620:2d:4002:1::196, nxt 6, rcvif ax1, outif pppoe0
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address	= 0xd00028
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80c13be7
stack pointer	        = 0x28:0xfffffe001d71aaa0
frame pointer	        = 0x28:0xfffffe001d71aac0
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= resume, IOPL = 0
current process		= 11 (idle: cpu0)
rdi: fffff80001e58380 rsi: fffff80001e28000 rdx: 0000000000000000
rcx: 0000000000000000  r8: fffff80001e28478  r9: 000000009bafc01b
rax: 0000000000d00000 rbx: fffff80001e28000 rbp: fffffe001d71aac0
r10: 00000000000007d0 r11: 0000000000002710 r12: fffff80001e58380
r13: 00000000819c9958 r14: 00000000819c9958 r15: 0000000000000000
trap number		= 12
panic: page fault
cpuid = 0
time = 1737947157
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe001d71a790
vpanic() at vpanic+0x131/frame 0xfffffe001d71a8c0
panic() at panic+0x43/frame 0xfffffe001d71a920
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe001d71a980
trap_pfault() at trap_pfault+0x46/frame 0xfffffe001d71a9d0
calltrap() at calltrap+0x8/frame 0xfffffe001d71a9d0
--- trap 0xc, rip = 0xffffffff80c13be7, rsp = 0xfffffe001d71aaa0, rbp = 0xfffffe001d71aac0 ---
sleepq_remove_thread() at sleepq_remove_thread+0x67/frame 0xfffffe001d71aac0
sleepq_resume_thread() at sleepq_resume_thread+0x45/frame 0xfffffe001d71ab00
sleepq_timeout() at sleepq_timeout+0x94/frame 0xfffffe001d71ab20
softclock_call_cc() at softclock_call_cc+0x12c/frame 0xfffffe001d71abd0
callout_process() at callout_process+0x1e7/frame 0xfffffe001d71ac40
handleevents() at handleevents+0x180/frame 0xfffffe001d71ac80
timercb() at timercb+0x24c/frame 0xfffffe001d71acd0
lapic_handle_timer() at lapic_handle_timer+0xab/frame 0xfffffe001d71acf0
Xtimerint() at Xtimerint+0xb1/frame 0xfffffe001d71acf0
--- interrupt, rip = 0xffffffff81065b02, rsp = 0xfffffe001d71adc0, rbp = 0xfffffe001d71add0 ---
cpu_idle_hlt() at cpu_idle_hlt+0x32/frame 0xfffffe001d71add0
cpu_idle() at cpu_idle+0x9d/frame 0xfffffe001d71adf0
sched_idletd() at sched_idletd+0x576/frame 0xfffffe001d71aef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe001d71af30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001d71af30
--- trap 0x93d3a881, rip = 0x1b94a12917d4a12d, rsp = 0xbca33a65b0e33a61, rbp = 0x9c164b0cdc164f ---
KDB: enter: panic
panic.txt0600001214745574025  7146 ustarrootwheelpage faultversion.txt0600007414745574025  7551 ustarrootwheelFreeBSD 14.1-RELEASE-p6 stable/24.7-n267939-fd5bc7f34e1 SMP

Environment

OPNsense 24.10.1-amd64
Official Deciso DEC3852 hardware

[AdSchellevis]

If your crashes are rather random (which at a first glance is the case), there are a couple of things to look at:

• check memory using memtest (efi payload can be downloaded from https://www.memtest86.com/download.htm)
• check machine grounding, there's a ground pin at the back of the machine, floating power rails can cause hard to find issues
• try a clean install to rule out file system issues

[melroy89]

These are good starting points, thank you Ad.

• I will do a memtest
• Machine is already grounded to earth fully. Grounded to PDU. Zie hier hoe
• I will do a reinstall if memtest is showing no errors

EDIT: Here is the system health check results:

***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 24.10.1 (amd64) at Tue Jan 28 20:56:01 CET 2025
Strict TLS 1.3 and CRL checking is enabled.
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 24.7.8 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 24.7.8 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
OPNsense (Priority: 11)
>>> Check installed plugins
os-OPNBEcore 1.4_3
os-iperf 1.0_2
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense-business" at 24.10.1 has 70 dependencies to check.
Checking packages: ....................................................................... done
***DONE***