Our new ER-8 is panicing and rebooting randomly with a new configuration on it.
We've tried:
- Soft/Hard Reset
- Re-install v1.8.5 firmware
We add a minimal configuration and nothing actually connected to the ports and the reboots still occur. Here's the console output:
ubnt@ubnt:~$ INFO: rcu_bh detected stalls on CPUs/tasks:
(detected by 1, t=4295199011 jiffies, g=18446744073709289172, c=18446744073709551316, q=2)
INFO: Stall ended before state dump start
Kernel bug detected[#1]:
CPU: 0 PID: 5886 Comm: ubnt-cfgd Tainted: P O 3.10.20-UBNT #1
task: 800000008c337400 ti: 800000008c130000 task.ti: 800000008c130000
$ 0 : 0000000000000000 0000000010109ce1 ffffffffffffffff 800000008c510000
$ 4 : 800000008c024850 800000008c01b988 00000000ffffff01 00000000ffffff02
$ 8 : 800000008c01b998 800000008c01b9a8 0000000000000001 000000000000000f
$12 : 000000000000000f 0000000000000000 0000000000000001 800000008c510000
$16 : 80000000895c4000 800000008c024800 800000008c01b988 800000008c01c380
$20 : 00000000000080d0 0000000000000000 0000000000000033 ffffffffc0590000
$24 : fffffffffffbffff 0000000000000008
$28 : 800000008c130000 800000008c133bc0 800000008c01b980 0000000000000010
Hi : 0000000000000003
Lo : 000000003ce678df
epc : ffffffffc016e47c cache_alloc_refill+0x15c/0xa38
Tainted: P O
ra : 0000000000000010 0x10
Status: 10109ce2 KX SX UX KERNEL EXL
Cause : 00800034
PrId : 000d9301 (Cavium Octeon II)
Modules linked in: xt_conntrack ipt_MASQUERADE xt_comment 8021q garp xt_set bridge stp llc ip6table_mangle ip6table_filter ip6table_raw ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_mangle xt_CT iptable_raw nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_h323 nf_conntrack_h323 nf_nat_sip nf_conntrack_sip nf_nat_proto_gre nf_nat_tftp nf_nat_ftp nf_nat nf_conntrack_tftp nf_conntrack_ftp nf_conntrack ip_set_hash_net ip_set nfnetlink iptable_filter ip_tables x_tables cvm_ipsec_kame(O) ipv6 imq cavium_ip_offload(PO) ubnt_nf_app(PO) tdts(PO) octeon_rng rng_core octeon_ethernet mdio_octeon of_mdio ethernet_mem octeon_common ubnt_platform(PO) libphy
Process ubnt-cfgd (pid: 5886, threadinfo=800000008c130000, task=800000008c337400, tls=00000000770b5490)
Stack : 800000008c133ca0 800000008c133ca0 800000008c133ca0 0000000000000000
0000000000000041 8000000089443020 800000008c133e00 0000000000000001
800000008c01c380 00000000000080d0 ffffffffc05c8b00 ffffffffffffff9c
00000000004e0cc0 000000007f7c1320 0000000000000000 ffffffffc016e2f8
0000000000000001 800000008c133e00 800000008d714a00 ffffffffc0590000
ffffffffc058cb00 ffffffffc0178ef4 8000000089443000 ffffffffc016f084
800000008c133e00 800000008c133d40 800000008c133e00 0000000000000041
8000000089443000 ffffffffc018aaf4 00000008f399e37f 8000000089443086
0000000000000000 800000008a022c80 0000000000000000 0000000000000002
0000000000000000 80000000897e1668 800000008c133e00 0000000000000001
...
Call Trace:
[<ffffffffc016e47c>] cache_alloc_refill+0x15c/0xa38
[<ffffffffc016e2f8>] kmem_cache_alloc+0xe8/0x110
[<ffffffffc0178ef4>] get_empty_filp+0x6c/0x198
[<ffffffffc018aaf4>] path_openat+0x4c/0x4c8
[<ffffffffc018b818>] do_filp_open+0x38/0xa8
[<ffffffffc0175ae8>] do_sys_open+0x100/0x1e0
[<ffffffffc008c4a0>] handle_sys+0x120/0x144
Code: 8e6c0018 016c682b 2dae0001 <000e0336> 11a00064 26cfffff 12c000de 01e0902d 8e030024
---[ end trace 5716fc97bcafdf8d ]---
[sched_delayed] sched: RT throttling activated
Fatal exception: panic in 5 seconds
*** NMI Watchdog interrupt on Core 0x00 ***
$0 0x0000000000000000 at 0x0000000010109ce1
v0 0x0000000000010000 v1 0x0000000000000016
a0 0x800000008c01b980 a1 0x0000000000002468
a2 0x0000000000000000 a3 0xffffffffc0178970
a4 0xffffffffc061c3c8 a5 0x800000008d7c8780
a6 0x0000000000000001 a7 0x8000000002c703c8
t0 0x0000000000000002 t1 0x0000000000000003
t2 0x0000000000000003 t3 0x800000008d7c3180
s0 0x800000008c024800 s1 0x000000000000003c
s2 0x0000000000000001 s3 0x800000008c01b980
s4 0x800000008c01c380 s5 0x800000008dd36880
s6 0x0000000000000003 s7 0xffffffffc04b0000
t8 0x0000000000000000 t9 0x0000000000000010
k0 0x0000000000000000 k1 0x0000000000000000
gp 0x800000008965c000 sp 0x800000008965f930
s8 0x8000000002c6fbc0 ra 0xffffffffc0499bc4
err_epc 0xffffffffc049f328 epc 0xffffffffc00f7bd8
status 0x0000000010589ce4 cause 0x0000000040808c00
sum0 0x0000000000008000 en0 0x0900200500008000
*** Chip soft reset soon ***
*** NMI Watchdog interrupt on Core 0x01 ***
$0 0x0000000000000000 at 0x0000000010109ce1
v0 0x0000000000010000 v1 0x0000000000000000
a0 0x800000008c01b980 a1 0x0000000000002467
a2 0x0000000000000000 a3 0x0000000000000001
a4 0x0000000000000001 a5 0x0000000000000070
a6 0xffffffffc0590000 a7 0x0000000000000001
t0 0x800000008c01c410 t1 0x000000000000003c
t2 0x0000000000000011 t3 0x000000000000003c
s0 0x800000008c667e00 s1 0x800000008c0bec00
s2 0x800000008c01c380 s3 0x800000008c01c380
s4 0x00000000000080d0 s5 0x0000000000000000
s6 0x000000000000003c s7 0x0000000000000000
t8 0x0000000000000000 t9 0x0000000000000001
k0 0x000000007f774d78 k1 0x800000008c667fe0
gp 0x800000008c664000 sp 0x800000008c667bc0
s8 0x800000008c01b980 ra 0xffffffffc016e3ac
err_epc 0xffffffffc049f320 epc 0x0000000077802904
status 0x0000000010589ce4 cause 0x0000000040808c20
sum0 0x0000000100008000 en0 0x0000000100000000
*** Chip soft reset soon ***
Jumping to start of image at address 0xbfca0000
Once it starts back up, portions of the config are usually missing. Example could be a missing eth interface. following the crash we get this line:
INIT: Entering runlevel: 2 [ ok ] Starting routing daemon: rib nsm ribd. [FAIL] Starting EdgeOS router: migrate rl-system configure failed! Welcome to EdgeOS cm-er01 ttyS0
Is the unit just defective?
Thanks.