This morning my ER-X SFP was not working. It wouldn't give out addresses via DHCP and even if you set a static IP it would take 10ms to respond to ping (usually 0.2ms).
After rebooting the log showed events this morning at 12:25 as though I had just done a firmware upgrade:
Jul 6 00:25:13 router rsyslogd: set SCM_CREDENTIALS failed on '/dev/log': Protocol not available Jul 6 00:25:13 router kernel: Linux version 3.10.14-UBNT (root@edgeos-builder2) (gcc version 4.6.3 (Buildroot 2012.11.1) ) #1 SMP Sat May 28 09:48:10 PDT 2016 Jul 6 00:25:13 router kernel: Jul 6 00:25:13 router kernel: The CPU feqenuce set to 880 MHz Jul 6 00:25:13 router kernel: GCMP present Jul 6 00:25:13 router kernel: Zone ranges: Jul 6 00:25:13 router kernel: Normal [mem 0x00000000-0x0fffffff] Jul 6 00:25:13 router kernel: Movable zone start for each node Jul 6 00:25:13 router kernel: Early memory node ranges Jul 6 00:25:13 router kernel: node 0: [mem 0x00000000-0x0fffffff] Jul 6 00:25:13 router kernel: Primary instruction cache 32kB, 4-way, VIPT, linesize 32 bytes. Jul 6 00:25:13 router kernel: Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes Jul 6 00:25:13 router kernel: MIPS secondary cache 256kB, 8-way, linesize 32 bytes. Jul 6 00:25:13 router kernel: Built 1 zonelists in Zone order, mobility grouping on. Total pages: 65024 Jul 6 00:25:13 router kernel: Kernel command line: console=ttyS1,57600n8 ubi.mtd=7 root=ubi0_0 rootfstype=ubifs rootsqimg=squashfs.img rootsqwdir=w rw Jul 6 00:25:13 router kernel: launch: starting cpu1 Jul 6 00:25:13 router kernel: launch: cpu1 gone! Jul 6 00:25:13 router kernel: Primary instruction cache 32kB, 4-way, VIPT, linesize 32 bytes. Jul 6 00:25:13 router kernel: Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes Jul 6 00:25:13 router kernel: MIPS secondary cache 256kB, 8-way, linesize 32 bytes. Jul 6 00:25:13 router kernel: launch: starting cpu2 Jul 6 00:25:13 router kernel: launch: cpu2 gone! Jul 6 00:25:13 router kernel: Primary instruction cache 32kB, 4-way, VIPT, linesize 32 bytes. Jul 6 00:25:13 router kernel: Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes Jul 6 00:25:13 router kernel: MIPS secondary cache 256kB, 8-way, linesize 32 bytes. Jul 6 00:25:13 router kernel: launch: starting cpu3 Jul 6 00:25:13 router kernel: launch: cpu3 gone! Jul 6 00:25:13 router kernel: Primary instruction cache 32kB, 4-way, VIPT, linesize 32 bytes. Jul 6 00:25:13 router kernel: Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes Jul 6 00:25:13 router kernel: MIPS secondary cache 256kB, 8-way, linesize 32 bytes. Jul 6 00:25:13 router kernel: 4 CPUs re-calibrate udelay(lpj = 1167360) Jul 6 00:25:13 router kernel: Ralink gpio driver initialized Jul 6 00:25:13 router kernel: i2cdrv_major = 218 Jul 6 00:25:13 router kernel: flash manufacture id: c2, device id 20 15 Jul 6 00:25:13 router kernel: MX25L1605D(c2 2015c220) (2048 Kbytes) Jul 6 00:25:13 router kernel: mtd .name = raspi, .size = 0x00200000 (2M) .erasesize = 0x00010000 (64K) .numeraseregions = 0 Jul 6 00:25:13 router kernel: Creating 1 MTD partitions on "raspi": Jul 6 00:25:13 router kernel: 0x000000000000-0x000000080000 : "SPI_FLASH" Jul 6 00:25:13 router kernel: MediaTek Nand driver init, version v2.1 Fix AHB virt2phys error Jul 6 00:25:13 router kernel: Enable NFI Clock Jul 6 00:25:13 router kernel: # MTK NAND # : Use HW ECC Jul 6 00:25:13 router kernel: NAND ID [01 DA 90 95 46, 00909546] Jul 6 00:25:13 router kernel: Support this Device in MTK table! 1da Jul 6 00:25:13 router kernel: [NAND]select ecc bit:12, sparesize :112 spare_per_sector=28 Jul 6 00:25:13 router kernel: Signature matched and data read! Jul 6 00:25:13 router kernel: load_fact_bbt success 2047 Jul 6 00:25:13 router kernel: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jul 6 00:25:13 router kernel: last message repeated 15 times Jul 6 00:25:13 router kernel: Creating 7 MTD partitions on "MT7621-NAND": Jul 6 00:25:13 router kernel: 0x000000000000-0x00000ff80000 : "ALL" Jul 6 00:25:13 router kernel: 0x000000000000-0x000000080000 : "Bootloader" Jul 6 00:25:13 router kernel: 0x000000080000-0x0000000e0000 : "Config" Jul 6 00:25:13 router kernel: 0x0000000e0000-0x000000140000 : "eeprom" Jul 6 00:25:13 router kernel: 0x000000140000-0x000000440000 : "Kernel" Jul 6 00:25:13 router kernel: 0x000000440000-0x000000740000 : "Kernel2" Jul 6 00:25:13 router kernel: 0x000000740000-0x00000ff00000 : "RootFS" Jul 6 00:25:13 router kernel: [mtk_nand] probe successfully! Jul 6 00:25:13 router kernel: UBNT BD mac 44d9e7501b03 kidx 1 mrev 12 serial 44D9E7501B03 type e51 Jul 6 00:25:13 router kernel: rdm_major = 253 Jul 6 00:25:13 router kernel: GMAC1_MAC_ADRH -- : 0x000044d9 Jul 6 00:25:13 router kernel: GMAC1_MAC_ADRL -- : 0xe7501b03 Jul 6 00:25:13 router kernel: Ralink APSoC Ethernet Driver Initilization. v3.1 512 rx/tx descriptors allocated, mtu = 1500! Jul 6 00:25:13 router kernel: GMAC1_MAC_ADRH -- : 0x000044d9 Jul 6 00:25:13 router kernel: GMAC1_MAC_ADRL -- : 0xe7501b03 Jul 6 00:25:13 router kernel: PROC INIT OK! Jul 6 00:25:13 router kernel: Ralink I2C Init Jul 6 00:25:13 router kernel: UBI: attaching mtd7 to ubi0 Jul 6 00:25:13 router kernel: UBI: scanning is finished Jul 6 00:25:13 router kernel: UBI: attached mtd7 (name "RootFS", size 247 MiB) to ubi0 Jul 6 00:25:13 router kernel: UBI: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes Jul 6 00:25:13 router kernel: UBI: min./max. I/O unit sizes: 2048/2048, sub-page size 2048 Jul 6 00:25:13 router kernel: UBI: VID header offset: 2048 (aligned 2048), data offset: 4096 Jul 6 00:25:13 router kernel: UBI: good PEBs: 1982, bad PEBs: 0, corrupted PEBs: 0 Jul 6 00:25:13 router kernel: UBI: user volume: 1, internal volumes: 1, max. volumes count: 128 Jul 6 00:25:13 router kernel: UBI: max/mean erase counter: 5/3, WL threshold: 4096, image sequence number: 1593827417 Jul 6 00:25:13 router kernel: UBI: available PEBs: 0, total reserved PEBs: 1982, PEBs reserved for bad PEB handling: 40 Jul 6 00:25:13 router kernel: UBI: background thread "ubi_bgt0d" started, PID 51 Jul 6 00:25:13 router kernel: UBIFS: background thread "ubifs_bgt0_0" started, PID 52 Jul 6 00:25:13 router kernel: UBIFS: recovery needed Jul 6 00:25:13 router kernel: UBIFS: recovery completed Jul 6 00:25:13 router kernel: UBIFS: mounted UBI device 0, volume 0, name "troot" Jul 6 00:25:13 router kernel: UBIFS: LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes Jul 6 00:25:13 router kernel: UBIFS: FS size: 244428800 bytes (233 MiB, 1925 LEBs), journal size 12189696 bytes (11 MiB, 96 LEBs) Jul 6 00:25:13 router kernel: UBIFS: reserved for root: 4952683 bytes (4836 KiB) Jul 6 00:25:13 router kernel: UBIFS: media format: w4/r0 (latest is w4/r0), UUID 60F1AA5C-2A5A-4A19-996F-F794D5BCAE9C, small LPT model Jul 6 00:25:13 router kernel: Algorithmics/MIPS FPU Emulator v1.5 Jul 6 00:25:13 router kernel: ubnt_platform: module license 'Proprietary' taints kernel. Jul 6 00:25:13 router kernel: Disabling lock debugging due to kernel taint Jul 6 00:25:13 router kernel: Registering char device flash0 (200) succeeds Jul 6 00:25:13 router kernel: Raeth v3.1 (Tasklet) Jul 6 00:25:13 router kernel: Jul 6 00:25:13 router kernel: phy_tx_ring = 0x0ffe8000, tx_ring = 0xaffe8000 Jul 6 00:25:13 router kernel: Jul 6 00:25:13 router kernel: phy_rx_ring0 = 0x0ffea000, rx_ring0 = 0xaffea000 Jul 6 00:25:13 router kernel: change HW-TRAP to 0x17c8f Jul 6 00:25:13 router kernel: GMAC1_MAC_ADRH -- : 0x000044d9 Jul 6 00:25:13 router kernel: GMAC1_MAC_ADRL -- : 0xe7501b03 Jul 6 00:25:13 router kernel: eth0: ===> VirtualIF_open Jul 6 00:25:13 router kernel: eth1: ===> VirtualIF_open Jul 6 00:25:13 router kernel: eth2: ===> VirtualIF_open Jul 6 00:25:13 router kernel: eth3: ===> VirtualIF_open Jul 6 00:25:13 router kernel: eth4: ===> VirtualIF_open Jul 6 00:25:13 router kernel: eth5: ===> VirtualIF_open Jul 6 00:25:13 router kernel: CDMA_CSG_CFG = 81000000 Jul 6 00:25:13 router kernel: GDMA1_FWD_CFG = 21710000 Jul 6 00:25:13 router kernel: ESW: Link Status Changed - Port1 Link UP Jul 6 00:25:13 router kernel: ESW: Link Status Changed - Port4 Link UP Jul 6 00:25:13 router kernel: ESW: Link Status Changed - Port2 Link UP Jul 6 00:25:14 router kernel: Type=Linux Jul 6 00:25:14 router NSM[355]: NSM-6: 1000 MB Jul 6 00:25:14 router NSM[355]: NSM-6: 10 MB Jul 6 00:25:14 router NSM[355]: NSM-6: 1000 MB Jul 6 00:25:14 router NSM[355]: NSM-6: 1000 MB Jul 6 00:25:14 router NSM[355]: NSM-6: 10 MB Jul 6 00:25:14 router NSM[355]: NSM-6: 1000 MB Jul 6 00:25:14 router NSM[355]: NSM-6: 1000 MB Jul 6 00:25:15 router NSM[355]: NSM-4: Could not create VRF table with identifier 1 in the MPLS Forwarder Jul 6 00:25:15 router RIB[365]: RIB-6: RIBd (1.2.0) starts Jul 6 00:25:15 router kernel: ip_set: protocol 6 Jul 6 00:25:15 router sudo: root : unable to resolve host router Jul 6 00:25:15 router sudo: unable to execute /usr/sbin/sendmail: No such file or directory Jul 6 00:25:15 router sudo: root : unable to resolve host router Jul 6 00:25:15 router sudo: unable to execute /usr/sbin/sendmail: No such file or directory Jul 6 00:25:17 router rl-system.init: Checking/creating SSH host keys. Jul 6 00:25:20 router IMI[350]: IMI-6: imi_server_send_config called (PM 1) Jul 6 00:25:20 router IMI[350]: IMI-6: imi_server_send_config called (PM 42) Jul 6 00:25:25 router sudo: root : unable to resolve host router Jul 6 00:25:25 router sudo: unable to execute /usr/sbin/sendmail: No such file or directory Jul 6 00:25:25 router sudo: root : unable to resolve host router Jul 6 00:25:25 router sudo: unable to execute /usr/sbin/sendmail: No such file or directory Jul 6 00:25:25 router rsyslogd: set SCM_CREDENTIALS failed on '/dev/log': Protocol not available Jul 6 00:25:26 router rsyslogd: set SCM_CREDENTIALS failed on '/dev/log': Protocol not available Jul 6 00:25:38 router kernel: eth3: ===> VirtualIF_open Jul 6 00:25:41 router kernel: eth0: ===> VirtualIF_open Jul 6 00:25:43 router kernel: eth2: ===> VirtualIF_open Jul 6 00:25:45 router kernel: eth1: ===> VirtualIF_open Jul 6 00:25:48 router kernel: eth4: ===> VirtualIF_open Jul 6 00:25:54 router kernel: ESW: Link Status Changed - Port3 Link UP Jul 6 00:25:54 router kernel: ESW: Link Status Changed - Port0 Link UP Jul 6 00:25:54 router NSM[355]: NSM-6: 1000 MB Jul 6 00:25:55 router wlb: intf-proto Config is locked (12 pending events in queue) Jul 6 00:25:56 router wlb: intf-proto Config is locked (12 pending events in queue) Jul 6 00:25:57 router kernel: ESW: Link Status Changed - Port0 Link Down Jul 6 00:25:59 router dhclient: can't create /var/db/dhclient6.leases: No such file or directory Jul 6 00:25:59 router dhclient: can't create /var/db/dhclient6.leases: No such file or directory Jul 6 00:25:59 router kernel: ESW: Link Status Changed - Port0 Link UP Jul 6 00:26:00 router wlb: intf-proto Config is locked (12 pending events in queue) Jul 6 00:26:03 router ntpd[1292]: ntpd 4.2.6p2@1.2194-o Sat Jan 30 00:56:04 UTC 2016 (1) Jul 6 00:26:03 router ntpd[1297]: proto: precision = 38.689 usec Jul 6 00:26:03 router ntpd[1297]: ntpd exiting on signal 15 Jul 6 00:26:06 router ntpd[1359]: ntpd 4.2.6p2@1.2194-o Sat Jan 30 00:56:04 UTC 2016 (1) Jul 6 00:26:06 router ntpd[1360]: proto: precision = 40.733 usec Jul 6 00:26:06 router ntpd[1360]: ntpd exiting on signal 15 Jul 6 00:26:06 router wlb: intf-proto Config is locked (12 pending events in queue) Jul 6 00:26:08 router ntpd[1430]: ntpd 4.2.6p2@1.2194-o Sat Jan 30 00:56:04 UTC 2016 (1) Jul 6 00:26:08 router ntpd[1431]: proto: precision = 43.129 usec Jul 6 00:26:09 router rsyslogd: set SCM_CREDENTIALS failed on '/dev/log': Protocol not available Jul 6 00:26:11 router wlb: intf-proto Config is locked (12 pending events in queue) Jul 6 00:26:12 router openvpn[1477]: OpenVPN 2.3.2 mipsel-unknown-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] [IPv6] built on Dec 5 2014 Jul 6 00:26:12 router openvpn[1477]: Static Encrypt: Cipher 'BF-CBC' initialized with 128 bit key Jul 6 00:26:12 router openvpn[1477]: Static Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication Jul 6 00:26:12 router openvpn[1477]: Static Decrypt: Cipher 'BF-CBC' initialized with 128 bit key Jul 6 00:26:12 router openvpn[1477]: Static Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication Jul 6 00:26:12 router openvpn[1477]: Socket Buffers: R=[180224->131072] S=[180224->131072] Jul 6 00:26:12 router openvpn[1477]: TUN/TAP device vtun2 opened Jul 6 00:26:12 router openvpn[1477]: TUN/TAP TX queue length set to 100 Jul 6 00:26:12 router openvpn[1477]: do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0 Jul 6 00:26:12 router openvpn[1477]: /sbin/ip link set dev vtun2 up mtu 1500 Jul 6 00:26:12 router openvpn[1477]: /sbin/ip addr add dev vtun2 local 10.99.99.5 peer 10.99.99.6 Jul 6 00:26:12 router openvpn[1480]: GID set to nogroup Jul 6 00:26:12 router openvpn[1480]: UID set to nobody Jul 6 00:26:12 router openvpn[1480]: UDPv4 link local (bound): [undef] Jul 6 00:26:12 router openvpn[1480]: UDPv4 link remote: [AF_INET]*removed*:1196 Jul 6 00:26:13 router openvpn[1507]: OpenVPN 2.3.2 mipsel-unknown-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] [IPv6] built on Dec 5 2014 Jul 6 00:26:13 router openvpn[1507]: Static Encrypt: Cipher 'BF-CBC' initialized with 128 bit key Jul 6 00:26:13 router openvpn[1507]: Static Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication Jul 6 00:26:13 router openvpn[1507]: Static Decrypt: Cipher 'BF-CBC' initialized with 128 bit key Jul 6 00:26:13 router openvpn[1507]: Static Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication Jul 6 00:26:13 router openvpn[1507]: Socket Buffers: R=[180224->131072] S=[180224->131072] Jul 6 00:26:14 router openvpn[1507]: TUN/TAP device vtun3 opened Jul 6 00:26:14 router openvpn[1507]: TUN/TAP TX queue length set to 100 Jul 6 00:26:14 router openvpn[1507]: do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0 Jul 6 00:26:14 router openvpn[1507]: /sbin/ip link set dev vtun3 up mtu 1500 Jul 6 00:26:14 router openvpn[1507]: /sbin/ip addr add dev vtun3 local 10.99.99.8 peer 10.99.99.7 Jul 6 00:26:14 router openvpn[1510]: GID set to nogroup Jul 6 00:26:14 router openvpn[1510]: UID set to nobody Jul 6 00:26:14 router openvpn[1510]: UDPv4 link local (bound): [undef] Jul 6 00:26:14 router openvpn[1510]: UDPv4 link remote: [AF_INET]*REMOVED*:1197
When I checked the two ER-X at the other end of the OpenVPN links, it looks like these events actually happened at 4:11 AM, not 12:26.
Any ideas where I should start looking for a cause? The logs were all deleted because the router seems to have gone through a firmware-upgrade type process. I checked though and it doesn't look like an upgrade was done. The system image shows the proper 1.8.5 current image and the release candidate I was using before as the backup:
poisonsnak@router:~$ show system image The system currently has the following image(s) installed: v1.8.5.4884695.160608.1057 (running image) (default boot) v1.8.5rc1.4882235.160530.2158
It's working fine after I power cycled it this morning, but if anyone has any ideas I would appreciate it.