Good afternoon to the community!
Hope everything is having a fab weekend thus far.
I have an issue that hopefully someone can help me with.
I have a EdgeRouter LTE EU - around 6 months old.
Its been running perfectly since it was installed with zero downtime and zero reboots.
Then, yesterday evening for no reason that I can see it just went offline and took out our datacenter server's and my wireless internet for my WISP.
Here is what I got from the log / console output.
<0D><0A>Looking for valid bootloader image….<0D><0A>Jumping to start of image at address 0xbfc80000<0D><0A><0D><0A><0D><0A>U-Boot 1.1.1 (UBNT Build ID: 4493936-g009d77b) (Build time: Sep 20 2012 - 15:48:51)<0D><0A><0D><0A>BIST check passed.<0D><0A>UBNT_E100 r1:2, r2:14, serial #: DC9FDB800FF0<0D><0A>Core clock: 500 MHz, DDR clock: 266 MHz (532 Mhz data rate)<0D><0A>DRAM: 512 MB<0D><0A>Clearing DRAM……. done<0D><0A>Flash: 4 MB<0D><0A>Net: octeth0, octeth1, octeth2<0D><0A><0D><0A>USB: (port 0) scanning bus for devices… 1 USB Devices found<0D><0A> scanning bus for storage devices…<0D><0A> Device 0: Vendor: Prod.: USB DISK 2.0 Rev: PMAP<0D><0A> Type: Removable Hard Disk<0D><0A> Capacity: 3700.6 MB = 3.6 GB (7579008 x 512)<0D><0A><08><08><08> 0 <0D><0A>reading vmlinux.64<0D><0A>…………………………….<0D><0A><0D><0A>6866056 bytes read<0D><0A>argv[2]: coremask=0x3<0D><0A>argv[3]: root=/dev/sda2<0D><0A>argv[4]: rootdelay=15<0D><0A>argv[5]: rw<0D><0A>argv[6]: rootsqimg=squashfs.img<0D><0A>argv[7]: rootsqwdir=w<0D><0A>argv[8]: mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@3072k(eeprom)<0D><0A>ELF file is 64 bit<0D><0A>Allocating memory for mapped kernel segment, alignment: 0x400000<0D><0A>Allocated memory for ELF segment: addr: 0x400000, size 0x7de180<0D><0A>Processing PHDR 0<0D><0A> Loading 68ae80 bytes at 400000<0D><0A> Clearing 153300 bytes at a8ae80<0D><0A>## Loading Linux kernel with entry point: 0x008e8130 …<0D><0A>Bootloader: Done loading app on coremask: 0x3<0D><0A>Linux version 3.10.20-UBNT (root@ubnt-builder2) (gcc version 4.7.0 (Cavium Inc. Version: SDK_3_1_0_p2 build 34) ) #1 SMP Fri Jan 29 20:11:31 PST 2016<0D><0A>CVMSEG size: 2 cache lines (256 bytes)<0D><0A>Cavium Inc. SDK-3.1<0D><0A>bootconsole [early0] enabled<0D><0A>CPU revision is: 000d0601 (Cavium Octeon+)<0D><0A>Checking for the multiply/shift bug… no.<0D><0A>Checking for the daddiu bug… no.<0D><0A>Determined physical RAM map:<0D><0A> memory: 0000000007400000 @ 0000000000c00000 (usable)<0D><0A> memory: 0000000007c00000 @ 0000000008200000 (usable)<0D><0A> memory: 000000000fc00000 @ 0000000410000000 (usable)<0D><0A> memory: 0000000000648000 @ 0000000000400000 (usable)<0D><0A> memory: 0000000000048000 @ 0000000000a48000 (usable after init)<0D><0A>Wasting 57344 bytes for tracking 1024 unused pages<0D><0A>software IO TLB [mem 0x01707000-0x01747000] (0MB) mapped at [8000000001707000-8000000001746fff]<0D><0A>Zone ranges:<0D><0A> DMA32 [mem 0x00400000-0xefffffff]<0D><0A> Normal [mem 0xf0000000-0x41fbfffff]<0D><0A>Movable zone start for each node<0D><0A>Early memory node ranges<0D><0A> node 0: [mem 0x00400000-0x00a8ffff]<0D><0A> node 0: [mem 0x00c00000-0x07ffffff]<0D><0A> node 0: [mem 0x08200000-0x0fdfffff]<0D><0A> node 0: [mem 0x410000000-0x41fbfffff]<0D><0A>Primary instruction cache 32kB, virtually tagged, 4 way, 64 sets, linesize 128 bytes.<0D><0A>Primary data cache 16kB, 64-way, 2 sets, linesize 128 bytes.<0D><0A>Secondary unified cache 128kB, 8-way, 128 sets, linesize 128 bytes.<0D><0A>PERCPU: Embedded 10 pages/cpu @8000000001784000 s11904 r8192 d20864 u40960<0D><0A>Built 1 zonelists in Zone order, mobility grouping on. Total pages: 125887<0D><0A>Kernel command line: bootoctlinux $loadaddr coremask=0x3 root=/dev/sda2 rootdelay=15 rw rootsqimg=squashfs.img rootsqwdir=w mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@3072k(eeprom) console=ttyS0,115200<0D><0A>PID hash table entries: 2048 (order: 2, 16384 bytes)<0D><0A>Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)<0D><0A>Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)<0D><0A>Memory: 495244k/510528k available (5074k kernel code, 15284k reserved, 1354k data, 288k init, 0k highmem)<0D><0A>Hierarchical RCU implementation.<0D><0A><09>Additional per-CPU info printed with stalls.<0D><0A>NR_IRQS:255<0D><0A>Calibrating delay loop (skipped) preset value.. 1000.00 BogoMIPS (lpj=5000000)<0D><0A>pid_max: default: 32768 minimum: 501<0D><0A>Security Framework initialized<0D><0A>Mount-cache hash table entries: 256<0D><0A>Checking for the daddi bug… no.<0D><0A>SMP: Booting CPU01 (CoreId 1)…<0D><0A>CPU revision is: 000d0601 (Cavium Octeon+)<0D><0A>Brought up 2 CPUs<0D><0A>NET: Registered protocol family 16<0D><0A>bio: create slab <bio-0> at 0<0D><0A>SCSI subsystem initialized<0D><0A>usbcore: registered new interface driver usbfs<0D><0A>usbcore: registered new interface driver hub<0D><0A>usbcore: registered new device driver usb<0D><0A>Switching to clocksource OCTEON_CVMCOUNT<0D><0A>NET: Registered protocol family 2<0D><0A>TCP established hash table entries: 4096 (order: 4, 65536 bytes)<0D><0A>TCP bind hash table entries: 4096 (order: 4, 65536 bytes)<0D><0A>TCP: Hash tables configured (established 4096 bind 4096)<0D><0A>TCP: reno registered<0D><0A>UDP hash table entries: 256 (order: 1, 8192 bytes)<0D><0A>UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)<0D><0A>NET: Registered protocol family 1<0D><0A>octeon_pci_console: Console not created.<0D><0A>/proc/octeon_perf: Octeon performance counter interface loaded<0D><0A>HugeTLB registered 2 MB page size, pre-allocated 0 pages<0D><0A>squashfs: version 4.0 (2009/01/31) Phillip Lougher<0D><0A>Registering unionfs 2.5.13 (for 3.10.34)<0D><0A>msgmni has been set to 967<0D><0A>io scheduler noop registered<0D><0A>io scheduler cfq registered (default)<0D><0A>Serial: 8250/16550 driver, 6 ports, IRQ sharing disabled<0D><0A>1180000000800.serial: ttyS0 at MMIO 0x1180000000800 (irq = 34) is a OCTEON<0D><0A>console [ttyS0] enabled, bootconsole disabled<0D><0A>console [ttyS0] enabled, bootconsole disabled<0D><0A>1180000000c00.serial: ttyS1 at MMIO 0x1180000000c00 (irq = 35) is a OCTEON<0D><0A>loop: module loaded<0D><0A>ehci_hcd: USB 2.0 ‘Enhanced’ Host Controller (EHCI) Driver<0D><0A>ohci_hcd: USB 1.1 ‘Open’ Host Controller (OHCI) Driver<0D><0A>OcteonUSB 16f0010000000.usbc: Octeon Host Controller<0D><0A>OcteonUSB 16f0010000000.usbc: new USB bus registered, assigned bus number 1<0D><0A>OcteonUSB 16f0010000000.usbc: irq 56, io mem 0x00000000<0D><0A>hub 1-0:1.0: USB hub found<0D><0A>hub 1-0:1.0: 1 port detected<0D><0A>OcteonUSB: Registered HCD for port 0 on irq 56<0D><0A>usbcore: registered new interface driver usb-storage<0D><0A>octeon_wdt: Initial granularity 5 Sec<0D><0A>TCP: cubic registered<0D><0A>NET: Registered protocol family 17<0D><0A>NET: Registered protocol family 15<0D><0A>Bootbus flash: Setting flash for 4MB flash at 0x1f800000<0D><0A>phys_mapped_flash: Found 1 x16 devices at 0x0 in 8-bit bank. Manufacturer ID 0x0000c2 Chip ID 0x0000a7<0D><0A>Amd/Fujitsu Extended Query Table at 0x0040<0D><0A> Amd/Fujitsu Extended Query version 1.1.<0D><0A>phys_mapped_flash: Swapping erase regions for top-boot CFI table.<0D><0A>number of CFI chips: 1<0D><0A>3 cmdlinepart partitions found on MTD device phys_mapped_flash<0D><0A>Creating 3 MTD partitions on “phys_mapped_flash”:<0D><0A>0x000000000000-0x000000080000 : “boot0”<0D><0A>0x000000080000-0x000000100000 : “boot1”<0D><0A>0x000000300000-0x000000310000 : “eeprom”<0D><0A>Waiting 15sec before mounting root device…<0D><0A>usb 1-1: new high-speed USB device number 2 using OcteonUSB<0D><0A>usb-storage 1-1:1.0: USB Mass Storage device detected<0D><0A>scsi0 : usb-storage 1-1:1.0<0D><0A>scsi 0:0:0:0: Direct-Access USB DISK 2.0 PMAP PQ: 0 ANSI: 4<0D><0A>sd 0:0:0:0: [sda] 7579008 512-byte logical blocks: (3.88 GB/3.61 GiB)<0D><0A>sd 0:0:0:0: [sda] Write Protect is off<0D><0A>sd 0:0:0:0: [sda] No Caching mode page found<0D><0A>sd 0:0:0:0: [sda] Assuming drive cache: write through<0D><0A>sd 0:0:0:0: [sda] No Caching mode page found<0D><0A>sd 0:0:0:0: [sda] Assuming drive cache: write through<0D><0A> sda: sda1 sda2<0D><0A>sd 0:0:0:0: [sda] No Caching mode page found<0D><0A>sd 0:0:0:0: [sda] Assuming drive cache: write through<0D><0A>sd 0:0:0:0: [sda] Attached SCSI removable disk<0D><0A>kjournald starting. Commit interval 3 seconds<0D><0A>EXT3-fs (sda2): warning: maximal mount count reached, running e2fsck is recommended<0D><0A>EXT3-fs (sda2): using internal journal<0D><0A>EXT3-fs (sda2): recovery complete<0D><0A>EXT3-fs (sda2): mounted filesystem with journal data mode<0D><0A>VFS: Mounted root (unionfs filesystem) on device 0:11.<0D><0A>Freeing unused kernel memory: 288K (ffffffffc0648000 - ffffffffc0690000)<0D><0A>SQUASHFS error: zlib_inflate error, data probably corrupt<0D><0A>SQUASHFS error: squashfs_read_data failed to read block 0xabe427<0D><0A>SQUASHFS error: Unable to read fragment cache entry [abe427]<0D><0A>SQUASHFS error: Unable to read page, block abe427, size b144<0D><0A>SQUASHFS error: Unable to read fragment cache entry [abe427]<0D><0A>SQUASHFS error: Unable to read page, block abe427, size b144<0D><0A>SQUASHFS error: Unable to read fragment cache entry [abe427]<0D><0A>SQUASHFS error: Unable to read page, block abe427, size b144<0D><0A>SQUASHFS error: Unable to read fragment cache entry [abe427]<0D><0A>SQUASHFS error: Unable to read page, block abe427, size b144<0D><0A>SQUASHFS error: Unable to read fragment cache entry [abe427]<0D><0A>SQUASHFS error: Unable to read page, block abe427, size b144<0D><0A>Algorithmics/MIPS FPU Emulator v1.5<0D><0A>/bin/sh: bootoctlinux: No such file or directory<0D><0A>
Looking for valid bootloader image….
Jumping to start of image at address 0xbfc80000
U-Boot 1.1.1 (UBNT Build ID: 4493936-g009d77b) (Build time: Sep 20 2012 - 15:48:51)
BIST check passed.
UBNT_E100 r1:2, r2:14, serial #: DC9FDB800FF0
Core clock: 500 MHz, DDR clock: 266 MHz (532 Mhz data rate)
DRAM: 512 MB
Clearing DRAM……. done
Flash: 4 MB
Net: octeth0, octeth1, octeth2
USB: (port 0) scanning bus for devices… 1 USB Devices found
scanning bus for storage devices…
Device 0: Vendor: Prod.: USB DISK 2.0 Rev: PMAP
Type: Removable Hard Disk
Capacity: 3700.6 MB = 3.6 GB (7579008 x 512)
0
reading vmlinux.64
…………………………….
6866056 bytes read
argv[2]: coremask=0x3
argv[3]: root=/dev/sda2
argv[4]: rootdelay=15
argv[5]: rw
argv[6]: rootsqimg=squashfs.img
argv[7]: rootsqwdir=w
argv[8]: mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@3072k(eeprom)
ELF file is 64 bit
Allocating memory for mapped kernel segment, alignment: 0x400000
Allocated memory for ELF segment: addr: 0x400000, size 0x7de180
Processing PHDR 0
Loading 68ae80 bytes at 400000
Clearing 153300 bytes at a8ae80
## Loading Linux kernel with entry point: 0x008e8130 …
Bootloader: Done loading app on coremask: 0x3
Linux version 3.10.20-UBNT (root@ubnt-builder2) (gcc version 4.7.0 (Cavium Inc. Version: SDK_3_1_0_p2 build 34) ) #1 SMP Fri Jan 29 20:11:31 PST 2016
CVMSEG size: 2 cache lines (256 bytes)
Cavium Inc. SDK-3.1
bootconsole [early0] enabled
CPU revision is: 000d0601 (Cavium Octeon+)
Checking for the multiply/shift bug… no.
Checking for the daddiu bug… no.
Determined physical RAM map:
memory: 0000000007400000 @ 0000000000c00000 (usable)
memory: 0000000007c00000 @ 0000000008200000 (usable)
memory: 000000000fc00000 @ 0000000410000000 (usable)
memory: 0000000000648000 @ 0000000000400000 (usable)
memory: 0000000000048000 @ 0000000000a48000 (usable after init)
Wasting 57344 bytes for tracking 1024 unused pages
software IO TLB [mem 0x01707000-0x01747000] (0MB) mapped at [8000000001707000-8000000001746fff]
Zone ranges:
DMA32 [mem 0x00400000-0xefffffff]
Normal [mem 0xf0000000-0x41fbfffff]
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x00400000-0x00a8ffff]
node 0: [mem 0x00c00000-0x07ffffff]
node 0: [mem 0x08200000-0x0fdfffff]
node 0: [mem 0x410000000-0x41fbfffff]
Primary instruction cache 32kB, virtually tagged, 4 way, 64 sets, linesize 128 bytes.
Primary data cache 16kB, 64-way, 2 sets, linesize 128 bytes.
Secondary unified cache 128kB, 8-way, 128 sets, linesize 128 bytes.
PERCPU: Embedded 10 pages/cpu @8000000001784000 s11904 r8192 d20864 u40960
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 125887
Kernel command line: bootoctlinux $loadaddr coremask=0x3 root=/dev/sda2 rootdelay=15 rw rootsqimg=squashfs.img rootsqwdir=w mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@3072k(eeprom) console=ttyS0,115200
PID hash table entries: 2048 (order: 2, 16384 bytes)
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
Memory: 495244k/510528k available (5074k kernel code, 15284k reserved, 1354k data, 288k init, 0k highmem)
Hierarchical RCU implementation.
Additional per-CPU info printed with stalls.
NR_IRQS:255
Calibrating delay loop (skipped) preset value.. 1000.00 BogoMIPS (lpj=5000000)
pid_max: default: 32768 minimum: 501
Security Framework initialized
Mount-cache hash table entries: 256
Checking for the daddi bug… no.
SMP: Booting CPU01 (CoreId 1)…
CPU revision is: 000d0601 (Cavium Octeon+)
Brought up 2 CPUs
NET: Registered protocol family 16
bio: create slab <bio-0> at 0
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Switching to clocksource OCTEON_CVMCOUNT
NET: Registered protocol family 2
TCP established hash table entries: 4096 (order: 4, 65536 bytes)
TCP bind hash table entries: 4096 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP: reno registered
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
NET: Registered protocol family 1
octeon_pci_console: Console not created.
/proc/octeon_perf: Octeon performance counter interface loaded
HugeTLB registered 2 MB page size, pre-allocated 0 pages
squashfs: version 4.0 (2009/01/31) Phillip Lougher
Registering unionfs 2.5.13 (for 3.10.34)
msgmni has been set to 967
io scheduler noop registered
io scheduler cfq registered (default)
Serial: 8250/16550 driver, 6 ports, IRQ sharing disabled
1180000000800.serial: ttyS0 at MMIO 0x1180000000800 (irq = 34) is a OCTEON
console [ttyS0] enabled, bootconsole disabled
console [ttyS0] enabled, bootconsole disabled
1180000000c00.serial: ttyS1 at MMIO 0x1180000000c00 (irq = 35) is a OCTEON
loop: module loaded
ehci_hcd: USB 2.0 ‘Enhanced’ Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 ‘Open’ Host Controller (OHCI) Driver
OcteonUSB 16f0010000000.usbc: Octeon Host Controller
OcteonUSB 16f0010000000.usbc: new USB bus registered, assigned bus number 1
OcteonUSB 16f0010000000.usbc: irq 56, io mem 0x00000000
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
OcteonUSB: Registered HCD for port 0 on irq 56
usbcore: registered new interface driver usb-storage
octeon_wdt: Initial granularity 5 Sec
TCP: cubic registered
NET: Registered protocol family 17
NET: Registered protocol family 15
Bootbus flash: Setting flash for 4MB flash at 0x1f800000
phys_mapped_flash: Found 1 x16 devices at 0x0 in 8-bit bank. Manufacturer ID 0x0000c2 Chip ID 0x0000a7
Amd/Fujitsu Extended Query Table at 0x0040
Amd/Fujitsu Extended Query version 1.1.
phys_mapped_flash: Swapping erase regions for top-boot CFI table.
number of CFI chips: 1
3 cmdlinepart partitions found on MTD device phys_mapped_flash
Creating 3 MTD partitions on “phys_mapped_flash”:
0x000000000000-0x000000080000 : “boot0”
0x000000080000-0x000000100000 : “boot1”
0x000000300000-0x000000310000 : “eeprom”
Waiting 15sec before mounting root device…
usb 1-1: new high-speed USB device number 2 using OcteonUSB
usb-storage 1-1:1.0: USB Mass Storage device detected
scsi0 : usb-storage 1-1:1.0
scsi 0:0:0:0: Direct-Access USB DISK 2.0 PMAP PQ: 0 ANSI: 4
sd 0:0:0:0: [sda] 7579008 512-byte logical blocks: (3.88 GB/3.61 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] No Caching mode page found
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] No Caching mode page found
sd 0:0:0:0: [sda] Assuming drive cache: write through
sda: sda1 sda2
sd 0:0:0:0: [sda] No Caching mode page found
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] Attached SCSI removable disk
kjournald starting. Commit interval 3 seconds
EXT3-fs (sda2): warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs (sda2): using internal journal
EXT3-fs (sda2): recovery complete
EXT3-fs (sda2): mounted filesystem with journal data mode
VFS: Mounted root (unionfs filesystem) on device 0:11.
Freeing unused kernel memory: 288K (ffffffffc0648000 - ffffffffc0690000)
SQUASHFS error: zlib_inflate error, data probably corrupt
SQUASHFS error: squashfs_read_data failed to read block 0xabe427
SQUASHFS error: Unable to read fragment cache entry [abe427]
SQUASHFS error: Unable to read page, block abe427, size b144
SQUASHFS error: Unable to read fragment cache entry [abe427]
SQUASHFS error: Unable to read page, block abe427, size b144
SQUASHFS error: Unable to read fragment cache entry [abe427]
SQUASHFS error: Unable to read page, block abe427, size b144
SQUASHFS error: Unable to read fragment cache entry [abe427]
SQUASHFS error: Unable to read page, block abe427, size b144
SQUASHFS error: Unable to read fragment cache entry [abe427]
SQUASHFS error: Unable to read page, block abe427, size b144
Algorithmics/MIPS FPU Emulator v1.5
/bin/sh: bootoctlinux: No such file or directory
I think the USB stick or onboard memory went bad?
Is there anything I can do to get this unit back online ? I cannot get a replacement for like 5 days which will hurt our business badly.
Thanks in advance for any assistance.
Jamie