SRX Services Gateway
Highlighted
SRX Services Gateway

SRX240H Anyway to Recover from dwc_otg_nandflash_bulk_flush+0x54 error?

‎08-11-2016 10:50 AM

My lab 240H rebooted for some reason and came up with the following error. From searching this forum it seems the flash died? It's out of support so I'm hoping there may be SOME way to recocover it? Thankfully I log all my console's, but all I get now is a db> prompt.

 

After reboot(s):

 

dwc_otg_nandflash_bulk_flush+0x54 (0x10000,0,0,0x80010700) ra 0x808120e4 sz 64
dwcshutdown+0x84 (0x10000,0,0,0x80010700) ra 0x8083712c sz 40
cpld_device_intr+0x2b4 (0x10000,0,0,0x80010700) ra 0x807b5068 sz 56
mips_handle_this_interrupt+0x8c (0x10000,0,0,0x80010700) ra 0x807b50fc sz 40
mips_handle_interrupts+0x60 (0x10000,0,0,0x80010700) ra 0x807b5528 sz 48
mips_interrupt+0x22c (0x10000,0,0,0x80010700) ra 0x80a420c4 sz 32
MipsKernIntr+0x140 (0x1,0x80d44dc8,0xffff8010,0x80d44dc8) ra 0x807b4940 sz 368
spinlock_exit+0x84 (0x1,0x80d44dc8,0xffff8010,0x80d44dc8) ra 0x801fd2f0 sz 24
fork_exit+0x250 (0x1,0x80d44dc8,0xffff8010,0x80d44dc8) ra 0x80a42d50 sz 48
MipsNMIException+0x34 (0x1,0x80d44dc8,0xffff8010,0x80d44dc8) ra 0 sz 0
pid 10, process: ktrace
cpu:0-Trap cause = 2 (TLB miss (load or instr. fetch) - kernel mode)
badvaddr = 0, pc = 0x80812a48, ra = 0x80812a1c, sr = 0x508008a3
[thread pid 10 tid 100006 ]
Stopped at      dwc_otg_nandflash_bulk_flush+0x54:      lw      s4,0(v0)
db>

Console Log of event:

 

 

root> 

NMI Exception on core:0
Watchdog status, core 0: 0xfffe64ffffb
FPA INT Summery: 0x0
Err EPC: 0x8083719c
Trapframe Register Dump:
zero: 0000000000000000  at: fffffffffffffffe  v0: 000000001fbf0006  v1: 0000000000000006
  a0: ffffffff80010000  a1: 000000001fbf0006  a2: 0000000000000080  a3: 00000000508008a1
  t0: 800100001fbf0006  t1: 0000000000000000  t2: ffffffff80016f00  t3: 0000000010000508
 ta0: 0000000010000500 ta1: ffffffffc1cc16e0 ta2: ffffffffc1cc3640 ta3: 0000000000000001
  t8: 0000000057acac80  t9: 0000000000001388  s0: 0000000000000080  s1: ffffffffc1d03d80
  s2: 0000000000000000  s3: ffffffffc2aa0440  s4: 0000000000000000  s5: 0000000000000007
  s6: ffffffffd66dfb30  s7: ffffffff80cd0000  k0: 1a00000080c099dc  k1: 808042a80000000a
  gp: ffffffff80c197b0  sp: ffffffffd66dfa50  s8: 0000000000000000  ra: ffffffff8083712c
  sr: 0000000050c808a5 mullo: 0000000000000001    mulhi: 000000000000000c
  pc: ffffffff80811834 cause: 0000000040008408 badvaddr: ffffffffc1d7dd54
ErrPC: 0000000000000840
Current ticks/softticks 2517262/2517262, curproc [22] idle: cpu0
Core0: CacheErr(I/D: current: 0x7f7f0000000000/0x1330)

PCPU dump:
cpuid        = 0
curthread    = 0xc1ce5420: pid 22 "idle: cpu0"
ipis         = 0x0
cpuid        = 1
curthread    = 0xc2c09a50: pid 1356 "flowd_octeon_hm"
ipis         = 0x0
cpuid        = 2
curthread    = 0xc2c09840: pid 1356 "flowd_octeon_hm"
ipis         = 0x0
cpuid        = 3
curthread    = 0xc2c09630: pid 1356 "flowd_octeon_hm"
ipis         = 0x0
cpuid        = 4
curthread    = none
ipis         = 0x0
cpuid        = 5
curthread    = none
ipis         = 0x0
cpuid        = 6
curthread    = none
ipis         = 0x0
cpuid        = 7
curthread    = none
ipis         = 0x0
cpuid        = 8
curthread    = none
ipis         = 0x0
cpuid        = 9
curthread    = none
ipis         = 0x0
cpuid        = 10
curthread    = none
ipis         = 0x0
cpuid        = 11
curthread    = none
ipis         = 0x0
Memory dump of 1024 words starting at 0x80000000
0x80000000: 082905e3 401a4000 00000000 80056560 
0x80000010: 800565c4 800728f8 aaaaaaaa aaaaaaaa 
0x80000020: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000030: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000040: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000050: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000060: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000070: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000080: 082905e3 401a4000 00000000 aaaaaaaa 
0x80000090: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800000a0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800000b0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800000c0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800000d0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800000e0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800000f0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000100: 3c1b80d5 277bae68 7c1a003b 001ad0c0 
0x80000110: 035bd821 403ad801 ff7a0000 401a6000 
0x80000120: 335a0002 17400005 00000000 3c1a80a4 
0x80000130: 275a2af0 03400008 00000000 3c1a807e 
0x80000140: 275a70b0 03400008 00000000 1000ffff 
0x80000150: 00000000 42000018 aaaaaaaa aaaaaaaa 
0x80000160: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000170: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000180: 401a6000 401b6800 335a0010 001ad0c0 
0x80000190: 337b007c 037ad825 3c1a80c0 275a5130 
0x800001a0: 035bd021 8f5a0000 00000000 03400008 
0x800001b0: 00000000 aaaaaaaa aaaaaaaa aaaaaaaa 
0x800001c0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800001d0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800001e0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800001f0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000200: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000210: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000220: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000230: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000240: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000250: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000260: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000270: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000280: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000290: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800002a0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800002b0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800002c0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800002d0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800002e0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800002f0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000300: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000310: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000320: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000330: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000340: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000350: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000360: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000370: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000380: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000390: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800003a0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800003b0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800003c0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800003d0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800003e0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800003f0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
Stack trace:
cpld_device_intr+0x324 (0x80010000,0x1fbf0006,0x80,0x508008a1) ra 0x807b5068 sz 56
mips_handle_this_interrupt+0x8c (0x80010000,0x1fbf0006,0x80,0x508008a1) ra 0x807b50fc sz 40
mips_handle_interrupts+0x60 (0x80010000,0x1fbf0006,0x80,0x508008a1) ra 0x807b5528 sz 48
mips_interrupt+0x22c (0x80010000,0x1fbf0006,0x80,0x508008a1) ra 0x80a420c4 sz 32
MipsKernIntr+0x140 (0x80ae0000,0x80d44dc8,0xffff8010,0xffff8010) ra 0x807b2f70 sz 368
cpu_idle+0x10 (0x80ae0000,0x80d44dc8,0xffff8010,0xffff8010) ra 0x80202c2c sz 24
idle_proc+0x22c (0x80ae0000,0x80d44dc8,0xffff8010,0xffff8010) ra 0x801fd2fc sz 64
fork_exit+0x25c (0x80ae0000,0x80d44dc8,0xffff8010,0xffff8010) ra 0x80a42d50 sz 48
MipsNMIException+0x34 (0x80ae0000,0x80d44dc8,0xffff8010,0xffff8010) ra 0 sz 0
pid 22, process: idle: cpu0
Resetting the  system now...
cpu_reset: Stopping other CPUs
timeout stopping cpus

 

11 REPLIES 11
Highlighted
SRX Services Gateway

Re: SRX240H Anyway to Recover from dwc_otg_nandflash_bulk_flush+0x54 error?

‎08-11-2016 11:23 AM

Hi,

 

Please follwo the stepns mentione in the following KB to recover the SRX :-

 

https://kb.juniper.net/InfoCenter/index?page=content&id=KB20046&actp=search&viewlocale=en_US&searchi...

 

Regards,

Sahil Sharma

---------------------------------------------------

Please mark my solution as accepted if it helped, Kudos are appreciated as well.

 

Highlighted
SRX Services Gateway

Re: SRX240H Anyway to Recover from dwc_otg_nandflash_bulk_flush+0x54 error?

‎08-11-2016 12:02 PM

Performed, but failed:

 

Press SPACE to abort autoboot in 1 seconds
=> setenv oct.net.init
=> saveenv
Saving Environment to Flash...
Un-Protected 1 sectors
Erasing Flash...
. done
Erased 1 sectors
Writing to Flash... writing to flash...
done
Protected 1 sectors
=> reset
pid 10, process: ktrace
cpu:0-Trap cause = 2 (TLB miss (load or instr. fetch) - kernel mode)
badvaddr = 0, pc = 0x80805538, ra = 0x8080550c, sr = 0x50800883
[thread pid 10 tid 100006 ]
Stopped at      dwc_otg_nandflash_bulk_flush+0x54:      lw      s4,0(v0)
db> 

Any other thoughts? Thanks for the quick response!

Highlighted
SRX Services Gateway

Re: SRX240H Anyway to Recover from dwc_otg_nandflash_bulk_flush+0x54 error?

‎08-11-2016 01:00 PM
Try reflashing the file system according to:
http://www.juniper.net/techpubs/en_US/junos14.2/topics/task/installation/security-junos-os-tftp-srx-...
Just have a backup of the configuration as everything on the flash will be deleted.
If this doesn't help, seems the flash memory is dead and you'll need to RMA the SRX.
Highlighted
SRX Services Gateway

Re: SRX240H Anyway to Recover from dwc_otg_nandflash_bulk_flush+0x54 error?

‎08-11-2016 01:55 PM

Attempted install:

 

loader> install file:///junos-srxsme-12.1X46-D55.3-domestic.tgz
/kernel data=0xb16d5c+0x134b2c syms=[0x4+0x8bbd0+0x4+0xcadc3]
Kernel entry at 0x801000e0 ...
init regular console
Primary ICache: Sets 64 Size 128 Asso 4
Primary DCache: Sets 1 Size 128 Asso 64
Secondary DCache: Sets 512 Size 128 Asso 8
GDB: debug ports: uart
GDB: current port: uart
KDB: debugger backends: ddb gdb
KDB: current backend: ddb
kld_map_v: 0x8ff80000, kld_map_p: 0x0

System restarted and same thing:

pid 10, process: ktrace
cpu:0-Trap cause = 2 (TLB miss (load or instr. fetch) - kernel mode)
badvaddr = 0, pc = 0x80812a48, ra = 0x80812a1c, sr = 0x508008a3
[thread pid 10 tid 100006 ]
Stopped at      dwc_otg_nandflash_bulk_flush+0x54:      lw      s4,0(v0)
db> 

I think it's dead Jim....and no more warranty/support agreement on this one. Can I run an image OFF the USB drive? Or will it be forced to boot from internal regardless?

 

Highlighted
SRX Services Gateway

Re: SRX240H Anyway to Recover from dwc_otg_nandflash_bulk_flush+0x54 error?

‎08-11-2016 03:22 PM

The SRX will boot from external USB first.  So if you can create a snapshot to USB on a working SRX of the same model.  Then insert that here you should be back in business.

Steve Puluka BSEET - Juniper Ambassador
IP Architect - DQE Communications Pittsburgh, PA (Metro Ethernet & ISP)
http://puluka.com/home
Highlighted
SRX Services Gateway

Re: SRX240H Anyway to Recover from dwc_otg_nandflash_bulk_flush+0x54 error?

‎08-11-2016 08:47 PM

Hi,

 

Please try the 2 methods explained in the following link :-

 

http://www.juniper.net/documentation/en_US/junos15.1/topics/topic-map/security-software-installation...

 

  1. Boot using TFTP.
  2. Boot from USB without a snapshot.

 

Regards,

Sahil Sharma

---------------------------------------------------

Please mark my solution as accepted if it helped, Kudos are appreciated as well.

Highlighted
SRX Services Gateway

Re: SRX240H Anyway to Recover from dwc_otg_nandflash_bulk_flush+0x54 error?

‎08-12-2016 10:48 AM

I only have an SRX650 sitting here, I put junos-srxsme-12.1X46-D55.3-domestic.tgz on the 650, and did the snapshot to the USB drive, plugged into the 240, it tried to boot from internal. Went into U-Boot and tried usbboot and it says * No boot device *. The SRX650 won't boot from it either though. But it seems like it was built correctly:

 

root> show system snapshot media usb        
Information for snapshot on       usb (/dev/da0s1a) (primary)
Creation date: Aug 12 18:36:11 2016
JUNOS version on snapshot:
  junos  : 12.1X46-D55.3-domestic
Information for snapshot on       usb (/dev/da0s2a) (backup)
Creation date: Aug 12 18:38:10 2016
JUNOS version on snapshot:
  junos  : 12.1X46-D55.3-domestic

I'll try another USB stick this afternoon.

Highlighted
SRX Services Gateway

Re: SRX240H Anyway to Recover from dwc_otg_nandflash_bulk_flush+0x54 error?

‎08-12-2016 10:59 AM

Sorry, I was wrong, I went into loader and did nextboot usb and it booted from USB but I still get the error:

 

FreeBSD/MIPS U-Boot bootstrap loader, Revision 1.7
(builder@shoth.juniper.net, Tue May  4 07:15:51 UTC 2010)
Memory: 1024MB
[1]Booting from usb slice 1
Un-Protected 1 sectors
writing to flash...
Protected 1 sectors
Loading /boot/defaults/loader.conf 
/kernel data=0xb16d5c+0x134b2c syms=[0x4+0x8bbd0+0x4+0xcadc3]

...

pid 10, process: ktrace
cpu:0-Trap cause = 2 (TLB miss (load or instr. fetch) - kernel mode)
badvaddr = 0, pc = 0x80812a48, ra = 0x80812a1c, sr = 0x508008a3
[thread pid 10 tid 100006 ]
Stopped at      dwc_otg_nandflash_bulk_flush+0x54:      lw      s4,0(v0)
db> 
Highlighted
SRX Services Gateway

Re: SRX240H Anyway to Recover from dwc_otg_nandflash_bulk_flush+0x54 error?

‎08-12-2016 04:54 PM

I had similar issues trying to create boot USB cross hardware for the SRX.  Pretty sure you need to create the snapshot on the same hardware for the device to boot correctly.

Steve Puluka BSEET - Juniper Ambassador
IP Architect - DQE Communications Pittsburgh, PA (Metro Ethernet & ISP)
http://puluka.com/home
Highlighted
SRX Services Gateway

Re: SRX240H Anyway to Recover from dwc_otg_nandflash_bulk_flush+0x54 error?

‎08-16-2016 08:03 AM

I know this is probably not possible, but is there anyway someone can post a snapshot that I can write to a usb drive? Or maybe less difficult, anyway I can dd an image (junos-srxsme-12.1X44-D10.4-domestic.tgz) directly to a USB drive? I can't find any of the old recovery files on the Juni downloads like I used to be able to use on M series. Can't even find them for those chassis anymore in fact.

 

Thank you for all the suggestions though, I really do appreciate it!

Highlighted
SRX Services Gateway

Re: SRX240H Anyway to Recover from dwc_otg_nandflash_bulk_flush+0x54 error?

2 weeks ago

 Try replacing the CMOS battery on the motherboard - a standard CR2032 coin cell. Worked for me with my SRX210.