My vintage 2013 SRX210 recently went offline unexpectedly. During all reboots, the kernel startup halts at the same point:
...
PCI Status: PCI 32-bit: 0xc041b
pcib0: Initialized controller
pci0: <PCI bus> on pcib0
pci0: <simple comms> at device 1.0 (no driver attached)
pci0: <serial bus, USB> at device 2.0 (no driver attached)
pci0: <serial bus, USB> at device 2.1 (no driver attached)
pci0: <serial bus, USB> at device 2.2 (no driver attached)
gblmem0 on obio0
octpkt0: <Octeon RGMII> on obio0
cfi0: <AMD/Fujitsu - 4MB> on obio0
Timecounter "mips" frequency 600000000 Hz quality 0
###PCB Group initialized for udppcbgroup
###PCB Group initialized for tcppcbgroup
dwc_otg_nandflash_bulk_flush+0x54 (0x10000,0,0,0x80010700) ra 0x80872f10 sz 64
dwcshutdown+0x84 (0x10000,0,0,0x80010700) ra 0x80893820 sz 40
gpio_srx210_poweroff_intr_handler+0xa0 (0x10000,0,0,0x80010700) ra 0x808151dc sz 24
mips_handle_this_interrupt+0x8c (0x10000,0,0,0x80010700) ra 0x80815268 sz 40
mips_handle_interrupts+0x58 (0x10000,0,0,0x80010700) ra 0x8081568c sz 48
mips_interrupt+0x224 (0x10000,0,0,0x80010700) ra 0x80aaf4c4 sz 32
MipsKernIntr+0x140 (0x1,0x80da89e8,0xffff8010,0x80da89e8) ra 0x80814ab4 sz 368
spinlock_exit+0x84 (0x1,0x80da89e8,0xffff8010,0x80da89e8) ra 0x801ffee0 sz 24
fork_exit+0x24c (0x1,0x80da89e8,0xffff8010,0x80da89e8) ra 0x80ab0150 sz 48
MipsNMIException+0x34 (0x1,0x80da89e8,0xffff8010,0x80da89e8) ra 0 sz 0
pid 23, process: swi7: clock
cpu:0-Trap cause = 2 (TLB miss (load or instr. fetch) - kernel mode)
badvaddr = 0, pc = 0x8087389c, ra = 0x80873870, sr = 0x508008a3
###Not a panic, mastership retained
[thread pid 23 tid 100007 ]
Stopped at dwc_otg_nandflash_bulk_flush+0x54: lw s4,0(v0)
db>
Dumping all of the processes shows that pid 23 is failing in the clock(3) kernel call.
db> alltrace
Tracing command swi1: ipfwd pid 37 tid 100043 td 0xc380e440
SP 0: not in kernel
uart_sab82532_class+0x0 (0,0,0,0) ra 0 sz 0
pid 37, process: swi1: ipfwd
Tracing command swi4: ip6mismatch+ pid 36 tid 100044 td 0xc380e220
SP 0: not in kernel
uart_sab82532_class+0x0 (0,0,0,0) ra 0 sz 0
pid 36, process: swi4: ip6mismatch+
...
Tracing command swi2: netisr 0 pid 25 tid 100022 td 0xc36cd000
SP 0: not in kernel
uart_sab82532_class+0x0 (0,0,0,0) ra 0 sz 0
pid 25, process: swi2: netisr 0
Tracing command swi6: vm pid 24 tid 100023 td 0xc36bfcc0
SP 0: not in kernel
uart_sab82532_class+0x0 (0,0,0,0) ra 0 sz 0
pid 24, process: swi6: vm
Tracing command swi7: clock pid 23 tid 100007 td 0xc36ba000
dwc_otg_nandflash_bulk_flush+0x54 (0,0,0,0) ra 0x808757c0 sz 64
dwcshutdown+0x84 (0,0,0,0) ra 0x808960d0 sz 40
gpio_srx210_poweroff_intr_handler+0xa0 (0,0,0,0) ra 0x80817a8c sz 24
mips_handle_this_interrupt+0x8c (0,0,0,0) ra 0x80817b18 sz 40
mips_handle_interrupts+0x58 (0,0,0,0) ra 0x80817f3c sz 48
mips_interrupt+0x224 (0,0,0,0) ra 0x80ab1d44 sz 32
MipsKernIntr+0x140 (0x1,0x80dab9e8,0xffff8010,0x80dab9e8) ra 0x80817364 sz 368
spinlock_exit+0x84 (0x1,0x80dab9e8,0xffff8010,0x80dab9e8) ra 0x801ffee0 sz 24
fork_exit+0x24c (0x1,0x80dab9e8,0xffff8010,0x80dab9e8) ra 0x80ab29d0 sz 48
MipsNMIException+0x34 (0x1,0x80dab9e8,0xffff8010,0x80dab9e8) ra 0 sz 0
pid 23, process: swi7: clock
Tracing command idle: cpu0 pid 22 tid 100008 td 0xc36bf440
SP 0: not in kernel
uart_sab82532_class+0x0 (0,0,0,0) ra 0 sz 0
pid 22, process: idle: cpu0
Tracing command idle: cpu1 pid 21 tid 100009 td 0xc36bf220
SP 0: not in kernel
uart_sab82532_class+0x0 (0,0,0,0) ra 0 sz 0
pid 21, process: idle: cpu1
The mention of clock, combined with the age of the machine and the fact that is had been switched off for an extended period time led me to suspect the CMOS battery was flat.
Replacing the CMOS battery - a standard CR2032 coin cell - brought the machine fully back to life.