Screen OS

Hi All,

I have  problem with my SSG550M 

 We  found some commands to apply.

av_key exec license key delete-

anti_spam_key exec license key delete-

Now with this change the ssg550m random reboot in 4 days .

Before it restarted in during in  the day.

We think it's a firmware problem and not hardware.

get sys log saved

get log sys saved System version:6.3.0r18-cu2.0

###Crash Time: 27Apr2015:17:04:06###

*********************************************************

                Exception Dump

*********************************************************

System up time: 96 hours 39 minutes 50 seconds Version 6.3.0r18-cu2.0

Exception Reason: Page Fault

Error number:2

Crash in task 49

  eax = 0x0b1be800, ebx = 0x0b1be800, ecx = 0x00000000, edx = 0x00000000

  esi = 0x00000000, edi = 0x0a6e9ee4, ebp = 0x8bffedf8, esp = 0x8bffedf8

  eip = 0x00c573dd, ds = 0x0010,  gs = 0x0010,  ss = 0x0010,

  cs = 0x0008,   es = 0x0010,  fs = 0x0010

  flags = 0x00010246, cr0 = 0x80010019 cr2 = 0x00000000

SAN_JOSE_333_CD3001-> get memory pool

Global memory pools:

NAME                         SYS_MEM   ALLOCMEM NALLOC  NFREE OVERSZ     QUOTA

==============================================================================

Routing                        16436       6168    464    627      0        -1

SSHv2 String Pool                  0          0      0      0      0        -1

idp                         10668636    9515392 134612  18834      1  53886976

JPS Notify                         0          0      0      0      0        -1

JPS Context                     8212         48      2    290      0        -1

defrag pool                   390104          0      0    680      0    975000

net                            24572        352     11    703      0        -1

Auth Id Table                      0          0      0      0      0        -1

CAVIUM                       9433088    9184000  30733    330     10        -1

NET-PAK                       455292      43520    128    584      0 536870912

PKI-IKE                      1723632    1416512   9151   1600  17656        -1

sys                        141516004  109080320 7760162   1248      0        -1

In the sys memory is constantly increases

Why I can see in memory register using the IDP.

We are not using Deep Inspection.

Please can you help me please

Thanks in advance

Regards

Could you please provide the entire output of "get log sys save"

 get log sys saved
System version:6.3.0r14.0. ( downgrade)
###Crash Time: 08May2015:22:16:41###

*********************************************************
Exception Dump
*********************************************************
System up time: 95 hours 29 minutes 10 seconds
Version 6.3.0r14.0

Exception Reason: Crash Dump called
Error number:0
Crash in task 89
eax = 0x017088f8, ebx = 0x0168d4dc, ecx = 0x00000000, edx = 0x0000000d
esi = 0x2bee7f84, edi = 0x29cd9dcc, ebp = 0x8bfffa78, esp = 0x8bfffa6c
eip = 0x00108dd8, ds = 0x0010, gs = 0x0010, ss = 0x0010,
cs = 0x0008, es = 0x0010, fs = 0x0010
flags = 0x00000283, cr0 = 0x80010019 cr2 = 0x80007000

Stack dump:
8bfffa6c: 01 70 88 f6 01 68 d4 dc 2b ee 7f 64 8b ff fa a0
8bfffa7c: 00 12 7d 37 01 68 d4 dc 2b ee 7f 84 2a a5 36 74
8bfffa8c: 01 14 0d 40 00 00 00 00 2b ee 7f 84 00 00 08 af
8bfffa9c: 29 cd 9d cc 8b ff fa bc 01 13 15 f4 2b ee 7f 84
8bfffaac: 00 00 00 00 29 cd 9d cc ff ff ff ff 00 00 00 00
8bfffabc: 8b ff fa d4 01 14 0d da 29 cd 9d cc 01 14 0d 90
8bfffacc: 00 00 00 00 00 00 00 00 8b ff fb 14 00 7e bf 76
8bfffadc: 29 cd 9d cc 0b 5d ba 34 00 16 ac 3a 29 cd 9d cc
8bfffaec: 00 80 36 e0 00 00 00 00 00 00 00 00 29 cd 9d cc
8bfffafc: ff ff ff ff 29 f3 d2 e4 00 00 00 0c 29 cc a3 b4
8bfffb0c: 00 16 ac 3a 00 00 00 3f 8b ff fb 58 00 80 1f 70
8bfffb1c: 29 cc a3 b4 00 80 36 e0 00 00 00 00 00 00 00 10
8bfffb2c: 8b ff fb 78 70 ff fb 58 00 e6 45 88 0e 00 00 0b
8bfffb3c: 01 6e bb 10 01 6e bb 17 01 6e bb 1e 00 00 00 00
8bfffb4c: 18 8a e9 8c 03 cf 4d f8 18 8a ed 0c 8b ff fb 84
8bfffb5c: 01 15 b8 fc 0b 47 78 c4 03 cf 4d f8 8b ff fb b0

Trace Dump:
00108dd8 00127d37 011315f4 01140dda 007ebf76 00801f70 0115b8fc 0115e791
0115b8fc 0113a5cf 0113ab74 0115b8fc 01158fe3 01153cf3 0113e3b1 00802fc2
00109cbe

Crash dump, the system will reboot...

-----------
OS Context:
-----------
Died Flow/bootup Module
Cur Task Context: sme

Cavium chip enabled.
----------------------------------------------------------
Queue(Hi) Queue(Low) Total
Requests 265 76205676 76205941
Completed 265 76205676 76205941
Timed out 0 0 0
Length 0 0 0
----------------------------------------------------------
Pending queue length: 0
Interrupt: 0 WatchDog Timeout: 0 Last ISR: 00000000
----------------------------------------------------------

Cavium Registers:
----------------------------------------------------------
COMMAND_STATUS 0000b205
UNIT_ENABLE 1000000f
INTERRUPT MASK 001fdfff
INTERRUPT STATUS 00000000 *
FAILING_SEQ_REG 18fe9618
FAILING_EXEC_REG 00000000 *
PCI_ERR_REG 00000000 *
REQ0_BASE_HIGH 00000000
REQ0_BASE_LOW 03092b40
REQ0_SIZE 00002800
REQ1_BASE_HIGH 00000000
REQ1_BASE_LOW 030e2be0
REQ1_SIZE 00002800
REQ0_DOOR_BELL 00000000 *
REQ1_DOOR_BELL 00000000 *
----------------------------------------------------------

Cur Task Context: sme

This indicates that the task running at the time of the crash was NSM.  Are you using NSM to manage the device?  Also, were there any changes or delta configs ran during this time?

Yes I manage the device with NSM Xpress.

Yes, but do not apply config delta.

I saw in the memory is red from WEBUI.

When applying get mem pool command and i saw sys constantly increasing

Regards

I concur with you sense that this is a firmware issue.  In my experience any issue that creates the exception dump relates ultimately to a software bug.

You could upgrade to the newer version to see if the issue clears.  Or read the latest release notes to see if there are NSM related bugs fixed first.

I update to 6.3.0r18 firmware version.  

For now this 6 days ago without random rebooting.

memory is full 

sys 452663744 358410784 22494242 4349 0 -1

get mem pool
Global memory pools:

NAME SYS_MEM ALLOCMEM NALLOC NFREE OVERSZ QUOTA
==============================================================================
Routing 16436 6480 485 606 0 -1
SSHv2 String Pool 0 0 0 0 0 -1
idp 10685032 9770752 159061 3068 3 53886976
JPS Notify 0 0 0 0 0 -1
JPS Context 8212 48 2 290 0 -1
defrag pool 390104 0 0 680 0 975000
net 16424 224 7 629 0 -1
Auth Id Table 0 0 0 0 0 -1
CAVIUM 9433088 9184000 30733 330 10 -1
NET-PAK 455292 33320 98 614 0 536870912
PKI-IKE 1692608 1464128 10673 963 215075 -1
sys 452663744 358410784 22494242 4349 0 -1

Thanks in advance

Regards 

Your memory looks pretty much the same as some samples in my lab.

Do you still see evidence of a memory leak after the upgrade?

The equipment random reboot again in the 7 day.
The memory increase constantly in SYS option.

I think you are hitting this bug as listed in the 6.3R19 release notes as fixed.

Addressed Issues in ScreenOS 6.3.0r19

1012257 - An exception dump occurred when memory availability was zero as a result of memory leakage.

So if you upgrade again to 6.3R19 (just released a two weeks ago) you should be fixed.

http://www.juniper.net/techpubs/software/screenos/screenos6.3.0/rn-630r19-rev01.pdf

Hi

I have the same case (firewalls rebooting, master and passive), upgraded to version 6.3R21, but still has the same issue.

Do I have to downgrade to 6.3R19?

Thanks

You do not need to downgrade, issues fixed in lower releases are maintained going forward.  What this means is you are hitting a different bug.  This case the symptom was the slowly rising memory usage over time.

What are the symptoms when your reboot occurs?

Do you get a core dump on the device?  get log sys saved

Any messages in syslog right before the reboots?

Hello Steve

the two firewalls are rebooted almost in the same time and every day once or twice.

And yes there is Core Dump as below output;

But there is no related syslog messages right before the firewalls reboot...

System version:6.3.0r21.0.
###Crash Time: 19May2016:07:51:43###

*********************************************************
Exception Dump
*********************************************************
System up time: 48 hours 3 minutes 4 seconds
Version 6.3.0r21.0

Exception Reason: Page Fault
Error number:2
Crash in flow
eax = 0x00009000, ebx = 0x0000003e, ecx = 0x00000000, edx = 0x1b44d0c8
esi = 0x08edbd70, edi = 0x00000016, ebp = 0x01d49a5b, esp = 0x01d49967
eip = 0x0019f95e, ds = 0x0010, gs = 0x0000, ss = 0x0010,
cs = 0x0008, es = 0x0010, fs = 0x0000
flags = 0x00014006, cr0 = 0x80010019 cr2 = 0x00000000

Stack dump:
01d49967: 08 ed bd 70 02 99 53 60 00 00 08 00 00 00 00 00
01d49977: 08 f7 63 d0 01 d4 99 c7 00 00 20 00 09 11 40 e4
01d49987: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01d49997: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01d499a7: 00 00 00 00 1b 4a c1 bc 00 00 00 00 03 d9 a9 30
01d499b7: 08 ed bd 70 00 00 00 00 00 00 00 00 02 99 53 60
01d499c7: 69 00 90 d9 01 d4 99 e7 00 11 53 2b 00 00 00 00
01d499d7: 00 00 00 00 01 97 f0 04 01 97 f0 04 00 00 00 06
01d499e7: 01 d4 9a 0b 00 10 d5 6b 00 00 00 01 01 97 f0 04
01d499f7: 01 99 1c 84 00 00 00 01 01 97 f0 04 00 00 00 01
01d49a07: 01 97 f2 4c 01 d4 9a 23 00 11 69 6f 01 97 f0 04
01d49a17: 00 00 00 03 1b 39 a4 a4 01 97 f0 04 01 d4 9a 43
01d49a27: 00 12 05 08 00 00 00 01 02 41 d5 44 80 00 00 00
01d49a37: 02 41 d5 4c 00 00 00 00 09 11 40 b0 02 41 d5 04
01d49a47: 08 eb c0 00 02 41 d5 3c 08 f7 63 d0 00 00 00 00
01d49a57: 09 11 40 b0 01 d4 9a 97 00 17 5d ec 08 f7 63 d0

Trace Dump:
0019f95e 00175dec 00176237 0016b5a2 0019c42f 00164895 001789d2 0010ad0f
00000000

Crash dump, the system will reboot...

-----------
OS Context:
-----------
Died Flow/bootup Module
Cur Task Context: idle task

Cavium chip enabled.
----------------------------------------------------------
Queue(Hi) Queue(Low) Total
Requests 36 1 37
Completed 36 1 37
Timed out 0 0 0
Length 0 0 0
----------------------------------------------------------
Pending queue length: 0
Interrupt: 0 WatchDog Timeout: 0 Last ISR: 00000000
----------------------------------------------------------

Cavium Registers:
----------------------------------------------------------
COMMAND_STATUS 0000b205
UNIT_ENABLE 1000000f
INTERRUPT MASK 001fdfff
INTERRUPT STATUS 00000000 *
FAILING_SEQ_REG fd9effc7
FAILING_EXEC_REG 00000000 *
PCI_ERR_REG 00000000 *
REQ0_BASE_HIGH 00000000
REQ0_BASE_LOW 0311bd20
REQ0_SIZE 00002800
REQ1_BASE_HIGH 00000000
REQ1_BASE_LOW 0316bda0
REQ1_SIZE 00002800
REQ0_DOOR_BELL 00000000 *
REQ1_DOOR_BELL 00000000 *
----------------------------------------------------------

This confirms you are hitting a software bug, Juniper calls these PR (problem reports).  You can try to upgrade to the newest release as there is r22 posted now in the download sites to see if this corrects the issue.  If you read the release notes for this version you can see if any of the listed corrected PR match circumstances on your network.

Or you can open an official JTAC case now with the tech support and core dump information.  JTAC should be able to tell from these if you are hitting an existing PR or have discovered a new one.

This looks to be a new issue.  Please open a JTAC case.