Junos
Highlighted
Junos

mx 480 17.3R3-S2.2 krt queue problem

‎06-20-2019 10:08 PM

mx480

Junos: 17.3R3-S2.2

 

this krt queue problem is in the logical system.

 

ls:rpd[14349]: %DAEMON-3-RPD_KRT_Q_RETRIES: nexthop DELETE: Operation not permitted

 

show krt queue
Routing table add queue: 0 queued
Interface add/delete/change queue: 0 queued
Top-priority deletion queue: 0 queued
Top-priority change queue: 0 queued
Top-priority add queue: 0 queued
high priority V4oV6 tcnh delete queue: 0 queued
high prioriy anchor gencfg delete queue: 0 queued
High-priority multicast add/change: 0 queued
Indirect next hop top priority add/change: 0 queued
Indirect next hop add/change: 0 queued
high prioriy anchor gencfg add-change queue: 0 queued
MPLS add queue: 0 queued
Indirect next hop delete: 1 queued
DELETE nhtype Router index 7769 (49590)
error 'EPERM -- Jtree walk in progress'
kqp '0xb8a1540'
High-priority deletion queue: 0 queued
MPLS change queue: 0 queued
High-priority change queue: 0 queued
High-priority add queue: 0 queued
Normal-priority indirect next hop queue: 0 queued
Normal-priority deletion queue: 0 queued
Normal-priority composite next hop deletion queue: 0 queued
Low prioriy Statistics-id-group deletion queue: 0 queued
Normal-priority change queue: 0 queued
Normal-priority add queue: 0 queued
Least-priority delete queue: 0 queued
Least-priority change queue: 0 queued
Least-priority add queue: 0 queued
Normal-priority pfe table nexthop queue: 0 queued
EVPN gencfg queue: 0 queued
Normal-priority gmp queue: 0 queued
Routing table delete queue: 0 queued
Low priority route retry queue: 0 queued

{master}

 

anyone can help me figure out the root cause? when I clear the arp, it can be fixed for a short time.

JNCIE x 3 (SP DC ENT)
9 REPLIES 9
Highlighted
Junos

Re: mx 480 17.3R3-S2.2 krt queue problem

‎06-20-2019 10:23 PM

Next-hop type: Router Index: 7769 Address: 0xb64b4b0 Reference count 1 Kernel Table Id 5 Next hop: 23.248.248.18 via ae0.1456 Session Id: 0x349 Flags: explicit-add on-nhid-tree

 

when I clear this arp cache. the issue can be fixed for s short time. 

JNCIE x 3 (SP DC ENT)
Highlighted
Junos

Re: mx 480 17.3R3-S2.2 krt queue problem

‎06-20-2019 10:25 PM

Hello Xinhui,

 

This kind of issue is seen due to sequence of events. Is there load-balancing ECMP configured on this box?

Error EPERM means operation not permitted. There would be multiple reasons for the error which you are seeing, this could be a bug or PR.

 

I would suggest you to open a case with JTAC for further investigation.

 

Can you share below output?

 

show log messages | match jtree

Regards,

Akash

'Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too'

Highlighted
Junos

Re: mx 480 17.3R3-S2.2 krt queue problem

‎06-20-2019 10:55 PM

 Hi Xinhui,

 

are you seeing any core generated along with this? krt queue getting stuck is not uncommon.

 

Tks,

Abhishek.

Highlighted
Junos
Solution
Accepted by topic author xinhui jiang
‎06-24-2019 06:54 PM

Re: mx 480 17.3R3-S2.2 krt queue problem

‎06-21-2019 12:01 AM

Is this index part of any VRF  ? 

If not we may need to perform " restart routing " to clear it permanently 

 

Thx

Anish

Highlighted
Junos

Re: mx 480 17.3R3-S2.2 krt queue problem

‎06-21-2019 09:28 AM

For most of KRT stuck issue, you may want to open a JTAC case to find out a bug/PR.

 

However, there're a few common things to look at:

1. If ARP is resolved for next-hop 

2. If you have reachability to the protocol next-hop 

3. In some static route configuration, you have used an invalid next-hop, for example, a broadcast address, like 192.168.1.255/24. You may want to check your configuration for the IP  23.248.248.18 

 

 


Mengzhe Hu
JNCIE x 3 (SP DC ENT)
Highlighted
Junos

Re: mx 480 17.3R3-S2.2 krt queue problem

‎06-22-2019 03:34 AM

It's in logical system not vrf.

JNCIE x 3 (SP DC ENT)
Highlighted
Junos

Re: mx 480 17.3R3-S2.2 krt queue problem

‎06-22-2019 03:37 AM

1. If ARP is resolved for next-hop 

 

it needs arp resolved, sometimes when I clear arp, it can be fixed by a short time.

JNCIE x 3 (SP DC ENT)
Highlighted
Junos

Re: mx 480 17.3R3-S2.2 krt queue problem

‎06-24-2019 06:55 PM

I restarted the routing process in the logical system. it got fixed. thanks so much

JNCIE x 3 (SP DC ENT)
Highlighted
Junos

Re: mx 480 17.3R3-S2.2 krt queue problem

‎06-24-2019 10:16 PM

Thanks,  good to know it worked ,kindly mark it as accepted solution .

 

Thx

Anish k t 

Feedback