Junos OS

mx480

Junos: 17.3R3-S2.2

this krt queue problem is in the logical system.

ls:rpd[14349]: %DAEMON-3-RPD_KRT_Q_RETRIES: nexthop DELETE: Operation not permitted

show krt queue
Routing table add queue: 0 queued
Interface add/delete/change queue: 0 queued
Top-priority deletion queue: 0 queued
Top-priority change queue: 0 queued
Top-priority add queue: 0 queued
high priority V4oV6 tcnh delete queue: 0 queued
high prioriy anchor gencfg delete queue: 0 queued
High-priority multicast add/change: 0 queued
Indirect next hop top priority add/change: 0 queued
Indirect next hop add/change: 0 queued
high prioriy anchor gencfg add-change queue: 0 queued
MPLS add queue: 0 queued
Indirect next hop delete: 1 queued
DELETE nhtype Router index 7769 (49590)
error 'EPERM -- Jtree walk in progress'
kqp '0xb8a1540'
High-priority deletion queue: 0 queued
MPLS change queue: 0 queued
High-priority change queue: 0 queued
High-priority add queue: 0 queued
Normal-priority indirect next hop queue: 0 queued
Normal-priority deletion queue: 0 queued
Normal-priority composite next hop deletion queue: 0 queued
Low prioriy Statistics-id-group deletion queue: 0 queued
Normal-priority change queue: 0 queued
Normal-priority add queue: 0 queued
Least-priority delete queue: 0 queued
Least-priority change queue: 0 queued
Least-priority add queue: 0 queued
Normal-priority pfe table nexthop queue: 0 queued
EVPN gencfg queue: 0 queued
Normal-priority gmp queue: 0 queued
Routing table delete queue: 0 queued
Low priority route retry queue: 0 queued

{master}

anyone can help me figure out the root cause? when I clear the arp, it can be fixed for a short time.

Next-hop type: Router Index: 7769 Address: 0xb64b4b0 Reference count 1 Kernel Table Id 5 Next hop: 23.248.248.18 via ae0.1456 Session Id: 0x349 Flags: explicit-add on-nhid-tree

when I clear this arp cache. the issue can be fixed for s short time. 

Is this index part of any VRF  ? 

If not we may need to perform " restart routing " to clear it permanently 

Thx

Anish

It's in logical system not vrf.

I restarted the routing process in the logical system. it got fixed. thanks so much

Thanks,  good to know it worked ,kindly mark it as accepted solution .

Thx

Anish k t 

Hello Xinhui,

This kind of issue is seen due to sequence of events. Is there load-balancing ECMP configured on this box?

Error EPERM means operation not permitted. There would be multiple reasons for the error which you are seeing, this could be a bug or PR.

I would suggest you to open a case with JTAC for further investigation.

Can you share below output?

show log messages | match jtree

Regards,

Akash

'Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too'

 Hi Xinhui,

are you seeing any core generated along with this? krt queue getting stuck is not uncommon.

Tks,

Abhishek.

For most of KRT stuck issue, you may want to open a JTAC case to find out a bug/PR.

However, there're a few common things to look at:

1. If ARP is resolved for next-hop 

2. If you have reachability to the protocol next-hop 

3. In some static route configuration, you have used an invalid next-hop, for example, a broadcast address, like 192.168.1.255/24. You may want to check your configuration for the IP  23.248.248.18 

1. If ARP is resolved for next-hop 

it needs arp resolved, sometimes when I clear arp, it can be fixed by a short time.