Problem with NAT service (MX80)

[#1]

 Hello. Using Juniper MX80 as BRAS. Junos version: 13.3R9.13.

Clients go in  internet via NAT. Clients count about 4.2k. In last time faced with high load of MS-MIC-16G card, which affect response time of resource in internet ( from example from ~10 sec to ~40 sec).

After rebooting ms-mic-16g, cpu load return to normal and response time improved. But after a while time cpu load grows and response time also. This is shown in the graph (red line ms-mic-16g cpu utilization)

I checked NAT config  and did not find anything suspicious.

Here config:

pool NAT-POOL-1 {
    address-range low XXX.XXX.XXX.1 high XXX.XXX.XXX.254
    port {
        automatic {
            random-allocation;
        }
    }
}
rule NAT-RULE {
    match-direction input;
    term EIM {
        from {
            source-prefix-list {
                NAT-PREFIX-LIST;
            }
            applications [ junos-pptp junos-ipsec-esp ];
            application-sets APP;
        }
        then {
            translated {
                source-pool NAT-POOL-1;
                translation-type {
                    napt-44;
                }
                address-pooling paired;
            }
        }
    }
    term SIMPLY {
        from {
            source-prefix-list {
                NAT-PREFIX-LIST;
            }
        }
        then {
            translated {
                source-pool NAT-POOL-1;
                translation-type {
                    napt-44;
                }
                address-pooling paired;
            }
        }
    }
}

NAT statistics

router_name>show services nat statistics 
Interface: ms-0/2/0
 
Session statistics 
 
Session statistics 
    Total Session Interest events			    :487733053
    Total Session Create events				    :245970868
    Total Session Destroy events			    :499617371
    Total Session Pub Req events			    :24
    Total Session Accepts				    :245956398
    Total Session Discards				    :241762154
    Total Session Ignores				    :14501
    Session interest thru pub event			    :0
    ALG Session interest				    :48
    ALG Session Create					    :48
    Packet  Dst in NAT route				    :241757812
    Packet drop in backup state				    :0
    Session Ext Alloc Failures				    :0
    Session Ext Set Failures				    :0
    Session Created for EIF				    :0
    Session Created for EIM				    :0
    NAT rule lookup failures				    :241772313
    Pool session count update failed on create              :0
    Pool session count update failed on close               :0

NAT Allocation statistics
    NAT allocation Successes				    :245956350
    NAT allocation Failures				    :0
    NAT Free Successes					    :245826422
    NAT Free Failures					    :0
    NAT EIM mapping reused				    :0
    NAT EIM mapping allocation failures			    :0
    NAT EIM mapping Duplicate entry			    :0
    NAT EIM mapping create failed			    :0
    NAT EIM mapping Created				    :0
    NAT EIM mapping Updated				    :0
    NAT EIF mapping Free				    :0
    NAT EIM mapping Free				    :0
    NAT EIM waiting for init				    :0
    NAT EIM waiting for init failed			    :0
    NAT EIM lookup and hold success			    :0
    NAT EIM lookup entry in timeout			    :0
    NAT EIM lookup timer cleared for timeout entry	    :0
    NAT EIM lookup timeout entry without timer		    :0
    NAT EIM release without entry			    :0
    NAT EIM release entry in timeout    		    :0
    NAT EIM release race				    :0
    NAT EIM release set entry for timeout		    :0
    NAT EIM timer entry refreshed			    :0
    NAT EIM timer invalid timer started			    :0
    NAT EIM timer entry freed				    :0
    NAT EIM timer entry updated				    :0
    NAT EIM entry drained                                   :0

Packet statistics
    Total Packets Processed				    :2801621451
    Total Packets Forwarded				    :2801621442
    Total Packets Discarded				    :9
    Total Packets Translated				    :1773706062
    Total Packets Restored				    :996650560

Translation statistics
    Src  IPv4   Translations				    :1768704401
    Src  IPv4   Restorations				    :0
    Dst  IPv4   Translations				    :5001661
    Dst  IPv4   Restorations				    :996650560
    Src  IPv6   Translations				    :0
    Src  IPv6   Restorations				    :0
    Dst  IPv6   Translations				    :0
    Dst  IPv6   Restorations				    :0
    Src  Port   Translations				    :1756790969
    Src  Port   Restorations				    :0
    Dst  Port   Translations				    :0
    Dst  Port   Restorations				    :996153639
    ICMP ID     Translations				    :1025754
    ICMP ID     Restorations				    :496921
    ICMP Error  Translations				    :31264820
    TCP  Port   Translations				    :1828529677
    TCP  Port   Restorations				    :3321986633
    UDP  Port   Translations				    :4223228588
    UDP  Port   Restorations				    :1969134302
    NAT Unexpected Protocol With Port Xlation               :0
    GRE  CallID Translations				    :5001661
    GRE  CallID Restorations				    :0
    GRE  Wrong protocol value                               :0
    SRC IP restored in ICMP Error			    :0
    DST IP restored in ICMP Error			    :28198026
    SRC IP translated in ICMP Error			    :3066794
    DST IP translated in ICMP Error			    :0
    New SRC IP translated in ICMP Error			    :0
    Inner SRC IP restored in ICMP Error 		    :28198026
    Inner SRC port restored in ICMP Error		    :28198014
    Inner DST port restored in ICMP Error		    :0
    Inner DST IP restored in ICMP Error			    :0
    Inner SRC IP translated in ICMP Error		    :3066794
    Inner SRC port translated in ICMP Error		    :3066794
    Inner DST port translated in ICMP Error		    :0
    Inner DST IP translated in ICMP Error		    :0

Misc Errors
    NAT error - no policy                                   :0
    NAT error - IP version                                  :0
    NAT error - xlate free called with null ext             :0
    NAT error - ext free failed                             :0
    NAT error - policy add failed                           :0
    NAT error - policy delete failed                        :0
    NAT error - prefix filter allocation failed             :0
    NAT error - prefix filter name failed                   :0
    NAT error - prefix list create failed                   :0
    NAT error - prefix filter tree add failed               :0

Misc Counters 
    NAT prefix filter created                               :0
    NAT prefix filter changed                               :0
    NAT prefix filter control free                          :0
    NAT prefix filter match                                 :0
    NAT prefix filter no match                              :0
    NAT prefix filter mapping add                           :0
    NAT prefix filter mapping remove                        :0
    NAT prefix filter mapping free                          :0
    NAT prefix filter unsupported IP version                :0
    NAT unsupported layer-4 header for port translation     :0
    NAT unsupported icmp id for port translation            :0

NAT64 Counters
    NAT64 - IP options drop                                 :0
    NAT64 - UDP checksum zero drop                          :0
    NAT64 - Unsupported ICMP type drop                      :0
    NAT64 - Unsupported ICMP code drop                      :0
    NAT64 - Unsupported header drop                         :0
    NAT64 - Unsupported L4 drop                             :0
    NAT64 - MTU exceeded                                    :0
    NAT64 - TTL exceeded                                    :0
    NAT64 - dfbit set                                       :0
    NAT64 - Unsupported ICMP error                          :0
    NAT64 error - mapping ipv4 source                       :0
    NAT64 error - mapping ipv6 destination                  :0
    NAT64 error - MTU exceed build                          :0
    NAT64 error - TTL exceed build                          :0
    NAT64 error - MTU exceed send                           :0
    NAT64 error - TTL exceed send                           :0

Somebody faced a similar degradation of NAT? If so, please,  let me know solution to improve my service.

[#2]

Hello,

JUNOS 13.3 is not recommended for use with MS-MIC/MS-MPC CGNAT.

Please use 14.2R7-S2 or newer, this release has numerous bug fixes specifically for MS-MIC/MS-MPC.

Also, it seems that You have lots of traffic not matching Your NAT rules:

  NAT rule lookup failures				    :241772313

Could be either a mistake in Your prefix-list, or NAT hairpinning, or attacks from internet.

Could take a repeated "show services nat statistics" printout during Your troubles with MS-MIC to see if this counter goes up a lot at the same time?

HTH

Thx

Alex

[#3]

Hi all , small correction, 14.2 is not qualified for subscriber management, so you need 15.1. 

You can try with 15.1R5 , which released few days ago

[#4]

Thank all. I will try to update Junos. Hope this will help.

[#5]

Hello everyone again

JTAC has recomended to use 15.1R5.5 so we stoped on this decision. It inspected our MX80 and didnt find any misconfiguration.

After inspection, It was said that 15.1R5.5 is really best decision to build BRAS.

But in our case migrating to 15.1R5.5 didnt change anything, but brought a new problem. Sometimes (only when more then 500+ clients try to connect via pppoe), MX80 stops to handle connections. Proccess bbe-smgd takes almost all cpu resources. Helps restart of this proccess several times only. On Junos 13.9 this problem doesnt appear.

arseniev, you have said to look on counter "NAT rule lookup failures ". It grows, but is there method to know what kind of failures are exactly?

[#6]

Just an update, BRAS+CGNAT JTAC recommendation is 16.1R5. 16.1R5-S2 will be available soon.

There is a major archietecture change for subcriber management in next generation release.

bbe-smgd now handle most of the subscriber-managment part. For CPU issue, please open JTAC case.

[#7]

New junos version is good news.

But I think if counter "NAT rule lookup failures" is growing I have some misconfiguration or unwanted traffic.

So at first, have to resolve this issue.