Junos
Highlighted
Junos

"show | compare" and "commit" commands taking about 3 minutes to complete

2 weeks ago

We've got an MX5 router on where it takes about 3 minutes do complete its "show | compare" and "commit" commands.

I've traced the commit process on where the slowness is happening.

The config is quite large at the moment with a couple of thousands IFLs, but I would imagine that 3 minutes is a bit too long.

 

marijn@ams-nik-er2# commit | display detail

2020-05-20 14:16:41 CEST: updating commit revision

2020-05-20 14:16:41 CEST: start loading commit script changes

2020-05-20 14:16:41 CEST: no commit script changes

2020-05-20 14:16:41 CEST: no transient commit script changes

2020-05-20 14:16:41 CEST: finished loading commit script changes

2020-05-20 14:16:41 CEST: copying juniper.db to juniper.data+

... 16 seconds ...

2020-05-20 14:16:57 CEST: finished copying juniper.db to juniper.data+

2020-05-20 14:16:57 CEST: exporting juniper.conf

2020-05-20 14:16:59 CEST: expanding interface-ranges

2020-05-20 14:16:59 CEST: finished expanding interface-ranges

2020-05-20 14:16:59 CEST: expanding groups

2020-05-20 14:16:59 CEST: finished expanding groups

2020-05-20 14:16:59 CEST: setup foreign files

2020-05-20 14:16:59 CEST: update license counters

2020-05-20 14:16:59 CEST: finish license counters

2020-05-20 14:16:59 CEST: propagating foreign files

2020-05-20 14:16:59 CEST: cdg returns = 8(Can not execute binary)

... 37 seconds ...

2020-05-20 14:17:37 CEST: complete foreign files

2020-05-20 14:17:38 CEST: dropping unchanged foreign files

2020-05-20 14:17:38 CEST: executing 'ffp propagate'

... 25 seconds ...

2020-05-20 14:18:03 CEST: daemons checking new configuration

2020-05-20 14:18:03 CEST: Routing protocols process checking new configuration

2020-05-20 14:18:05 CEST: Chassis control process checking new configuration

2020-05-20 14:18:06 CEST: Firewall process checking new configuration

2020-05-20 14:18:06 CEST: sending commit-check command to dfwd(1857)

2020-05-20 14:18:08 CEST: Interface control process checking new configuration

2020-05-20 14:18:08 CEST: sending commit-check command to dcd(1924)

2020-05-20 14:18:11 CEST: Simple Network Management Protocol process checking new configuration

2020-05-20 14:18:11 CEST: Management Information Base II process checking new configuration

2020-05-20 14:18:12 CEST: SONET Automatic Protection Switching process checking new configuration

2020-05-20 14:18:12 CEST: Virtual Router Redundancy Protocol process checking new configuration

2020-05-20 14:18:12 CEST: Packet Forwarding Engine statistics management process checking new configuration

2020-05-20 14:18:12 CEST: Remote operations process checking new configuration

2020-05-20 14:18:12 CEST: Class-of-service process checking new configuration

2020-05-20 14:18:13 CEST: Redundancy interface management process checking new configuration

2020-05-20 14:18:13 CEST: Dynamic flow capture service checking new configuration

2020-05-20 14:18:13 CEST: Connectivity fault management process checking new configuration

2020-05-20 14:18:13 CEST: Dynamic Host Configuration Protocol process checking new configuration

2020-05-20 14:18:14 CEST: General authentication process checking new configuration

2020-05-20 14:18:14 CEST: Interface Auto-configuration checking new configuration

2020-05-20 14:18:14 CEST: Ethernet Clock Synchronization Process checking new configuration

2020-05-20 14:18:14 CEST: SCC Chassisd Process checking new configuration

2020-05-20 14:18:14 CEST: Layer 2 address flooding and learning process checking new configuration

2020-05-20 14:18:19 CEST: Layer 2 Control Protocol process checking new configuration

2020-05-20 14:18:19 CEST: commit wrapup...

2020-05-20 14:18:19 CEST: activating '/var/etc/ntp.conf'

2020-05-20 14:18:19 CEST: activating '/var/etc/cosd.conf'

2020-05-20 14:18:19 CEST: start ffp activate

2020-05-20 14:18:19 CEST: executing 'ffp activate'

... 35 seconds ...

2020-05-20 14:18:54 CEST: activating '/var/etc/pam.conf'

2020-05-20 14:18:54 CEST: activating '/var/etc/pam_radius.conf'

2020-05-20 14:18:54 CEST: activating '/var/etc/pam_tacplus.conf'

2020-05-20 14:18:54 CEST: activating '/var/etc/ifinfo.conf'

2020-05-20 14:18:54 CEST: activating '/var/etc/issue'

2020-05-20 14:18:54 CEST: activating '/var/etc/certs'

2020-05-20 14:18:54 CEST: activating '/var/etc/ifmdata.conf'

2020-05-20 14:18:54 CEST: activating '/var/etc/vmm.conf'

2020-05-20 14:18:54 CEST: activating '/var/etc/db_ext-cfg'

2020-05-20 14:18:54 CEST: executing foreign_commands

2020-05-20 14:18:54 CEST: /bin/sh /etc/rc.ui ui_setup_users (sh)

2020-05-20 14:18:54 CEST: not executing ui_commit in rc.ui

2020-05-20 14:18:54 CEST: finish ffp activate

2020-05-20 14:18:54 CEST: copying configuration to juniper.save

2020-05-20 14:19:07 CEST: db_check_constraint_ids_clear start

2020-05-20 14:19:07 CEST: db_check_constraint_ids_clear done

2020-05-20 14:19:07 CEST: db_groups_info_clear start

2020-05-20 14:19:07 CEST: db_groups_info_clear done

2020-05-20 14:19:07 CEST: activating '/var/run/db/juniper.data'

2020-05-20 14:19:07 CEST: Rotate backup configs

2020-05-20 14:19:08 CEST: ssync begins

2020-05-20 14:19:10 CEST: ssync ends

2020-05-20 14:19:10 CEST: notifying daemons of new configuration

2020-05-20 14:19:10 CEST:  notifying none(0)

2020-05-20 14:19:10 CEST: cannot signal 'None process', signal 1, no pid

2020-05-20 14:19:10 CEST:  notifying rpd(2)

2020-05-20 14:19:10 CEST: signaling 'Routing protocols process', pid 1817, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:10 CEST:  notifying chassisd(5)

2020-05-20 14:19:10 CEST: signaling 'Chassis control process', pid 1655, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:10 CEST:  notifying xntpd(6)

2020-05-20 14:19:10 CEST: signaling 'Network time process', pid 1816, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:10 CEST:  notifying dfwd(12)

2020-05-20 14:19:10 CEST: signaling 'Firewall process', pid 1857, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:10 CEST:  notifying dcd(13)

2020-05-20 14:19:10 CEST: signaling 'Interface control process', pid 1924, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:11 CEST:  notifying snmpd(14)

2020-05-20 14:19:11 CEST: signaling 'Simple Network Management Protocol process', pid 36009, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:11 CEST:  notifying mib2d(16)

2020-05-20 14:19:12 CEST: signaling 'Management Information Base II process', pid 36007, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:12 CEST:  notifying apsd(17)

2020-05-20 14:19:12 CEST: signaling 'SONET Automatic Protection Switching process', pid 1819, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:12 CEST:  notifying vrrpd(18)

2020-05-20 14:19:13 CEST: signaling 'Virtual Router Redundancy Protocol process', pid 1664, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:13 CEST:  notifying pfed(20)

2020-05-20 14:19:13 CEST: signaling 'Packet Forwarding Engine statistics management process', pid 1858, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:13 CEST:  notifying rmopd(28)

2020-05-20 14:19:15 CEST: signaling 'Remote operations process', pid 1821, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:15 CEST:  notifying cosd(29)

2020-05-20 14:19:16 CEST: signaling 'Class-of-service process', pid 1822, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:16 CEST:  notifying rdd(41)

2020-05-20 14:19:16 CEST: cannot signal 'Redundancy interface management process', signal 1, no pid

2020-05-20 14:19:16 CEST:  notifying dfcd(43)

2020-05-20 14:19:16 CEST: signaling 'Dynamic flow capture service', pid 1828, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:16 CEST:  notifying cfmd(48)

2020-05-20 14:19:17 CEST: signaling 'Connectivity fault management process', pid 1855, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:17 CEST:  notifying jdhcpd(49)

2020-05-20 14:19:17 CEST: signaling 'Dynamic Host Configuration Protocol process', pid 1832, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:17 CEST:  notifying authd(55)

2020-05-20 14:19:18 CEST: signaling 'General authentication process', pid 1833, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:18 CEST:  notifying autoconfd(59)

2020-05-20 14:19:18 CEST: signaling 'Interface Auto-configuration', pid 1848, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:18 CEST:  notifying transportd(76)

2020-05-20 14:19:18 CEST: cannot signal 'Transport control process', signal 1, no pid

2020-05-20 14:19:18 CEST:  notifying clksyncd(77)

2020-05-20 14:19:18 CEST: signaling 'Ethernet Clock Synchronization Process', pid 1669, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:18 CEST:  notifying l2ald(95)

2020-05-20 14:19:19 CEST: signaling 'Layer 2 address flooding and learning process', pid 1818, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:19 CEST:  notifying l2cpd(97)

2020-05-20 14:19:19 CEST: signaling 'Layer 2 Control Protocol process', pid 1829, signal 1, status 0 with notification errors enabled

2020-05-20 14:19:19 CEST: ssync begins

2020-05-20 14:19:19 CEST: ssync ends

2020-05-20 14:19:20 CEST: commit complete

commit complete

 

10 REPLIES 10
Highlighted
Junos

Re: "show | compare" and "commit" commands taking about 3 minutes to complete

[ Edited ]
2 weeks ago

Hi Beeelzebub,

 

May I know what version you are running on this device? If you are running a very old version, you might consider upgrading to JTAC Recommended version for this device.

 

We can try clearing out some space from the device to see if commits start behaving in a normal way with a reduction in delay time. Does your "show system storage detail" show the reasonable percent of available space?

Please perform the below and try to commit :
>request system storage cleanup dry-run
This command will list you the number of files and folders that are eligible for delete. If you find any file necessary or important from the list, you can keep a backup of those files just in case we might need those for later use. Please verify the list of files thoroughly after taking a backup, run the below command:
>request system storage cleanup
Now, try to perform a commit and make a note of the time taken for the commit to succeed.

Now, please try again to perform a commit of config by running the below and share the output: 

>commit | display detail

 

We might also want to check with respect to CPU consumption. Check the RE CPU in the below command output: 

>show chassis routing-engine

 

Also, check process CPU hogging output: 

>show system processes extensive

 

Hope this helps Smiley Happy

 

Please mark "Accept as Solution" if this helps you.

Kudos are always appreciated!

 

Highlighted
Junos

Re: "show | compare" and "commit" commands taking about 3 minutes to complete

2 weeks ago

Hi,

 

1. What the CPU on RE and whats the system process CPU, collect the following:
show chassis routing-engine
show system processes extensive

show configuration system services

show configuration chassis

 

 

 

2. Do you have large number traceoption enabled? if so, disable them:
show configuration | match trace |display set
 

3. clean-up the system file:
request system storage cleanup

 

and Set followings knob to impove  your commit performance:


set system commit persist-groups-inheritance
set system commit delta-export

 

 

/Karan Dhanak
Highlighted
Junos

Re: "show | compare" and "commit" commands taking about 3 minutes to complete

2 weeks ago

Hi Beeelzebub, 

 

This is expected if you have a lot of configuration being committed as the this is MX 5 with less scale.

 

Also if you are configuring interface information then, during commit all the interface description and other details are being propagated to /var/etc/ifinfo.conf file. Propagating this file may be taking time, and that may be reason why you see the time getting worse as the number of interfaces grow.

 

Refer this thread: Extremely long commit times on MX240/MX480  

 

Please mark "Accept as solution" if this answers your query. 

 

Kudos are appreciated too! 

 

Regards, 

Sharat Ainapur

Highlighted
Junos

Re: "show | compare" and "commit" commands taking about 3 minutes to complete

2 weeks ago

Hello,

 

The version we are running is quite old: 14.2R7-S9

We are planning to upgrade, but that would not be fore a couple of months.

Other devices are unning the same version and are planned to be upgraded in the near future, but these do not take very long to commit. Those devices have a way less IFL's configured.

 

I issued the the 'request system storage clean-up' command and it deleted some log files, but not that many.

The commit times after that were significantly faster, like under 1 minute, but later on it went back to 3 minutes.

I immediately checked the 'chassis routing-engine' commands and it was on 70% for User. 

The process that takes up the most CPU % was 'mgd' and later on 'mib2d'.

Highlighted
Junos

Re: "show | compare" and "commit" commands taking about 3 minutes to complete

2 weeks ago

'show chassis routing-engine' : usually idle for the most part, sometimes high on 'User'

'show system processes extensive | except 0.00' : usually idle for the most part, sometimes high on 'mgd' and 'mib2d' processes

'show configuration system services' : see below

'show configuration chassis' : see below

 

2. Do you have large number traceoption enabled? if so, disable them:
show configuration | match trace |display set

A: No, I don't.

 

3. clean-up the system file:
request system storage cleanup

A: Already done, no change.

 

4.

set system commit persist-groups-inheritance
set system commit delta-export

A: Are these safe to commit? I read about 'persist-groups-inheritance'  --> This can improve commit time performance. However, it can also increase the database size.

 

 

beelze@ams-nik-er2> show configuration system services 
ssh {
    protocol-version v2;
}
netconf {
    ssh;
}
dhcp-local-server {
    authentication {
        username-include {
            user-prefix core-nkh-02;
            interface-name;
        }
    }
    group local {
        interface ge-1/1/3.0;
    }
    group local-eurofiber {
        interface ge-1/1/4.0;
    }
    group local-mikrotik-test {
        interface ge-1/0/0.0;
    }
    group local-tele2-wba-test {
        authentication {
            username-include {
                user-prefix core-nkh-02;
                option-82;
                interface-name;
            }
        }
        interface ge-1/1/6.0;
    }
    group local-tele2-VT158463 {
        interface xe-1/3/0.0;
    }
}

beelze@ams-nik-er2> show configuration chassis 
aggregated-devices {
    ethernet {
        device-count 12;
    }
}
alarm {
    management-ethernet {
        link-down ignore;
    }
}
network-services enhanced-ip;

 

 

Highlighted
Junos

Re: "show | compare" and "commit" commands taking about 3 minutes to complete

2 weeks ago

Hi Beeelzebub,

 

As per what I understand in your situation, I would ideally recommend you to check with the commit issue after you have upgraded to the latest recommended release. And if you are still seeing issues, you can try clean up dry run again and check if that works for you. 

 

You can refer the below KB for CPU troubleshooting on RE:

https://kb.juniper.net/InfoCenter/index?page=content&id=KB26261

 

In situations when you see mgd process, you can refer the below link:

https://kb.juniper.net/InfoCenter/index?page=content&id=KB25382&actp=METADATA

 

Hope this helps Smiley Happy

 

Please mark "Accept as Solution" for this post if this serves your purpose.

Kudos are always appreciated!

 

 

Highlighted
Junos

Re: "show | compare" and "commit" commands taking about 3 minutes to complete

2 weeks ago

Hi Beeelzebub, 

 

As I already mentioned this behaviour is directly linked to the interface configuration. The mgd process is responsible for the communication between the other processes and an interface to the configuration database.

 

I would say you can try using interface-range or interface-set and group the interface with similar properties and apply configuration. Also this helps reduce the time and effort in configuring interfaces and configurations common to all the interfaces can be included in the interface range definition. This might improve the commit performance. 

 

Link : Interface Ranges and Configuring Interface Sets 

 

Hope this helps, Please mark "Accept as solution" if this answers your query. 

Kudos are appreciated too! 

 

Regards, 

Sharat Ainapur

Highlighted
Junos

Re: "show | compare" and "commit" commands taking about 3 minutes to complete

2 weeks ago

Amount of descriptions:

 

beelze@ams-nik-er2# show | match description | count

Count: 221 lines

 

What other details are propagated to the/var/etc/ifinfo.conf file?

Highlighted
Junos

Re: "show | compare" and "commit" commands taking about 3 minutes to complete

2 weeks ago

Is "I issued the the 'request system storage clean-up' command and it deleted some log files, but not that many.

The commit times after that were significantly faster, like under 1 minute, " reproducible? If so it might be a file-system / disk issue and not so much driven by the CPU resources available.  Ulf

--
If this worked for you please flag my post as an 'Accepted Solution' so others can benefit. A kudo would be cool if you think I earned it.
Highlighted
Junos

Re: "show | compare" and "commit" commands taking about 3 minutes to complete

2 weeks ago

Hi Beeelzebub

 

The ifinfo basically has all the interface related properties like the mtu,speed,status,interface-stats etc.

If you want to understand more on this, do try : 

> start shell user root
# ifinfo xe-0/1/0

Please mark "Accept as solution" if this answers your query. 

 

Kudos are appreciated too! 

 

Regards, 

Sharat Ainapur