Network OS
Network OS

commit comment "Why is this taking so long?"

by Juniper Employee on ‎05-19-2017 03:19 PM

We have all suffered through the wait time of committing configurations. It doesn’t matter which OS or hardware platform, we all despise waiting for five minutes or MORE only to end up receiving some sort of error. I believe Einstein’s theory of relativity covers how commit times are perceived longer during those midnight maintenance windows in production… or I could be misinterpreting time dilation.

 

The good news on this topic is that Juniper has been continuously working on ways to improve the commit times of configurations.  The catch is that you will need to upgrade to get these features as they have been trickling into the code for the previous five years. 

 

IMAGE 1: Commit and Configuration Visual

Commit.png

The commit above may be done via CLI or NETCONF.  The first step within the commit, the commit check process, invokes all underlaying infrastructure processes to validate the configuration.  Each process has the power to reject the configuration.  Once all processes pass the "check", the Junos OS continues to process the candidate configuration into the active configuration.  The commit check is the most time consuming part of the commit and is the focus of this article. If one wants to bypass the commit check entirely, please learn more about JET and fast programmatic configuration

 

COMMIT OPTIMIZATION 1: FAST SYNCHRONIZE

Command: set system commit fast-synchronize

Release: Junos OS 12.2

Summary: This command allows commits to run in parallel on the master and backup routing engine. Recommended use when the REs have almost identical configurations.

 

COMMIT OPTIMIZATION 2: GROUPS INHERITANCE

Command: set system commit persist-groups-inheritance

Release: Junos OS 13.2

Summary: Full inheritance paths of config groups will be built into the internal database so commit times are decreased when wildcards are used in config groups.

 

COMMIT OPTIMIZATION 3: DAEMON VALIDATION WITH DELTAS

Default behavior. No command.

Release: Junos OS 13.3

Summary: MGD (Management daemon) passes configurations to the various other daemons within the Junos OS and they individually perform validations on the delta configurations before reporting back to MGD.

 

COMMIT OPTIMIZATION 4: PRIVATE COMMIT

Default behavior. Command to turn off: set system commit private no-fast-diff

Release: Junos OS 13.3

Summary: Private commit was optimized by including the use of binary databases. Now a patch is generated and applied to the shared database at commit.

 

COMMIT OPTIMIZATION 5: juniper.conf TEXT FILE CREATION

Command: set system commit delta-export

Release: Junos OS 14.2

Summary: Only the delta in the candidate configuration, rather than the entire candidate configuration, are exported.   

 

COMMIT OPTIMIZATION 6: SYNC BOTH ROUTING ENGINES

Default behavior. Command to turn off: set system commit no-delta-synchronize

Release: Junos OS 14.2

Summary: For both commit synchronize and commit fast-synchronize, the master routing engine generates only the delta to synchronize across both REs.

 

COMMIT OPTIMIZATION 7: CONSTRAINT VALIDATION

Default behavior. Command to turn off: set system commit no-delta-constraints

Release: Junos OS 14.2

Summary: Related to commit optimization 2, groups inheritance. Once persist-groups is configured in 14.2 and forward, the check is performed on the delta which have constraints instead of the entire configuration.  

 

COMMIT OPTIMIZATION 8: DELTA OF GROUP INHERITANCE

Default behavior. Command to turn off: set system commit no-delta-build-persist-groups

Release: Junos OS 14.2

Summary: Also related to commit optimization 2, groups inheritance. At commit time, only the changed objects would need to be updated in the database.

 

COMMIT OPTIMIZATION 9: MULTI-PROCESSING DAEMON VALIDATION

Default behavior. Command to turn off: set system commit no-parallel-daemon-validation

Release: Junos OS 15.1

Summary: If one runs an image with FreeBSD 10.X, then SMP (symmetric multi-processing) allows the leveraging of multiple cores to validate sections of the configuration in parallel.

 

COMMIT OPTIMIZATION 10: FILE PROPAGATION

Default behavior. No command.

Release: Junos OS 16.1

Summary: Junos performs a FFP (foreign file propagation) on the configuration delta as opposed to the entire configuration.

 

The most troublesome configurations with respect to commit times are BNG configurations.  The chart below takes a sample configuration and shows the difference our Juniper test engineers saw. 

 

IMAGE 2: Time Improvements with a BNG ConfigurationImprovements.png

 

Beyond being annoyed by the wait, commit times are increasingly important as we move forward with automation and event-driven networks. We will continue to find ways to improve the Junos OS and commit times are just one example of what we are doing here at Juniper. 

 

WARNING: Virtual devices lag behind our hardware devices in these "commit" improvements.  However, we are working on that, as well. 

'
Comments
May 20, 2017
mountainrescuer

Hey @jgarrison this is a great one-stop shop for info around the commit times, and breaking down the various functions. You are not wrong, the time when performing a commit is at least doubled when working in a maintenance window, or when you have been called out ;)

 

Great post,

Graham

Jun 2, 2017
Duaneo

Fast/er commit times are nice, but we're seeing control plane flapping (BFD) during commits now on 15.1X code (4G SRX) that we don't see on our 12.1 or 12.3x (2G SRX) boxes..   ;-(

Jun 5, 2017
Juniper Employee

Hi Duaneo,

 

Some enhancements related to BFD are added in next SRX release (15.1X49-D100) and will fix this behavior. This release will be availble end of this month.

Regards,

/Fabien

Jun 6, 2017

This is awesome, Jessica!

Feb 9, 2018
Nehul Patel

Thanks for sharing it !!

Feb 11, 2018
mountainrescuer

Hey @Duaneo did you see any imporvements to your SRX after running D100 or later?

Feb 12, 2018
Duaneo

Hi @mountainrescuer,

 

For us w the srx550M (4G model), we have the ability to pull a core from the vPFE and dedicate it to "control plane" ( really ppmd and/or bfdd - BGP runs in the other control plane core) and in the future maybe LACP and some other bits.   While this is working for the control plane flapping, the price was losing 1 of the 5 cores bundled together for the vPFE so we've lost a fair bit of forwarding performance.  

 

We need 4 bfd sessions (3x500ms) and i was really hoping for something like they made available for the smaller (less core SRX boxes), but on the 550, we only got the pull a core from the vPFE option.   We're testing the new code now (550m got the improvement in d110) to verify the NAT/PAT tput at our pkt sizes (market data).  We're hoping we can achieve the forwarding performance we need (wo the core) so we don't have to impliment rules about when to deploy the 2G and 4G boxes.   Below are the two options available for the SRX branch line.  

 

  • To enable dedicated BFD on SRX240, SRX550, SRX650, SRX340, SRX345, SRX550M, and SRX1500 devices, use the set chassis dedicated-ukern-cpu command.
  • To enable real-time BFD on SRX100, SRX110, SRX210, SRX220, SRX300, and SRX320 devices, use the set chassis realtime-ukern-thread command.

I'm still working to see if we can get both of the options above for the 550M boxes we're using, which hopefully will get us similar tput without any control plane instability.

Jul 12, 2018
bvisaria

how do we verify and make sure delta-export and persist-groups-inheritance are working correctly. Is there a way to make sure apart from log in "commit | dispay detail" stating "using delta export to export juniper.conf" ?