Ethernet Switching
Highlighted
Ethernet Switching

finding a traceoptions file that created previously on EX 4600

a week ago

hi all,

I am trying to find  a traceoptions file that my colleague created for troubleshooting purpose.  How can be found this file via Junos CLI?

>file list / detail ----->didn't help!

 

Thanks

6 REPLIES 6
Ethernet Switching

Re: finding a traceoptions file that created previously on EX 4600

[ Edited ]
a week ago

Hello Erdix,

 

If the traceoptions are still enabled, "show configuration | display set | match traceoptions" will give you the filename. This file will be created in /var/log/ directory.

For example:

root@mx> show configuration | display set | match traceoptions
set protocols isis traceoptions file isis.log

root@mx> file list /var/log/isis* detail
-rw-r----- 1 root wheel 65 Mar 14 23:21 /var/log/isis.log

If these traceoptions are not configured anymore, you can check system commit history to identify the file name:

root@mx> show system commit
0 2019-03-14 13:21:11 CET by root via cli
1 2019-03-11 10:27:41 CET by regress via cli

root@mx> show system rollback compare 1 0
[edit protocols isis]
+ traceoptions {
+ file isis.log;
+ }

Best regards,

Sergii

----------------------------------------------------------------------------------------

Please mark this post as "Accepted solution" if your problem is resolved Smiley Wink

----------------------------------------------------------------------------------------

Ethernet Switching

Re: finding a traceoptions file that created previously on EX 4600

a week ago

Hi Sergii,

Thanks for reply.

In line with your directions, I did further look at the case...

 

Master> show configuration system | display set | match traceoptions

set system processes dhcp-service traceoptions file dihescipi_logfile
set system processes dhcp-service traceoptions file size 10m
set system processes dhcp-service traceoptions level all
set system processes dhcp-service traceoptions flag all
set system processes app-engine-virtual-machine-management-service traceoptions level notice
set system processes app-engine-virtual-machine-management-service traceoptions flag all

 

Master> file list /var/log/*_log* detail
-rw-r----- 1 root wheel 7170715 Mar 15 13:07 /var/log/dihescipi_logfile
-rw-r----- 1 root wheel 796877 Mar 15 13:06 /var/log/dihescipi_logfile.0.gz
-rw-r----- 1 root wheel 787786 Mar 15 13:05 /var/log/dihescipi_logfile.1.gz

 

I have some qs to you:

Q1) Clearly this traceoption is still enabled.  How to make sure this traceoption is not working for any automation purpose before making it deactivated or delete.

Q2) I don't think so this file was created today (Mar 15) as last commited date 5 month ago but not related to this file at that time. So how to find when this traceoption file was created and who? (my applogy this file was not created previously! We don't know when and who created).  ">sh sys rollback compara 0 1" with this command, each time needs to performed until  commit 49 to see change conf that related to this traceoption file. This way is really spending waste of time. Is there a efficient way or scripting to figure out the case? If so, can you explain pls.

Q3) As this traceoption is still running, why we are not seeing any warning or error that says storage exceeding volume or many creating files ext as we are seeing one of the Juniper strong recommendations is disabling traceoption when it is not using or capturing the packet. 

 

Thx.

Ethernet Switching

Re: finding a traceoptions file that created previously on EX 4600

[ Edited ]
a week ago

Hi Erdix,

 

Please see my answers inline.


@Erdix wrote:

Q1) Clearly this traceoption is still enabled.  How to make sure this traceoption is not working for any automation purpose before making it deactivated or delete.

[Sergii] The sole purpose of traceoptions is to give you more insight into what the process is doing, e.g. troubleshooting. Traceoptions are not used for anything else in JUNOS, so please feel free to deactivate or delete these lines.  In your case you should use:

delete processes dhcp-service traceoptions
delete system processes app-engine-virtual-machine-management-service traceoptions 

Q2) I don't think so this file was created today (Mar 15) as last commited date 5 month ago but not related to this file at that time. So how to find when this traceoption file was created and who? (my applogy this file was not created previously! We don't know when and who created).  ">sh sys rollback compara 0 1" with this command, each time needs to performed until  commit 49 to see change conf that related to this traceoption file. This way is really spending waste of time. Is there a efficient way or scripting to figure out the case? If so, can you explain pls.

[Sergii] If the process is idle, or not used in your configuration, it will not write anything into the trace logs. AFAIK there is no CLI command that can provide you with all configuration changes performed during last 50 commits, but a simple shell script will easily do it (I'll provide it below).

Q3) As this traceoption is still running, why we are not seeing any warning or error that says storage exceeding volume or many creating files ext as we are seeing one of the Juniper strong recommendations is disabling traceoption when it is not using or capturing the packet. 

[Sergii] Unfortunately, there is no such warning in JUNOS. I fully agree with the recommendation to disable traceoptions during normal operations because they might interfere with normal operation of the kernel and daemons (depending on the configuration). For example, you might see adjacencies timing out if you configure verbose trace logs for RPD on PE router with millions of routes, because you will force RPD to write all its activity to the disk which will slow it down. Traceoptions should only be enabled during troubleshooting and disabled when they're not needed anymore. Of course, another risk is that all available space will be consumed overtime (but most traceoptions allow you to configure number and size of log files).


This is the simple script that should do the job:

root@mx1:/var/tmp # cat chk.sh
#!/bin/bash
i=49
echo "================================"
echo "Checking system commit history"
echo "================================"
cli -c "show system commit | no-more"
while [ $i -ge 1 ]
do
    j=$(( i - 1 ))
    echo "================================"
    echo "Comparing commit #${i} and #${j}"
    echo "================================"
    cli -c "show system rollback compare ${i} ${j} | no-more | except \"Cannot open configuration file\""
    i=$j
done

root@mx1:/var/tmp # sh chk.sh
================================
Checking system commit history
================================
0   2019-03-14 13:21:11 CET by root via cli
1   2019-03-11 10:27:41 CET by regress via cli
================================
Comparing commit #49 and #48
================================
[...]
================================
Comparing commit #1 and #0
================================
[edit protocols isis]
+    traceoptions {
+        file isis.log;
+    }
root@mx1:/var/tmp #

Best regards,

Sergii

----------------------------------------------------------------------------------------

Please mark this post as "Accepted solution" if your problem is resolved Smiley Wink

----------------------------------------------------------------------------------------

Ethernet Switching

Re: finding a traceoptions file that created previously on EX 4600

Sunday

Hi Sergii,

Thanks for answers...I got your point.

 

Another thing that I like to ask is about your previous script is for on the box, isn't it? I like to do same thing via pyhton script remotely like what your shell script does. For instance, when running a pyhton script on my laptop, first it should be connected to the box (Virtual Chassis) over SSH/netconf and executing the commands and bringing all output into my laptop? Is this practible? If so,  could I please ask you about you have a working simlple pyhton script for achieving this task?

 

Thanks, 

 

Ethernet Switching

Re: finding a traceoptions file that created previously on EX 4600

Sunday

Hello Erdix,

 

Yes, this script is supposed to be run on the box, and, unfortunately, I don't have a Python script for your task handy. I hope that your original problem is resolved, and I'll recommend asking your question regarding the Python script in the Automation section. It's more appropriate place for such questions and scripts.

 

Best regards,

Sergii

----------------------------------------------------------------------------------------

Please mark this post as "Accepted solution" if your problem is resolved Smiley Wink

----------------------------------------------------------------------------------------

Ethernet Switching

Re: finding a traceoptions file that created previously on EX 4600

[ Edited ]
Monday

Hi Erdix.

 


@Erdix wrote:

Hi Sergii,

Thanks for answers...I got your point.

 

Another thing that I like to ask is about your previous script is for on the box, isn't it? I like to do same thing via pyhton script remotely like what your shell script does. For instance, when running a pyhton script on my laptop, first it should be connected to the box (Virtual Chassis) over SSH/netconf and executing the commands and bringing all output into my laptop? Is this practible? If so,  could I please ask you about you have a working simlple pyhton script for achieving this task?

 

Thanks, 

 


I have transformed the Sergii's shell script to Python script. See the code snippet below. The same is attached in ZIP file.

 

mat@ubuntu-autom:~/python$ cat chk.py
#!/usr/bin/python3
# import needed libraries
import sys
import re
from getpass import getpass
from jnpr.junos import Device
from jnpr.junos.exception import RpcError
from lxml import etree

# Get device details from user
hostname = input("Device hostname: ")
username = input("Device username: ")
password = getpass("Device password: ")
rollbacks = input("Starting rollback: ")

# initialize rollback number index with input from user
i=int(rollbacks)
print("==============================")
print("Checking system commit history")
print("==============================")
# create new device instance
dev=Device(host=hostname, user=username, passwd=password)
#open connection to device
dev.open()
# get information about commits "show system commit" via RPC and convert to string
commit_info=etree.tostring(dev.rpc.get_commit_information({'format':'text'}), encoding='unicode')
# trim unneeded tags and print the commit information
print(re.sub('<output>\n|\n</output>', "", commit_info))
# cycle through rollbacks until we compare rollback 1 and 0
while i >= 1:
   #secondary index
   j=i-1
   # print infor about rollback versions we want to compare at this iteration step
   print("==============================")
   print("Comparing commit #%d and #%d" % (i, j))
   print("==============================")
   # try to get rollback compare , try is to catch error when user inputs rollback value > 49
   try:
      #  get "show system rollback compare i j" via RPC as type "lxml.etree._Element"
      cmpr=dev.rpc.get_rollback_information({'format':'text'}, compare=str(i), rollback=str(j))
      # if rollback number is <= 49  but doesn't exist on device, the bool value is returned
      if isinstance(cmpr, bool):
         # inform user the rollback #i doesn't exist
         print("Rollback number #%d does not exist." % i)
      # else we have valid output from RPC
      else:
         # convert the output to string
         cmpr_str=etree.tostring(cmpr,encoding='unicode')
         #trim unneeded tags and print the rollback difference information
         print(re.sub(r'<rpc.*\">\n|</rpc-reply>', "", cmpr_str))
   # error caught - rollback number was >  49
   except RpcError as err:
      # print error message
      print ("Unable to retrieve rollback compare: {0}".format(err))
   # decrease rollback index
   i=i-1
# close connection to device
dev.close()

 

I tested script using Python3 on Ubuntu VM against QFX5100 Virtual Chassis  and SRX100 single node . The ouput from QFX5100 VC is following ( output truncated):

mat@ubuntu-autom:~/python$ python3 chk.py
Device hostname: QFX
Device username: mat
Device password:
Starting rollback: 49
==============================
Checking system commit history
==============================
0   2019-03-04 11:33:47 UTC by mat via cli commit synchronize
1   2019-03-04 11:32:22 UTC by mat via cli commit synchronize
2   2019-03-04 11:11:14 UTC by mat via cli commit synchronize
<omitted for brevity>
==============================
Comparing commit #49 and #48
==============================
[edit interfaces]
+   xe-1/0/3 {
+       flexible-vlan-tagging;
+       encapsulation extended-vlan-bridge;
+       unit 400 {
+           vlan-id 400;
+       }
+   }
[edit vlans 351869bd-e8b9-36ad-9e30-90bd173f600d]
+    interface xe-1/0/3.400;

<omitted for brevity>
==============================
Comparing commit #1 and #0
==============================
[edit interfaces ge-1/0/36 unit 0 family inet]
+       filter {
+           input VYPADEK-BKP;
+           output VYPADEK-BKP;
+       }

 

There is built-in some elementary error handling related to rollback number in the script.

 

OS and library versions:  Ubuntu 18.04.1 LTS, Python 3.6.7,  PyEZ 2.2.0 , lxml 4.3.2

 

Many thanks at @Sergii for initial script and all the replies.


Let us know if it helped.

Regards
Luděk Matoušek
JNCIS-ENT, JNCIS-SP, JNCIP-SEC, JNCIA-DevOps

Attachments