Ethernet Switching
Ethernet Switching

DHCP issue on EX

[ Edited ]
2 weeks ago

Hi all,

I read the this article:

https://kb.juniper.net/InfoCenter/index?page=content&id=KB23335&actp=METADATA

But I like to ask you guys maybe experienced in the same situation...

 

DHCP server and Clients are on the same VC consist of 9 4200ex series. There are 11 L3 vlans with different subnets including WiFi.... Each L3 Vlan has various number of clients. 

topology is: 

Servers<----Local traffic-----4200ex VC--------->Firewall--------->Internet...

But there is a predicament issue that time to time the same things are occuring....

 

One subnet's clients are experiencing no LAN connectivity as some WiFi users are unable to connect to the Wifi. This is happening at the same time.

When this is happening another clients from different subnet are starting to have connectivity LAN issue where they are directly connected to.

 

During the event of this behavours, internet access from the Firewall is fine. No issue. how about from VC switch? There is significant latency happening.

 How about from clients?

From non-complaining clients initially are fine, but when time goes, these clients are unable to connect to Lan.

 

Accedently when DHCP process on the VC is restarted, all users are reporting no-more issue....

 

During the issue I couldn't find any significant log messages/chassisd. and resource utilization etc...

 

To find/identify the root cause, what troublehoting should be quickly done? 

 

 

Thanks

Ar.

 

 

 

 

 

12 REPLIES 12
Ethernet Switching

Re: DHCP issue on EX

2 weeks ago

Hi Arix,

 

Those kind of issues sounds like a bug, memory or corruption.

 

Some troubleshooting steps.

 

- Enable traceoptions.

set system processes dhcp-service traceoptions file dhcp-logs

set system processes dhcp-service traceoptions flag all

 

- Clear statistics and check which packets are incrementing when DHCP is working fine.  Then check again when the issue is happening and compare the outputs.  That will let you know if the issue with discover packets that are not reaching the server or if the server is not sending the offer or also if the client is not sending the request.

 

clear dhcp server statistics

show dhcp server statistics

 

- Check for core-dumps

set system core-dumps

 

- Check CPU utilization and see if any process is spiking.

set system processes extensive all-members | except 0.00

 

 

Randall

Ethernet Switching

Re: DHCP issue on EX

2 weeks ago

Hi Randero,

Thanks for help...

 

I read your post ....

 

When this issue occurs, it is very stresfull as many clients suddenly are unable to access the network resources... It also takes for while to finish troubleshooting and then analysis case and find actual root cause, as the issue itself disapears sometimes... --LAst time when restarting the dhcp, issue stopped...

 

1-) In order to be ready for very quickly doing troubleshooting, is it possible to provide a list of cli command including shell cli. Only for dhcp issue.

2-) When doing traceoptions on dhcp, what exact thing should I be looking for traceoption file?

 

I want to make a specifically framework for this issue... 

Thanks

 

A.

Ethernet Switching

Re: DHCP issue on EX

2 weeks ago

I'm not sure on the details of the issue.  

 

Are the clients first reporting problems only the newly connecting clients looking for dhcp addresses?

And then the existing clients also start to be impacted?

 

or

 

do the initial reports start with the connected already clients?

 

Are the vlan subnet gateways configured on the ex VC or on the upstream firewall?

If they are on the upstream firewall how is the dhcp forwarding configured on the ex vc stack?

 

Steve Puluka BSEET - Juniper Ambassador
IP Architect - DQE Communications Pittsburgh, PA (Metro Ethernet & ISP)
http://puluka.com/home
Ethernet Switching

Re: DHCP issue on EX

2 weeks ago

Hi,

Thanks for reply.

 

Are the clients first reporting problems only the newly connecting clients looking for dhcp addresses?

And then the existing clients also start to be impacted?

Answer: There is no newly connecting clients...All  clients are already existing. The issue is reported by exiting clients in different subnets... 

 

or

 

do the initial reports start with the connected already clients?

    Answer: Yes, initial reports are coming from already connected clients. There is no new client.

Are the vlan subnet gateways configured on the ex VC or on the upstream firewall?

    Answer: On the ex VC.

If they are on the upstream firewall how is the dhcp forwarding configured on the ex vc stack?

Answer: There is no any dhcp service on the firewall. Dhcp service is only on the only ex VC and all clients' GW(s) are sit on the ex VC as well.

Ethernet Switching

Re: DHCP issue on EX

2 weeks ago

It sounds to me that you're simply running out of leases. 

 

show dhcp server binding interface vlan.xxx summary will give you a count of leases that you can compare against the pool size.

Ethernet Switching

Re: DHCP issue on EX

2 weeks ago

Couple of additional questions:

 

1.  In which VLAN is DHCP Server located, and can I assume all other VLAN/Subnets have DHCP Relay configured?  Maybe send the VC configuration, . . .

 

2.  Do you have any port-level Security features enabled, like DHCP-Snooping, IP Source Guard, Dynamic ARP Inspection, etc.  Again configuration would tell this.

Ethernet Switching

Re: DHCP issue on EX

2 weeks ago

Addiotional FYI, to use Trace Options these need to be enable prior to the situation, not after.  It is recommended to not leave this running during normal operation, but only during troubleshooting, as these use extra system resources.

 

When the situation occurs, do the affect clients have an IP address and correct Gateway?  Do you know what your DHCP refresh interval is set to on DHCP Server?

Ethernet Switching

Re: DHCP issue on EX

2 weeks ago

Thanks for the updates.

 

Since the issue surfaces with existing connected clients this will likely be one of two scenarios.

1-software bug during the renew process or a memory leak type issue

What is the junos version?

 

2-some interaction with dhcp or port security

Do you have any of these enabled

dhcp snooping

802.1x

IP source guard

arp inspection

 

 

Steve Puluka BSEET - Juniper Ambassador
IP Architect - DQE Communications Pittsburgh, PA (Metro Ethernet & ISP)
http://puluka.com/home
Ethernet Switching

Re: DHCP issue on EX

[ Edited ]
2 weeks ago

Hi all,

Thanks replies:

 

questions from Spuluka:

1-software bug during the renew process or a memory leak type issue

What is the junos version?
        Answer: 4200ex, version: 15.1R6.7

2-some interaction with dhcp or port security

Do you have any of these enabled
dhcp snooping
802.1x
IP source guard
arp inspection

  Answers:
      > show configuration | display set | match snooping---Yes, there is.
            set protocols igmp-snooping vlan all

      >show configuration | display set | match 802.1x ---No conf.
      >show configuration | display set | match "guard"------No guard conf.
      >show configuration | display set | match "arp-inspection" ----No inspection config.

============================================

> show system processes extensive | except 0.00
last pid: 6078; load averages: 0.59, 0.51, 0.52 up 288+14:16:31 08:01:39
153 processes: 5 running, 127 sleeping, 21 waiting

Mem: 577M Active, 99M Inact, 153M Wired, 69M Cache, 109M Buf, 79M Free
Swap:


PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
10 root 1 155 52 0K 16K RUN 4458.7 49.71% idle
1299 root 2 -52 -52 77028K 22092K select 767.6H 16.36% sfid
1330 root 1 4 0 58688K 42880K pfesta 383.8H 12.45% mib2d
1301 root 1 55 0 87884K 37588K RUN 458.3H 6.05% pfem
41 root 1 12 0 0K 16K sleep 96.5H 1.86% netdaemon
1298 root 1 4 0 95520K 15208K kqread 193.4H 0.34% chassism
1336 root 1 42 0 22888K 15072K select 120.6H 0.05% ppmd
931 root 1 42 0 16372K 9228K RUN 39.9H 0.05% eventd
20 root 1 -84 -187 0K 16K WAIT 30.7H 0.05% mpfe_drv_taskq16: +
===========================================================

 

 

Ethernet Switching

Re: DHCP issue on EX

2 weeks ago

Could you please try using 12.3R12-S14 instead of 15.1?

Ethernet Switching

Re: DHCP issue on EX

2 weeks ago

1-) In order to be ready for very quickly doing troubleshooting, is it possible to provide a list of cli command including shell cli. Only for dhcp issue.

 

>monitor traffic interface <client interface> no-resolve size 1500 matching "port 67||68"

 

This is to monitor the DORA process.  Here you can see if the conversation is complete or which packets are misssing.

 

 

2-) When doing traceoptions on dhcp, what exact thing should I be looking for traceoption file?

Traceoptions is a huge file, but you can trace the DORA process and if some packet is not seen, you can see the reason of the error of a timeout.

 

By the way, how you noticed any core-dump generated during the issue.  This sounds like a memory leak, it should be good a format install, an upgrade or a reboot of all members at least.

 

Ethernet Switching

Re: DHCP issue on EX

a week ago

Thanks for those who replied... I took your points...

 

Can I further ask, about any other ideas please? really appreciated...

 

Thx,

A