Ethernet Switching
Highlighted
Ethernet Switching

EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

‎01-06-2020 07:17 AM

Hello,

 

I'm working with a few EX3400-48P that are exhibiting some odd behavior around PoE. Every few hours, one of the poe devices (there doesn't look to be any pattern to which device, it seems evenly distributed) will reboot indicating a momentary loss of PoE. At the same time, the following message appears in the syslog:

 

fpc0 PoE: Failed to read voltage measurements

My searching hasn't found what that message means so I'm at a bit of a loss to where to look for a resolution. This is happening on multiple switches in the facility. I have already updated the PoE controller. Every port/switch is well below the PoE budget.

 

root@switch1> show chassis firmware detail 
FPC 0                     
   PoE firmware            2.1.1.19.3
   Boot Firmware          
      U-Boot               U-Boot 2016.01-rc1 (Sep 01 2016 - 16:00:13 -0700)  1.3.0
   Boot Firmware          
      loader                       FreeBSD/armv6 U-Boot loader 1.2
FPC 1                     
   Boot Firmware          
      U-Boot               U-Boot 2016.01-rc1 (Sep 01 2016 - 16:00:13 -0700)  1.3.0
   Boot Firmware          
      loader                       FreeBSD/armv6 U-Boot loader 1.2
root@switch1> show version 
fpc0:
--------------------------------------------------------------------------
Hostname: switch1
Model: ex3400-48p
Junos: 18.2R3.4
JUNOS OS Kernel 32-bit  [20190605.30b921f_builder_stable_11]
JUNOS OS libs [20190605.30b921f_builder_stable_11]
JUNOS OS runtime [20190605.30b921f_builder_stable_11]
JUNOS OS time zone information [20190605.30b921f_builder_stable_11]
JUNOS py extensions [20190620.215446_builder_junos_182_r3]
JUNOS py base [20190620.215446_builder_junos_182_r3]
JUNOS OS crypto [20190605.30b921f_builder_stable_11]
JUNOS network stack and utilities [20190620.215446_builder_junos_182_r3]
JUNOS libs [20190620.215446_builder_junos_182_r3]
JUNOS runtime [20190620.215446_builder_junos_182_r3]
JUNOS Web Management Platform Package [20190620.215446_builder_junos_182_r3]
JUNOS ex libs [20190620.215446_builder_junos_182_r3]
JUNOS ex runtime [20190620.215446_builder_junos_182_r3]
JUNOS ex platform support [20190620.215446_builder_junos_182_r3]
JUNOS dcp network modules [20190620.215446_builder_junos_182_r3]
JUNOS modules [20190620.215446_builder_junos_182_r3]
JUNOS ex modules [20190620.215446_builder_junos_182_r3]
JUNOS ex Data Plane Crypto Support [20190620.215446_builder_junos_182_r3]
JUNOS daemons [20190620.215446_builder_junos_182_r3]
JUNOS SDN Software Suite [20190620.215446_builder_junos_182_r3]
JUNOS Extension Toolkit [20190620.215446_builder_junos_182_r3]
JUNOS Phone-home [20190620.215446_builder_junos_182_r3]
JUNOS Packet Forwarding Engine Support (EX34XX) [20190620.215446_builder_junos_182_r3]
JUNOS jdocs ex [20190620.215446_builder_junos_182_r3]
JUNOS jail runtime [20190605.30b921f_builder_stable_11]
JUNOS FIPS mode utilities [20190620.215446_builder_junos_182_r3]

fpc1:
--------------------------------------------------------------------------
Hostname: switch1
Model: ex3400-48t
Junos: 18.2R3.4
JUNOS OS Kernel 32-bit  [20190605.30b921f_builder_stable_11]
JUNOS OS libs [20190605.30b921f_builder_stable_11]
JUNOS OS runtime [20190605.30b921f_builder_stable_11]
JUNOS OS time zone information [20190605.30b921f_builder_stable_11]
JUNOS py extensions [20190620.215446_builder_junos_182_r3]
JUNOS py base [20190620.215446_builder_junos_182_r3]
JUNOS OS crypto [20190605.30b921f_builder_stable_11]
JUNOS network stack and utilities [20190620.215446_builder_junos_182_r3]
JUNOS libs [20190620.215446_builder_junos_182_r3]
JUNOS runtime [20190620.215446_builder_junos_182_r3]
JUNOS Web Management Platform Package [20190620.215446_builder_junos_182_r3]
JUNOS ex libs [20190620.215446_builder_junos_182_r3]
JUNOS ex runtime [20190620.215446_builder_junos_182_r3]
JUNOS ex platform support [20190620.215446_builder_junos_182_r3]
JUNOS dcp network modules [20190620.215446_builder_junos_182_r3]
JUNOS modules [20190620.215446_builder_junos_182_r3]
JUNOS ex modules [20190620.215446_builder_junos_182_r3]
JUNOS ex Data Plane Crypto Support [20190620.215446_builder_junos_182_r3]
JUNOS daemons [20190620.215446_builder_junos_182_r3]
JUNOS SDN Software Suite [20190620.215446_builder_junos_182_r3]
JUNOS Extension Toolkit [20190620.215446_builder_junos_182_r3]
JUNOS Phone-home [20190620.215446_builder_junos_182_r3]
JUNOS Packet Forwarding Engine Support (EX34XX) [20190620.215446_builder_junos_182_r3]
JUNOS jdocs ex [20190620.215446_builder_junos_182_r3]
JUNOS jail runtime [20190605.30b921f_builder_stable_11]
JUNOS FIPS mode utilities [20190620.215446_builder_junos_182_r3]
root@switch1> show poe controller 
Controller  Maximum   Power         Guard    Management   Status        Lldp
index       power     consumption   band                                Priority 
   0        740W      67.00W          0W     Class        AT_MODE       Disabled

Any insight into this would be greatly appreciated.

11 REPLIES 11
Highlighted
Ethernet Switching

Re: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

‎01-06-2020 07:44 AM

Are you connecting your switches (which have failed) using the POE ports by the way? That's a usual issue where they both try to power each other and it causes a crash.

KR
Adam

~~~~~~~~~~~~~~~~~~~~~~~
- Please Kudos if you found my response helpful
- Please accept my response as a 'Accepted Solution' if it solved your query
Highlighted
Ethernet Switching

Re: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

[ Edited ]
‎01-06-2020 08:03 AM

How to troubleshoot when POE is input to a POE enabled port (back to back POE) is called out in the KB below:

 

https://kb.juniper.net/InfoCenter/index?page=content&id=KB31588&cat=EX2300&actp=LIST

Highlighted
Ethernet Switching

Re: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

‎01-06-2020 08:36 AM

Each switch is only connected to other PoE switches via DAC or fiber. I'll verify once more to be sure.

Highlighted
Ethernet Switching

Re: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

‎01-06-2020 08:40 AM

None of the ports are indicating the status 36 so I have high confidence that there isn't any mis-patch anywhere:

 

Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/0 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/1 status 1
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/2 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/3 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/4 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/5 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/6 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/7 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/8 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/9 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/10 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/11 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/12 status 1
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/13 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/14 status 1
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/15 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/16 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/17 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/18 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/19 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/20 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/21 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/22 status 1
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/23 status 1
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/24 status 1
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/25 status 1
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/26 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/27 status 1
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/28 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/29 status 1
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/30 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/31 status 1
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/32 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/33 status 1
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/34 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/35 status 1
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/36 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/37 status 1
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/38 status 1
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/39 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/40 status 1
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/41 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/42 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/43 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/44 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/45 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/46 status 27
Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/47 status 27
Highlighted
Ethernet Switching

Re: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

‎01-06-2020 09:26 AM

Strange...

 

It might be a firmware issue with the 3rd party POE code.

When you ran 'show poe controller' was there a notice in the CLI response to update the POE Firmware?

 

Generally I'd say now open a ticket with JTAC though and confirm with them it isnt a JUNOS issue but if that isn't an issue it's likely be the POE firmware and you can update via CLI using 'request system firmware upgrade poe fpc-slot 0'.

That's a last resort one though.

KR
Adam

~~~~~~~~~~~~~~~~~~~~~~~
- Please Kudos if you found my response helpful
- Please accept my response as a 'Accepted Solution' if it solved your query
Highlighted
Ethernet Switching

Re: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

‎01-06-2020 11:26 AM

I've already updated the PoE firmware as it was one of the few things that I could do. This happened across multiple different brands of devices which have never seen this problem before on either Juniper or other switch vendors.

 

I'll see about opening JTAC but I'm not hopeful that it'll be too fruitful.

Highlighted
Ethernet Switching

Re: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

‎01-06-2020 12:44 PM

I would think a TAC case where they can tell you meaning of Status = 1 and probably more importantly Status = 27.  I might 'guess' 1 is good, and 27 (like 36) may indicate what is going on.

 

Just IMHO.

Highlighted
Ethernet Switching

Re: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

‎01-07-2020 03:51 AM

@RCCPGM 1 is POE off and 27 is POE on flag codes. Normal Codes all is fine codes.

 

What is discerning is the fact this isn't a vendor specific issue. This is making me thing theres a power issue into these boxes and not the boxes themselves (just my opinion).

~~~~~~~~~~~~~~~~~~~~~~~
- Please Kudos if you found my response helpful
- Please accept my response as a 'Accepted Solution' if it solved your query
Highlighted
Ethernet Switching

Re: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

‎01-07-2020 05:21 AM

That's exactly my thinking and mostly what I was hoping someone would know what "Failed to read voltage measurements" meant. It's too much of a concidence for the power to be dropped at the same time as those log entries. And even more strange that it's not all of the power, just a port or two at a time.

 

I'll see about moving some connections around and adjusting the poe enable/disable to see if that helps.

 

And thank you all for the replies. Very welcoming for my first post here.

Highlighted
Ethernet Switching

Re: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

[ Edited ]
‎01-07-2020 05:30 AM

No problem, it's why the community is great and filled with some great people. We all can't know everything.

Please do update the thread when you find out what the issue was though haha! Sparked my curiousity!

KR
Adam

~~~~~~~~~~~~~~~~~~~~~~~
- Please Kudos if you found my response helpful
- Please accept my response as a 'Accepted Solution' if it solved your query
Highlighted
Ethernet Switching

Re: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

Wednesday

hello Guys,

someone found solution to this error? i have the same issue with ex3400