vMX can't connect fpc,slot 0 Absent

[#1]

Hello,

I'm running VMX on my Ubuntu(Install Ubuntu on EXSI 5.5). VCP et VFP are running (cf below),but can't find interface ge port ,did any help to check it.

Thanks

Attach for reference:

admin@vmx1> show chassis fpc
Temp CPU Utilization (%) CPU Utilization (%) Memory Utilization (%)
Slot State (C) Total Interrupt 1min 5min 15min DRAM (MB) Heap Buffer
0 Present Absent

admin@vmx1> show chassis hardware
Hardware inventory:
Item Version Part number Serial number Description
Chassis VMX755c VMX
Midplane
Routing Engine 0 RE-VMX
CB 0 VMX SCB
CB 1 VMX SCB
FPC 0 Virtual FPC
CPU

vodatel@vmx-dayone-1:~/vmx-15.1F4-3$ sudo virsh net-dumpxml br-ext
<network>
<name>br-ext</name>
<uuid>0e0ca463-a184-497c-868e-e127beecc308</uuid>
<forward mode='route'/>
<bridge name='br-ext' stp='on' delay='0'/>
<mac address='52:54:00:9f:a0:77'/>
<ip address='192.168.88.194' netmask='255.255.255.0'>
<dhcp>
<host mac='0A:00D:C0E:0E' name='vcp-vmx1' ip='192.168.88.11'/>
<host mac='0A:00D:C0E:10' name='vfp-vmx1' ip='192.168.88.15'/>
</dhcp>
</ip>
</network>

[#2]

Try this:

"set chassis fpc 0 lite-mode"

If this works, problem is with Memory or/and CPU.

[#3]

still offline ，but ping vfp ip okay

[#4]

Follow this link for minimum requirements

https://www.juniper.net/techpubs/en_US/vmx16.1/topics/reference/general/vmx-hw-sw-minimums-vmware.ht...

[#5]

Hi,

Are you able to ping the VFP from the VCP over the internal bridge (not management link)  please can you provide output showing this

On 15.1 / 16.1 the command is

ping 128.0.0.16 routing-instance __juniper_private1__

Thanks

[#6]

Dear mdinham,

Ping is okay 

[#7]

Please can you confirm how much memory you have allocated to the VCP and VFP?

[#8]

Having exactly the same issue running under KVM (Linux kernel 4.4) running in lite mode, 1 vCPU allocated to vCP and 3 vCPU allocated to vFP with 2GB/8GB memory allocated respectively.

Basic connectivity between them seems to be fine, the vCP and vFP just never connect. Physical host is a dual Xeon E5-2630v2 with 256GB memory, nothing else running on it. Problem only seems to have started after someone did an apt-get -y update && apt-get -y upgrade on the box.

root> ping 128.0.0.16 routing-instance __juniper_private1__
PING 128.0.0.16 (128.0.0.16): 56 data bytes
64 bytes from 128.0.0.16: icmp_seq=0 ttl=64 time=0.329 ms
64 bytes from 128.0.0.16: icmp_seq=1 ttl=64 time=0.437 ms
64 bytes from 128.0.0.16: icmp_seq=2 ttl=64 time=0.416 ms
^C
--- 128.0.0.16 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.329/0.394/0.437/0.047 ms

root>

[#9]

If you log in to the VFP console, what do you see in the crash logs (/var/crash) ?

Also if you look in the VFP syslog file in /var/log/messages do you see the below?

RPIO: Accepted connection from 128.0.0.1:50896 <-> vPFE:3000
RPIO: Accepted connection from 128.0.0.1:56098 <-> vPFE:3000
HOSTIF: Accepted connection

Also note that JTAC will only support the following OS/Kernel versions for vMX

Ubuntu 14.04 LTS (recommended host OS)
Linux 3.13.0-32-generic

QEMU-KVM 2.0.0+dfsg-2ubuntu1.11 or later

Red Hat Enterprise Linux 7.2
Linux 3.10.0-327.4.5 or later

QEMU-KVM 1.5.3 or later

[#10]

If you may share /var/log on both vCP and vFP?

[#11]

Ok so have found the underlying issue, the vMX configuration has the following parameters in it;

FORWARDING_PLANE:
memory-mb : 8192
vcpus : 3

Yet it seems to only be booting with 1 vCPU;

root@localhost:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 1

On-line CPU(s) list: 0
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 1
NUMA node(s): 1

root@localhost:~# cat /var/log/messages | grep CPU

Oct 26 15:49:22 localhost user.notice root: vFP Error! Insufficient number of CPUs 1

Same issue present on an Ubuntu 14.04 LTS 3.13.0 kernel box running QEMU 2.6.0.

[#12]

I see a similar problem - although I am running directly on ESXi (using the OVAs supplied by Juniper). 

My /var/log/messages | grep CPU also shows this message:

vFP Error! Insufficient CPUs

However, I have 7 cores assigned to it in ESXi (lscpu confirms this).. Not sure what the issue _really_ is, but I seem to have had more success moving the assigned RAM for the vFPC up to 16GB (it was at 8GB). This is all on a trial license, that could be something to do with it as well. 

I first had an issue in that I could ping between the VCP and VFPC just fine, but I was always seeing FPC 0 as 'Absent' in 'show chassis fpc'. Now (having added double the RAM) - I can see my FPC 0 there, in 'Online   Testing' state.. ge-0/0/0 appears to be up, so I'll move on carefully from here.

Thanks for the tips in this thread. 

[#13]

Add vMX 60-days trial license and check. 

[#14]

I have a similiar issue on vMX 14.1R1.10 .

On my Fusion I can see PIC 0 being listed on show chassis fpc pic-status  and on show chassis hardware .

Then exported the vMX vm with cli ovftool and imported into Cloudshare cloud provider, that pretty much is a huge ESX server.

There I cannot see PIC 0 and no ge-* interfaces are present.

There's no lite command available and vMX trial from Juniper site cannot be installed as being detected as invalid.

[#15]

Trying to restart an old thread before starting a new one...

Did you find a solution to this?  I have the same issue with a vMX running 17.4R3-S2.  The FPC shows as present but absent.  I can ping between the two VMs and I have tried to increase the RAM to 8G (doc say 2G is enough for lite-mode).  A have a vMX running on 15.1 with the same set up and that is working so not sure what else to try.

<edit>

In the log messages on the VCP I can see the following:

.....
Apr 27 08:32:48   fpc0 afi-server: luchip_client_start: trying server connection, attempt 586
Apr 27 08:32:49   fpc0 afi-server: luchip_client_start: trying server connection, attempt 587
Apr 27 08:32:50   fpc0 afi-server: luchip_client_start: trying server connection, attempt 588
Apr 27 08:32:51   fpc0 afi-server: luchip_client_start: trying server connection, attempt 589
Apr 27 08:32:52   fpc0 afi-server: luchip_client_start: trying server connection, attempt 590
Apr 27 08:32:53   chassisd[5430]: CHASSISD_FRU_UNRESPONSIVE: Error for FPC 0: unresponsive to attempts to start it; left it offline
Apr 27 08:32:53   chassisd[5430]: CHASSISD_FRU_OFFLINE_NOTICE: Taking FPC 0 offline: Error
Apr 27 08:32:53   chassisd[5430]: CHASSISD_SNMP_TRAP3: ENTITY trap generated: entStateOperDisabled (entPhysicalIndex 20, entStateAdmin 4, entStateAlarm 6)
Apr 27 08:32:53   chassisd[5430]: CHASSISD_SNMP_TRAP0: ENTITY trap generated: entConfigChange
Apr 27 08:32:53   chassisd[5430]: CHASSISD_SNMP_TRAP10: SNMP trap generated: FRU power off (jnxFruContentsIndex 7, jnxFruL1Index 1, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: Virtual FPC @ 0/*/*, jnxFruType 3, jnxFruSlot 0, jnxFruOfflineReason 3, jnxFruLastPowerOff 275026, jnxFruLastPowerOn 215026)
Apr 27 08:32:53   chassisd[5430]: CHASSISD_SNMP_TRAP10: SNMP trap generated: Fru Offline (jnxFruContentsIndex 7, jnxFruL1Index 1, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: Virtual FPC @ 0/*/*, jnxFruType 3, jnxFruSlot 0, jnxFruOfflineReason 3, jnxFruLastPowerOff 275026, jnxFruLastPowerOn 215026)
Apr 27 08:32:53   chassisd[5430]: CHASSISD_SNMP_TRAP3: ENTITY trap generated: entStateOperDisabled (entPhysicalIndex 20, entStateAdmin 2, entStateAlarm 6)
Apr 27 08:32:53   chassisd[5430]: CHASSISD_SNMP_TRAP7: SNMP trap generated: Fru Failed (jnxFruContentsIndex 7, jnxFruL1Index 1, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: Virtual FPC @ 0/*/*, jnxFruType 3, jnxFruSlot 0)
Apr 27 08:32:53   alarmd[5636]: Alarm set: FPC color=RED, class=CHASSIS, reason=FPC 0 Hard errors
Apr 27 08:57:21   mgd[5717]: UI_DBASE_LOGIN_EVENT: User 'root' entering configuration mode
Apr 27 08:58:17   mgd[5717]: UI_COMMIT: User 'root' requested 'commit' operation (comment: none)
Apr 27 08:58:17   mgd[5717]: UI_CHILD_EXITED: Child exited: PID 6496, status 7, command '/usr/sbin/mustd'
Apr 27 08:58:18   chassisd[6498]: JAM:PL: mod info  i2cc4a
Apr 27 08:58:18   chassisd[6498]: JAM:PL: asy info  i2cc4a
Apr 27 08:58:18   chassisd[6498]: JAM:PL: Reg Attribs done for c4a
Apr 27 08:58:18   chassisd[6498]: JAM:PL: vectors_set done for c4a
Apr 27 08:58:18   chassisd[6498]: JAM:PL: ds_prepare done for c4a
Apr 27 08:58:18   chassisd[6498]: JAM:PL: mod info  i2ccd8
Apr 27 08:58:18   chassisd[6498]: JAM:PL: asy info  i2ccd8
Apr 27 08:58:18   chassisd[6498]: JAM:PL: Reg Attribs done for cd8
Apr 27 08:58:18   chassisd[6498]: JAM:PL: vectors_set done for cd8
Apr 27 08:58:18   chassisd[6498]: JAM:PL: ds_prepare done for cd8
Apr 27 08:58:18   chassisd[6498]: VMX Platform. cbd_virtual_init
Apr 27 08:58:18   chassisd[6498]: VMX Platform. cbd_virtual_init
Apr 27 08:58:18   mobiled: main Neither BNG LIC nor JMOBILE package is present,exit mobiled
Apr 27 08:58:18   l2cpd[6523]: ppmlite_var_init: iri instance = 36736
Apr 27 08:58:19   ffp[6524]: "dynamic-profiles": No change to profiles
Apr 27 08:58:20   mgd[5717]: UI_COMMIT_COMPLETED: commit complete
Apr 27 08:58:20   chassisd[5430]: fpc_atlas_ok_to_powerup() FPC slot 0 i2c_id 0xbaa is_vir [1] data_valid[1]is_jam[0] is_plug[0]
Apr 27 08:58:20   chassisd[5430]: CHASSISD_SNMP_TRAP10: SNMP trap generated: FRU power on (jnxFruContentsIndex 7, jnxFruL1Index 1, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: Virtual FPC @ 0/*/*, jnxFruType 3, jnxFruSlot 0, jnxFruOfflineReason 2, jnxFruLastPowerOff 275026, jnxFruLastPowerOn 427708)
Apr 27 08:58:22   fpc0 LOGIN: : SETTING UP DPDK ENVIRONMENT ON
Apr 27 08:58:22   fpc0 sudo:     root : TTY=unknown ; PWD=/home/pfe/riot/dpdk ; USER=root ; COMMAND=/sbin/modprobe uio
Apr 27 08:58:22   fpc0 sudo:     root : TTY=unknown ; PWD=/home/pfe/riot/dpdk ; USER=root ; COMMAND=/sbin/insmod /home/pfe/riot/dpdk/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko
Apr 27 08:58:22   fpc0 LOGIN: : Creating /mnt/huge and mounting as hugetlbfs
Apr 27 08:58:22   fpc0 sudo:     root : TTY=unknown ; PWD=/home/pfe/riot/dpdk ; USER=root ; COMMAND=/bin/mkdir -p /mnt/huge
Apr 27 08:58:23   fpc0 LOGIN: : Detected Junos Device type vmxnet3
Apr 27 08:58:23   fpc0 sudo:     root : TTY=unknown ; PWD=/home/pfe/riot/dpdk ; USER=root ; COMMAND=/home/pfe/riot/dpdk/tools/dpdk_nic_bind.py -u 0b:00.0
Apr 27 08:58:23   fpc0 sudo:     root : TTY=unknown ; PWD=/home/pfe/riot/dpdk ; USER=root ; COMMAND=/home/pfe/riot/dpdk/tools/dpdk_nic_bind.py -b igb_uio 0b:00.0
Apr 27 08:58:23   fpc0 LOGIN: start_afi_server.sh : Found /usr/share/pfe/afi-server.tgz tarball
Apr 27 08:58:23   fpc0 LOGIN: : Detected Junos Device type vmxnet3
Apr 27 08:58:23   fpc0 LOGIN: : Number of CPUs available : 3
Apr 27 08:58:23   fpc0 LOGIN: : Number of PORTs to service : 1, loopbacks: 0
Apr 27 08:58:23   fpc0 LOGIN: : vdev_str:
Apr 27 08:58:23   fpc0 LOGIN: : /home/pfe/riot/build/app/riot -c 0x3 -n 2 --lcores='(1-2)@1' --log-level=5  -w 0b:00.0 -- --rx "(0,0,0,1)," --tx "(0,1)," --w "2"   --rpio "local,3000,3001" --hostif "local,3002" --bsz "(32,32),(32,32),(32,32)"
Apr 27 08:58:24   fpc0 LOGIN: start_afi_server.sh : tar -xzf /usr/share/pfe/afi-server.tgz : SUCCESS
Apr 27 08:58:24   fpc0 LOGIN: start_afi_server.sh : Running /usr/share/pfe/afi-server/usr/bin/run-afi-server.sh ...
Apr 27 08:58:24   fpc0 riot[10537]: rte_eth_dev_config_restore: port 0: MAC address array not supported
Apr 27 08:58:24   fpc0 riot[10537]: rte_eth_stats_reset: Function not supported
Apr 27 08:58:32   fpc0 afi-server: luchip_client_start: trying server connection, attempt 1
.....

I cant see anything online about the luchip message although the message is descriptive, I'm unsure how to fix it.

Thanks

Paul

[#16]

Hi Paul,

What is the ESXi version used?

Could you please share "*.vmx" file for both vCP & vFP. [It will be under datastore folder of both VM's]