Junos Space Developer
Junos Space Developer

Trouble With Junos Space Install

‎08-21-2014 06:15 PM

Got all installed (virtual machine), expanded the HDD. Now unable to access the webui. 

 

I rebooted just in case, and when I login to the CLI I get above the main menu:

 

#####################################################

WARNING: WATCHDOG IS DISABLED, ENABLE IT IF NECESSARY

#####################################################

 

Unfortunately, Google isn't helping me much here.

 

Here's my interface configs:

 

[root@space-0050568d4d27 ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:50:56:8D:4D:27
inet addr:10.201.17.23 Bcast:10.201.17.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:933 errors:0 dropped:0 overruns:0 frame:0
TX packets:245 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:72496 (70.7 KiB) TX bytes:34142 (33.3 KiB)

eth0:0 Link encap:Ethernet HWaddr 00:50:56:8D:4D:27
inet addr:10.201.17.93 Bcast:10.201.17.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth1 Link encap:Ethernet HWaddr 00:50:56:8D:67:C8
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:695 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:43973 (42.9 KiB) TX bytes:0 (0.0 b)

eth2 Link encap:Ethernet HWaddr 00:50:56:8D:49:8D
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:694 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:43913 (42.8 KiB) TX bytes:0 (0.0 b)

eth3 Link encap:Ethernet HWaddr 00:50:56:8D:55:89
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:693 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:43853 (42.8 KiB) TX bytes:0 (0.0 b)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:206 errors:0 dropped:0 overruns:0 frame:0
TX packets:206 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:45654 (44.5 KiB) TX bytes:45654 (44.5 KiB)

 

And here is my routing table:

 

[root@space-0050568d4d27 ~]# netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
10.201.17.0 * 255.255.255.0 U 0 0 0 eth0
169.254.0.0 * 255.255.0.0 U 0 0 0 eth3
default 10.201.17.1 0.0.0.0 UG 0 0 0 eth0

 

I don't even know where to begin troubleshooting. This is my first experience with Space. 

--
Chris Jones
JNCIE-ENT #272
Former JNCIP-SP
Former CCIE #25655
Former Juniper Ambassador
23 REPLIES 23
Junos Space Developer

Re: Trouble With Junos Space Install

‎08-21-2014 07:02 PM

What address are you trying to connect to the Space UI with? The Space UI responds to the VIP address on eth0:0, but not eth0. You can ssh to either of those, so that can be confusing initially.  Make sure you are using the URL https://10.201.17.93 in your case.

There are a couple of other things that have me scratching my head a little. I have mostly worked with a Space fabric, and have not looked closely at standalone configs, so this could just be I've not seen this before. I am surprised to find the multicast subnet on the eth3 interface and yet eth3 has no unicast IP address. It might mean nothing, but if the VIP address does not work, let me know.

 

Bruce Bowen

RE Junos Space

Junos Space Developer

Re: Trouble With Junos Space Install

‎08-21-2014 07:05 PM

@bhbowen wrote:

What address are you trying to connect to the Space UI with? The Space UI responds to the VIP address on eth0:0, but not eth0. You can ssh to either of those, so that can be confusing initially.  Make sure you are using the URL https://10.201.17.93 in your case.

There are a couple of other things that have me scratching my head a little. I have mostly worked with a Space fabric, and have not looked closely at standalone configs, so this could just be I've not seen this before. I am surprised to find the multicast subnet on the eth3 interface and yet eth3 has no unicast IP address. It might mean nothing, but if the VIP address does not work, let me know.

 

Bruce Bowen

RE Junos Space


I honestly have no idea what Space fabric is. 

 

Yes, been using the .93 URL. Not sure where to go now. 

--
Chris Jones
JNCIE-ENT #272
Former JNCIP-SP
Former CCIE #25655
Former Juniper Ambassador
Junos Space Developer

Re: Trouble With Junos Space Install

‎08-21-2014 07:16 PM

/var/log/messages has a crapton of:

 

Aug 22 01:14:29 space-0050568d4d27 kernel: martian source 10.201.17.255 from 10.201.17.38, on dev eth1
Aug 22 01:14:29 space-0050568d4d27 kernel: ll header: ff:ff:ff:ff:ff:ff:00:50:56:8d:14:0c:08:00
Aug 22 01:14:29 space-0050568d4d27 kernel: martian source 10.201.17.255 from 10.201.17.38, on dev eth3
Aug 22 01:14:29 space-0050568d4d27 kernel: ll header: ff:ff:ff:ff:ff:ff:00:50:56:8d:14:0c:08:00
Aug 22 01:14:30 space-0050568d4d27 kernel: martian source 10.201.17.255 from 10.201.17.38, on dev eth2
Aug 22 01:14:30 space-0050568d4d27 kernel: ll header: ff:ff:ff:ff:ff:ff:00:50:56:8d:14:0c:08:00
Aug 22 01:14:30 space-0050568d4d27 kernel: martian source 10.201.17.255 from 10.201.17.38, on dev eth1
Aug 22 01:14:30 space-0050568d4d27 kernel: ll header: ff:ff:ff:ff:ff:ff:00:50:56:8d:14:0c:08:00
Aug 22 01:14:30 space-0050568d4d27 kernel: martian source 10.201.17.255 from 10.201.17.38, on dev eth3
Aug 22 01:14:30 space-0050568d4d27 kernel: ll header: ff:ff:ff:ff:ff:ff:00:50:56:8d:14:0c:08:00
Aug 22 01:14:31 space-0050568d4d27 kernel: martian source 10.201.17.255 from 10.201.17.38, on dev eth2
Aug 22 01:14:31 space-0050568d4d27 kernel: ll header: ff:ff:ff:ff:ff:ff:00:50:56:8d:14:0c:08:00
Aug 22 01:14:31 space-0050568d4d27 kernel: martian source 10.201.17.255 from 10.201.17.38, on dev eth1
Aug 22 01:14:31 space-0050568d4d27 kernel: ll header: ff:ff:ff:ff:ff:ff:00:50:56:8d:14:0c:08:00
Aug 22 01:14:31 space-0050568d4d27 kernel: martian source 10.201.17.255 from 10.201.17.38, on dev eth3
Aug 22 01:14:31 space-0050568d4d27 kernel: ll header: ff:ff:ff:ff:ff:ff:00:50:56:8d:14:0c:08:00
Aug 22 01:14:31 space-0050568d4d27 kernel: martian source 10.201.17.255 from 10.201.17.38, on dev eth2

--
Chris Jones
JNCIE-ENT #272
Former JNCIP-SP
Former CCIE #25655
Former Juniper Ambassador
Junos Space Developer

Re: Trouble With Junos Space Install

‎08-21-2014 07:25 PM

Space fabric <==> cluster of multiple servers acting as one unit.

 

How long did you wait from answering the initial install questions and doing the reboot? Space can take 20+ minutes to finish configuring the DB and the web server before it is finished installing. I wonder if you may have interrupted the configuration scripts with the reboot.

 

Run a 'service --status-all' to see if there are any services stopped. The output will list the whole firewall config, but after that is some important stuff. Here is an exmple output from one of my systems:

 

iscsid (pid  1525) is running...
jboss (pid 4626) is running
jboss domain controller (pid 3948) running
jmp-firewall is running
jmp-heartbeat is running...
jmp-jboss-dc-monitor (pid  3397 ) is running
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64/jre/bin/java = cap_net_bind_service,cap_net_raw+eip
opennms is running..
OpenNMS is running or OpenNMS service can't be run on this non-VIP node
jmp-watchdog running
Kdump is not operational
libvirtd is stopped
started
mcstransd is stopped
dbus-daemon (pid 2048) is running...
multipathd is stopped
 SUCCESS! MySQL running (1348)
 SUCCESS! MySQL running (1348)

 

If there are some services not running, it may be simpler to reinstall than try to repair the failed configuration process.

 

I would recommed the following install process:

Install VM, do not set to boot after install finished.

Reconfigure VM hardware settings (cpus, memory, add one disk)

Boot VM and answer installation questions.

Open browser to VIP IP and monitor configuration process, when done the normal Space login banner will appear.

Go back to CLI and expand first partition/mount point.

For each other partition to be expanded:

   Go to VM config and add disk

   Go to admin cli menu and expand partition.

 

 

Hope this helps,

Bruce Bowen

RE Space

Junos Space Developer

Re: Trouble With Junos Space Install

‎08-21-2014 07:26 PM

You have all your VM interfaces connected to the same virtual switch. The firewall sees this as packets coming in on interfaces that they should not and complains like mad.

Junos Space Developer

Re: Trouble With Junos Space Install

‎08-21-2014 07:44 PM

If this does not resolve things, contact me directly tomorrow, I thing a 15 minute conversation will straighten things out.

 

Later,

Bruce Bowen

RE Junos Space

Junos Space Developer

Re: Trouble With Junos Space Install

‎08-22-2014 04:00 PM

[root@space-0050568d4d27 ~]# service --status-all
acpid (pid 2360) is running...
anacron is stopped
atd (pid 4176) is running...
auditd is stopped
Avahi daemon is running
Avahi DNS daemon is not running
cpuspeed is stopped
crond (pid 4133) is running...
dc_client is stopped
dc_server is stopped
dnsmasq is stopped
# Generated by ebtables-save v1.0 on Fri Aug 22 22:59:59 UTC 2014
gpm (pid 3607) is running...
hald (pid 2368) is running...
heartbeat OK [pid 3583 et al] is running on space-0050568d4d27 [space-0050568d4d27]...
webProxy (pid 3620) is running...
Firewall is stopped.
ipmi_msghandler module not loaded.
ipmi_si module not loaded.
ipmi_devintf module not loaded.
/dev/ipmi0 does not exist.
ipmievd is stopped
Firewall is stopped.
irqbalance (pid 2316) is running...
iscsid (pid 1676) is running...
iscsid (pid 1676) is running...
jboss is stopped
jboss domain controller is stopped
jmp-firewall is stopped
jmp-heartbeat is running...
jmp-jboss-dc-monitor (pid 3657 ) is running
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64/jre/bin/java = cap_net_bind_service,cap_net_raw+eip
opennms is running..
OpenNMS is running or OpenNMS service can't be run on this non-VIP node
jmp-watchdog running
Kdump is not operational
libvirtd is stopped
started
mcstransd is stopped
dbus-daemon (pid 2334) is running...
multipathd is stopped
SUCCESS! MySQL running (3237)
SUCCESS! MySQL running (3237)
netconsole module not loaded
netplugd is stopped
Configured devices:
lo eth0 eth1 eth2 eth3
Currently active devices:
lo eth0
rpc.mountd is stopped
nfsd is stopped
rpc.statd is stopped
nma (pid 3648) is running...
ntpd (pid 2461) is running...
Error: you must run this script as opennms, not 'root'
portmap is stopped
(pid 3908) is running...
Process accounting is disabled.
rdisc is stopped
rpc.idmapd is stopped
rsyslogd (pid 2421) is running...
saslauthd is stopped
smartd (pid 4235) is running...
snmpd is stopped
snmptrapd is stopped
openssh-daemon (pid 6283) is running...
Usage: /etc/init.d/stor_agent {start|stop|restart}
syslogd is stopped
klogd is stopped
xfs (pid 4161) is running...
xinetd (pid 2450) is running...
yum-updatesd is stopped

 

I also noticed when I tried to start jboss manually:

 

[root@space-0050568d4d27 ~]# service jboss start
PING jmp-CLUSTER (10.201.17.93) 56(84) bytes of data.
64 bytes from jmp-CLUSTER (10.201.17.93): icmp_seq=1 ttl=64 time=0.047 ms
64 bytes from jmp-CLUSTER (10.201.17.93): icmp_seq=2 ttl=64 time=0.050 ms
64 bytes from jmp-CLUSTER (10.201.17.93): icmp_seq=3 ttl=64 time=0.043 ms

--- jmp-CLUSTER ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.043/0.046/0.050/0.008 ms
in master node
domain controller is stopped, cannot start

 

This is frustrating, not exactly a good first impression of Space. 

--
Chris Jones
JNCIE-ENT #272
Former JNCIP-SP
Former CCIE #25655
Former Juniper Ambassador
Junos Space Developer

Re: Trouble With Junos Space Install

‎08-25-2014 03:25 AM

Hi,

 

Please start watchdog service as below :

 

service jmp-watchdog start

 

--PL

Junos Space Developer

Re: Trouble With Junos Space Install

‎08-25-2014 09:54 AM

How long should I wait after doing that? 

--
Chris Jones
JNCIE-ENT #272
Former JNCIP-SP
Former CCIE #25655
Former Juniper Ambassador
Junos Space Developer

Re: Trouble With Junos Space Install

‎08-25-2014 12:03 PM

The VM install can be a little tricky since the VM resources can be easily mis-allocated. I suspect the initial configuration of JBoss may have been messed up after you installed the VM. The good thing is it does not take long to do the initial VM install. At this point, here is the high-level process I suggest.

 

  • Install the VM image (do not boot after install complete)
  • Edit VM settings (add CPU, memory. Disconnect any network interfaces not being used [default is they all connect to the same vswitch which results in the 'crapton' of filewall messages you reported last week ], don't add any disk space at this point).
  • Start the VM
  • Log into the console as admin and answer the initial install questions (network IP, name service, ntp service, etc.)
  • After you commit your answers, start a 'debug shell' on the console and check that your networking is OK: ifconfig eth0 and eth0:0, ping the default gateway, etc.
  • Use a browser to connect to Space on the VIP address you used during install. At this point, Space will probably still be in 'maintenance mode' and will not present a 'login' screen but instead show the configuration process status. Depending on how many resources your VM has, it may take 20 - 30 minutes to finish the install and finally show the login screen.
  • At this point, you can grow the file systems of the VM. Add one disk at a time and expand the file systems via the admin menus after logging out of the 'debug shell' on the console. If you add more than one disk to the VM before expanding a file system, the expansion will use all of the new disks it finds for the one partition.

Hope that helps,

Bruce Bowen

RE Junos Space

Junos Space Developer

Re: Trouble With Junos Space Install

‎08-26-2014 02:07 PM

@bhbowen wrote:

The VM install can be a little tricky since the VM resources can be easily mis-allocated. I suspect the initial configuration of JBoss may have been messed up after you installed the VM. The good thing is it does not take long to do the initial VM install. At this point, here is the high-level process I suggest.

 

  • Install the VM image (do not boot after install complete)
  • Edit VM settings (add CPU, memory. Disconnect any network interfaces not being used [default is they all connect to the same vswitch which results in the 'crapton' of filewall messages you reported last week ], don't add any disk space at this point).
  • Start the VM
  • Log into the console as admin and answer the initial install questions (network IP, name service, ntp service, etc.)
  • After you commit your answers, start a 'debug shell' on the console and check that your networking is OK: ifconfig eth0 and eth0:0, ping the default gateway, etc.
  • Use a browser to connect to Space on the VIP address you used during install. At this point, Space will probably still be in 'maintenance mode' and will not present a 'login' screen but instead show the configuration process status. Depending on how many resources your VM has, it may take 20 - 30 minutes to finish the install and finally show the login screen.
  • At this point, you can grow the file systems of the VM. Add one disk at a time and expand the file systems via the admin menus after logging out of the 'debug shell' on the console. If you add more than one disk to the VM before expanding a file system, the expansion will use all of the new disks it finds for the one partition.

Hope that helps,

Bruce Bowen

RE Junos Space


Bruce,

 

Thank you for the response. I'm going  to try re-installing the VM as you suggest. I'm a bit confused about the last bullet point though. 

 

The Junos Space install docs say to add a 100G drive to the VM and then "expand the filesystem". When you actually go to do that, it asks which filesystem to expand. I tried just "/" but none of the others. Do I need to add a disk for each of the filesystems listed? And exand each of them? I'm a bit confused on that process and the Space docs aren't very good in that regard.

 

Thanks,

 

Chris

--
Chris Jones
JNCIE-ENT #272
Former JNCIP-SP
Former CCIE #25655
Former Juniper Ambassador
Junos Space Developer

Re: Trouble With Junos Space Install

‎08-27-2014 05:38 AM

That is one place where the documentation did not keep up with the changes to the software. Space 13.3 for the VM changed from a single file system to multiple. The file systems that need to be 'expanded' are the ones where the logs and databases are kept (/var /var/log /opt). I had to put in a doc change request when I found that (it has been completed online) and I'll see if I can find the reference to it and post it here later.

If all you are doing is simple lab testing without many devices, I just go with the default disk layout and only do thin provisioning of the disk space. That lets me run a system with a handful of devices under management.

 

This may be TMI, but if you ever get to building a large Space fabric/cluster with VMs, and you want to create a 'Specialized Node' for fault management/performance management, you have to expand the /var file system before adding it to the fabric. A FM/PM node is one that has been set up to only run the OpenNMS code that shows up under the 'Network Monitoring' link on main menu.

 

Later,

Bruce Bowen

RE Junos Space

 

Junos Space Developer

Re: Trouble With Junos Space Install

‎08-27-2014 02:05 PM

 


@bhbowen wrote:

That is one place where the documentation did not keep up with the changes to the software. Space 13.3 for the VM changed from a single file system to multiple. The file systems that need to be 'expanded' are the ones where the logs and databases are kept (/var /var/log /opt). I had to put in a doc change request when I found that (it has been completed online) and I'll see if I can find the reference to it and post it here later.

If all you are doing is simple lab testing without many devices, I just go with the default disk layout and only do thin provisioning of the disk space. That lets me run a system with a handful of devices under management.

 

This may be TMI, but if you ever get to building a large Space fabric/cluster with VMs, and you want to create a 'Specialized Node' for fault management/performance management, you have to expand the /var file system before adding it to the fabric. A FM/PM node is one that has been set up to only run the OpenNMS code that shows up under the 'Network Monitoring' link on main menu.

 

Later,

Bruce Bowen

RE Junos Space

 


This is a production space install.

 

So do I need to expand each filesystem all onto the 100G disk I created? Or do I need seperate virtual disks for each filesystem I expand?

--
Chris Jones
JNCIE-ENT #272
Former JNCIP-SP
Former CCIE #25655
Former Juniper Ambassador
Junos Space Developer

Re: Trouble With Junos Space Install

‎08-27-2014 06:44 PM
You'll most likely have to add disks to expand them. I recently just did that with two of our fabric nodes. The /var directory was getting a bit full, so I added another 20 GB disk and expanded onto it.
Scott Ware
Security Engineer
Juniper Ambassador
Twitter: @scottdware
Skype: scottdware
scottdware@gmail.com

"Do. Or do not. There is no try." - Yoda
Junos Space Developer

Re: Trouble With Junos Space Install

‎08-27-2014 06:47 PM

@scottdware wrote:
You'll most likely have to add disks to expand them. I recently just did that with two of our fabric nodes. The /var directory was getting a bit full, so I added another 20 GB disk and expanded onto it.

So I DO have to add a new disk for each partition? Can you give me some idea of how big the added four disks should be? Seems really stupid that you have to add a whole seperate virtual disk for each partition. 

--
Chris Jones
JNCIE-ENT #272
Former JNCIP-SP
Former CCIE #25655
Former Juniper Ambassador
Junos Space Developer

Re: Trouble With Junos Space Install

‎08-28-2014 04:01 AM

I agree, it does seem a little wonky. The initial disks we setup are a 32 GB and 100 GB. Here's what our layout looks like on one of our nodes currently (after I expanded the /var directory the other day):

 

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/jmpvgnocf-lvroot
                       87G  8.4G   74G  11% /
/dev/mapper/jmpvgnocf-lvtmp
                      1.9G   49M  1.8G   3% /tmp
/dev/mapper/jmpvgnocf-lvvar
                       24G  5.0G   17G  23% /var
/dev/mapper/jmpvgnocf-lvlog
                      4.8G  2.1G  2.5G  46% /var/log
/dev/sda1              99M   17M   78M  18% /boot
tmpfs                  16G     0   16G   0% /dev/shm

 

The /var directory seems to be the one where you want to have the most space. I'd make that anywhere from 60-100 GB, and possibly more if you are going to be storing backups locally. Even the /var/log partition you could make 20 GB or so. I haven't found that the logs take up too much space, though, as we've had these running for quite some time and it really hasn't grown all that much.

 

Now, there might be a better way to partition these up, maybe someone with more Space experience that I have can chime in, but from the initial setup, I haven't seen anything that would make this the case.

Scott Ware
Security Engineer
Juniper Ambassador
Twitter: @scottdware
Skype: scottdware
scottdware@gmail.com

"Do. Or do not. There is no try." - Yoda
Junos Space Developer
Solution
Accepted by topic author JunosChris
‎08-26-2015 01:27 AM

Re: Trouble With Junos Space Install

‎09-25-2014 01:13 PM

I don't know if this has been fully answered her but I took this out of the latest release documents.  The sizes below I doubled for my last install due to their environment. You have to create a seperate virtual disk and then go and expand the volume under space.

 

80 GB for /var

• 60 GB for /var/log

• 30 GB for /tmp

• 20 GB for /

Junos Space Developer

Re: Trouble With Junos Space Install

‎09-25-2014 02:18 PM

@ghaug wrote:

I don't know if this has been fully answered her but I took this out of the latest release documents.  The sizes below I doubled for my last install due to their environment. You have to create a seperate virtual disk and then go and expand the volume under space.

 

80 GB for /var

• 60 GB for /var/log

• 30 GB for /tmp

• 20 GB for /


Thanks, this was the part that I found to be confusing. I wasn't sure if you had to add a separate virtual disk for each partition or not. This makes more sense now. Thank you!

--
Chris Jones
JNCIE-ENT #272
Former JNCIP-SP
Former CCIE #25655
Former Juniper Ambassador
Junos Space Developer

Re: Trouble With Junos Space Install

[ Edited ]
‎12-11-2014 04:09 PM

Why would i get this message when trying to expand the partitions?

 ERROR: Did not find any drives to add!

 

I added a 32GB RAM and 100GB memory to each junos space node in a fabric but I just can't still figure out why it wouldn't reflect when I login to the CLI the node.

 

 

 

Space.PNG

JNCIE-SEC #240, JNCIE-ENT #557, CCIE-RS #51416