Junos Space Developer
Showing results for 
Search instead for 
Do you mean 
Reply
Contributor
Posts: 54
Registered: ‎07-26-2011
0 Kudos

Kernel Panic error while shutting down JSVA

Hi,

 

Once JSVA is up and running for 5-6 hours and all of us are using it, then JSVA becomes very slow. Sometime it creshes with kernel panic error or when I try to shutodnw manually, it gives kernel panic error while trying to shutdown Jboss processes.

 

This is daily activity for us. Is this bug in 11.2? Is there patch which soleves this problem so that we can use JSAv continuously without having to restart quite often.

 

Regards

Satish

Recognized Expert
Posts: 456
Registered: ‎09-29-2009
0 Kudos

Re: Kernel Panic error while shutting down JSVA

HI Satish,

 

 

I am assuming you are using WIndows 2007. How many users are using this single JSVA and what is the memory you have allocated to it?

 

Please send me the Vbox log for the machine.

 

Regards

Yesh

 

Recognized Expert
Posts: 456
Registered: ‎09-29-2009
0 Kudos

Re: Kernel Panic error while shutting down JSVA

Hi Satish,

 

Also send /var/log/messages and /var/log/mysqld.log.

 

Thanks

Yesh

Contributor
Posts: 54
Registered: ‎07-26-2011
0 Kudos

Re: Kernel Panic error while shutting down JSVA

Attached Vbox logs.

 

Will send other requested logs shortly.

 

Only 2 users are connected. Memory allocated is by default 2 GB. Yes Machine is windows 7

Super Contributor
Posts: 209
Registered: ‎10-07-2010
0 Kudos

Re: Kernel Panic error while shutting down JSVA

Hi Satish,

 

      As per the recommendations we recommend having a minmum RAM of 4GB.  2 users connecting onto 2GB is very less. 

 

     If there is any way to upgrade with 4GB.  Pl. upgrade and one more question is that i hope you haven't opened any other applications like outlook etc.,

 

 

 

Regards,

 

Vimala.

Recognized Expert
Posts: 456
Registered: ‎09-29-2009
0 Kudos

Re: Kernel Panic error while shutting down JSVA

HI Satish,

 

What Vimala specified is for 4GB for the machine itself.  2GB is enough for the VM if the activities are not intensive. Since two users are using it, you may want to increase the VM memory to 3GB if possible to reduce the slowness you encounter.

I am waiting for the logs to look at the kernel panic issue.

 

Regards

Yesh

Contributor
Posts: 54
Registered: ‎07-26-2011
0 Kudos

Re: Kernel Panic error while shutting down JSVA

Hi Vimala, Yesh,

 

Total machine RAM is 4GB. 2GB allocated to JSVA.

 

please find attached logs - mysqld.log and messages

 

We are developing our sample apps. We uninstall and install apps as we progress. So after 4-5 hours, it crashes. Now also it crashed.

 

Regards

Satish

Recognized Expert
Posts: 456
Registered: ‎09-29-2009
0 Kudos

Re: Kernel Panic error while shutting down JSVA

Hi Satish,

 

Thanks for the logs. I need to investigate a little bit further. Seems like the cause is a Memory Crash. 

Can you check the /var/log/jboss/server.log and see if there are IOTHREAD issues reported around this time(log time) of the crash?

 

 

Aug 10 09:37:48 space-080027b5335d kernel: Memory for crash kernel (0x0 to 0x0) notwithin permissible range

 

My guess for now is , you are trying to install/uninstall when some of the components are still being loaded. Even after jboss is started , you have to give another 10 mins for Junos Space to be all set and ready for your application.

For now,  once you restart jboss, please wait till you are able to access the Junos Space Web UI to continue with installation and uninstallation of application. 

Let me know how it goes.

 

Regards

Yesh

 

 

Contributor
Posts: 54
Registered: ‎07-26-2011
0 Kudos

Re: Kernel Panic error while shutting down JSVA

Yesh, After restarting JSVA, we wait until login page is displayed. Once we are able to login to mainui site. then we start deploying our apps. Sometime in the middle of our development, kernel crashes.

 


Log entries from Jboss as below




2011-08-10 06:58:36,495 WARN  [org.jboss.ejb.txtimer.EJBTimerServiceImpl] (mainSmiley Happy Cannot obtain the implementation of a PersistencePolicy, using NoopPersistencePolicy: java.lang.RuntimeException: Error creating MBeanProxy: jboss.ejb:service=EJBTimerService,persistencePolicy=database
2011-08-10 06:58:38,223 ERROR [STDERR] (mainSmiley Happy Aug 10, 2011 6:58:38 AM org.hornetq.core.logging.impl.JULLogDelegate info
INFO: live server is starting..
2011-08-10 06:58:41,002 ERROR [STDERR] (mainSmiley Happy Aug 10, 2011 6:58:41 AM org.hornetq.core.logging.impl.JULLogDelegate info
INFO: HornetQ Server version 2.1.2.Final (Colmeia, 120) started
2011-08-10 07:00:44,585 ERROR [net.juniper.jmp.cmp.deviceManager.ejb.MessageDeviceServiceBean] (Thread-3 (group:HornetQ-client-global-threads-1891955678)Smiley Happy Failed to lookup DeviceService EJB
2011-08-10 07:01:00,468 ERROR [STDERR] (mainSmiley Happy jsFile ./js/extension/gridfilters/menu/RageMenu.js not found!
2011-08-10 07:01:00,478 ERROR [STDERR] (mainSmiley Happy jsFile ./js/uicommon/notification/NotificationUtils.js not found!
2011-08-10 07:01:00,483 ERROR [STDERR] (mainSmiley Happy jsFile ./js/uicommon/ilp/JxEmbeddedScriptLandingPage.js not found!
2011-08-10 07:01:00,556 ERROR [STDERR] (mainSmiley Happy cssFile ./js/extension/gridfilters/css/RageMenu.css not found!
2011-08-10 07:01:11,877 WARN  [org.apache.solr.core.SolrCore] (mainSmiley Happy Solr index directory '/usr/local/jboss/server/all/data/solr/index' doesn't exist. Creating new index...
2011-08-10 07:01:14,667 WARN  [org.apache.solr.handler.component.SpellCheckComponent] (mainSmiley Happy No queryConverter defined, using default converter
2011-08-10 07:01:33,954 WARN  [org.hibernate.engine.loading.LoadContexts] (mainSmiley Happy fail-safe cleanup (collections) : org.hibernate.engine.loading.CollectionLoadContext@43365e86<rs=com.mysql.jdbc.JDBC4ResultSet@23a2823a>
2011-08-10 07:01:33,954 WARN  [org.hibernate.engine.loading.LoadContexts] (mainSmiley Happy fail-safe cleanup (collections) : org.hibernate.engine.loading.CollectionLoadContext@43365e86<rs=com.mysql.jdbc.JDBC4ResultSet@23a2823a>
INFO: live server is starting..
2011-08-10 06:58:41,002 ERROR [STDERR] (mainSmiley Happy Aug 10, 2011 6:58:41 AM org.hornetq.core.logging.impl.JULLogDelegate info
INFO: HornetQ Server version 2.1.2.Final (Colmeia, 120) started
2011-08-10 07:00:44,585 ERROR [net.juniper.jmp.cmp.deviceManager.ejb.MessageDeviceServiceBean] (Thread-3 (group:HornetQ-client-global-threads-1891955678)Smiley Happy Failed to lookup DeviceService EJB
2011-08-10 07:01:00,468 ERROR [STDERR] (mainSmiley Happy jsFile ./js/extension/gridfilters/menu/RageMenu.js not found!
2011-08-10 07:01:00,478 ERROR [STDERR] (mainSmiley Happy jsFile ./js/uicommon/notification/NotificationUtils.js not found!
2011-08-10 07:01:00,483 ERROR [STDERR] (mainSmiley Happy jsFile ./js/uicommon/ilp/JxEmbeddedScriptLandingPage.js not found!
2011-08-10 07:01:00,556 ERROR [STDERR] (mainSmiley Happy cssFile ./js/extension/gridfilters/css/RageMenu.css not found!
2011-08-10 07:01:11,877 WARN  [org.apache.solr.core.SolrCore] (mainSmiley Happy Solr index directory '/usr/local/jboss/server/all/data/solr/index' doesn't exist. Creating new index...
2011-08-10 07:01:14,667 WARN  [org.apache.solr.handler.component.SpellCheckComponent] (mainSmiley Happy No queryConverter defined, using default converter
2011-08-10 07:01:33,954 WARN  [org.hibernate.engine.loading.LoadContexts] (mainSmiley Happy fail-safe cleanup (collections) : org.hibernate.engine.loading.CollectionLoadContext@43365e86<rs=com.mysql.jdbc.JDBC4ResultSet@23a2823a>
2011-08-10 07:01:33,954 WARN  [org.hibernate.engine.loading.LoadContexts] (mainSmiley Happy fail-safe cleanup (collections) : org.hibernate.engine.loading.CollectionLoadContext@43365e86<rs=com.mysql.jdbc.JDBC4ResultSet@23a2823a>
INFO: live server is starting..
2011-08-10 06:58:41,002 ERROR [STDERR] (mainSmiley Happy Aug 10, 2011 6:58:41 AM org.hornetq.core.logging.impl.JULLogDelegate info
INFO: HornetQ Server version 2.1.2.Final (Colmeia, 120) started
2011-08-10 07:00:44,585 ERROR [net.juniper.jmp.cmp.deviceManager.ejb.MessageDeviceServiceBean] (Thread-3 (group:HornetQ-client-global-threads-1891955678)Smiley Happy Failed to lookup DeviceService EJB
2011-08-10 07:01:00,468 ERROR [STDERR] (mainSmiley Happy jsFile ./js/extension/gridfilters/menu/RageMenu.js not found!
2011-08-10 07:01:00,478 ERROR [STDERR] (mainSmiley Happy jsFile ./js/uicommon/notification/NotificationUtils.js not found!
2011-08-10 07:01:00,483 ERROR [STDERR] (mainSmiley Happy jsFile ./js/uicommon/ilp/JxEmbeddedScriptLandingPage.js not found!
2011-08-10 07:01:00,556 ERROR [STDERR] (mainSmiley Happy cssFile ./js/extension/gridfilters/css/RageMenu.css not found!
2011-08-10 07:01:11,877 WARN  [org.apache.solr.core.SolrCore] (mainSmiley Happy Solr index directory '/usr/local/jboss/server/all/data/solr/index' doesn't exist. Creating new index...
2011-08-10 07:01:14,667 WARN  [org.apache.solr.handler.component.SpellCheckComponent] (mainSmiley Happy No queryConverter defined, using default converter
2011-08-10 07:01:33,954 WARN  [org.hibernate.engine.loading.LoadContexts] (mainSmiley Happy fail-safe cleanup (collections) : org.hibernate.engine.loading.CollectionLoadContext@43365e86<rs=com.mysql.jdbc.JDBC4ResultSet@23a2823a>
2011-08-10 07:01:33,954 WARN  [org.hibernate.engine.loading.LoadContexts] (mainSmiley Happy fail-safe cleanup (collections) : org.hibernate.engine.loading.CollectionLoadContext@43365e86<rs=com.mysql.jdbc.JDBC4ResultSet@23a2823a>
2011-08-10 09:53:01,532 WARN  [org.hibernate.engine.loading.LoadContexts] (mainSmiley Happy fail-safe cleanup (collections) : org.hibernate.engine.loading.CollectionLoadContext@6bd66fa5<rs=com.mysql.jdbc.JDBC4ResultSet@468b9227>
2011-08-10 09:53:01,532 WARN  [org.hibernate.engine.loading.LoadContexts] (mainSmiley Happy fail-safe cleanup (collections) : org.hibernate.engine.loading.CollectionLoadContext@6bd66fa5<rs=com.mysql.jdbc.JDBC4ResultSet@468b9227>
2011-08-10 09:53:02,036 WARN  [net.juniper.jmp.cmp.resourceDistributionManager.ejb.ResourceDistributionManagerBean] (mainSmiley Happy Reset all device connection status to down for node 10.0.2.15
2011-08-10 09:53:02,168 WARN  [net.juniper.jmp.cmp.resourceDistributionManager.ejb.ResourceDistributionManagerBean] (mainSmiley Happy flip config bit 12
2011-08-10 09:53:02,215 WARN  [org.hibernate.engine.loading.LoadContexts] (mainSmiley Happy fail-safe cleanup (collections) : org.hibernate.engine.loading.CollectionLoadContext@23a32b01<rs=com.mysql.jdbc.JDBC4ResultSet@59480073>
2011-08-10 09:53:02,215 WARN  [org.hibernate.engine.loading.LoadContexts] (mainSmiley Happy fail-safe cleanup (collections) : org.hibernate.engine.loading.CollectionLoadContext@23a32b01<rs=com.mysql.jdbc.JDBC4ResultSet@59480073>
2011-08-10 09:53:03,417 WARN  [net.juniper.jmp.cmp.resourceDistributionManager.ejb.ResourceDistributionManagerBean] (mainSmiley Happy syncFilesToNewlyComingNode: this is local node: 10.0.2.15 local ip: 10.0.2.15 skip csync ...
2011-08-10 09:53:03,428 WARN  [org.hibernate.engine.loading.LoadContexts] (mainSmiley Happy fail-safe cleanup (collections) : org.hibernate.engine.loading.CollectionLoadContext@369e247b<rs=com.mysql.jdbc.JDBC4ResultSet@72e0e90a>
2011-08-10 09:53:03,428 WARN  [org.hibernate.engine.loading.LoadContexts] (mainSmiley Happy fail-safe cleanup (collections) : org.hibernate.engine.loading.CollectionLoadContext@369e247b<rs=com.mysql.jdbc.JDBC4ResultSet@72e0e90a>
2011-08-10 09:53:04,819 ERROR [net.juniper.jmp.apps.datastorage.attr.xsd.XSDComplexType] (mainSmiley Happy Warning: Recursive type definition for element 'subcomponent'!!!
2011-08-10 09:53:05,012 WARN  [net.juniper.jmp.cmp.schema.SchemaServiceBean] (mainSmiley Happy Cannot find any SchemaNodeBlockEntities for schema: cmp-settings
2011-08-10 09:53:05,397 WARN  [net.juniper.jmp.cmp.system.AppInitializer] (mainSmiley Happy AppInitializer before version check: cmp.ear
2011-08-10 09:53:05,402 WARN  [net.juniper.jmp.cmp.system.AppInitializer] (mainSmiley Happy AppInitializer is enabling app: cmp.ear
2011-08-10 09:53:19,710 WARN  [net.juniper.jmp.cmp.deviceManager.ejb.MessageDeviceServiceBean] (Thread-0 (group:HornetQ-client-global-threads-1815427789)Smiley Happy ################### 

Recognized Expert
Posts: 456
Registered: ‎09-29-2009
0 Kudos

Re: Kernel Panic error while shutting down JSVA

HI Satish,

 

When you say "Sometime in the middle of our development, kernel crashes." , what exactly are you doing especially accessing the JSVA other than installing /uninstalling?  Are you trying Rest API calls, testing your application ?

 

Also, do you restart Jboss often? and everytime you restart do you wait for the MainUI to come up before you install or uninstall the application?

 

Thanks

Yesh 

Contributor
Posts: 54
Registered: ‎07-26-2011
0 Kudos

Re: Kernel Panic error while shutting down JSVA

When I say sometime in the middle, I meant that we are coding in local Eclipse. Now when we want to deploy app, then we find Kernel is crashed. This does not happen  every hour. Since today morning, kernel crashed once. Kernel cashes after JSVA is up and running for 4-5 hours.

 

We do wait for mainui screen to come up.

 

We do not touch jboss at all. We shutdown JSVA gracfully and then restart. Jboss and mysql are automatically started when JSVA starts.

Highlighted
Recognized Expert
Posts: 456
Registered: ‎09-29-2009
0 Kudos

Re: Kernel Panic error while shutting down JSVA

HI Satish,
I was able to reproduce the issue with different steps but my crash trace matches yours. I have found a solution for the crash which is working in my environment, please try and let me know.Which is your processor and RAM?  
I reproduced the issue as below, 1.      Started JSVA

2.      Waited for  MainUI to show up

3.      Restarted Jboss

4.      Tried App installation from Eclipse before Jboss loaded all the components

5.     It goes to kernel panic mode


Solution: 


Open the file /boot/grub/menu.lst and modify kernel boot parameters as below. Here I am disable SMP and APIC since you are using single CPU which seems to be causing the issue. 


 kernel  /vmlinuz-2.6.18-92.el5  ro root=/dev/VolGroup00/LogVol00 nosmp noapic

Reboot the system. Please work as usual and let me know how it goes.
Regards

Yesh

Contributor
Posts: 54
Registered: ‎07-26-2011
0 Kudos

Re: Kernel Panic error while shutting down JSVA

Thanks Yesh. I have made the change as you suggested. Now we will monitor for next 2-3 days and will let you know.

 

Regards

Satish