Hyperion - Essbase Production Server crash

Issue:Essbase Production Server crash: Unable to connect to Essbase Server

Solution:Removethe essbase.lck file and reset the JVMMODULELOCATION back to the location of the .so file.

Essbase now starts up.

Cause Determination 

The root cause was 2 things

1)  JVMMODULELOCATION was commented out

2) essbase.lck file was present in the bin directory.

The LCK file is present when essbase is started and should get deleted when essbase stops normally.That file just has to be renamed or deleted to allow essbase to start 

Action Plan

Essbase was not able to start up.

Perform the below steps

1.  Check the essbase.cfg file. The JVMMODULLOCATION was commentd out and not set to NULL. The JVMMODULELOCATION must be  set to the SO file or at minimum set to NULL.
2. I placed it back to the default value. There is no need to set this to NULL if they are not hanging or crashing.
3.  I use the startEsscmd.sh and created a new script to start essbase in foreground.
4.  Remove the essbase.lck file
5. Use the new script Essbase was able to start without the message that it is already loaded.

Proposed Solution 

• Remove the essbase.lck file and reset the JVMMODULELOCATION back to the location of the .so file.
• Essbase now starts up

 ===Cause Justification ===

Essbase uses the LCK file to determine if it's running.

If there is a termination that file may not get removed, hence preventing essbase from starting up.

 ===Issue Clarification ====

EPM 11.1.2.0.00 version, Server Administration

When attempting to start Essbase they get a message that essbase is already loaded.

the following error occurs.

ERROR

-----------------------

essbase is already loaded

STEPS

-----------------------

The issue can be reproduced at will with the following steps:

1.  starting essbase using startessbase.sh if the essbase.lck file is present. . Essbase Server log
2. Attached all the logs from location MIDDLEWARE_HOME/user_projects/domains/domain name/servers/managed server name/logs
3.  Run \Oracle\Middleware\user_projects\epmsystem1\bin\ziplogs.sh 
4. The output zip file will go to \Oracle\Middleware\user_projects\epmsystem1\diagnostics\ziplogs
5. Run \Oracle\Middleware\user_projects\epmsystem1\bin\validate.bat 
6. The HTML output will go \Oracle\Middleware\user_projects\epmsystem1\diagnostics\reports
7. \instance_report_20110415_025021.html
8. "instance_report_20110415_025021.html" Name of the File Might Change.
9. Confirmed the issue. Essbase do not start up. -

The following Error occurs when trying to access Essbase Server from EAS  Console -

ERROR - 111 - Unable to connect to Essbase Server - Confirmed that there is no issues with Shared Services & EAS. 

Steps Performed:

• From the machine, Launched EAS Console and logged into EAS, and it was successful.
• Tried to access Essbase from EAS Console, the following error occurs -
• ERROR[111] - Unable to connect to Essbase Server :1423. The client timed out waiting to connect to Essbase Agent using 
• TCP/IP. Check your network connections. Also make sure that server and port values are correct.
• Pinged the Essbase from the client machine and it was successful
• Executed the telnet command from the  machine and it failed. Informed the team  that Essbase is not started.

 Edited the essbase.cfg file with the following -

AGENTPORT 1423

SERVERPORTBEGIN 42768

SERVERPORTEND 43768

AGENTDELAY 60

NETDELAY 600

NETRETRYCOUNT 800

NETTCPCONNECTRETRYCOUNT 100

AGENTTHREADS 60

SERVERTHREADS 60

AGTSVRCONNECTIONS 7

CALCCACHE TRUE

CALCCACHEHIGH 199000000

CALCCACHEDEFAULT 10000000

CALCCACHELOW 5000000

CALCLOCKBLOCKHIGH 10000

CALCLOCKBLOCKDEFAULT 5000 

CALCLOCKBLOCKLOW 2500

MULTIPLEBITMAPMEMCHECK TRUE

PARCALCMULTIPLEBITMAPMEMOPT TRUE

6.Replaced the Essbase security file (essbase.sec) from essbase.bak
============================================


Hyperion Essbase Service was running at 100% server utilization. This occurred while the Essbase Reporting cube and Hyperion Planning Database were in shutdown mode. The following steps were taken: 

As can be seen in the screen shot below, the Essbase Service is running at 100%. 

 ended up finding out that the issue was caused due to an authentication corruption in Shared Service. To fix this, first make a copy of the essbase.sec file for backup purposes. Once that is completed remove it and replace it with a copy of the essbase.bak_postUPM. 

 

After the backup security file has been moved into place, stop and start the Essbase service. Once the Essbase service is backup, we can see we have fixed the issue as the Essbase service is now running idle rather than the 100% it was previously. 

Since the security file was replaced, you will need to perform a security refresh from the Hyperion Planning application to resync the security from Planning to Essbase.

One issue that was found after fixing the corrupted Hyperion Shared Services security file, was specific Hyperion Essbase Application Properties were reset for each of the Essbase Applications. This caused the pag files to explode from one pag file to 82 pag files.

The 0 level file size was 500mb for these Hyperion Applications, with only two aggregating dimensions. Due to the explosion of the pag files we immediately knew that the Data Compression was flipped to No Compression. After the security file was replaced and a new data file was loaded into the application, it was found that the compression was turned off for the app. (Located in the database properties under the Storage tab) 

 

The remedy for this is to switch the compression from No Compression to Bitmap encoding 

 

Stop and restart the Hyperion Essbase application. After clearing All Data, reload the data file into the Hyperion Essbase application. The result should be one index file and one pag file with the compression turned on. 

===============================================