Basic Operational Checks

See below a set of basic Operational checks that need to be carried out when building an instance of Ooyala Flex.

1. Check that the Ooyala Flex process is running

Check that the Java process for Ooyala Flex is running on the system:

ps –ef | grep mio

If the relevant Java process is not running then the JVM may not be running or has crashed. In this case, please provide with the relevant /tmp/hs_err_pid*.log for investigation.

Use ‘top’ on the server to check the state of processes and any which might be tying up the CPU.

2. CPU's and RAM

If performance is significantly affected, check whether the number of workflows/jobs running has increased significantly according to the number of CPUs and amount of RAM assigned to the server. It may be that more CPUs/RAM will need to be added.

3. Check log files for any exceptions or errors

On the Ooyala Flex master server, check for any exceptions or error messages appearing in the server log file.

/nem/logs/jboss/logs/server.log

If there are exceptions, check back through the history of logs to see if you can ascertain when they first began and if you can identify any regularity/pattern.

grep “Exception” /nem/logs/jboss/logs/server.log

4. Check the Logging Level

If JBoss logging has been set to DEBUG mode, this can adversely affect performance.

/nem/mio/jboss-eap-6.3/standalone/configuration/mio-*.xml

5. Check the status of Workflows

From the Ooyala Flex Console check that the number of created workflows are running or whether they have failed in the last 3-24 hours.

1) Log into the Ooyala Flex Console

2) Click on the Workflows tab

3) Click on the Advanced button on the right hand side (just below Search Workflows)

4) Use General or Workflow options for the searching

If most jobs are in a ‘scheduled’ state, the Ooyala Flex master will require restarting. If the master is clustered then identify which master is currently serving the live website. Log onto one of the master servers and check the number of interfaces displayed:

IP Address

If two IP addresses are listed, then that server is serving the live website and should NOT be restarted. Restart Ooyala Flex on the other server.

Ensure that ‘ucarp’ is running on the restarted server for automatic IP failover:

cd /nem/carp

sudo nem status

Go to the first master server and disable NEM

cd /nem/carp

sudo nem disable

6. Take relevant Java dumps of the affected Java Process

While Ooyala Flex is still affected and the Java process is running, it is worthwhile taking a few dumps from the process for further investigation.

Java Stack Trace Dump

Take a Java stack trace dump if the Ooyala Flex process is still running and send it to the support team for investigation. This will take a dump of active Java threads for a given Java process. It might be worthwhile taking a few dumps over a given time period for comparison.