We have upgraded from 8.1.02 to 9.1.03 (RHEL 7, Oracle), Install failed at post install activities since the server couldn't come up in a timely manner. Yet, the "select dbversion from control" suggests that DB is upgraded to 9.1.03. I needed to fix ar.conf (with few custom changes) and tried to bring up the server, server and monitor comes up (but not fork) and fails shuts down with Error creating bean with name 'serverInitializer' defined in URL [bundleentry://162.fwk66233253/spring-context/application/domain/init_context.xml]: Invocation of init method failed; nested exception is ERROR (8552): Message not in catalog; Message number = 8552. Tried looking up, but i couldn't find what 8552 means.

Going by the debug logs, import of few system forms failed with Internal Exception: java.sql.SQLIntegrityConstraintViolationException: ORA-00001: unique constraint (ARADMIN.SYS_C00226042) violated. That seems to be about overlays and we don't use overlays in our system.

Elapsed: 00:00:00.05

SQL> SELECT column_name, position FROM all_cons_columns WHERE constraint_name = 'SYS_C00226042';

SCHEMAID

1

FIELDID

2

OVERLAYGROUP

3

Above error appears multiple times for all the system form imports.

We encountered this error earlier as well and BMC support couldn't suggest what might have caused this and suggested to re-run the install in 'Pause' mode which didn't work for us ( we are using silent mode). To get through this error, we tried creating overlays (though we don't use them) and re-run the install, and ended up in the same scenario. Just the unique_constraint number varied, but about the same columns. Have You see ever seen this kind of error before? Any pointers are highly appreciated.

The 8552 error means AR_ERROR_NO_CONFIGURATION_NAME so there appears to be a problem looking up the CCS config data for the server. What value do you have for Configuration-Name: in the ar.cfg and does the value appear in this output?

Find the schemaID of the AR System Configuration Component form

> select schemaid from arschema where name='AR System Configuration Component'

We literally corrected this last night. Added Configuration-Name: to ar.conf and a different error pops up now & AR server is still not up. We were back to the same error (from the previous install attempt)

Thanks for attaching the logs. Can you attach the .zip from maintenance tool too?

Its strange that the arerror.log (why are there two of them?) reports errors while importing system forms but the sql log doesn't show any hint why (or ony statement refering to a form/workflow import). The only inserts I found seem to be configuration data.

Apologies for the confusion. Based on the interactions with BMC support, we tried few things to see if the server comes up

Earlier the start ups were failing with unique constraint violation errors. Server tries to import the system forms, instead of overwriting it tries to INSERT new rows in field table for existing columns and gets the below error and (after 7-8 minutes) fails with an initialization error. Neither We nor BMC have an explanation for this behavior.

An example here. Please note that this just for one of the forms and this error repeats for all almost all the system/install form imports (50+ forms) as part of the start up.

SQL> SELECT column_name, position FROM all_cons_columns WHERE constraint_name = 'SYS_C00226017';

SCHEMAID 1FIELDID 2OVERLAYGROUP 3

To isolate the root cause and in an attempt to get the server up

(1) BMC suggested to remove/rename the system & install forms so that installer can't find them to import. Hence arerror.log reports since it can't find the def files, while the SQL log wont have any trace since the actual import did NOT happen

(2) Since we have so many group records, we suspected that could be another reason for the server to choke, so we cleaned up all custom group and just retained the bare minimum. This caused the server to fail fast on start up ( instead of waiting for 7-8 mins)

(4) Currently, start up logging is enabled as well (arserver.jar -t -s) , to get any clue for the start up failures.

And thanks for the hint on bundle cache, this is the error in see in the latest log under bundle cache

!ENTRY com.bmc.arsys.restapi 4 0 2018-02-02 10:40:21.886

!MESSAGE FrameworkEvent ERROR

!STACK 0

java.lang.NullPointerException

at com.bmc.arsys.restapi.resourcemanager.RestfulConnector.removeResource(RestfulConnector.java:85)

at com.bmc.arsys.restapi.resourcemanager.ResourceTracker.removedService(ResourceTracker.java:57)

at org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:956)

at org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1)

at org.osgi.util.tracker.AbstractTracked.untrack(AbstractTracked.java:341)

at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:902)

at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)

at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)

at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)

at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)

at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)

at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)

at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:225)

at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.unregisterServices(ServiceRegistry.java:635)

at org.eclipse.osgi.framework.internal.core.BundleContextImpl.close(BundleContextImpl.java:88)

at org.eclipse.osgi.framework.internal.core.BundleHost.stopWorker(BundleHost.java:514)

at org.eclipse.osgi.framework.internal.core.AbstractBundle.suspend(AbstractBundle.java:566)

at org.eclipse.osgi.framework.internal.core.Framework.suspendBundle(Framework.java:1206)

at org.eclipse.osgi.framework.internal.core.StartLevelManager.decFWSL(StartLevelManager.java:592)

at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:257)

at org.eclipse.osgi.framework.internal.core.StartLevelManager.shutdown(StartLevelManager.java:215)

at org.eclipse.osgi.framework.internal.core.InternalSystemBundle.suspend(InternalSystemBundle.java:284)

at org.eclipse.osgi.framework.internal.core.Framework.shutdown(Framework.java:692)

at org.eclipse.osgi.framework.internal.core.Framework.close(Framework.java:600)

at org.eclipse.osgi.framework.internal.core.InternalSystemBundle$1.run(InternalSystemBundle.java:261)

at java.lang.Thread.run(Thread.java:748)

I will try to clear the bundle cache and rerun the db checker and see if i can find any more info.

On a different note, we have a weird confusion, our DB character set is UTF8 and we store and some unicode chars as well, but BMC thinks that we are non-unicode ( to decide which dbchecker has to be run, since unicode and nonunicode has different versions of checker). Any inputs?

But the control table doesn't have some of these columns (associationId, useSHA256 ) & there are no rollback statements as well - just to rule out that the control may have been altered but rolled back later.