Today, I will tell you why my last posting with the video of how to capture http traffic is so important.

You all probably know the symptoms - you run your Coach and it's loading. You check the news in a separate tab, you jump back to the Coach, still loading,

loading aaaaaaand loading. Okay, I am exaggerating a bit, but sometimes the loading times feel like they are taking forever.

Now, before contacting IBM Support to help you solving your performance issue, I wanted to share some steps you can use to further diagnose your problem.

And for that, we will need this *.HAR (HTTP Archive) file that is being generated.

Now, to look at this HAR file, you will require an HAR Viewer (http://www.softwareishard.com/blog/har-viewer/).

Now simply drag&drop your HAR file onto the page and click on the Preview tab.

Here you will see all the requests with request/response, as well as header information, the response codes to the requests and most importantly - the timings.

Since we have long loading times for our Coach, we are particularly interested in those loading times.

A good indicator is the bar on the right hand side. If it is purple and the number given next to it is quite small (it's the actual time this request took in milliseconds), that's good.

Just some general remarks:

The purple bar indicates the wait time, i.e. the time the client waited for a response from the server. The smaller, the better. Of course it depends on the size of the request, network latency, server load etc).

You may see a brownish bar that indicates Blocking time. This is the time the request spent waiting for a network connection.

Blue bars indicate the time the DNS lookup took, green is the time it took to create the actual TCP connection.

Red bars show the time spent on sending the HTTP request to the server, and as said earlier, purple is the waiting time for the response.

They grey bar stands for the time it took to receive and read the complete response from the server.

In some cases you might even see SSL timings, which show how long the handshake etc took.

Just open the twisties at the bottom named "Cuzillion" and you can review these timings.

Or you can reload this page, grap the HAR from your browser console and it might look something like this:

I had a patient the other day, whose HAR file showed something very interesting. The actual waiting times were pretty small, all only a couple of milliseconds. Yet he still told me his Coach was ailing. There were pages and pages of requests. I stopped counting after 200. The farther i scrolled down in the list, I noticed that the brown bar continuously increased.

Brown bar? Yes! Blocking time! What was that again?

Exactly! The time waiting for a connection. I am not a plumber, but this smelled like a clogged drain. To use the medical term - the patient was probably suffering of constipation.

I asked my patient to take the longest taking call (it was a REST call - I could see that from the request details) - and run it against the server from the REST-API tester while capturing the HTTP traffic again. In isolation, no other requests, just this one.

The result: It finished in no time. So we knew it was not the DB or Server that had a problem with that request. This supported the drain theory.

What can cause such a constipation are the connection- and thread-pool settings for your server. In other words: If you eat more than you can egest, at some point you will feel constipated.

I gave him the advice to increase the pool sizes in an iterative process, always monitoring the performance of his Coach while doing so.

And voilá - the clogged drain was free again.

And if none of this helps, take two of these and call me in the morning.

Some of you may have already encountered such a situation. You get your morning coffee, boot up your workstation, you start your IBM Integration Designer (IID) and bang - there it is. A crash notification. What a good start in your day.
Right after you started the application, it crashes and presents you an Eclipse pop-up window like the following:

Java was started but returned exit code=1

If you have not encountered this yet - Congratulations! I would still recommend reading this post, maybe some days you will need this .

So, what's next?

Step 1:Prepare yourself for some debug fun. If your coffee is gone, I recommend getting another one, then continuing with the below steps. If you still have coffee left, move right on to the next step.

Step 2: Did this installation ever work or did you just install it? If this error shows up after a new installation of the Eclipse based product (here IID), check out the installation logs to make sure it was installed without any exceptions. To do so, have a look at the IBM Installation Manager log files under <Installation_Manager_Install_Dir>/logs

In my case:

Open the index.xml file and review all logs linked in there for any exceptions or failures during the installation.

Failed or partially failed installations may cause such statup issues.

Step 3: Did anybody mess with your Eclipse startup parameters? If this is not a new installation and the exception happened all of a sudden, maybe some JVM arguments were changed and are hence causing this crash. Always a good candidate is the -Xmx value, which specifies the maximum memory allocation pool for the Java Virtual Machine (JVM). A lot of users still think: The more, the better! Unfortunately they overtune the JVM and the memory allocation is simply too large for the JVM to be successfully created. If changes were made to the JVM settings, please revert those changes back to the default values (you can for example compare them to a working or untouched system). Then try to launch your Eclipse based application again.

Step 4: Create Javacore log files to gain more insight into the root cause of the crash. As Eclipse is still in the start-up process when the crash occurs, the JVM errors cannot be written to the Eclipse error logs under <workspace_root>/.metadata/.log
Hence we need another way to get more information about the root cause. Open your eclipse.ini file (located in the <install_root> directory) and add the following line under -vmargs opt:

-Xdump:java:events=vmstop

This should enforce the creation of a javacore.txt file in case of a JVM crash. In our case here, you will find it in the IID root directory after the next restart.

The other day I ran into a problem, where my IBM Integration Designer (IID) did not start properly. I tried to start the application, but the startup never finished and was hung. Frozen - no way to stop it except killing the process. Alternatively I could have waited for it to crash..

If it does crash during the workspace initialization phase, a log file is written to the workspace/.metadata/ directory. The file is named ".log" and is always worth a look. In my case, I saw the following exception:

I looked at the LSW_SERVER table of my BPM database and the server entries were listed.

My first thought was that I was hitting APAR JR42774 - as described in this Technote.
However, a look at the 7.5.1.2 fixlist showed the iFix for this APAR is included in 7.5.1.2.

Reviewing the table definition of my LSW_SERVER table, I saw that the password column had a definition of varchar(256), although after the installation of FP2 (or JR42774) it was supposed to have varchar(1000). So something must have went wrong during the installation of the fixpack - most likely during the post-install steps where the db schema was upgraded.

After I updated the schema (as described in the post install steps) I nulled the passwords in the LSW_SERVER table, as the heartbeat would simply re-add them afterwards.
And voilá - the servers were listed again.

And if you are facing the same issue but the above steps do not help, take two of these and call me in the morning.
Your Doc D.

Today I would like to show you, how you can set a specific editor to display a certain file type.

As you all may already know, the IBM Integration Designer (IID) is a development tool where you can use a Graphical User Interface to create and test your applications. For this GUI there is a set of editors available that will display your artifacts in a graphical view instead of a file view.

An example would be the mediation flow, which is opened with the "Mediation Flow Editor" by default when you open the flow.

So, you may run into situations where the view is not available, so the mediation flow might open in the XML file view. First attempt would be to open it from the context menu (BI view >> right click on your mediation flow name >> Open With >> Mediation Flow Editor). If you are as unlucky as I am, this option is not available, and the context menu looks like this:

How can I get my Mediation Flow Editor back? Well, it's not hard if you know how. And I am going to show you how.

All you need to do is to register the editor for the related file type again. In our case the mediation flow (associated to the .mfc file type).

To do so, click on Window >> Preferences and click on the little eraser icon to remove all filters.

Then click on General >> Editors >> File Associations

You will find a list of file types where you can associate the correct editor to the according selected file type.

In our example, we will select the file type .mfc. You will now see the editors in the "Associated editors" section. With the "Add..." button we can now add additional editors for this file type. And we will now add the 'Mediation Flow Editor".

All we need to do now is save our changed and go back to the BI view where we will open the context menu to verify the Mediation Flow Editor was added:

And we're there. It worked.

And if all of that does not help, take two of these and call me in the morning.

Last week I came accross an interesting issue and I thought I would share my findings with you.

In BPM version 8.5.5 the new Client-Side Human Service (CSHS) was introduced.
In a fairly simple usecase, we had a CSHS with a Coach, and that Coach basically had 2 buttons - Button A and Button B. Each button had a certain value assigned to a variable. Upon a click of the button, the value of this variable was changed in a Client-Side Script using Javascript. We then ran a service (General System Service) to throw a validation error. The following picture will give you an idea of the whole scenario:

The Service looked like this:

It is important to understand, that the validation step was a script named "Validation Script" by the user. The validation service has a green icon (V) leading to the script step. If you don't see it, this is just a regular script step (not true validation). See this Knowledge Center link.

From the Knowledge Center:

Please note the green checkmarks or Vs.

However, to configure this validation service, you would have to set the "fire validation property" to "before" for the line to the boundary event. But the CSHS does not support this kind of validation. This is only available for Heritage Coaches in v8.5.5.

In our usecase, when we clicked the button the values of the variables should have been changed when the validation error was triggered, however, the variables still showed the "old" values.

So why was that?

Well, in v8.5.5 for the CSHS, when validation errors occur, ONLY the validation errors info is returned back to Coach for display. Any changed variables are NOT returned to the Coach. This explains why our variables showed the old values, and not the ones we assigned to them in our Client-Side Script.Note that this behavior will be enhanced in v8.5.6 where both validation errors AND changed variables are to be returned.

With this in mind, we can now understand why our variables remained unchanged.And if this does not help, take two of these and call me in the morning.

As the login was done with an anonymous UserID, no file listing is allowed on the server. You will not be able to check what files you have already uploaded, which means that you will need to do this very carefully.

If this does not work for you, take two of these and call me in the morning.

A few days ago, I was presented an interesting issue. One of my customers experienced an unacceptable wait problem while starting their BPM AppCluster.
As always, the first step is to look at the SystemOut.log to see, if there is anything interesting in there.
In this case, there were a vast numer of DB queries, similar to this:

select data,version_id from lsw_bpd where version_id in (select distinct ver.po_version_id from lsw_po_versions ver inner join lsw_snapshot snap on (ver.branch_id=snap.branch_id and ver.start_seq_num<=snap.seq_num and ver.end_seq_num > snap.seq_num) where ver.po_type=25 and snap.name is not null)

Yep, yet again another Performance debugging.

As my customer mentioned the delay starts right then, when cache settings were loaded, I checked the SystemOut.log for anything Cache related during the start-up.

I found the following line:

000000ae wle I CWLLG2155I: Cache settings read have been from file file:

As you can see, the threadID for that activity is 000000ae. So I grepped for 000000ae to see the whole thread.

And I was quite surprised to see a !9 MINUTE! delay in the start-up. You know, most of the performance cases we see are on a very high level, meaning we are talking of improvements of a few seconds. But this one, I have to tell you, had a good point. Now 9 minutes is an awefully long time to wait. 9 minutes - that is about 2 cups of coffee. Too long, as you will most likely agree.

From the information in the logs it seemed that the delay was related to the Cache, as it occured between the cache settings being loaded and the cache being initialized.

So, where could this be coming from? Maybe some slow DB causing this delay? Well, we do not yet have enough information to tell, hence we need some traces.

This APAR was introduced for BPM 7.5.1.2 with a new property named <use-business-aliases-for-process-instances>.
The default value for that property is " T R U E ".

TRUE means that the list of searchable Business data aliases are returned only when they are used in actual process instances.

If FALSE is selected, the Business data aliases will be created at the server startup already BEFORE the first access by process instances. If there is a large number of BPDs, searching all the BPDs can take a very long time.

The more process applications are deployed on server the more aliases will be created, the longer the cache load time takes.

So, if this was introduced in 7.5.1.2 why did I still see this in 8.5.5? APAR JR47818 is part of BPM v8.5.5.

Hm, what could have happened here?

As my grandmother would have said: Never trust a running config (if she had known what a running config was). Hence lets have a look at the 99local.xml file on my BPM 8.5.5 setup to verify the default configuration for that attribute.

Indeed, the value is set to " F A L S E " !

Checking the customer's 99local.xml and 100Custom.xml showed the same.

The patient needs to report an urgent issue to IBM BPM Support and wants to provide valuable information that expedites the resolution right from scratch.

Diagnosis

For Dr. Debug and the IBM Support team it is of utmost importance to have as much information as possible in order for being able to diagnose the problem. Remember: You might know pretty much all about the error and the environment, the steps to recreate and what was done prior to the error. But this is all unknown and new to the support folks.

Potential treatments

Should you require Support's assistance, the following steps may help you providing some basic information that should help Support to start investigating your problem.

Ask yourself: What steps do I take in order for the error to occur? What are the symptoms? What was the expected result? Was it working before? Have I changed anything before the problem occurred for the first time?
Note the answers to these questions and provide them to the support folks.

Provide as many screenshots as possible. The more detailed the better. If you know how to recreate the problem, take a screenshot for each step. It will make Support's life easier. It will enable them to understand and see what you are referring to. Or even better: Record a quick video of how you recreate the problem.

Gather your versioninfo output. It gives an excellent overview of the patients healthstatus and previous treatments if you will. Sort of like: Who is the patient? How old is the patient? What previous issues did the patient have? What Support can pull from these information is which ifixes were installed yet, which exact version are you running? Is there a known issue that has already been fixed and the iFix was not yet installed on this system?

Now, you might be asking yourself: OK, how do I gather the versioninfo? That is fairly easy:
Just open a command prompt and run the following command from <install_root>/bin :

In BPM v7.5.X:versionInfo -maintenancenPackages > versioninfo.txt

In BPM v8.X:versionInfo -fixpacks -ifixes > versioninfo.txt

or:

versioninfo -long > versioninfo.txt

This will redirect the output of the versioninfo command to a textfile in the bin directory. You can also specify any other custom path for the output if you wish.

Gather information about the underlying database, the OS and hardware specifications that might help.

Does the patient have a fever? Is the patient sneezing or coughing? In other words: Do you see any errors in the logs?
First thing to look at is the SystemOut.log file which you will find under:

<install_root>/profiles/profile_name/logs/server_name

These logs are created per profile and server. Note the timestamp of the occurence of the issue before sending the logs to IBM Support. That will expedite their review of the log files.
If you spot an exception in the logs, you can try to search for potential known issues and how to address them. Good resources are dwAnswers, the BPM Fixlists, Technotes or sometimes even Google. In some cases the solution is already available.

Taking a blood sample to gain further details and insight. Now we do not poke into our workstation with a needle, but we use traces. There are traces for almost anything. The tricky part is to define the correct one. IBM Support will provide you with the trace string to use. At the bottom of this Technote you will find links to component-specific diagnostic information. Most of those include component specific traces you may use.
To set the trace string IBM Support or the Technote provided, follow these steps:

- Log on to your WAS Administrative Console
- Expand the Troubleshooting section
- Select Logs and Trace
- Select the application server to be traces and click on the Diagnostric Trace link
- Select the Runtime tab
- Click on Change Log Detail Levels under the Additional Properties section on the right hand side panel
- Under Trace Specification enter the trace string
- Click Apply and OK. Then Save your configuration.

The trace will be saved in a file named trace.log by default. Do not forget to note the timestamp and provide the trace.log, SystemOut.log and timestamp to IBM Support. Follow these steps or the steps listed in this technote to get your traces set up.
You can also gather traces in a static manner. The difference is that you need to restart BPM when using static tracing. And that usually mean high impact.
Please keep in mind that our patient is temporarily limited in his actions while the trace is enabled, meaning that you may encounter performance degradation while the trace is enabled. Remember to remove the trace string once you are done.

Upload all the data and information you gathered to your PMR.

Be proud of yourself, you helped IBM Support a lot to expedite the resolution of your problem.

Reward yourself with a nice cup of coffee or tea.

In general it is quite handy when all the information Support needs is available right from the beginning. If anything is missing, do not worry. Support will let you know what they need in addition to what you have provided.

And if this article does not help, take two of these and call me in the morning.
Yours,
Dr. Debug

The other day I came accross an interesting new problem while working with BPM in the SmartCloud Orchestrator (SCO). It was in a multi-staging environment and my client was trying to export a change made in their test environment to import it in their production environment.

When you are trying to export your freshly developed code from a test system and import it on a production system you would usually create a snapshot on test, export it and import it on production. Pretty much straight forward, no magic.

Now if the Process Application or the Toolkit that you are trying to export/import has dependecies on theSCOrchestrator_Scripting_Utilities_Toolkit (SCOSUTK) and SCOrchestrator_Support_IaaS_Toolkit (SCOIAAS) (both shipped with SCO) and these Toolkits are already present in production, you may receive a notification during the import stating that these Toolkits already exist and will not be imported.

If you confirm this with 'OK' you may receive the following error message indicating the import failed:

Now, what went wrong?
In the installation guide for SCO IF0004 and IF0005 some of the manual steps described can lead to this problem.

For the mentioned toolkits SCOrchestrator_Scripting_Utilities_Toolkit (SCOSUTK) and SCOrchestrator_Support_IaaS_Toolkit (SCOIAAS) you have to update the dependencies and you need to create a new snapshot.
By doing this on both systems (test and production) individually, the internal Snapshot ID will differ in both environments due to the algorithm that is used to create these IDs. Although the Snapshot-Name as well as the acronym are identically on both systems, the import process will not identify them being the same due to the different IDs. Which does make sense.

Hence, a correct approach in a multi-staging environment is to perform these actions (updating the dependencies and creating a new snapshot) on one system and then export/import these to the other systems. Do not perform these steps on the target environment as this will lead to the above error.

If you encounter this issue, and the above steps did not help, take two of these and call me in the morning.
Yours truly,
Dr. D

Last week I stumbled across an interesting question in BPM. As it took me some time to figure out why BPM was behaving the way it did, I thouhgt I'd share my findings with you, as it may help you some day as well.

The scenario was the following:
I was using a Human Service (HS) with a Coach that consisted of a Document List and a Document Viewer. My BPM version was 8.5.5 but the exposed behavior is the same in 8.5.0.1 and 8.5.6 as well.

Here is my HS and the Document List Configuration:

As you can see, I used a custom Search Service. My goal was to filter the document list based on the Content Type of the documents stored. I wanted to accomplish that my filter looked up the ContentType in a local variable, so I could simply change the value of the variable without touching my search criteria to adapt my searches.

To do so, I added a search criterion in the Search Service by browsing the Content Filters tab and clicking the "Add Search Criterion..." button:

This will add a new criteria. As I wanted to check for the Mime Type, I selected Mime Type in the first field, and selected "is equal to" in the second field:

In the third field as a first test I wrote "application/pdf" (without the quotes) to see, if it would work:

I ran the HS and it did work:

Now I wanted to use a variable instead of hardcoding the MimeType in my Search Criteria. So I created a new Local Input variable named "ContentType" and set it's default value to "application/pdf":

And I adapted my Search Criteria by entering the name of my variable (tw.local.ContentType) into the third field:

Here an interesting read that our community member O.J. sent in. Thank you O.J. for your contribution. This is good to know!

Enjoy this read,

Yours
Dr. D

Environments updated to BPM version 8.0.1.3 or 8.5.6 may have issues in the Process Portal or in a custom-written Coach, where data is not displayed as expected. Related SystemOut.logs will show:

[31/03/15 16:33:52:855 BST] 0000015a CallServiceAc E The servlet, callservice.do, is not configured to call Example; configuration option
"callservice-valid-services" is not configured to call services of type General System Service.

This is due to APAR JR50215 which is included in BPM 8.0.1.3 and 8.5.6
Because there is no access restriction based on service type when invoking a service using the callService URL, services that were meant for internal use only are exposed.
The fix enables administrators to restrict callable services by type. The fix is secure by default and only allows AJAX services to be invoked via the callService.do URL. If you have custom client applications relying on callService.do exposing service types other than Ajax Services, you need to add configuration as described below.

Edit your 100custom.xml to contain desired service screening. Administrators are given the ability to change this to allow any permutations of different services that will be allowed.
Be advised that opening multiple services to be executed by callService.do will introduce a possible security issue. In fact, even AJAX services could be used for an attack by an authenticated user, given they know specifics about the service.

Service developers should redesign any potential services to use something that will not be allowed to be executed.

If the <callservice-valid-services> tag is not added to 100custom.xml, Process Portal will default to only using AJAX services. Should you need to invoke callService.do to launch a service, you need to ensure there is a <valid-service-entry> tag (inside the <callservice-valid-services> tag) containing the type of the service you need to launch. An example to only run integration services is shown below:

Users may use any permutation of the service ID to allow specific types of services to be executed by callService.do.
Another example would be shown below, which will block everything except for Regular Services, AJAX Services, and SCA Services.

The patient has recently installed a fix pack (in this particular case it was on WLE 7.2.0. FP5)

After installation, the patient discovered that changes to environment variables were not picked up by the process app

For instance, our patient had an environment variable called „testVar“ which was set to 25 for DEVELOPMENT before the upgrade was performed. Now after changing it to another value (like 100, see screenshot), the value of 25 was picked up every time.

At the patient the symptom can be confirmed by our excellent diagnosis tool - the Execution Evaluator:

Even creating a new snapshot did not help the patient

The patient was quite suprised as there were no issues in that respect before WLE/BPM had been upgraded

Diagnosis

The reproduction attempt of the symptom at a test person (test environment – not upgraded) showed a working scenario. Here, the environment variables switched to the proper values.

A further test on the patient was to create and test another environment variable. At this test the patient showed no symptom.

Summarized, Dr. Debug faced the following situation:

The symptom is related to process applications which existed before the upgrade

After the upgrade the environment variables revealed an inconsistency

This inconsistency was not saved in the .twx-file as a further test person showed no symptoms

Hence, it seemed to belong to the database and a specific references there

Dr. Debug's assumption was that some references of the affected process app might have gotten corrupted by the upgrade. He further thought, an export of the process application before the upgrade and an import back after the upgrade might have prevented this issue.

Potential treatments

Now, what could Dr. Debug try to heal the patient?

Right, he can try to treat this disease by creating new database entries and references.

For instance, a potential treatment can be the cloning of the affected process application. By doing so, new references in the database are created and all environment variables should work as expected.

Another option would be to remove the affected artifacts and create new ones. However, this needs to comprimise all affected artifacts.

It seems that the better option is the cloning since an entirely new process application is created.

Hence the cloning approach was Dr. Debug's Advice in this case. The result was very positive. All environment variables took the proper values according to the environment type used. Hence, the patient no longer showed the symptoms and Dr. Debug's Advice helped another patient back on his feet.

And if this does not help in your case, take two of these and call me in the morning.

Today I want to share and IBM Integration Designer deployment issue with you.

To share some background: The goal of the task described here was to migrate an SCA (Service Component Architecture) project created in WebSphere Integration Developer (WID) v7.0.x to IBM Integration Designer (IID) v8.5.5. The migration itself worked pretty well, but afterwards, the deployment from IID to the Process Server failed with the below error message:

Reviewing the server log, I saw a lot of scatterted exceptions ending with the deployment error:

java.lang.IllegalStateException: Library NISSP_DK_Lib at version could not be resolved from the ear, current BLA or globally at com.ibm.ws.al.scope.profile.SharedLibraryLoader.getLibrary(SharedLibraryLoader.java:166) at com.ibm.ws.al.scope.profile.SharedLibraryLoader.computeRefLib(SharedLibraryLoader.java:94) at com.ibm.ws.al.scope.profile.SharedLibraryLoader.getReferencedLibraries(SharedLibraryLoader.java:78) at com.ibm.ws.al.scope.profile.SharedLibraryProfileHelper.addSharedLibraries(SharedLibraryProfileHelper.java:288) at com.ibm.ws.al.scope.profile.ArchiveProfile.getCatalog(ArchiveProfile.java:65) at com.ibm.ws.al.scope.ALScopeImpl$1.run(ALScopeImpl.java:250) at com.ibm.ws.al.scope.ALScopeImpl$1.run(ALScopeImpl.java:248) at java.security.AccessController.doPrivileged(AccessController.java:273) at com.ibm.ws.al.scope.ALScopeImpl.getCatalog(ALScopeImpl.java:248)[...]

And indeed, the module had dependencies to 3 libraries:

And this, my dear friends, is why I always ask for the complete log folder containing the SystemOut.log, trace.log and FFDC folder as well. It is very helpful to review ALL logs and traces available as they sometimes provide different information and additional hints. Sometimes we need to pull our information together piece by piece - just like doing a jigsaw puzzle. And in the end, we will get a clear(er) picture of the issue and maybe even the root cause.

But back to debugging this one - the focus was now on the libraries. Why did they prevent a successful deployment?
A shot check with my friends in development (whom I would like to thank again hereby) shed some light:

Under some circumstances the .settings/org.eclipse.wst.common.component file loses the source folder entry for the project root, and exporting the project using the J2EE tools will not export anything from the root level.

As the libraries were affected here, the following 'workaround' did the trick here:

In IID switch to the Navigator View (Windows >> Show View >> Other >> Open Navigator View)

For the first library, open the settings/org.eclipse.wst.common.component fileand add this line:

<wb-resource deploy-path="/" source-path="/"/>

Do the same for all other libraries used by this module as well.

Save your changes and try to deploy the application again.

And this is it. Now it should work. And in case it does not, take two of these and call me in the morning.

Today I would like to share with you an interesting case that I recently worked on.

Symptoms

Process Portal in BPM 7.5.1.1 showed several completed process instances, all with a certain task in state "new". Clicking the "Run" button for that task did not work. No Task was started, nothing happened. No new window opened, nothing changed.

Diagnosis

To further diagnose the problem, we looked at the Process Inspector in the Process Admin Console. Here we saw something pretty amazing:

The instance was marked as completed, and so was the task. At least in the left pane of the Process Inspector. The right frame showed the task as "new". Which is fairly interesting as they contradict each other.

As a next step we had a look at the LSW_TASK table in the BPM database. This showed a status of "11" - which is "new".

So the right hand side and the Process Portal were correct. This task still was in the status "new". So we were facing an inconsistency. As all these bogus instances were listed in Process Portal, our main goal was to get all those tasks completed and get rid of the instances.

Potential Treatment

Well, I could think of 2 possible options to mark those tasks as completed.

1. Manipulate the database - too risky! Never do that unless IBM Support advised you to and provided clear instructions.

2. Use the REST API - Bingo! As described in the IBM Knowledge Center we can use the REST API to finish tasks. We simply need to use this POST method:

POST /rest/bpm/wle/v1/task/{taskId}?action={complete}

Guess what?! It worked! We simply performed this step for all the tasks that were still marked as "new"and were able to resolve the problem.

However, we were not able to find the cause, why these inconsistent data were present

I hope that one day this may help you solving a problem in your environment. And if it does not, take two of these and call me in the morning.

This week, I found something interesting. While the official fixlist for BPM 8.5.5 states that the iFix for JR50008 - MALFORMEDURLEXCEPTION OCCURS WHEN YOU ACCESS A MANAGED ASSET FILE - is included, the "versioninfo -long" command output run on any v8.5.5 system does not show it listed.

After some research I found, that JR50008 is not included in v855 and it is accidentally listed in the fixlist for this product version. We are working on getting this corrected.

If you need this fix on v855, you may simply install iFix JR50653 for v855. It does contain the code change for JR50008 and is available for v855.

And if JR50653 does not help you, take two of these and call me in the morning.

The other day I was asked by a colleague: Where do we get the IID for BPM on Cloud?

To get IID for BPM on Cloud, you will have to get in touch with the Operations team. You can do so by opening a PMR.

The operations team will enable the IID tool download in the InfoSys instance of BPM on Cloud (BPMoC). In the BPMoC portal, where in the DEV environment's Process Center you had previously seen one link to download PD, you should now also see a link to download IID.

After downloading IID, you will install it to your local machine.

In addition you will need the latest auth plugin. (This is also the same plugin you use in PD, as seen in your <PD_Install_Dir>\teamworks\eclipse\plugins directory)

This authentication plug-in (this jar) is to be copied into the following directory: <IID_Install_Dir>/dropins/plugins
Note: This directory might not exist on your system. If it does not, create the directory and then put the JAR file in the dropins/plugins directory.

After the plug-in has been copied into the IID installation, restart IID.

You will tell your IID how to communicate with the Process Center so that you can work with process applications and create Advanced Integration Services

The IID download from the BPM on Cloud portal does not include the Unit Test Environment (UTE) for IID.
If you want to install IID plus the UTE, then you can obtain this from your IBM Passport Advantage account.http://www-01.ibm.com/software/passportadvantage/pao_customer.html
The UTE is a small local Process Server server which does not communicate with the Process Center.

I hope this will help you getting IID on BPMoC. If it does not, take two of these and call me in the morning.

They now wanted to escape the '&' character with it's percent-encoded octet '%26', i.e. replacing '&' with '%26'. This however returned an error, stating "Project with name ... not found".

Now, why is this? The percent-encoded octet represents the '&' and they should be equivalent, shouldn't they?

Well, yes and no. The percent-encoded octet is equivalent to the '&' sign, correct.

However, as you will see in RFC3986 Section 2.2 the '&' is a reserved character in a URL. Escaping it will most likely change the way the application is interpreting the URL. And this is the case here. The '&' is a control character, a so-called sub-delimiter. And in our case here, it changes the way BPM interprets this URL.

So you should not escape the reserved characters in the above mentioned URL.

I hope this helps you overcoming problems in the future. And if it does not, take two of these and call me in the morning.

If you ever wondered how you can use server variables used in your BPM processes inside of your client-side coach this quick-tip is for you. At least when you’re using BPM 8.5.5 or higher.

First of all let's clarify some things to make sure we are on the same side: We have to divide between server-side JavaScript which runs on the BPM Process Server / Process Center and client-side JavaScript code which runs in the actual browser e.g. when using a coach.

Since those are 2 different JavaScript Engines one cannot easily access the other one’s variables or information. In BPM 8.5.5 a new feature was introduced that tackles this problem. It is called substitution.

Using substitution you can easily access server-side variables and render them into your client-side coach using custom html.

And here is how it works:

Set up a variable for your BPD (server-side) that will be used as an input for your human service.

Don’t forget to map the variable from your BPD as an input variable to your coach within your human service. In the sample the BPD variable is called “BPDtext” and contains the value: “This is a text saved in a server variable within your BPD”.

Don’t forget to create your input variable (in this case I called it BPDInput), this is where the value from the calling BPD gets passed to.

Within the human service create a coach with a custom html snippet. And here the magic happens.

Within the custom html’s HTML section you can access the server-side variable using doubled curly brackets: {{tw.local.BPDInput}} will render your server-side variable into your html code when the parser passes it.

Now this also means that this variable will only be renderedonce when the page loads. So this will not dynamically update itself when it’s changed. In order to change the variable a new html rendering operation would have to be triggered.

You can see the result in the last screenshot.

I hope this will help you developing your Coaches in the future. And if it does not - take two of these and call me in the morning.

Let's take a look at that exception. The „install did not complete successfully“ (CWLLG2164E) because „Tracking definitions were not sent successfully“ (CWLLG0433E). I see.

So that makes me think... Hmm, tracking definitions... Isn't that stuff related to Performance Data Warehouse (PDW) and data defined in the process app/toolkit? And those are tracked there if I am not mistaken.

Oh well, PDW-related data might be logged in the Support-Cluster. Worth a shot.

The analysis in the SystemOut.log of the PS-Support-Cluster reveals the following:

[10/27/14 23:36:39:133 EDT] 00000035 XATransaction E J2CA0027E: An exception occurred while invoking prepare on an XA Resource Adapter from DataSource jms/DataDefLoaderConnectionFactory, within transaction ID {XidImpl: formatId(), gtrid_length(), bqual_length(), data()} : javax.transaction.xa.XAException: CWSIC8007E: An exception was caught from the remote server with Probe Id 3-013-0010. Exception: CWSIC2029E: This transaction cannot commit as an operation that was performed within the transaction boundary failed. The first operation that failed generated the following exception: com.ibm.ws.sib.processor.exceptions.SIMPLimitExceededException: CWSIK0025E: The destination DataDefLoaderQueueDestination.PSMECluster on messaging engine PSME.PERFDW.BPMCell.Bus is not available because the high limit for the number of messages for this destination has already been reached... ... Caused by: com.ibm.wsspi.sib.core.exception.SILimitExceededException: CWSIK0025E: The destination DataDefLoaderQueueDestination.PSMECluster on messaging engine PSME.PERFDW.BPMCell.Bus is not available because the high limit for the number of messages for this destination has already been reached.

To sum it up the exception says that the destinationDataDefLoaderQueueDestination.PSMEClusteron messaging enginePSME.PERFDW.BPMCell.Bus is not available because the high limit for the number of messages for this destination has already been reached. Aha!

Now THAT is the information I looked for. It seems that the limit of the DataDefLoaderQueue has been reached, hence tracking definitions could not be sent, and therefor the deployment failed.

That does make sense, I guess.

Potential treatments

In order to heal the patient I'd recommend to clean up messages from the DataDefLoader-Queue. This should generate space for tracking definitions and deployments should not fail anymore with this exception.

To do so, browse your WAS Admin console, open the Runtime tab in Service Integration>Buses>bus_name>Bus members>Messaging engines for SingleCluster>cluster_name>Queue points>DataDefLoaderQueueDestination_Bus>messages (running messaging engine is required).

There are usually messages with content "Request PDW transfer" (approx. 20 Bytes large). These messages are just trigger messages for the PDW application to start tracking data transfers, and it is usually safe to delete them.

In additon we can go all the way with antibiotics which accelerates the treatment:

The patient might identify BPDs and toolkits where autotracking is enabled, see: