Notes about web and mobile technologies, devops, software architecture, and life

Menu

Author: Denny Pichardo

If you’re implementing a portal site, chances are that you are also implementing (or at least planning to implement) Web Analytics in order to understand user behavior, measure what works and what doesn’t work on your site, perform A/B testing, and improve your site’s performance/conversion rates. The information gained by successfully implementing Web Analytics can be invaluable in improving your user experience, assisting with online marketing strategies, and maximizing sales generation leads, which are all key aspects of successfully running your online business.

IBM WebSphere Portal has a built-in framework that integrates with various market leaders in the Web Analytics space, such as IBM Digital Analytics (aka Coremetrics), Adobe Analytics (aka Omniture), and WebTrends. Portal’s Active Site Analytics (ASA) framework generates page/application metadata which can then be aggregated and sent to your web analytics solution. You can read more about their framework and technical implementation information on IBM’s Knowledge Center.

Here are 5 considerations that can help you successfully implement Web Analytics on IBM’s WebSphere Portal. The list itself is agnostic to the different Web Analytics solutions; however, the “Guidance” section on each step is based on an IBM Digital Analytics implementation. Please beware that Web analytics solutions differ in functionality and may have a different method to achieve the same thing. Additionally, IBM WebSphere Portal is known to have better integration with IBM Digital Analytics, an example being their Analytics Overlay Reports.

Pre-requisites:

Your Web Analytics solutions has already been installed and configured

Basic knowledge of JavaScript and HTML

IBM WebSphere Portal version 8.0.0.1 or Later

1. Understand who your users really are

A key aspect of a Web Analytics solution is to be able to do customer segmentation / profiling. Customer segmentation is the practice of grouping users based on similar characteristics (such as location, age, gender, etc). By default IBM WebSphere Portal doesn’t share much user information when it sends registration/visitor data, so this is likely to get missed. As part of your implementation process you should first understand how your Web Analytics solution leverages user data, then extend your Portal solution to populate the data needed to segment your customers. If you’re using the content targeting feature and have already created segments in your Portal solution, ideally you should have the same data to recreate the same segments in your analytics solution.

Guidance for IBM Digital Analytics

IBM Digital Analytics tracks user information using “Registration Tags”. The aggregator provided by IBM uses this type of tag and sends a visitor ID along with the userName. My recommendation is to modify the aggregator and add your own user data there via Registration Attributes. You can retrieve user data using EL beans for example, and then add pass it to the aggregator.

Each registration attribute specified in the aggregator will also need to be specified in IBM Digital Analytics admin section (in order for it to become available for segmentation / Explorer reporting). Given that you can only use up to 50 registration attributes and the first 15 are special attributes for creating segments, you should pay close attention to which attributes should be in the first 15. As an example, you wouldn’t create a segment based on first name or last name so they would be good candidates for assigning an attribute number between 16 and 50.

Bellow is a brief sample of how to update the CoremetricsAggregator.js to accomplish this. This assumes that you have implemented a JavaScript function that retrieves user data. Also note that I’ve changed what the cmReg cookie stores; instead of a simple “Y” value, I’ve specified the actual visitor ID, this really helps during testing as we’re often using the same browser but using different test accounts.

Within IBM Digital Analytics “Admin” UI, you would then need to set these registrations attributes in the Explorer Attributes section. Please note that Email/City/State/Postal Code/Country are built-in attributes and hence there’s really nothing to set in the Admin UI.

Figure 1: Setting Explorer registration attributes

Registration attributes 1-15 should also be specified in the “Extra Fields” section. This would allow that data to be exported via Standard Data Export. See IBM Digital Analytics Implementation Guide for more details.

2. Differentiate between a Portal page and a functional page

Depending on how your Portal site was designed, a portal page might not necessarily equate to a single functional page. Say for example you have an “Product Detail” page, which takes an “Product ID” as a parameter and renders the product details and associated content based on the ID. To IBM WebSphere Portal, this is a single portal page and that’s how it’ll likely be reported, however to your business users it could be interpreted as multiple functional pages. From a web analytics perspective, you care more about the actual functional pages since that’s what the visitors see and interact with.

Guidance for IBM Digital Analytics

All portal page titles are reported to IBM Digital Analytics out of the box. If those page titles matches all your functional pages, you’ll just need to ensure that the title you see on the browser is what’s being reported to IBM Digital Analytics. There are other situations though, in which you should override the page title with a portlet-generated page title. For these situations, I recommend to implement a page title override system.

In order to implement a page title override system, you have a couple of options:

Use two-phase rendering on necessary Portlets to override the Page title. This might be cleanest approach, but it’s a bit complex and offers little flexibility. For more information please read: “Modifying the HTML head section of a JSR 286 portlet” in the IBM Knowledge Center

Output override metadata from necessary Portlets. This approach consists of updating the portlets to write metadata that overrides the page metadata and then processing the new tags in CoremetricsAggregator.js. This approach is very robust as you can override any page metadata (not just title) and simple to implement.
Sample override metadata:

In the example above, we allowed four tags to be overridden by a Portlet: asa.page.title, asa.page.breadcrumb, asa.search.query, asa.search.results. Also note that the individual page tag would’ve been used in the case that it was not overridden.

3. . Support business driven categorization of pages and applications

Categorizing pages (or creating Content Groups) is essential in understand user behavior on the sites. This is specially true when your site has a large amount of pages that have similar functional areas. The default categorization provided by IBM WebSphere Portal is a single generic value that doesn’t provide much value to understanding user behavior. You should empower your business users or content authors to easily categorize pages and applications, thus improving your analysis of functional areas on your site, improving speed to market, and minimizing the impact on the I/T organization.

Guidance for IBM Digital Analytics

The only category reported within the CoremetricsAggregator is “asa.page”. This category gets reported by every page. Ideally content authors are able to change categories without I/T involvement, since categorization is driven by functionality or business use cases and can be volatile. In order to do this you can leverage Portal’s Page Properties, which get syndicated as part of your WCM syndication (assumption that you’re leveraging Portal’s Managed Pages). This information is specified in the IBM Digital Analytics Implementation Guide (page 106), however the aggregator sample provided does not implement this.

Note that the cmCreatePageviewTag has been updated to send the new category variable.

4. Allow for applications integrated on the glass to report analytics data.

Arguably some of the most powerful features of IBM WebSphere Portal are the integration capabilities. One of my favorite is Web Application Bridge (WAB), which allows you to surface existing web applications into your site rapidly and seamlessly. The ASA framework however does not offer APIs or built-in mechanisms for these existing applications to take part of your analytics reporting. If you have analytics requirements involving these types of applications that integrate on the glass, you should plan on creating some customization to enable reporting of analytics data. WAB leverages iFrame technologies and each web analytics solution vendor might provide a different way to handle these types of scenario.

Guidance for IBM Digital Analytics

The steps needed to implement this are well documented as part of the IBM Digital Analytics Implementation Guide. Please refer to section 2.8 “Tagging Frames” and follow the documented instructions. The key aspect of the solution is that each application being rendered within an iFrame must include the eluminate.js library and cmSetClientID script blocks. Do keep in mind that you might have multiple environments with corresponding domains/client IDs, therefore the embedded applications would need keep those in-sync with your Portal site.

5. Hidden applications should not be reported unless used

IBM WebSphere Portal provides a hidden portlet layout container that can host portlets that are meant to be hidden until invoke explicitly by an user for example. Your developers might have also developed portlets that are essentially containers for modal dialogs. These set of hidden portlets automatically populate the analytics metadata and are reported. In my opinion, you should not report these portlets until the users starts interacting with them. From a web analytics standpoint there’s very little value in knowing that the application was loaded on the page; however, there’s significant value in understanding how your users interacted with it.

Guidance for IBM Digital Analytics

This is actually a bit of a challenge, mostly due to how the microformats work today (i.e. they’re specified in the html DOM and parsed/processed at page load time). There are also a few ways to solve this problem, the one describe below is the simplest but does require tinkering with your applications a bit.

Manually report the portlet element tag directly using the Coremetrics JavaScript API
From your portlet you would call the following Javascript function when a user opens your hidden portlet or interacts it with it.

cmCreateElementTag("Portlet Title", "Portlet Category", attributes);

Summary

I have just provided 5 unique implementation considerations when integrating IBM WebSphere Portal with your analytics solution. In a feature post, I will discuss additional considerations when specifically using IBM Digital Analytic as your solution, along with some more code samples. The topics will include: Configuration management, Virtual Portals and IBM Digital Analytics Multisite addition, Testing tools, Video Tracking, and Search.

In this article, I’ll describe an approach to provisioning using Powershell Desired State Configuration (DSC) in Vagrant. I’ll deploy a static website to IIS on Windows Server 2012 to showcase this approach. At the end of the article, I’ve included the final Vagrantfile as well as a DSC Configuration script, but would recommend to read through my steps in order to get a sense on why it was done as such.

On another note, I’m currently employing this setup to develop and test custom DSC modules. Using Vagrant for this purpose has saved me quite a bit of time and given me and fellow developers a nice development workflow.

Pre-requisites:

Create Windows Server 2012 Vagrant Box

The focus of this article is on provisioning. However I’m including a summary of steps taken to create a WS2012 Virtual Box using Packer (For information on manually creating Vagrant Windows Boxes see my previous post)

Clone packer-windows Git Repository and copy your WS2012 ISO to “iso” folder in packer-windows (If you’re not using Git, you can just download the latest release of packer-windows)

Update each builder in packer-windows\windows_2012_r2.json with a relative path to your iso and your respective checksum (I used the SHA1 from MSDN)

If not using the Windows Evaluation ISO, Add your product key to packer-windows\answer_files\2012_r2\Autounattend.xml.

If installing a different Windows Server 2012 other than Standard or Standard Core, you may want to set headless to true in the json file and use the UI to finish the installation (In my case I did this with WS2012 R2 Essentials)

Remove the VMWare builder in the json file if you’re just interested in the Virtual Box VM (or vice versa)

Setup WMF 5.0 and PowerShellGet

In order to use the new Package Manager “PowerShellGet” we need to install the latest WMF 5.0 Preview and configure a PowerShellGet repository. For the purpose of the exercise, we’ll configure the current central repository called “PowerShell Gallery”. These steps assume you have followed the steps in the previous section or you have a windows vagrant box with WinRM communications already configured and working.

Create a new Vagrant project based on your WS2012 vagrant box and enable WinRM

Configure the guest VM to be part of the same Windows Domain as the host or add your host computer as a trusted host on WinRM or use a wildcard (please note using wildcard is not recommended for security reasons):

Create a new DSC/WebSite Project and Initialize Vagrant

Create folder structure for storing your website files and DSC config/mof/custom module files. The structure below is a good starting point, but is only meant to be an example. You may want to do things a bit differently based on your needs.

Using Third-Party DSC Modules

The true power of DSC comes with the large amount of DSC modules that are being published and shared everyday by Microsoft and the PowerShell community. Many of these modules are being uploaded to the central repository: PowerShell Gallery. In this section, we’ll leverage the package manager PowerShellGet to install the xWebAdministration DSC module, that allows us to manage IIS Web Sites, App Pools, etc.

Install xWebAdministration module. We’ll create another provisioner script to take care of any DSC module dependencies for us (similar to how Berkshelf works with Chef, although we’ll just create a simple script to do it).
In the Vagrantfile define the script prior to Vagrant.configure

Note: Get-DscResource is a nice way to confirm you’ve loaded all the necessary modules. It’ll print all modules loaded on the Vagrant output screen. You only have to call it once.
Call the script (prior to any DSC provisioner)

config.vm.provision "shell" do |s|
s.inline = $dscModDepScript
end

Import the DSC module. Now that we have a DSC module installed, we can start using their resources in our DSC configuration.
Import it prior to the Node script block

Import-DscResource -Module xWebAdministration
Node $NodeName {...}

Use any DSC resource from that module. In my case I’ll use the xWebsite DSC Resource. In addition to this resource I’ll use the File DSC resource to copy any files/folders in the MySite folder to the default IIS website.

Run vagrant provision again (or vagrant up if your vagrant VM is down/destroyed)

Test your new site (If using my sample, you should see Hello DSC! by launching a browser on your host machine and going to http://localhost:8080)

Food for thought and final files

As I stated earlier, I used this setup to develop/test custom DSC modules. The dscModDepScript in the Vagrantfile below shows how to install your custom DSC modules (Get-DscResource would take care of loading them as well).

The DSC configuration file might need a bit of tinkering if you were to use it with something like Microsoft Release Management (this file is optimized to work with Vagrant)

The latest version of Vagrant addressed some issues around passing arguments to PowerShell scripts as well as RDP. Highly recommend that you install the latest version, as the sample Vagrantfile would not work. In addition I’ve used all 3 types of shell provisioning (inline, inline+variable, path) and passed arguments to it (as an example of how this works with Vagrant)

Vagrantfile:

DSC Config File: DSC/Config/MySiteConfig.ps1

In this article, I’ll describe how to install the IBM Script Portlet on IBM WebSphere Portal 8.5 and create scripts that leverage some of the key features of the Script Portlet. The script portlet allows developers with primarily client-side development skills (HTML/CSS/JavaScript) to develop portlets rapidly, both offline on their favorite editors/IDEs and within WebSphere Portal itself with a jsfiddle-like experience. To find out more information about it, visit the IBM Greenhouse Solution Catalog and the IBM Knowledge Center.

Click on the Edit link inside the Script Portlet and Verify it comes up

Notable WCM Tags in Script Portlet Development

The IBM Script Portlet components (HTML/JavaScript/CSS) are all stored within WCM, which means script portlets can be included as part of projects, deployed to other environments as part of syndication, and developers can leverage many of the WCM tags available. I used a few tags in combination with the AngularJS TodoMVC sample application to showcase how it can be used on common portlet development tasks, as well as other interesting use cases. To see a list of all the tags available, please visit the IBM WebSphere Portal Product Documentation.

URL Generation

Plugin to generate portlet render URLs. See documentation on how you can set parameters within the plugin itself. In the example below I use a FORM GET to add the parameter as query strings in the URL which sets it as a private render parameter by default. In general the approach below works well for parameters with small values, since there’s a limitation of how large URLs can get. The example below is for illustration purposes only, you may not want a large amount of Todos as part of the URL or to store Todos in Session 😉

This plugin allows you to construct URLs with query parameters as well as proxy the resources through WebSphere Portal’s AJAX proxy. This is very useful when consuming external REST services while complying with the browser’s same origin policy.

Use VBoxManage list ostypes In order to get the possible OS types you can use. Also note that 64bit OS may not show up on this list if Virtualization Technology (Intel VT-x, AMD-V, etc) is not enabled in the BIOS.

Vagrant Up

Edit Vagrantfile
By default winrm is not setup (ssh is the default); you need to explicitly set it. In addition, I didn’t use the default vagrant user/password, so I set those as well. Lastly make sure to forward the RDP port.

* In Vagrant version 1.6.3 there was a regression introduced around the rdp command (See issue #3973). I used my regular Remote Desktop client manually and all worked fine.

At this point you should have a Windows Box all set with Vagrant. In a future post, I intend to discuss provisioning software with Chef. If you ran into any issues following these instructions, please add those as comments below. Thanks for reading!

Success, after all, loves a witness, but failure can’t exist without one.

– Junot Diaz, The Brief Wondrous Life of Oscar Wao

Kicking off with a great quote from the book that inspired the title the blog, “The Brief Wondrous Life of Oscar Wao“, written by my friend, Junot. If you haven’t already, go buy yourself a copy; it’s indeed a terrific read that you won’t regret.

As a software architect, this quote really resonates with me. It’s inevitable that software projects will have issues; there are just too many factors that influence this. Yes, there are issues that architects have little control of, but let’s put those aside for now. What matters is to recognize the issues we do cause, being the witness to our failures.

Today’s technology gives us almost endless choices, and is constantly evolving. There’s always room for improvement. There are unexpected changes. This is what we have to deal with. You could say failure is always around the block, waiting for us. But when we turn into that street, which will happen, what differentiates us is to quickly realize that we just made the wrong turn, understand the path that got us there, and avoid that street later. If we’re lucky, we are the first witness to our failures, but when we aren’t, we need to take ownership of it. Understanding this plays a critical role in our success as software architects.