ADEs: The Rise of Buy and Extend BI Tools

This is an excerpt from Wayne Eckerson's "Development Techniques for Creating Analytic Applications" (TDWI Report Series), The Data Warehousing Institute, March 2005.

Analytic development environments (ADEs) are the newest development technique on the block. ADEs are typically component-based extensions of business intelligence (BI) tools that let developers and power users create sophisticated analytic applications by dragging analytical objects onto a graphical workbench, where they can be connected and configured to create an analytic application without writing much code, if any at all.

ADEs are the analytical complement of integrated development environments or IDEs, which are used to build transaction applications. Examples of IDEs are Microsoft Visual Studio.NET, Borland's JBuilder, Eclipse, IBM's WebSphere Studio and BEA's WebLogic Workshop. ADEs are the spiritual heir to IDEs, both in functionality and name.

A Promising Future. ADEs promise to accelerate the development of custom-built analytic applications, as well as make it easier and faster to customize packaged analytic applications. An ADE enables developers to drag and drop analytic components onto a screen to rapidly create analytic applications. More than a report designer, ADEs give developers precise control over the look and feel, functionality and workflow of an application.

Today, organizations spend way too much time customizing and extending BI tools and packages to create analytic applications that meet user requirements. On average, organizations customize approximately 33 percent of every analytic application using mostly SQL and other hand-written code and spend 7.5 months to deliver a final product -- way too much time to meet fast-changing user needs.

As a result, users will soon be using ADEs to "buy and extend" BI tools or packaged analytic applications. In fact, the drag-and-drop nature of ADEs will further shift development responsibilities from IT developers to power users in the field. With an ADE, a power user can easily modify a packaged analytic application, flesh out a report definition or create a new application or report from scratch (once IT has established data connections and BI query objects). Thus, ADEs will once and for all get the IT staff out of the business of creating reports so they can focus on what they are best at: building robust data architectures and abstraction layers for end users.

Rapid Prototyping. ADE tools will also accelerate the trend toward rapid prototyping. Developers and power users can use an ADE tool in a joint application design session to get immediate feedback from users on data, application screens, metrics and report designs. This iterative process results in better designed applications that are delivered more rapidly. Many vendors are shipping ADEs for specific applications to facilitate rapid prototyping. For example, many dashboard and scorecard solutions are ADEs.

Service-Oriented Architecture. The real power behind ADEs comes from the fact that vendors have componentized the functionality of their BI tools. In the past, vendors hard-wired presentation, logic and data functionality together. However, the advent of object-oriented programming and service-oriented architectures has enabled vendors to open up their products, componentizing functionality within a services-oriented framework. The upshot is that ADEs enable developers to create multiple instances of components, store them centrally and reuse them in other applications.

Vendor Offerings

Tool Extensions. In most cases, ADEs are extensions of existing BI tools. Many BI vendors, under pressure from users, now offer an ADE version of their BI tool. (See http://www.tdwi.org/adeapp for a list of criteria to evaluate vendor ADEs.)

"We found that many users wanted more flexibility to create custom applications using the components in our toolset without dropping down into an SDK [software development toolkit]," says Clay Young, senior vice president of strategic marketing at ProClarity Corp. "So now we give users the option of getting our traditional BI tool out of the box or a developer's version [ProClarity Analytics Application Development Platform] in which we surface hundreds of our components in a graphical development environment."

ProClarity recently went one step further in the ADE camp when it released an Excel-based design and modeling environment that provides real-time access to multiple, distributed data sources, including ODBC/JDBC, CSV and Web pages. IT professionals set up persistent data connections and then step aside, allowing power users to create reports and screens inside Excel (see Figure 1).

Figure 1: Real-Time ADE. ProClarity's new Live Server product empowers business users to rapidly create read/write analytic applications in Excel after IT establishes persistent connections to multiple data sources. Here an anlyst at a distributor compares product pricing with direct competittors (whose prices were extracted in real time from their Web sites) and then inserts new prices to model the impact of a price change on margins and revenues (Courtesy of ProClarity Corp.)

ProClarity introduced Live Server because it found that traditional analytic application development cycles were too long for many users. According to Young: "For every business user who has been served by a traditional analytic application, there are 50 who haven't. Even tools like ProClarity [Analytic Applications Development Platform], which speeds development by an order of magnitude, are not fast enough to serve many users. With ProClarity Live Server, we put the tools in business users' hands so they can develop custom applications quickly."

However, Young points out that ProClarity Live Server is not a replacement for most types of analytic applications. For example, companies will still want to use ProClarity's development platform if they wish to deploy Web-based applications (versus Excel-based desktop applications) that provide guided analysis, rich OLAP navigation, and sophisticated visualization and calculation objects.

Internal Requirements. Other ADEs have evolved from practical necessity. When Business Objects decided to enter the market for packaged analytic applications, it leveraged numerous analytic engines, visualization components, and query/reporting tools it had built and acquired over the years. It wrapped these components together into a graphical development environment to build its suite of packaged applications. After completing the applications, Business Objects decided to commercialize the development environment as well. Much to its surprise, the toolset, then called Application Foundation, became as popular as the packaged applications it helped build.

"We discovered that there was a pent-up demand among customers to 'buy and extend' their BI tools and packaged applications, which is why [Application Foundation] has been one of our fastest-growing products in recent years," says Steve Wooledge, product marketing manager at Business Objects.

Business Objects now calls the ADE "Performance Manager" to reflect the way most organizations use it - as a specialized ADE for building dashboards and scorecards. Performance Manager contains metrics maps, strategy maps, goal setting, alerting and built-in annotations and discussions - all the elements a developer needs to create any type of dashboard or scorecard (see Figure 2).

Figure 2: Specialized ADE for Dashboards and Scorecards. BusinessObjects Performance Manager provides a host of components for building dashboards and balanced scorecards. This screenshot shows four tabs that let users define and view goals, thresholds, initiatives, strategy maps and metrics. The body of the screen shows color-coded performance against goals and a threaded discussion attached to those goals. (Courtesy Business Objects SA)

ADE Pure-Plays

Outside of mainline BI vendors, some companies have emerged as ADE pure-plays. AlphaBlox was the first company to evangelize the benefits of component-based analytical development. But like many first-movers, AlphaBlox mistimed the market and never gained significant traction. It was recently acquired by IBM and will be embedded in various IBM development environments.

Another ADE pure-play is arcplan, an 11-year-old German firm with 1,800 customers. The company provides a "codeless" development environment that lets developers or power users rapidly build robust analytic applications against OLAP, relational and legacy data sources. Initially, arcplan grew its business around Hyperion Essbase, which used Excel as its only front-end environment. Now, arcplan is gaining momentum as a front end to Microsoft Analysis Services. Unlike many ADEs, arcplan lets developers position objects anywhere on a screen to create a completely custom look and feel (see Figure 3).

Figure 3: arcplan's ADE.arcplan's ADE lets developers drag and drop graphical objects and widgets onto a screen - such as table, column, cell and button - and configure them in a WYSIWYG (what you see is what you get) environment. (Courtesy arcplan, Inc.)

Leveraging IDEs. Some ADEs are built inside IDEs. For example, Microsoft's ADE, BI Development Workbench, is a BI instantiation of Microsoft's IDE, Visual Studio.NET. With Microsoft's BI Development Workbench, developers can not only build reports, but also create ETL programs, design relational schemas and OLAP cubes, and build data mining models. This is a one-stop-shop ADE, running on one of the most popular IDEs in the industry today (see Figure 4).

In a similar fashion, Crystal Decisions (acquired by Business Objects in 2003) differentiates its toolset by embedding Crystal Reports and report components inside third-party IDEs, specifically Microsoft's Visual Studio.NET and Borland's JBuilder. This lets application developers create and embed Crystal reports into their applications without having to leave the comfort of their preferred IDE.

Like Microsoft, SAP delivers BI and data warehousing capabilities through its broad-based NetWeaver platform, but with a twist. SAP's BI-specific ADE - a Windows tool called Web Application Designer - integrates with every other service supported by NetWeaver, including SAP and non-SAP services, and generates zero-client analytic applications (see Figure 5).

Report Design Environments

Other vendors blur the lines between an ADE and a robust report or dashboard authoring environment.

For example, ADVIZOR Solutions enables power users to create interactive visual applications without coding. ADVIZOR Solutions supports 12 visualization components that users can query, drill into, filter and animate. These components include standard bar, pie and line charts as well as advanced visualization techniques such as histograms, scatter plots, data constellations and paraboxes (see Figure 6).

Figure 6: A Visual ADE.ADVIZOR Solutions enables power users to create visual dashboards, as shown. In this example, their Visual Discovery technology enables users to circle a cluster of data points that represent strong mutual fund performance. The scatter plot highlights the selected data points, graying out others, while the dynamically linked bar chart and data sheet automatically highlight the selected data. Users can click an icon to remove non-selected data and drill down to make fact-based decisions. (Courtesy ADVIZOR Solutions, Inc.)

"Our visualization components enable users to quickly identify patterns in complex data that would be nearly impossible to ferret out in traditional BI tools," says Doug Cogswell, chief executive officer of ADVIZOR Solutions. Because ADVIZOR uses an in-memory database, it excels when analyzing hundreds of thousands of records that are up to 40 columns wide.

Dancing on the edge of an ADE, MicroStrategy provides new report design tools that let users create a wide variety of report types, including dashboards and scorecards, master-detail management reports, invoices and statements, and compound reports that mix graphical and tabular elements. The finished HTML reports use drag-and-drop functionality, embedded OLAP grids and links to other reports to deliver an interactive end-user experience. Reports can also be output to Excel and PDF.

The firm's newest release, MicroStrategy 8, provides two report development environments. The first, MicroStrategy 8 Web Design Mode, is geared to professional developers who need to create complex, Web-based reports that offer document navigation, query prompts and OLAP grids to provide a high degree of interactivity for end users. The second, MicroStrategy 8 WYSIWYG Mode, is geared to power users who may take a report definition created in Web Design Mode and customize document formatting, grid objects and views, and other elements for a specific group of users.

"Our goal is to provide flexible report design without the traditional steps and backlog involved in designing and delivering a report," says Mark LaRow, head of marketing for MicroStrategy.

The Delivery

Clearly, BI vendors have recognized the need to deliver "buy and extend" capabilities. Most are starting to deliver ADEs or ADE-like capabilities. Pure ADEs, such as those from arcplan, Business Objects, Microsoft, ProClarity and SAP, give developers almost unlimited control over the look and feel of an application and the way users navigate through it.

Newer report authoring tools, such as those from MicroStrategy and ADVIZOR Solutions, give report developers much more flexibility to create a wide range of reports or dashboards to meet the unique requirements of a broad range of users. In both cases, the tools provide an easier-to-use authoring environment, which is helping to finally move development out of the hands of professional developers and into the hands of power users and business analysts.