March 2010

March 08, 2010

My nephew Dave is a major in the U.S. Army. Dave is risking life and limb serving in Iraq. I am intensely and justifiably proud of him. Everything that my nephew does in Iraq is difficult. The treatise “On War” rightly observes, “Everything in war is very simple, but the simplest thing is difficult. The difficulties accumulate and end by producing a kind of friction that is inconceivable unless one has experienced war.” The soldiers and marines in Iraq experience this “friction” daily. They address this friction with the phrase, “embrace the suck”. This phrase is indicative of their grim determination to succeed in a hostile environment where even the simplest tasks are difficult. “Embrace the suck” is both an implied order and wise advice. It means, “Face it, soldier. This ain't easy. Now let's deal with it.” With bravado and grit like that in the face of a daily and dangerous grind, you can see why I am proud of Dave.

Enterprise IT has its own kind of friction. In enterprise IT, a simple task, such as adding a data attribute to an information system, is difficult. The irony is that this friction is of our own making. Enterprise information systems do not have to suck. Instead of embracing the suck, we ought to make a deliberate effort to reject the suck of today’s enterprise information systems.

Front-end software that can understand new data and metadata without modification

Back-end software that that can help you create and recreate your metadata

The first thing you need is front-end software that can understand new data and metadata without modification. This is amazingly straightforward: web browsers and productivity applications such as Microsoft Office and Open Office perform this job very nicely.

When you add new data attributes to an enterprise information system, new data and metadata can be rendered simply as new sections in a Word document, new columns and rows in an Excel spreadsheet, new fields in an InfoPath form, and new hyperlinks in a browser. When you use web browsers and productivity applications as the UI for enterprise information systems, changing backend data and metadata requires few if any changes to the front-end software.

You might not have realized this, but Microsoft Word 2007 is a terrific XML editor. Try it. Click here to use Word 2007 to edit an XML file. (Word’s native file format is XML.) Actually, Word is more than an XML editor; it is a powerful UI into XML data. So is Excel. So is InfoPath. And web browsers are quite good at displaying XML data. When you have XML-savvy UI tools on the front-end of enterprise information systems, you are free to be flexible with XML data and metadata on the back-end.

The second thing you need to reject IT system stodginess is back-end software that that can help you create and recreate your metadata. This is also amazingly straightforward: XML databases perform this magic.

The data inside enterprise information systems is usually structured in a way that makes its metadata obvious. Metadata is usually defined via the native structure of the data itself (columns, rows, paragraphs, sections, etc.) However, it is vital to realize that metadata can also be defined through tags and annotations. If the data storage format supports it, existing data can be annotated with new tags. By using annotations, you can change the metadata of your data. In other words, you can use tags to add new metadata to your existing data.

Relational database technology is over 30 years old, and it shows. Relational databases do not readily enable annotations to existing data. The metadata in relational databases is defined exclusively through the native structure of the data – the columns and rows.

Unlike relational databases, XML is built around tags and annotations. XML data offers flexible metadata. XML is designed from the ground up to enable you to change the metadata. You are free to add and/or transform the annotations to XML data anytime you need to. A good XML database, such as MarkLogic Server, can automate the task of creating and recreating your metadata. In fact, a good XML database will do entity extraction, whereby you can tell the database to search through your existing data and insert new annotations around new entity types you define and want to find and process.

Relational databases can’t really do this. It is difficult to change the metadata of relational data. About the only thing you can do is change the structure and thereby the metadata of relational data (by using the DBMS’s DDL or by using ETL processes). And then you still have to change all of the software that sits in front of that relational data, because there are no nifty UI tools that render relational data the way that Office can render XML data. So information systems that have relational data on the backend tend to have lots of stodginess.

With XML, you can change the metadata on the backend without changing the software that implements the UI. Yes, you will likely be obligated to change the software that implements the business logic, but that is precisely where XQuery and keeping the data in XML all of the time makes things easier.

In a typical enterprise N-tier application, you have data in a relational database, with OO code (and probably some SQL stored procedures too) to implement the business logic, and more code of some kind to implement the UI. If you have adopted SOA, the app also has internal and external service interfaces. So a typical enterprise information system has a data model, an object model, a message model, and a UI model (which all must align with the business model). And then it has a bunch of code to deal with the impedance mismatches between all of those models. So, when you have to change the data model and its associated business logic, you are obligated to change lots of extraneous code to boot.

Now imagine an enterprise information system that has all of its data in XML and its business logic in XQuery. Adding a new data attribute is straightforward. Sure, you might have to change some XQuery code for the business logic, but the XQuery code is pure business logic, and is not encumbered by any of that extraneous code that translates the data between the various models in a traditional app. XML is used as the data model, the message model, and the UI model, and the data readily flows between the back-end and the front-end. I have defined a set of XML-based development tools that I call the "XQuery Development Stack". (The XQuery Dev Stack is ideally suited for MODS -- the Methodology for Overcoming Data Silos.) Information systems built with MODS and the XQuery Dev Stack are not stodgy. They are not stodgy because the metadata is easy to change and the data is not buried beneath a mountain of software.

You will notice that I did not mention implementing an object model in an XML-based information system. There should be no object model in an XML-based information system. Object-oriented code is largely responsible for the suck of enterprise information systems. Don’t embrace the suck of enterprise information systems, reject it. With OO development, the dose is the poison, and enterprise IT has overdosed on OO programming. My colleague Joe Maquire wrote about this in an earlier blog entry here. I will talk more about the poison that is OO development in a future blog entry.

March 07, 2010

Most organizations are aware of the value of their information/data assets; information has become a core asset to many organizations. If data is lost or stolen, there is a definite cost; and in most cases, a total data loss will bring an organization to its knees. Data is a representation of the business. The business uses this data to operate, track, report, manage, and predict the business; data is clearly a business asset that needs to be managed. Business assets are traditionally managed as part of the business operations. So what is the role/responsibility of an IT department in the management of this business asset?

IT has traditionally been responsible for designing and applying technology solutions to business problems, as well as support of those solutions. The focus of IT has been the application and management of technology. The same approach has been used for the data assets. IT has traditionally approached data by applying technology solutions to store, move, protect, and process the data.

A database is typically used to store the data. A role of a database administrator is to manage the technology that is used to store the data assets. This is clearly a technology role. What about the data assets stored within the database? This is where the roles/ responsibilities are unclear. The hype cycle of data governance and ownership has shed light on this grey line between the business role and IT roles regarding the management of the data assets.

Is the role of an IT department to support and manage what the data assets within the technology or only to apply and manage the technology that supports the delivery of information to the business? It is clear that the data assets that fall outside of technology (i.e. paper records and files) are typically outside the responsibility of an IT department.

Prior to computerization, the information/data of an organization was managed directly by the business people who produced/used the data. The data was managed by the people who knew what the data represented. There was no disconnect between the business and the data as there is today. With computerization, the business organization’s data assets were encapsulated deep within technology. This made it nearly impossible for a businessperson to manage their data assets as they had when the assets were in a paper file. The management of the data assets fell into IT by default and not by design, as it became an accepted practice for the business to rely on IT for the management of the data assets.

Technology has evolved over the years, to a point where there is no longer a clear distinct between the business and its technology, in both directions. Many techno-savvy business folks can effectively perform tasks that previously fell under the responsibility of the IT department. In fact, there has been a mushrooming of what is known as “IT shadow groups” within many business organizations. These groups are not part of the IT department, but perform many traditional IT type functions and thus indicate a desire of the business to take back the control/management of their data assets. Should the business organization, rather than IT, manage the data assets as they did prior to computerization?

Interestingly, there is an “I” in “IT. Does this mean IT’s role is intended to be 50% in support of the business’s information assets and 50% the technology that supports the information assets? Should IT play a significant role in management of the business’s information asset? Since data is a representation of the business organization, a significant information management role in an IT department would require a more business-focused rather than a technology-only-focused for an IT department.

The best fit for the role/responsibilities of information/data management within an organization is an interesting challenge many organizations face, knowingly or unknowingly. However, more importantly is assuring that the responsibility and accountability for the data assets are assigned, regardless of placement within the organization. Sadly, these often fall between the cracks and the important data assets are undermanaged or managed in a reactive manner.

Many organisations are preparing for economic recovery by rethinking assumptions, striving for regulatory compliance, charting alternative courses, and monitoring which business scenario might unfold. One thing is clear; navigating the new normal requires more than automation. It requires the ability to leverage data to develop insight that can inform and drive business decisions.

But knowing the path isn’t the same as walking the path. Business professionals are inundated with information, and it’s getting harder to know what is “true.” Business people struggle to share their knowledge and most organisations are mired in unreliable data. Business cannot thrive under these conditions. Simply put, IT must help the business determine how to make sense out of petabytes of information, and use it to make better decisions

Catalyst 2010 will help you create the information environment for the new normal. We’ll examine how to leverage data, content, social media, and analytics effectively and deliver business insight.

How enabling insight can build on and extend existing approaches to business intelligence (BI)

How organisations can address the eroded utility and quality of data

How attention management helps information overload

How organisations can effectively combine structured and unstructured data, and manage it as information

How to foster insight through the intersection of search, social media, information delivery, and visualisation

March 03, 2010

The standard recipe for creating enterprise information systems has been around for a long time: relational data behind application logic behind a user interface. This basic N-tier architecture has been and continues to be the de-facto standard for enterprise IT application development. Sure, service oriented architecture and Agile methods are intended to remedy several of the pesky problems that are inherent with N-tier development. But still, the fundamental architecture persists, and that architecture is data in a database behind software for application logic behind software for the UI. Unfortunately (and demonstrably), this is a recipe for stodginess.

In fact, the N-tier design paradigm just touches the surface of the stodginess. My colleague Jack Santos rightly pointed out to me that IT architecture today really is NxN-tiers. It is not only two dimensionally data-process-logic, but two dimensions multiplied by multiple different implementations, due to stove pipe approaches, varying off the shelf software implementations, or (now) external cloud solutions. Each have an N-tier approach, and each can stand on their own.

When enterprise IT groups are forced to spend multiple man-days changing code in several NxN-tier software layers just to add a data attribute, those enterprise information systems are patently stodgy.

Here is an example of enterprise stodginess that we have encountered right here at Burton Group. My colleague Joe Bugajski wrote a paper on governance that is quite popular with our clients. However, we have no way of knowing how many readers agree with Joe’s recommendations or how many readers have implemented or plan to implement Joe’s recommendations. We don’t even know how many client “key issues” or “problem statements” Joe addressed in that paper.

To find out how many readers agree with Joe and are implementing his recommendations, we will need to implement a way to better collaborate with our clients. We will also need to make sure that inside these collaboration system(s), we use the same customer identifiers that we use for customer data within our other information systems. IOW, it will not help us to have our collaboration systems tell us that “Customer 47” is implementing Joe’s recommendations and then have our CRM system tell us that it has no record of a “Customer 47”. Clearly, effective collaboration and proper data identifiers are a powerful combination.

To find out how many “key issues” or “problem statements” Joe addressed in his paper, we will need to go back and add that metadata to Joe’s document. And then we will need to change the code (in perhaps several layers of software) to make use of that new metadata.

We have to make those code changes because the metadata for our data is hard-coded and our data is trapped behind our software. These are problems with NxN-tier architecture that neither SOA nor Agile can remedy. Whenever you have to modify software in order to take advantage of new data and metadata, you have stodgy systems.

Eliminating this stodginess requires a different mindset. You have to stand enterprise IT’s traditional approach to creating enterprise information systems on its head. Basically, you have to use flexible metadata and you have to put the data in front of the software instead of behind it. You must not bury the data behind software that is hard-coded to only understand certain types of information. You have to use commodity software that can readily understand, process, and display lots of different information types.

In order to avoid in stodginess in enterprise information systems, you need:

Front-end software that can understand new data and metadata without modification

Back-end software that that can help you create and recreate your metadata

I don’t have room in this blog post to tell you what that software is -- I only have room to tell you what that software isn’t. That software isn’t a relational database trapped behind software for application logic behind yet more software for the UI. Avoiding stodginess requires something else entirely. I will describe this front-end and back-end software and its associated cures for enterprise IT stodginess in my next blog post.