SAP BI and the Conservation of Complexity

You may remember a rather obscure law of science from your high school days in physics called the Conservation of Energy:

The total amount of energy in a closed system remains constant. In other words, energy can be converted from one form to another, but it cannot be created or destroyed.

I don't claim to understand this principle fully, but essentially it's saying there is a finite amount of energy in any given system and that amount cannot be changed. All you can do is change the way it's packaged. I guess it's a little like the mess in my boys room. It never really goes away ~ it only changes form now and then.

I would like to offer a corollary to the Conservation of Energy that I call the Conservation of Complexity, which states:

The total amount of complexity in an information system remains constant. In other words, complexity can be transferred from one party to another, but it cannot be created or destroyed.

This principle states that in any given information system there is a finite amount of complexity that cannot be removed. All you can do is determine who deals with the complexity. With any given business intelligence system the options have traditionally been the BI software/system vendor, IT back end resources, IT front end resources, and the end user.

Going back to ancient history (10 ~ 20 years ago), the following graphic shows how the typical transactional system might have distributed complexity:

Mainframe Vendor: 20%
IT back end: 20%
IT front end: 50%
End user: 10%

What this is essentially saying is in this type of system the bulk of the complexity falls on the IT front end resources, or the report writer/developer. In most situations the IT back end resources are only responsible for providing simple database views or perhaps some stored procedures to simplify data retrieval, but in almost all cases it falls upon the report developer(s) to deal with most of the complexity in the system. The primary problem with this arrangement was the bottleneck created by the IT front end 's it simply wasn't feasible to keep enough resources on hand to keep up with demand.

I experienced this arrangement first hand when working at my first real job after college in the mid to late 80's. I worked at the Data Processing Center at the college I had attended and part of my job was to produce IBM VM/SP (mainframe) reports for college staff. I can assure you that there was a LOT more demand for information than I could possibly keep up with. And while the end user was certainly shielded from much of the complexity of the system, they certainly experienced more than their share of frustration waiting for their ration of data from the mainframe gods (this of course leads us to yet another corollary principle, the Conservation of Frustration, which we'll save for a future discussion).

Once SQL-based relational systems came of age one of the first attempts to resolve this issue was to provide the end-user with their own query and/or report writing tools with the hopes that they would become more self-sufficient (adhoc queries). The distribution in this type of system looks something like:

RDMS Vendor: 25%
IT back end: 25%
IT front end: 0%
End user: 50%

Essentially this arrangement eliminated the bottleneck created by the IT front end resource. It also shifted some of the complexity to the back end resources (primarily due to the need for additional and more complex views and reporting structures). The reason why this was a colossal failure for the vast majority of end users is easy to understand (at least now): the typical end user proved incapable of dealing with such a high level of complexity. It quickly became obvious that shifting the bulk of the complexity onto the end user was not the path to take.

Once again, I experienced this misguided effort first hand. I worked with more than one client in the mid to late 90's who had attempted to put reporting/query tools in the hands of their end users and almost died trying. I remember one particular client, a large hospital in Denver, CO, that had purchased over 400 copies of Crystal Reports and installed them on users PCs in the hopes they would somehow manage to find their way to their data. Wishful thinking. Fortunately for them we were able to later work a trade for a true enterprise reporting system. But back then a lot of end users simply put their "off the shelf" software back on the shelf.

About this same time the data warehouse was born. The idea behind the data warehouse was to provide a back end data structure that essentially would absorb much of the complexity associated with traditional SQL transaction systems. The distribution then became something like this:

Warehouse Vendor: 30%
IT back end: 30%
IT front end: 15%
End user: 25%

In this scenario, the warehouse vendor takes on some additional complexity by providing out of the box data structures and tools for gathering, cleansing, and organizing the data. In most cases it still requires IT front end resources to create queries (in the case of SAP BW) and other custom interfaces to allow users to access data stored in the data warehouse.

You may notice the end user is shown dealing with more complexity than the IT front end resource. On the surface this contradicts one of the stated benefits of a data warehouse � that the end user can easily access the data in the warehouse via a simple GUI interface. In other words, the user can be self-servicing. The reason for this is we are making the assumption that many end users are not only interested in accessing the data but also in presenting the data.

Using the standard BEx interface it is fairly simple for an end user to access SAP BW data. All the use needs to do is open the query, respond to the necessary prompts, and then run the query. The data is then retrieved into the Excel spreadsheet. If this were the end of the story there would be very little complexity for the end user to deal with.

However, in many cases the end user does not simply want to leave the data in the spreadsheet "as is". The most common need is to produce printed output. Because the BEx interface is basically a dump of data into a spreadsheet, it requires a considerable amount of work to format the output to make it "fit for print". Something as simple as page headers and footers are quite difficult to accomplish. Pagination and labeling are also a challenge. Therefore, when the user requires formatted output of SAP BW data, the level of complexity that the user must deal with is often quite high.

In the typical rollout of SAP BW the initial end user community is quite small and unusually advanced in both their data access requirements as well as capabilities. These are the "power users". While the standard BEx interface may prove quite useful for these more self-sufficient users, it is almost a certainty that as the reports are rolled out to a wider base of users that many users will find BEx to be insufficient in meeting their needs. For these users the distribution of complexity needs to be more like the following:

Warehouse Vendor: 30%
IT back end: 30%
"Information Broker": 30%
End user: 10%

This shift in complexity is primarily the result of moving the responsibility for the finished (or formatted) output from the end user back to IT front end resources, or as it says above, the "Information Broker". I use this term deliberately to get away from the mindset that it is up to official IT front end resources to help users produce finished output.

I have worked on a lot of BI projects throughout the years and one thing that is always puzzled me a bit is how little companies invest into front end resources in IT. In almost every case, even with the largest companies, you can count the front end resources on one hand � or finger. Even then they are almost never dedicated to assisting the end user in accessing and formatting data. On the contrary, they typically have other responsibilities that dominate their time and attention. The end users get the leftovers. And they get a lot of complexity dumped in their laps.

I will save this idea of an Information Broker for a later installment. This is partly because I have got to have some time to think it through a bit. For now my point is simply this: if your business users are looking to IT front end resources to take on some of the complexity they are dealing with, they are likely going to be waiting a long time. There has to be another way.

At first glance this can appear to be a step back to the days of transactional reporting where the bulk of the complexity fell upon the IT front end resource, and in some ways this is true. I have been reading a lot lately about the need for real ROI with the tremendous investments companies have been making in their information infrastructure, in particular with data warehouses like SAP BW. In order to insure maximum ROI it is imperative that companies find creative ways to reduce the amount of complexity presented to the broader end user base. I am convinced that a little extra investment in the right resources can, in this case, reap tremendous bottom-line benefits.

Information Brokers: The Missing Link?

In a previous installment, SAP BI and the Conservation of Complexity, I introduced the concept of an information broker. In that installment I make the point that a major reason why so many companies fail to realize the full potential of the information housed in their various application systems is because end users are unable to effectively deal with the high level of complexity presented to them. Even in a warehouse environment such as SAP BW where data has been cleansed, sanitized, de-normalized, and supposedly made fit for human consumption, it can be a daunting task for the rank-and-file user base to access mission critical information.

If we can accept the premise that users are being forced to deal with too much complexity when accessing corporate information, then what can and should be done to rectify the situation?

One obvious place to look for improvement would be in the technology being used - the 'front-end' tools. There has been no lack of effort over the past few years by all the big BI vendors to provide the holy grail of front-end tools - a tool that can be effectively used by the un-initiated end user. Some recent examples fall under the heading of natural language query tools, such as Intelligent Question from Business Objects. The promise of these tools is to provide a way for the ordinary business user to ask a question and actually receive an answer (What were my five top selling products in the southern region last quarter?).

The purpose of this installment is not to argue the merits of these tools and debate their relative effectiveness in providing real answers to business user's questions. I have not personally had the opportunity to use one of these tools at a customer site and therefore cannot provide any real-world observations. And while I certainly agree that it is important for all BI tools to become more 'natural' and simple to use, there's more to solving this puzzle than simply putting the right tools in the right hands. As with many business problems, it can be more of an organizational/people issue than a technological issue.

As I've worked with companies through years, I've noticed that certain people tend to function as 'information magnets' in any organization. They are usually fairly easy to spot: all you've got to do is follow the tracks worn in the carpet leading to their cube. One thing they have in common is everyone else around them looks to them for answers to their questions. These are the keepers of the infamous 'spreadmarts' and, quite frankly, we should all be thankful for them because without them commerce as we know it would grind to an immediate and grinding halt.

While I would agree that these spreadmarts are not the most ideal method of disseminating information, we should all recognize that they have served a very important function that, for whatever reason or reasons, was not (and is not) being performed elsewhere: they have functioned as information clearinghouses where raw data is brought in and usable information is produced and delivered to decision makers. If we think of this in terms of a transaction of goods or services, then we could say that the keepers of these spreadmarts operate as a sort of 'information broker', connecting the right piece of information with the right person in order to produce the best possible business outcome.

Although this method of collecting and disseminating information has proved somewhat effective and useful, it is not without significant issues. Perhaps the foremost is lack of control over corporate data. Because this process involves downloading data from corporate data stores into either Excel or Access the data can therefore be easily manipulated - a feature for some but a cause for concern for anyone responsible for corporate governance. There is also the issue of distribution of the information. These local spreadmarts and Access databases do not offer the secure distribution and auditing capabilities of a true enterprise reporting system.

The idea I would like to propose is for IT to deliberately identify, engage, and empower these individuals and help make them part of an enterprise reporting solution. In most of the reporting projects I've been a part of there is, at best, a sort of 'us vs. them' mentality among IT staff. Instead of being viewed as potential allies these 'keepers of the spreadmarts' are often seen as part of the problem and are often ignored or otherwise shoved to the side while the 'real' enterprise BI solution is being implemented. However, by leaving these individuals out of the loop and not giving them a central role in the design and implementation of the enterprise BI system the entire project can be jeopardized. They are the ones whole truly understand the business need and where to find the answers to questions in the data.

How would this work itself out practically as part of an enterprise reporting project?

* Early on in the process there needs to be a deliberate effort to identify these individuals in an organization. The scope of the search would depend on the scope of the reporting system deployment. This could be accomplished using a simple survey of end users: Who do you go to in your department to get answers to your business questions?

* Once these individuals have been identified, include them in the early stages of design. They can offer invaluable assistance in identifying deficiencies in the data and common analysis 'themes' requested by end users.

* By providing these individuals with early ownership of the new solution they will feel less threatened and therefore more likely to adopt and promote it rather than resist.

* It is important that these individuals be officially recognized as the critical link between IT and the user base at large. I would even recommend going as far as giving them an official internal title, such as 'HR Information Broker'.

* Lastly, these individuals need to be empowered with the training and guidance they need to fully understand and adopt an enterprise BI system. They need to be shown how they personally will benefit from moving to an enterprise system and how they can better serve their end users (through developing end user reports, creating meta-layer views, creating business logic/formulas, etc.).

One common flaw I've notice in many IT departments is a lack of real understanding of the business need and how information is really used in an organization.

This is understandable as IT does not work as part of the business, but rather in support of the business. By identifying existing 'information brokers' and bring them into the fold early on in the project, IT can help insure that the new BI system will be designed correctly and fully adopted by the end user community.

And you may also find the missing link between your data and your users.