Information Infrastructure EII TCO/ROI Hardware Uncategorized Green IT Development
Two weeks ago, Merv Adrian’s blog was filled with analysis of the recent TDWI conference, which had as a theme “Agile Business Intelligence.” Merv’s initial reaction was the same as mine: what does BI have to do with the agile development movement  In the title, ABI is short for Agile Business Intelligence, and WTF, as every fan of the TV show Battlestar Galactica knows, is short for What The Frak, while WTFN stands for Why The Frak Not. My confusion deepened as I tracked down the BI companies that he cited: It appeared that only one, Endeca, was marketing its solution as “agile BI” (Wherescape simply notes that its data-warehouse-building solution is increasing its built-in support for agile development practices). Endeca’s definition of agile BI appears from its web site to boil down to: BI is agile if it speeds ad-hoc querying, because that allows changes in pre-decision analysis that lead to better and quicker business decisions. It isn’t intuitively obvious that such a definition corresponds to development agility as defined by the Agile Manifesto or to the various definitions of business agility that have recently surfaced.
 In the title, ABI is short for Agile Business Intelligence, and WTF, as every fan of the TV show Battlestar Galactica knows, is short for What The Frak, while WTFN stands for Why The Frak Not.
My confusion deepened as I tracked down the BI companies that he cited: It appeared that only one, Endeca, was marketing its solution as “agile BI” (Wherescape simply notes that its data-warehouse-building solution is increasing its built-in support for agile development practices). Endeca’s definition of agile BI appears from its web site to boil down to: BI is agile if it speeds ad-hoc querying, because that allows changes in pre-decision analysis that lead to better and quicker business decisions. It isn’t intuitively obvious that such a definition corresponds to development agility as defined by the Agile Manifesto or to the various definitions of business agility that have recently surfaced.
Definitions really matter in this case, because, as I have argued in previous articles, improved agility (using the correct definition) has a permanent positive impact on the top line, the bottom line, and business risk. Data from my Aberdeen Group 2009 survey of local and global firms of a range of sizes and verticals suggests that increased agility decreases costs in the long term, on average, by at least 10% below their previous trend line, increases revenues by at least a similar 10% above trend, and decreases the risk of negative surprises by at least 5%. And, according to the same study, the only business/IT process being tried that clearly increased agility and produced such effects was agile development as defined by the Manifesto (“hybrid” open-source development and collaborative development may also improve agility, to a much smaller extent).
On further reflection, I have decided that agile BI is indeed a step forward in overall business agility. However, it is a very small step. It is quite possible for a smart organization to take what’s out there, combine it in new ways, and make some significant gains in business agility. But it’s not easy to do, and right now, they won’t get much help from any single vendor.
Key Points in the Definition of Agility
I define agility as the ability of an organization to handle events or implement strategies that change the functioning of key organizational processes. Agility can be further categorized as proactive and reactive; anticipated and unanticipated; internally or externally caused; new-product, operational, and disaster. That is, improved agility is improvement in one or all of these areas.Initial data suggest that improvements in new-product development (proactive, unanticipated, externally caused) have the greatest impact, since they have spill-over effects on the other categories (anticipated, internally-caused, operational, and disaster). However, improvements in operational and disaster agility can also deliver significant bottom-line long-term increases. Improved agility can be measured and detected from its effects on organizational speed, effectiveness, and “follow-on” metrics (TCO, ROI, customer satisfaction, business risk).The implications for Agile BI are:
- Unless improved BI agility improves new-product development, its business impact is smaller.
- Increased speed (faster reporting of results) without increased effectiveness (i.e., a more agile business decision-making process) has minimal impact on overall agility.
- Improvements to “reactive” decision-making deliver good immediate results, but have less long-term impact than improvements to “proactive” decision-making that anticipates rather than reacting to key environmental changes.
In summary, agile BI that is part of an overall agile decision-making and new-product-strategy-driving business process, and that emphasizes proactive search for extra-organizational data sources, should produce much better long-term bottom-line results than today’s reactive BI that depends on relatively static and intra-organizational data sources.The Fundamental Limit to Today’s Agile Decision-Making via BI
Question: Where do the greatest threats to the success of the organization lie, in its internal business processes or in external changes to its environment and markets? Answer: In most cases, external. Question: Which does better at allowing the business person to react fast to, and even anticipate, external changes – internally gathered data alone, or internal data plus external data that appears ahead of or gives context to internal data? Answer: Typically, external. Question: What percentage of BI data is external data imported immediately, directly to the data store? Answer: Usually, less than 0.1 %. Question: What is the average time for the average organization from when a significant new data source shows up on the Web to when it begins to be imported into internal databases, much less BI? Answer: more than half a year.
The fundamental limit to the agility and effectiveness of BI therefore lies not in any inability to speed up analysis, but in the fact that today’s BI and the business processes associated with it are designed to focus on internal data. Increasingly, your customers are moving to the Web; your regulatory environment is moving to the Web; mobile devices are streaming data across the Web; new communications media like Facebook and Twitter are popping up; and businesses are capturing a very small fraction of this data, primarily from sources (long-time customers) that are changing the least. As a result, the time lost from deducing a shift in customer behavior from weekly or monthly per-store buying instead of social-network movement from one fad to another dwarfs the time saved when BI detects the per-store shift in a day instead of a weekend; and a correct reaction to the shift is far less likely without external contextual data.
This is an area where agile new product development is far ahead of BI. Where is the BI equivalent of reaching out to external open-source and collaborative communities? Of holding “idea jams” across organizations? Of features/information as a Web collaboration between an external user and a code/query creator? Of “spiraling in on” a solution? Of measuring effectiveness by “time to customer value” instead of “time to complete” or “time to decide”?
A simple but major improvement in handling external data in BI is pretty much doable today. It might involve integrating RSS feeds as pop-ups and Google searches as complements to existing BI querying. But if a major BI vendor features this capability on the front page of its Web site, I have yet to find that vendor.Action Items
In the long run, therefore, users should expect that agile BI that delivers major bottom-line results will probably involve:
- Much greater use of external data to achieve more proactive decision-making.
- Major changes to business processes involving BI to make them more agile.
- Constant fine-tuning of the querying that BI offers, customized to the needs of the business, rather than feature addition and decision-process change gated by the next BI vendor release.
- Integration with New Product Development, so that customer insights based on historical context can supplement agile development’s right-now interaction with its Web communities.
Here are a few suggestions:
- Look at a product like the joint Composite Software/Kapow Technologies Composite Application Data Services for Web Content to semi-automatically inhale new Web-based external data.
- Look for major BI vendors that “walk the walk” in agile development, such as IBM with its in-house-used Jazz development environment, as a good indicator that the vendor’s BI services arm is up to the job of helping improving the agility of BI-related business processes; but be sure to check that the BI solution is also being developed that way.
- Look for BI vendor support for ad-hoc querying (as noted above, kudos to Endeca in this area), as this will likely make it easier to fine-tune querying constantly.
- Look for a BI vendor that can offer, in its own product line or via a third party, agile NPD software that includes collaborative tools to pass data between BI and the NPD project. Note that in most if not all cases you will still need to implement the actual BI-to-NPD link for your organization, and that if your organization does not do agile NPD you won’t get the full benefit of this. Also note that agile plus lean NPD, where the emphasis is on lean, does not qualify.
- Above all, change your metrics for agile BI success, from “increased speed” to “time to value”.
Today’s agile BI as touted by BI vendors is a very small, very delayed piece of a very good idea. Rather than patting them and yourself on the back for being five years behind development-tool vendors and three years behind NPD software vendors, why don’t you get moving on more ambitious stuff with real business impact? If not, WTFN?
 Answers based on Aberdeen Group data usefulness study, used by permission of Aberdeen Group.
 I am in disagreement with other commentators on this matter. I believe that lean cost-focused just-in-time processes work against agility as much as they work for it, since if product specs change there is less resource “slack” to accommodate the change.