Vision and leadership for developing and implementing information technology initiatives

Chief Information Officer

Subscribe to Chief Information Officer: eMailAlertsEmail Alerts newslettersWeekly Newsletters
Get Chief Information Officer: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


CIO Authors: Jason Bloomberg, Adrian Bridgwater, Ed Featherston, Jackie Kahle, Srinivasan Sundara Rajan

Related Topics: Cloud Computing, Agile Software Development, Change Leadership Journal, CIO, SOA Best Practices Digest, Microservices Journal

Agile Development: Blog Post

Streaming Analytics By @TheEbizWizard | @CloudExpo [#Cloud]

Our technology is advancing so quickly that we’re still struggling to move from batch thinking to real-time thinking

Streaming Analytics: Terminator or Star Trek?

Since the dawn of enterprise digital computing, managers have been looking to computer operators to run data processing jobs on those original workhorses of computation. The goal: to generate reports that the managers would use to make decisions on how to run their companies or government agencies.

Today, those large binders filled with perforated pages striped in glorious green and white may be long gone, but the batch job – paper report – human decision pattern remains engrained on our managerial consciousness, for better or worse.

Cut to half a century later. We now have big data. Cloud computing. A wholly revamped notion of digital that is mobile-centric, omnichannel, and customer-focused. And yet, old habits are slow to die.

Take, for example, the first version of Hadoop. It was a batch job processing platform that would generate reports (paper optional, thankfully) that managers might attempt to interpret in order to make better decisions.

Yes, today the hardware is faster, the software is better, the data sets bigger – but the human context for data analytics in many ways remains ensconced in the Mad Men days.

Fortunately, this perception is changing. Real-time analytics technology is now a reality. Analysis of data about the past is still important, but insight about what’s happening right now is increasingly available to the decision maker, and it is thus up to that decision maker to understand how to leverage such insight to make good decisions – right now.

The True Value of Real-Time Analytics
The reason that it has taken so long for real-time analytics to become a useful tool is because of the bottleneck problem: every step in the data lifecycle must perform in real-time or the end result falls short.

Today all the pieces are finally falling into place. From data collection to movement to aggregation to processing to analysis to visualization, vendors and open source communities are stepping up to the plate and solving each bottleneck on this path to insight.

The end result – when everything works properly – can be astounding. A simple example: imagine moving the “buy” button to different positions on a live ecommerce web page while watching real-time purchasing data go up and down until you find the optimal location.

Now do the same thing for every other aspect of your business. That’s the power of real-time analytics.

Sounds good, right? But if that example leaves you scratching your head, you’re not alone. Only in certain still-rare circumstances are the benefits of real-time analytics so cut-and-dried. It’s one thing to shift a button around, but quite another to deal with the broad complexities of an enterprise business environment.

Perhaps the greatest challenge in the simplistic ecommerce example above is establishing the feedback loop. The traditional batch job/report/decision pattern ends with a human decision, not with an automated decision leading to an action that changes the data in real-time. That part of the real-time analytics pattern is still uncomfortably unfamiliar.

From Real-Time Analytics to Streaming Analytics
On the one hand, the importance of a feedback loop indicates how we must shift our thinking away from the traditional batch job/report/decision pattern to the real-time pattern in order to maximize the value we can squeeze from our analytics.

On the other hand, the feedback examples above reveal one remaining bottleneck: the human itself. As long as the goal of big data analytics insight is to fuel human decision making, then all-too-human limitations will continue to be our limiting factor.

Once we establish a fully automated feedback loop, however, then we take this final remaining bottleneck out of the equation, unleashing a new level of speed and performance.

We’re not just talking real-time analytics at that point. We’ve moved to streaming analytics.

The field of streaming analytics takes as its starting point not simply large data sets, but a never-ending, 24 x 7 fire hose of data. All of a sudden, making sure we have no bottlenecks simply becomes the price of admission.

We can’t even begin to deal with streaming data if we have to pause even for a moment to move it, store it, process it, or analyze it. The fire hose just keeps on streaming.

Today’s technology is finally piecing together the end-to-end components that make streaming analytics a reality. And yet, making sense out of the results of our streaming analytics – in real-time, without introducing a bottleneck – presents perhaps the greatest challenge of this approach.

Rising to this challenge is the field of cognitive computing – a way of analyzing streaming data that are multistructured, ambiguous, and in a constant state of flux.

The advantages are profound. Fraud detection and prevention, dynamic product pricing, Internet-of-Things (IoT) data analysis, electronic trading, customer promotion triggering, and compliance monitoring are some of the early examples of the power of streaming analytics – bolstered by cognitive computing to establish real-time, machine learning-based feedback loops that drive business value with no bottlenecks.

And you ain’t seen nothin’ yet.

Putting the Human Back into the Streaming Analytics Picture
Most people’s exposure to cognitive computing consists of watching IBM Watson trounce the world’s best players at Jeopardy! The same kind of technology is the key to fully automated streaming analytics – technology that gets smarter and smarter by itself.

If the notion of an intelligent, constantly learning, Jeopardy!-playing computer running the largest companies and governments in the world doesn’t frighten you just a bit, then you probably never watched the Terminator movies, or any number of other dystopian AI-run-wild flicks.

Well, you can relax – at least for now. Establishing fully automated streaming analytics feedback loops won’t take humans out of the picture. Rather, it raises the role humans play to a new level. Instead of Terminator, think Star Trek.

People, after all, must still build and manage the entire kit and caboodle. And for all its power and velocity, streaming analytics is still just a tool – a tool in human hands.

In order to play at this new level, however, we must learn new skills. Just as we’re struggling to move from batch job/report/decision thinking to real-time thinking, we must now take the next step: working with never-ending torrents of multistructured, dynamic data.

The Intellyx Take: Thinking at the Meta-Insights Level
In other words, we must raise the insights we obtain from streaming analytics to the meta level. The insights we gain aren’t simply about how best to run our business, as we’re automating how we use those insights. Instead, the insights we humans must gather are meta-insights: insights into how to leverage the ongoing insights the technology provides us.

Most people aren’t working at this meta-insight level. Our technology is advancing so quickly that we’re still struggling to move from batch thinking to real-time thinking. The result is increased turbulence, both in the business and technology domains. After all, change is ubiquitous and accelerating across the digital spectrum. It’s virtually impossible to keep up.

Dealing with change as the fundamental business constant, after all, is what the Agile Architecture Revolution is all about. As we remove the bottlenecks from our data processing and leverage the fire hose of real-time, streaming analytics, this revolution becomes an unavoidable business reality. Get ready for warp speed!

Intellyx advises companies on their digital transformation initiatives and helps vendors communicate their agility stories. As of the time of writing, none of the organizations mentioned in this article are Intellyx customers. Image credit: US Navy.

More Stories By Jason Bloomberg

Jason Bloomberg is the leading expert on architecting agility for the enterprise. As president of Intellyx, Mr. Bloomberg brings his years of thought leadership in the areas of Cloud Computing, Enterprise Architecture, and Service-Oriented Architecture to a global clientele of business executives, architects, software vendors, and Cloud service providers looking to achieve technology-enabled business agility across their organizations and for their customers. His latest book, The Agile Architecture Revolution (John Wiley & Sons, 2013), sets the stage for Mr. Bloomberg’s groundbreaking Agile Architecture vision.

Mr. Bloomberg is perhaps best known for his twelve years at ZapThink, where he created and delivered the Licensed ZapThink Architect (LZA) SOA course and associated credential, certifying over 1,700 professionals worldwide. He is one of the original Managing Partners of ZapThink LLC, the leading SOA advisory and analysis firm, which was acquired by Dovel Technologies in 2011. He now runs the successor to the LZA program, the Bloomberg Agile Architecture Course, around the world.

Mr. Bloomberg is a frequent conference speaker and prolific writer. He has published over 500 articles, spoken at over 300 conferences, Webinars, and other events, and has been quoted in the press over 1,400 times as the leading expert on agile approaches to architecture in the enterprise.

Mr. Bloomberg’s previous book, Service Orient or Be Doomed! How Service Orientation Will Change Your Business (John Wiley & Sons, 2006, coauthored with Ron Schmelzer), is recognized as the leading business book on Service Orientation. He also co-authored the books XML and Web Services Unleashed (SAMS Publishing, 2002), and Web Page Scripting Techniques (Hayden Books, 1996).

Prior to ZapThink, Mr. Bloomberg built a diverse background in eBusiness technology management and industry analysis, including serving as a senior analyst in IDC’s eBusiness Advisory group, as well as holding eBusiness management positions at USWeb/CKS (later marchFIRST) and WaveBend Solutions (now Hitachi Consulting).