Decision Maker

Data, Done Big

The term is not just a buzzword. The trick is to figuring out how to get Big Data to work for you.

• By Don Jones
• 03/27/2015

Big Data. I literally haven't been to a single IT conference in the last two years where that term hasn't been bandied about. So much so, in fact, that I think everyone is just ignoring it, the way we do with terms like cloud or security these days. And that's a shame, because Big Data, believe it or not, is actually a useful thing, right here and right now.

Big Data kind of started out as a vague set of concepts applicable to the data warehousing and business intelligence trades. Its general mission statement was to help make practical use out of the metri butt-tones of data that every business has lying around in various sources. The problem is that early discussions were focused entirely on the technologies and techniques needed to manage massive data sets, and you started reading words like map reduce and Hadoop, neither of which meant anything to the average IT manager. Then the Big Service companies started jumping on Big Data, and marketing people at IBM, Oracle, HP and the like made Big Data as incomprehensible as possible.

But Big Data is a thing, and it's not just a thing for marketing demographics. IT operations people can use the heck out of it. Splunk, for example, is one vendor that's turned Big Data into a very real thing for making IT operations better. Even IBM is in the game for operations management, and Gartner Inc. invented IT Operations Analytics (ITOA) as an acronym.

So how can Big Data help you? Currently, the big focus is on historical analysis: scanning log files and other data repositories to collate and correlate data. You'll see spikes in overall workload across the organization, be able to better predict broad and specific trends, and so on. You'll be able to connect performance to specific events -- like the dropoff in productivity during the last office birthday celebration, or the specific activity patterns that accompanied the last DDoS attack on your extranet. Predictive analysis takes all of that and tells you what'll happen in the future, when you'll run out of computing power to handle the workload, and so on.

An emerging class of tools will focus instead on event processing, working with real-time data as it happens, rather than relying on what has happened. Think real-time alerts of activity patterns, proactive shuffling of workloads to accommodate near-future demand and that sort of thing. Crunching events from across your entire infrastructure in real time is absolutely Big Data in every sense of the word, and the tools that'll make it happen are impressive.

It's important to keep up with these developments because their enabling technologies -- things you'll end up using and supporting -- are markedly different from what's come in the past. That real-time operational data isn't going to live in a SQL Server RDBMS, because an RDBMS simply can't respond quickly enough to the bazillions of data points we're talking about. Instead, simpler, leaner, faster technologies -- NoSQL is a trend -- support Big Data storage and analysis.

This is also an important space because the skills and conceptual understanding needed to work with operational Big Data are a vast departure from what your team had probably gotten used to. You must take your best and brightest and skill them up.

There's a massive opportunity here. For the last decade, you've been spending money on a flexible infrastructure. You can live migrate VM workloads, spin up new databases, and provision new users -- hopefully with the press of a button. Operational Big Data will take that functionality -- the how of managing your environment -- and pair it with a why and when. Used properly, Big Data can intelligently tell your infrastructure what to do, and your existing investment will in large part be able to take it from there.

Yes, IT management has gotten a lot more complex, and it'll continue to do so. Whether you're in a very small environment -- where your ITOA may come from an outsourced cloud service -- or a huge one with massive on-premises investment, operational data is going to be a big part of your future.