Windows Azure HDInsight Now Available -- Redmondmag.com

Windows Azure HDInsight Now Available

By Kurt Mackie
10/28/2013

Microsoft on Monday announced the "general availability" of Windows Azure HDInsight, the company's cloud-based distribution of Hadoop.

Hadoop is an Apache open source project for so-called "big data" implementations using MapReduce technology. It uses clusters of commodity hardware to analyze heaps of structured or unstructured data, typically at large scale. Hadoop is used by companies to query petabytes of data to gain insights, such as sifting through clickstream Web data. The standard was originally fostered by Google, and further implemented by Yahoo and Facebook.

For its part, "Microsoft recognizes Hadoop as a standard," according to a blog post by Quentin Clark, corporate vice president of the Data Platform Group at Microsoft. The company worked with its partner, Hortonworks, to make HDInsight "100 percent Apache Hadoop offered as an Azure cloud service," according to Clark.

Multiple vendors offer various Hadoop-related products, but Microsoft is claiming full Hadoop compatibility with its HDInsight product. "HDInsight builds on top of the Hortonworks Data Platform (HDP) to ensure full compatibility with Apache Hadoop," Microsoft's SQL Server blog claimed. Hortonworks, in addition to partnering with Microsoft, offers its own Hadoop product, which includes Pig, Hive and Sqoop components, among others. Its HDP 2.0 product will be available for Windows next month, according to Clark.

Microsoft stopped developing its own "HDInsight Server for Windows" product in favor of Hortonworks' product, according to a post by veteran Microsoft watcher Mary Jo Foley.

Until today, Microsoft had just offered HDInsight as a preview test release. The product supports both Java and .NET, and Microsoft has released an SDK that allows dev-ops to use Visual Studio to write their Hadoop jobs. Early reviews, though, described the dev experience as a little bumpy. HDInsight supports PowerShell scripting and can tap the LINQ-to-Hive query language. Hive is an Apache project for data warehouses running atop Hadoop that supports querying and managing large data sets.

HDInsight notably works with Microsoft's Power BI tools. However, that's done by taking a small segment of Hadoop data and putting it into SQL Server. The amount of data that can be analyzed by Excel, one of Power BI's principal tools, is limited to 250 MB, which is a limitation of Excel's workbook size, Microsoft explained back in August.

Microsoft's HDInsight product rollout today is corresponding with opening of the Strata + Hadoop World event, which is ongoing this week in New York City. Clark is scheduled to speak at Strata + Hadoop World on Tuesday, and his talk will be streamed at 9:35 a.m. Eastern Time here.

About the Author

Kurt Mackie is senior news producer for 1105 Media's Converge360 group.

Featured

What's Inside Microsoft 365's Security Toolbox?

Microsoft provides plenty of native tools to secure Microsoft 365. IT pros just have to know where to look.
Microsoft Kills 30-Day Data Retention Policy for Copilot in Fabric

Microsoft this week announced several usability improvements coming to Copilot in Fabric in preparation of its general availability release later this year.
Using Microsoft Office to Build a Network Diagram 2

Now that we've set up our process, let's dig in with the actual execution.
Microsoft Graph 'Activity Logs' Feature Goes Live

Microsoft announced the general availability of the "activity logs" capability in Microsoft Graph, giving administrators more options to track user activity and identify patterns of potential misuse.
Fallout from Microsoft's 'Midnight Blizzard' Saga Hits Feds

When the Russian attack group Midnight Blizzard successfully breached Microsoft corporate e-mail accounts late last year, it apparently managed to steal U.S. government agency e-mails, too.