Windows Azure HDInsight Now Available

Microsoft on Monday announced the "general availability" of Windows Azure HDInsight, the company's cloud-based distribution of Hadoop.

Hadoop is an Apache open source project for so-called "big data" implementations using MapReduce technology. It uses clusters of commodity hardware to analyze heaps of structured or unstructured data, typically at large scale. Hadoop is used by companies to query petabytes of data to gain insights, such as sifting through clickstream Web data. The standard was originally fostered by Google, and further implemented by Yahoo and Facebook.  

For its part, "Microsoft recognizes Hadoop as a standard," according to a blog post by Quentin Clark, corporate vice president of the Data Platform Group at Microsoft. The company worked with its partner, Hortonworks, to make HDInsight "100 percent Apache Hadoop offered as an Azure cloud service," according to Clark.  

Multiple vendors offer various Hadoop-related products, but Microsoft is claiming full Hadoop compatibility with its HDInsight product. "HDInsight builds on top of the Hortonworks Data Platform (HDP) to ensure full compatibility with Apache Hadoop," Microsoft's SQL Server blog claimed. Hortonworks, in addition to partnering with Microsoft, offers its own Hadoop product, which includes Pig, Hive and Sqoop components, among others. Its HDP 2.0 product will be available for Windows next month, according to Clark.

Microsoft stopped developing its own "HDInsight Server for Windows" product in favor of Hortonworks' product, according to a post by veteran Microsoft watcher Mary Jo Foley.

Until today, Microsoft had just offered HDInsight as a preview test release. The product supports both Java and .NET, and Microsoft has released an SDK that allows dev-ops to use Visual Studio to write their Hadoop jobs. Early reviews, though, described the dev experience as a little bumpy. HDInsight supports PowerShell scripting and can tap the LINQ-to-Hive query language. Hive is an Apache project for data warehouses running atop Hadoop that supports querying and managing large data sets.

HDInsight notably works with Microsoft's Power BI tools. However, that's done by taking a small segment of Hadoop data and putting it into SQL Server. The amount of data that can be analyzed by Excel, one of Power BI's principal tools, is limited to 250 MB, which is a limitation of Excel's workbook size, Microsoft explained back in August.

Microsoft's HDInsight product rollout today is corresponding with opening of the Strata + Hadoop World event, which is ongoing this week in New York City. Clark is scheduled to speak at Strata + Hadoop World on Tuesday, and his talk will be streamed at 9:35 a.m. Eastern Time here.

About the Author

Kurt Mackie is senior news producer for 1105 Media's Converge360 group.


  • Microsoft Offers More 'Solorigate' Advice Using Microsoft 365 Defender Tools

    Microsoft issued yet another article with advice on how to use its Microsoft 365 Defender suite of tools to protect against "Solorigate" advanced persistent threat types of attacks in a Thursday announcement.

  • Microsoft FastTrack Support Extended to Microsoft 365 Defender Solutions

    The Microsoft FastTrack support program has been extended to Microsoft 365 Defender products for certain qualified subscribers, Microsoft indicated this week.

  • Microsoft 365 File-Size Support Expanding to 250GB

    Microsoft 365 users will be getting expanded file-size support, allowing files to be shared that are 250GB maximum in size, per a Microsoft announcement this week.

  • Windows Server Update Services Users Getting Proxy-Use Change This Month

    Microsoft on Tuesday notified Windows Server Update Services (WSUS) users that it's no longer going to automatically support "user proxies" to get patches from Microsoft's content delivery networks (CDNs), starting with this month's cumulative update release.

comments powered by Disqus