News

Microsoft Furthering Open Source Hadoop Interoperability With Windows

• By Kurt Mackie
• 03/05/2012

Microsoft last week described more about its plans for "big data" and Hadoop interoperability with Windows and Microsoft tools.

The details were described on Wednesday by David Campbell, a Microsoft technical fellow, at the Strata Conference in Santa Clara, Calif. Hadoop is an open source MapReduce framework sponsored by the Apache Software Foundation that lets users gather business intelligence (BI) from unstructured and structured data at petabyte levels. Campbell said during the talk that Microsoft is submitting two new proposals to the Apache Software Foundation for the effort.

One proposal concerns a new JavaScript framework for writing MapReduce programs. The second proposal is focused on the creation of a new open database connectivity (ODBC) driver for Hive, which is Hadoop's data warehouse system.

Tools for Big Data
The addition of the Hive driver will bring Microsoft's BI tools to bear on Hadoop. For instance, Microsoft is touting the use of PowerPivot for Excel, as well as Power View, which will arrive with SQL Server 2012 product this month (Microsoft is planning a SQL Server 2012 "launch event" on Wednesday). Both PowerPivot and Power View can be used to visually display Hadoop query results.

Campbell explained during the talk that having Microsoft's BI tools in place for Hadoop is important because researchers need common tools to share their results. He proposed Data Explorer as a solution for such sharing. Data Explorer, offered via Microsoft's SQL Azure Labs, accommodates multiple data formats and ties into the Windows Azure Marketplace, which provides data feeds for a price. Data Explorer for SQL Azure provides "capabilities for data curation, collaboration and mashup," according to Microsoft's datasheet description (PDF).

With Hadoop, the idea is to improve "time to insight" when sifting through masses of data, and data quantity is key. Campbell said that, when using Hadoop, having more data with a less sophisticated algorithm is better than having less data with a more sophisticated algorithm.

Hadoop, largely fostered by Yahoo, has been used for things like social media analytics and ad analytics. It's designed to handle big data at a low cost. For instance, Hadoop is designed to run on commodity hardware and it permits queries to be conducted on big piles of data on an ad hoc basis.

Partner Efforts
Microsoft's main partner on Hadoop is Hortonworks, a key contributor to the open source effort. Hortonworks is collaborating with Microsoft on the Hive ODBC driver and the JavaScript framework. In addition, patches are being contributed to the Apache Software Foundation to enable Apache Hadoop version 1.0 on Windows Server, according to an announcement issued by Hortonworks.

Other Microsoft Hadoop partner efforts were announced last week at the Strata Conference. Microsoft is collaborating with Karmasphere to enable Karmasphere's tools on Hadoop for Windows Server and Windows Azure, including Karmasphere Analyst and Karmasphere Studio. Microsoft is also working with Datameer to help make that company's BI tools work with Hadoop on Windows Azure, according to Datameer's announcement. HStreaming and Microsoft have formed a "strategic relationship" that will enable HStreaming's real-time analytics tools to work with Hadoop on Windows Server and Windows Azure. The HStreaming effort is currently open for testing through a Microsoft Community Technology Preview program, according to HStreaming's announcement.

Microsoft has its own SQL Server technology called StreamInsight that's used for complex event processing. Possibly, StreamInsight could be used with Hadoop MapReduce jobs during the reducer phase, according to a Microsoft blog post. The reducer step is part of a three-tier Hadoop structure. Hadoop consists of the Hadoop Distributed File System at its base. Data are mapped via MapReduce. Lastly, the data undergo a reduce operation, which produces a summary of the data after they have been processed in parallel.

"Isotope" Release Plans
Microsoft uses an internal code name, called "Isotope," which describes its Hadoop interoperability efforts with Windows Server and Windows Azure, according to Alex Stojanovic, Microsoft's general manager of Hadoop on Azure and Windows, in a Channel 9 video. In that video, Stojanovic said that Microsoft will deliver Isotope on Azure in March 2012. In June 2012, Microsoft will deliver the general availability release of the "enterprise edition," he added. Microsoft also plans to provide "deep integration" with System Center, he said.

According to a tip given to veteran Microsoft observer Mary Jo Foley, the specific delivery dates have been disclosed. Microsoft will deliver Hadoop on Windows Azure on March 30, whereas Hadoop on Windows Server is expected to arrive on June 29, according to Foley's source.