The Schwartz Report

Blog archive

Microsoft Extends Analytics Platform Portfolio with Databricks Spark Service in Azure

Microsoft and Apache Spark creator Databricks are building a globally distributed streaming analytics service natively integrated with Azure for machine learning, graph processing and AI-based applications.

The new Datrabricks Spark as a service was introduced at Microsoft's annual Connect developer conference, which kicked off today in New York. The new service, available in preview, is among an extensive list of announcements focused on its various SQL and NoSQL database products and services, as well as productivity, cross-platform and added language improvements to Visual Studio and VSCode developer tools, as well as new DevOps capabilities, new machine learning, AI and IoT tooling.

During the opening keynote, Scott Guthrie, Microsoft's executive VP for Cloud and Enterprise, emphasized that Databricks is the creator of, and steward of, Apache Spark, and the new service will enable organizations to build modern data warehouses that support self-service analytics and machine learning using all data types in a secure and compliant architecture.

Databricks has engineered a first-party Spark-as-a-service platform for Azure. "It allows you to quickly launch and scale up the Spark service inside the cloud on Azure," Guthrie said. "It includes an incredibly rich, interactive workspace that makes it easy to build Spark-based workflows, and it integrates deeply across our other Azure services."

Those services include Azure SQL Data Warehouse, Azure Storage, Azure Cosmos DB, Azure Active Directory, Power BI and Azure Machine Learning, Guthrie said. It also provides integration with Azure Data Lake stores, Azure Blob storage and Azure Event Hub.  "It's an incredibly easy way to integrate Spark deeply across your apps and drive richer intelligence from it," he said.

Databricks customers have been pushing the company to build its Spark platform as a native Azure service, said Ali Ghodsi, the company's cofounder and CEO, who joined Guthrie on stage. "We've been hearing overwhelming demand from our customer base that they want the security, they want the compliance and they want the scalability of Azure," Ghodsi said. "We think it can make AI and big data much simpler."

In addition to integrating with the various Azure services, it's designed to let those who want to create new data models to do so. According to Databricks, a user can target data regardless of size or create projects with various analytics services including Power BI, SQL, Streaming, MLlib and Graph. "Once you manage data at scale in the cloud, you open up massive possibilities for predictive analytics, AI, and real-time applications," according to a technical overview of the Azure Databricks service. "Over the past five years, the platform of choice for building these applications has been Apache Spark. With a massive community at thousands of enterprises worldwide, Spark makes it possible to run powerful analytics algorithms at scale and in real time to drive business insights."

However, deploying, managing and securing Spark at scale has remained a challenge, which Databricks believes will make the Azure service compelling.

Internally, Databricks is using the Azure Container Services to run the Azure Databricks control-plane and data planes using containers, according to the company's technical primer. It's also using accelerated networking services to improve performance on the latest Azure hardware specs.  

Posted by Jeffrey Schwartz on 11/15/2017 at 1:33 PM


Featured

  • AI, IoT and Machine Learning To Challenge Traditional Networking

    The next phase of networking will depend on IT learning to wrangle modern technologies in ways that simplify operations and help humans make decisions, according to a new report by Cisco.

  • Coming in 2020: .NET 5, The Next Phase of Microsoft's .NET Framework

    .NET 5 (no "Core" and no "Framework") will mark the transition from the aging, proprietary, Windows-only .NET Framework to a modern, open source, cross-platform .NET.

  • What Computing Will Look Like in 2030: Top 5 Tech Predictions for the Next Decade

    For better or worse, the next 10 years will bring more intelligent devices to more areas of our daily lives. From the proliferation of AI to what that means for user privacy, here are Brien's tech predictions for 2020 and beyond.

  • Azure Arc: A Deeper Look at Microsoft's Multicloud Play

    Arguably one of Microsoft's biggest announcements this year was the introduction of Azure Arc at Ignite. But is this really a game-changer or is Microsoft just falling for the multicloud buzz?

comments powered by Disqus

Office 365 Watch

Sign up for our newsletter.

Terms and Privacy Policy consent

I agree to this site's Privacy Policy.