Joey on SQL Server

Microsoft Fabric Conference Europe Recap: Copilot, Real-Time Intelligence and More

The firehose of Microsoft Fabric announcements shows a once-clunky product finally hitting its stride. Here are the highlights.

• By Joey D'Antoni
• 09/27/2024

This week, I traveled to Sweden to speak at the European Microsoft Fabric Community Conference (Fabcon EU), which was a showcase of Microsoft's continued investments in the Fabric software-as-a-service platform.

Microsoft gave a very long list of announcements about new features and many components of the Fabric service that are becoming generally available. In this column, I wanted to focus on a few of the more interesting announcements and dive deeper into one of the most notable components of Fabric: real-time intelligence.

The Top Features
One of the more surprising announcements from Fabcon EU is that Fabric now supports mirroring an Azure Databricks Unity Catalog. Unity Catalog is Databricks' popular governance solution for data and AI that aims to reduce friction around both data governance processes and, in turn, security.

Databricks is a direct competitor to Fabric, which makes this announcement somewhat surprising. However, there are several points of interoperability between the two platforms. I frequently see large enterprises buying overlapping solutions to take advantage of specific functionality in each platform. This integration with Unity Catalog will make a lot of larger organizations who use both Databricks and Fabric happy.

Microsoft also gave a number of announcements around Purview, including the general availability of its new data governance experience.

And, of course, it wouldn't be a technology event in 2024 if there were no announcements related to generative AI. To that end, Microsoft introduced an update to Copilot for Data Warehouse and made the Copilot feature for Dataflow Gen 2 (an ETL tool) generally available.

Copilot in Dataflow will help customers generate mashup language (also known as M) to transform data using natural-language queries, significantly simplifying the task. Meanwhile, the Copilot experience for Data Warehouse will be similar to the experience in Azure SQL Database for generating and remediating T-SQL code. In addition to the Data Warehouse experience, Microsoft also promised Copilot for the SQL endpoint (which is somewhat analogous to the Synapse serverless service in Azure) will be coming soon.

One thing to note about these AI features is that they require a minimum Fabric SKU of F64.

Inside Real-Time Intelligence
Real-time intelligence (RTI) is one of the more compelling features of Fabric, simply because it represents a more mature implementation of features around handling streaming and log data than anything else on the market. It’s not that there are no other solutions that do this; there are several open source, commercial and even as-a-service offerings that can handle streaming data. However, they are challenging and complex.

As an example, I was involved in a project a few years ago where a customer was using Amazon’s Data Migration Service (DMS) to connect to a SQL Server, with Change Data Capture enabled. The change stream data streamed into Amazon S3 from DMS, where it was then picked up into a Kafka topic. This project was ultimately foiled by DMS using an undocumented SQL Server procedure to read the transaction log, which eliminated the guarantee of the order of data.

As you can see from this example, capturing and parsing streaming data is challenging. Even for a simpler example, just creating a streaming demo in Azure typically requires the use of a custom application, an Event Hub and Stream Analytics. Then you connect that to Power BI.

RTI makes this all much easier. All of the tools I named above are aimed at developers and IT pros. With RTI, Microsoft announced a new user experience that allows “citizen developers” to build streaming data flows and dashboards within Fabric.

This experience interfaces with many sources, including nearly all Azure Data Platform options (Azure SQL Database, Managed Instance, Virtual Machines, PostgreSQL, MySQL, Cosmos DB) and those from Google Cloud and Amazon. Once you have defined a data source, you define an "Eventhouse" (I am not in love with this name, but Microsoft wants to maintain the "lakehouse"/"warehouse" naming convention). The data is then analyzed using Microsoft's Kusto Query Language (KQL), which is quite friendly to time-series data.

Streaming data use cases and demos typically follow a few patterns. A common one is engine temperature, where you're notified if the temperature goes above or below range -- or, in a more straightforward scenario, the time series data for the temperature of my BBQ along with the beef brisket smoking within. However, there's another Fabric component that makes real-time intelligence scenarios more interesting: Data Activator.

Data Activator is a trigger-based service with Reflex alerts, which can trigger based on real-time data patterns or conditions. Commonly, you may see this used in those "out-of-band" data scenarios mentioned above, though some customers use them in conjunction with Power BI dashboard to eliminate the need for sales representatives to look at a dashboard daily. Instead, they simply get an e-mail when underlying data about their sales direction changes. One firm captured its ETL processes into a table, loaded them into a Power BI dashboard, and received a data-driven alert when the number of NULL rows exceeded a certain percentage, indicating anomalies with the source system.

It's not that any of these scenarios weren't possible before. However, building solutions like these requires much development time and effort. With RTI and Data Activator, complex event-driven scenarios and responses can be built with a few mouse clicks. This simplicity makes them some of the more compelling features in Fabric -- a new, much easier way of doing something that used to be very difficult.

Microsoft continues to make heavy investments into Fabric; it is a key area for growth of its data platform segment. While Fabric has had its fits and starts, it is gradually becoming a more mature product, with Microsoft adding governance features through mirroring Unity Catalog for Databricks users, as well as the Purview Data Catalog and Data Loss Prevention features. And, unsurprisingly, Microsoft continues to add Copilot features across the Fabric environment, while real-time intelligence features are maturing quickly, allowing more organizations to take advantage of streaming data and make decisions faster.