Microsoft Fabric Real Time Analytics

In a recent Smartbridge Hackathon, a team explored using Microsoft Fabric to create a real time streaming analytics solution.

To stay ahead of the curve and cultivate a culture of innovation, Smartbridge conducts in-house hackathons throughout the year to test new technologies or case studies that could be beneficial to solving current client problems or to build up our consultants’ tool kits.

This past event consisted of a project that focused on the consumption of streaming data while transforming it and making it usable for a visualization tool to produce stunning graphics. Earlier this year, Microsoft rolled out their end-to-end analytics solution called Microsoft Fabric (note: some of their services are still in preview), and in the spirit of venturing into the unknown, the team at Smartbridge created a proof-of-concept solution using Microsoft Fabric to solve the streaming data analytics challenge.

Objective: Real Time Analytics using Fabric and Streaming Data

The need to stream data and having the ability to consume data in near real-time is an ever-growing requirement for competitive business industries as data becomes more and more available to companies than ever before. Gone are the days when it was deemed reasonable to take hours to massage data and get it to a stage where a visualization tool can consume it and create reports or visuals off of it.

Luckily, Microsoft is at the forefront of this need and their response to unifying the traditional Extract-Transform-Load process is Microsoft Fabric. The main objective for this exercise is to explore the functionality and usability of Microsoft’s newest offering along with utilizing some of the preview features such as setting up reflexes in Data Activator to add value to the solution.

In theory, we should be able to simulate a streaming dataset modeled after an IoT use case using an Azure event streams, then stage the data in an Azure Lakehouse, transform it in an SQL endpoint, and have a Power BI report consume it.

Challenges from Fabric, Power BI and other Real Time Analytics Difficulties

To give some background, event streams in Azure can bring high-frequency data into the SQL endpoint of the Lakehouse from four main sources: Sample Data, Event Hubs, IoT Hubs, or a Custom App. We created an event stream object using the “Stock Market” sample dataset to explore its enhanced UI/UX capabilities to stream data to a Lakehouse. Some of the key challenges we ran into while completing this solution were the following:

  • Custom sample data from the Oil and Gas industry could not be used in an event stream object, so a Jupyter notebook with a readStream function was used to simulate the streaming object.

  • Due to the nature of some of the features still being in preview, there were some bugs in the usability that were out of the user’s control. For example, we set up a reflex on Data Activator for a specific KPI to trigger an email once it got past a certain threshold. The rate at which the emails were being sent ended up being a little inconsistent and that was due to the server response.

  • Some of the key features needed were also locked behind a substantial paywall, in this case, the paywall being Power BI Premium. One of the features not available is the automatic refresh of the visuals. For a Pro license, it keeps it at 30 minutes while a Premium capacity would have the option to refresh every 1 second.

Why focus on Oil & Gas data for our sample scenario?

Smartbridge has 20+ years of experience working with energy and oil & gas companies to modernize their data foundation.

The Smartbridge Solution

microsoft fabric real time analytics
Fabric Enhanced Workspace UI/UX makes it way easier to keep track of lineage in its architecture.

With our objective in mind, we went to work on creating the simulated data following what would be our mock-up oil pipeline pressure readings. We created a script that fed CSV files with these made-up readings in a staggered 5-second interval to ensure continuous data was being fed into the solution. Then, we created an Azure Lakehouse to store the data that was captured from these CSVs. Next, the Lakehouse was connected to an Azure SQL endpoint.

The only transformations we did at this stage were appending the data from the different CSVs into one table. Once the data was established and running as expected, we created a Power BI report that connects directly to this SQL endpoint in Direct Query mode to get the latest data every time the report was interacted with along with troubleshooting the automatic refresh on the visuals.

microsoft fabric real time analytics
Final Output: A Power BI report with updates at a user-defined frequency and user-defined event triggers

The Results: Real Time Data!

By taking advantage of Microsoft Fabric, the team was able to create an end-to-end workflow solution from ingesting raw data, all the way to visualizing information and even creating email alerts for when said information has an anomaly.

The ETL process is not new, and we have had the capability of extracting data, transforming it, and loading it for some years now, but what Microsoft Fabric did differently is that it is all found in one centralized location.

The ability to view a single lineage from start to end is of high value, especially when needing to troubleshoot why a certain section of the pipeline is not behaving as expected.

Share This:

Looking for more on data and analytics?

Explore more insights and expertise at

There’s more to explore at!

Sign up to be notified when we publish articles, news, videos and more!