Real-Time Intelligence in Microsoft Fabric

Overview of Real-Time Intelligence

On 15^th Nov 2023, Microsoft announced General Availability of Microsoft Fabric. For Data professionals who are already working on real-time data analytics solutions, here is small effort to introduce real-time intelligence capability of Microsoft Fabric. Microsoft Fabric is Cloud-centric, all-in-one large-scale data analytics service which can be used to design, implement, and manage systems to collect, store, process and analyse large volumes of data. Fabric offers a comprehensive suite of services through experiences such as Synapse Data Engineering, Synapse Data Science, Synapse Data Warehouse, Synapse real-time analytics, Data Factory, SQL Databases and Power BI.

Real-Time Intelligence service offers end-to-end solution for event-driven scenarios, streaming data, and data logs. Microsoft Fabric’s Real-Time Intelligence has revolutionized how organizations efficiently create comprehensive solutions by leveraging detailed, real-time data at scale. This all-encompassing data platform enables businesses to effortlessly ingest, process, analyze, visualize, monitor, alert, and respond to events. Recently Microsoft announced General Availability of Real-Time Intelligence in Fabric service. Which comes with Real-Time Hub, Eventstream, Eventhouse, Real-Time Dashboards, and Activator. With General Availability of Real-time intelligence, customers can maximize the powerful features of Real-Time Intelligence in their production solutions, improving operations and achieving their business objectives.

How Real-Time Intelligence works

Image source: https://learn.microsoft.com/en-us

• Real-Time hub: Centralized catalog that facilitates easy access, addition, exploration, and data sharing within your organization. Hub ensures that data is not only available but also accessible to everyone, fostering quick decision-making and informed actions.
• Eventstream: Eventstreams are no-code experience to capture, transform, and route high volumes of real-time events to various destinations. Eventstreams supports various data sources and destinations, offering a broad array of connectors to external sources such as Apache Kafka clusters, AWS streaming sources (Kinesis), and Google (GCP Pub/Sub).
• EventHouse: Eventhouses are the perfect analytics engine for processing real-time data. In Eventhouse, we store real-time data. They are designed for time-based, streaming events with structured, semi-structured, and unstructured data. This data is automatically indexed and partitioned by ingestion time, providing exceptionally fast and complex analytic querying capabilities on high-granularity data.

Within an eventhouse, you can create:

• KQL databases: Real-time optimized data store that contain a collection of tables, stored functions, materialized views, and shortcuts
• KQL querysets: A KQL queryset supports queries written using Kusto Query Language (KQL) and a subset of the Transact-SQL language that you can use to work with data in KQL database tables.
• Real-time dashboard: Real-time dashboards allow you to pin data visualizations to a unified visual interface, making it easy to display real-time insights briefly.
• Activator: Activator in Microsoft Fabric monitors changing data and enables automated processing of events that trigger actions when certain patterns or conditions are detected.

New Capabilities:

• OneLake Query Acceleration: OneLake serves as the central repository for all your data in Fabric, while Eventhouse offers a highly scalable and efficient engine for handling time-series data. If you have historical data in OneLake and wish to combine it with real-time streams or simply query it at lightning-fast speeds without natively landing it in Eventhouse, OneLake Query Acceleration makes this possible. With query acceleration, you can achieve up to 50x faster query performance compared to unaccelerated queries, and even greater improvements compared to other engines like Spark.
• Fabric Events: Facilitating seamless integration and management of event data, including OneLake and Job events, these capabilities enable a transition from schedule-driven to event-driven data solutions, reducing latency and saving time and resources.
• New Connectors: Adding support for Azure Service Bus, fulfilling a major customer request to stream data from Azure Service Bus topics or queues into Eventstream for real-time processing.

Conclusion

Real-Time Intelligence in Microsoft Fabric is now generally available, Real-Time Intelligence provides both pro-dev and no-code tools to ingest high-volume, high-granularity streaming data, dynamically transform it, query it in real-time for immediate insights, and trigger automated actions based on the data.

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is the first Indian Company to win the prestigious Microsoft Partner 2024 Award and is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, Microsoft Gold Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, AWS GenAI Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner,  AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, Amazon ECS Service Delivery Partner, AWS Glue Service Delivery Partner, Amazon Redshift Service Delivery Partner, AWS Control Tower Service Delivery Partner, AWS WAF Service Delivery Partner and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.