Microsoft FabCon Day 3: Keynote and Artificial Intelligence Updates

For Day 1 Microsoft FabCon Blog and Updates: Read here.

Read Day 2 Blog Here for Microsoft Fabric Featured Partner Insights.

Nosebleeds.   I’d forgotten about the nosebleeds.  Not sure why it happens but the day after a long-haul flight my nose starts bleeding.   Anyway, this morning my room looks like the aftermath of a Hillary Briss BBQ.   

I left my coffee run until 7am this morning.  The casino floor was humming.  Loud techno music pumping out, lots of people sat at slot machines and even one or two at the bar.  Fair play.  I tried out a Bagel store which was pretty good, but my order came with no butter and an object that even after eating it I’m struggling to identify. 

A few of points worthy of note before I get into the meat of the day: 

1. Why do people here use travelators to slow down?  Surely the idea of them is to accelerate your journey into the casino but people get on them and stop. 

2. There was an out of order sign on an escalator this morning meaning I had to use the stairs.  Escalators are one of the few things in the world that can’t really break down – surely a broken escalator *is* stairs? 

3. My room is on the 26th floor.  When I got in the lift this morning an American lady also got in and her opening gambit was “I am sweating – A LOT!”.   The journey down to the lobby felt like about an hour and a half. 

Right, this is it keynote day. Time to unpack my Fabric branded merchandise and make my way over to the T-Mobile Arena.

As expected, this was a far more razzmatazz day. There wasn’t room to open my laptop so I had to take notes by hand and the pace was incredibly fast, but I will try and translate my scrawl.

Highlights from the opening session:

• There are 6,500 of Microsoft’s “closest partners and customers” at the conference.

• Microsoft are the only partner to appear in the Leaders quadrant of Gartner’s Magic Quadrants for Cloud DBMS, Cloud AI Development, Data Integration and Analytics & BI.

• The Microsoft Fabric trial makes implementation easy to get started and there is the equivalent of $17,000 of capacity available to trial customers.

• There are 322 user groups in 62 countries with 267,000 members. Every 99 seconds somebody answers a question on a community forum, and expertise is shared.

• 30,000 DP600 and DP700 exams have been taken in the last 12 months.

The keynote was oriented around three pillars, AI Enabled Data Platform, Open & AI ready Data Lake and AI Enabled Business Users. Let’s dive into these:

AI Enabled Data Platform

Microsoft started this part of the keynote announcing that 87% of business leaders think that AI could give them a competitive edge.  That left me wondering who the 13% were that didn’t and whether I could find out who they are so that I could divert any of my pension that is invested in their stock elsewhere.   

There was a significant emphasis on professional developers. This included:

• Extended support for Service Principals.

• Terraform provider for Fabric.

• Variable libraries in Public Preview giving the ability to manage variables at Workspace level.

• User Data Functions – created in code and a SSAS version of Azure functions.

• A command line interface (CLI) – this looks at Fabric through the lens of an operating system using O/S familiar commands.

No complaints here on any of those developments, the acknowledgement of the need for features like these is very welcome, and showcases the innovative opportunities available.

SQL Data Warehouse has had significant focus. Given that this was a brand-new product in the market, there has been development and process, to overcome initial barriers. Key points include:

• 100+ features have shipped in the last 12 months

• There has been a 40% improvement in query performance.

Co-Pilot is now built into the development environment with intelligent completion of syntax, the ability to translate natural language requests into SQL and the ability to generate reports directly from SQL results (and save them).   

Warehouse supports notebooks for those that prefer them, but they come with the ability to switch between compute engines so it will not be compulsory for the users who know SQL to use Spark and Python.

A new and exciting feature includes – Synapse Gen2 Warehouse Migration ‘Seamless Warehouse Migration’. A new “migrate” button will appear in workspaces from today which will run a workflow which connects to the Synapse warehouse, allowing the user to select the objects to migrate (views, procs, tables etc.), go through a validation pane and finally start the migration process. 

For database objects that don’t migrate seamlessly, Co-Pilot can be used to create suggestions for Fabric-ready alternatives.   Data can be migrated manually or via the assistant.  Interestingly, part of the migration process is to create copies of all existing Synapse connections to the migrated database in Fabric so that application migration is also as seamless as possible. 

On the data engineering side, Spark compute has grown 50x in the last 12 months, and Fabric Spark runs 400% faster than Open-Source Spark. Notebooks now contain “T-SQL Magic” where T-SQL can be inserted into a Python data frame with the results available to be operated on.  The newly announced Data Functions can be called from notebooks, and the new Variable Library is also accessible to allow for use of Development, Test and Production parameter sets.  Variable objects can be created in the notebook with IntelliSense enabled to make it easy to browse through the variable library.   Microsoft expect a lot of Warehouse instances to be maintained through notebooks and so new Co-Pilot capabilities are in public preview from today. 

Materialised views can now be created in Spark / Lakehouse.  This includes a management portal which allows view precedence to be visualised and allows errors during the creation of the view to be diagnosed.  Views can contain data quality constraints so failure of a view could be the violation of a DQ constraint.   From a user perspective, materialised views just look like tables. 

Auto-scale for Spark is in public preview – customers will be billed for what they use with up and down auto-scale which should be of benefit in scenarios where Spark workloads are “spiky”. 

An important message which may slip under the radar is that Microsoft Fabric is evolving from an “analytics platform” to a “unified data platform“. There is a clear focus on bringing more database technology under the Fabric umbrella, and I’m excited to see future developments and updates.

The Real Time Intelligence Part of the keynote began with a few statistics:

• 5.1 billion real time queries per day on “planet scale infrastructure”, with almost 9,000 of Fabric customers using Real Time Intelligence.

Microsoft has recognised a need to associate and model real world objects with streams of data.  The “Digital Twin Builder” has been created to accommodate this.  It allows entities and relationships to be modelled with data streams being mapped into the entities. This looks interesting and could be an attempt to bring the capabilities of Fabric closer to those of a full Historian product.  It’s been a few years since I worked with products like OSIsoft PI but in those market leaders it was useful to be able to visualise an asset hierarchy in a schematic and visualise streams of data in context.  Well worthy of further investigation. 

Open & AI Ready Data Lake

This segment begins with more impressive statistics, showcasing the development and future of Microsoft Fabric Implementation.

• 3 million shortcuts have been created and the volume of data in OneLake doubles every 15 weeks.

Additionally, there is now support for mirroring sources behind a firewall or private links.  The plan is to build this support into all future mirroring sources added to the platform going forwards with key points coming soon including, Mirroring for Azure SQL Database, SQL Server 2025, Oracle and Dataverse.

There was excitement amongst the audience for the announcement of OneLake security which can now be defined in one place in OneLake (using table, row or column security rules) and it will apply in every workload both inside and outside of Fabric – e.g. these security roles are respected if you connect to data in Onelake from Excel or Power BI desktop.  This makes far more sense than maintaining data level security in individual workloads so looks like another win. 

Data Factory has also had some of the professional developer treatment, particularly around the ability to create secure and dynamic pipelines.  Parameterisation works across Fabric so is available in Data Factory which can now also request credentials directly from Key Vault.   

A new “Dynamic Content” option on data sources allows parameterisation of data flows, data sources and destinations so that e.g. data can be pushed to different tables based on parameters passed into the pipeline.    There was a round of applause for the ability to save a data flow without having to automatically publish it. 

In what was something of a surprise, Amir was joined on stage by Chris Child, VP of Product for Snowflake.  Iceberg data can now be used on both platforms without the need to move or copy it.   

OneLake Catalog in Excel was also announced, with additional work being invested into network security, in particular data exfiltration protection for Fabric and Private Link for Fabric workspaces with both anticipated to be available by Spring, watch this space for any further updates.

AI Enabled Business Users

Power BI is the undeniable success story for Microsoft, with the current customer count standing at 375,000+. With key notes to mention, showcasing clear innovation and developments, these include:

• Extension of Direct Lake
  Direct Lake has been extended to work in Power BI Desktop.  This also includes composite models – the ability to have multiple direct query sources inside the same Semantic Model, enabling organisations to effectively accelerate their data journey.  

• Tabular Model Development Language (TMDL) will soon include Co-Pilot support.

• “Chat” and Co-Pilot for Power BI
  This upcoming development will allow developers to “educate Co-Pilot”, and train it to understand the content of a Power BI report including the ability to slice and dice directly from the Co-Pilot experience. Enabled across all paid SKUs, the Co-Pilot button will appear in the Power BI tenant in the new few weeks. This innovate feature demonstrates the fast-moving world of AI, and how organisations must stay ahead of the data curve.

The trajectory of Power BI suggests that the role of a Power BI Developer will shift from hands-on data wrangling and repetitive report-building to a blend of strategic oversight and technical adaptation. TMDL with Co-Pilot pushes more of a code-first approach which will need familiarity with scripting and AI-assisted model refinement rather than GUI-driven DAX development.  The Chat/Co-Pilot capabilities go some way to offloading basic report design to AI, so it will be important for report developers to understand how to educate it to understand a data model. It’s an evolution in role from solo DAX-and-design specialist to a hybrid part-developer, part-AI-curator.  Less manual slog, more tech fluency and AI collaboration – great news to future progression. 

Summary and Key Takeaways

Did you make it this far? I did say it was fast and furious.

Unsurprisingly, the Keynote majored on Microsoft’s progress with Fabric, with a strong emphasis on AI’s potential – tempered by the need for high quality data to support it.    

Highlights for me included the firm nod to professional developers – Terraform support, a CLI, parameterisation, OneLake security and variable libraries are significant steps towards an enterprise-ready backbone, alongside the welcome inclusion of Co-Pilot AI across all paid SKUs.   

The Snowflake partnership and expanded mirroring, reinforces Fabric’s positioning as a complementary player. The long-overdue Synapse migration tool, with its metadata workflow and preserved connections, directly tackles a key point from Blog #1 and will ease customer transitions. 

If you take the comments from the partner day together with the keynote announcements it does look like Microsoft have got their direction of travel right

Blog Author: Mick Horne, Data Analytics Practice Manager at Simpson Associates.

Read the full series here and dive into the latest FabCon updates via our LinkedIn and social channels.

If you would like to explore the potential of Microsoft Fabric for your organisation, get in touch via our live chat now.