BI Pixie on Power BI over Azure Databricks
The BI Pixie ETL is the Spark job that transforms the raw telemetry BI Pixie collects into the compact, pre-aggregated tables the BI Pixie dashboard reads. On Microsoft Fabric it runs as a notebook feeding a Direct Lake semantic model. If you run BI Pixie on Azure but do not have a Fabric capacity, you can run that same ETL on your own Azure Databricks workspace instead: the ETL reads the telemetry BI Pixie already stores, transforms it with Spark, and writes the tables the BI Pixie report connects to. You keep the same report and semantic model you already know, connected to Databricks rather than imported into Power BI Pro from storage.
The Databricks ETL is an option on the Enterprise plan, and our team sets it up for your environment. This page explains what the Databricks ETL is and how it works, so that you can decide whether it fits. When you are ready, you can open a support ticket, and our team scopes and sets up the Databricks ETL with you.
When the Databricks ETL is the right fit
- You already run Azure Databricks. If Databricks is part of your data platform, the Databricks ETL reuses compute and governance you already operate. Databricks also reads and writes the same Delta table format BI Pixie uses.
- You are on Azure without Fabric. Your BI Pixie telemetry lives in an Azure Storage account, but you have no Fabric capacity, so the Fabric Notebook + Direct Lake path is not available to you.
- You are hitting Power BI Pro refresh limits. Large data volumes push the dashboard's import refresh past the two-hour Power BI Pro cap, and you want Spark compute to do the heavy lifting instead.
If you are open to adopting Fabric, a small Fabric capacity with the Notebook + Direct Lake path is usually simpler than running Databricks, and it has no import refresh timeouts. A Databricks workspace is one more platform to administer, so the Databricks ETL suits you best when you already run Databricks, or cannot use Fabric.
What you get
- The same report and dashboard. There is no new report to learn. The published semantic model is generated from the same BI Pixie semantic model, with only its data connection swapped to Databricks, so the report, its pages, and its measures are unchanged.
- The same transformation logic. The Spark ETL is the same logic BI Pixie runs on Fabric, adapted to run on Databricks. Your dashboard shows the same metrics, computed the same way.
- Incremental processing. Each run reads only new telemetry and writes compact, pre-aggregated tables, so the amount of data Power BI has to handle stays small even as your history grows.
Choosing how Power BI connects
Three connection modes are available. The right one depends on your Power BI license and how fresh the data needs to be. All three use the same generated semantic model and the same report.
| Mode | How it behaves | Best for |
|---|---|---|
| Import (default) | Power BI caches the data; the Databricks SQL warehouse runs only at refresh time. Fast, in-memory reports. | Power BI Pro; the recommended default. |
| DirectQuery | Power BI queries Databricks live, so the report is always current with no refresh. Needs the warehouse warm while reports are open. | Teams that need near-live data. |
| Direct Lake | Power BI reads the Databricks tables through a Fabric mirror: no warehouse, near-real-time. Requires a Fabric capacity and a mirrorable catalog. | Teams that also have Fabric and want the fastest, lowest-overhead path. |
Microsoft's general guidance is to use Import by default and move beyond it only when a specific constraint requires it. On a Power BI Pro license, Import avoids DirectQuery's row limits and does not need a warehouse kept warm while reports are open. Our team helps you pick the right mode when we scope the project.
How it works (high level)
The setup is a short pipeline our team runs against your Databricks workspace. It does not require Power BI Desktop.
- Governance setup. We register your BI Pixie storage container with Databricks Unity Catalog (an access credential, an external location, and a volume) so Spark can read your telemetry under your own governance.
- Run the ETL. A Spark job reads the new telemetry and writes the pre-aggregated dashboard tables into a Unity Catalog schema you own. This is the step that repeats on a schedule to keep the dashboard current.
- Provision compute. A serverless Databricks SQL warehouse is created for Power BI to connect through. Serverless means it starts on demand and stops when idle.
- Publish. The semantic model and the reused report are published to your Power BI workspace, connected to Databricks, and (for Import) refreshed.
After setup, keeping the dashboard current is a single scheduled ETL run; the whole flow can run as one command in your environment. Both the ETL and the SQL warehouse run on serverless compute, so neither keeps a virtual machine running between uses.
What you will need
- An Azure Databricks workspace with serverless compute enabled, and a Databricks access connector.
- Your existing BI Pixie Azure Storage container (the telemetry BI Pixie already collects).
- A Power BI workspace to publish into, on a capacity (Premium, Premium Per User, or Fabric) if you want the semantic model refreshed automatically.
- Permission to assign a few Azure roles to the access connector's identity so Databricks can read your storage under Unity Catalog governance.
We provide the exact prerequisites, roles, and a runbook as part of the project.
Get started
The Databricks ETL is an option on the Enterprise plan. When you open a support ticket, our team confirms the fit, scopes the work, and sets up the Databricks ETL for your environment. If you are weighing the Databricks ETL against adopting Fabric, our team can help you compare the two paths for your data volume and licensing.