AI to Test

The first choice on the benchmark screen is which AI answers the questions. Two AI layers are involved, and the screen says so: BI Pixie creates the questions and computes the correct answers from your data on its own, and the AI you choose here only has to answer them. Your AI provider is never the thing being tested, unless you choose the generic AI agent, which a bring-your-own provider plays.

Choose the AI

When you start from New benchmark in the page header, the screen opens on the workspace and semantic model pickers; both lists can be typed into. Starting from a semantic model's row skips that step. Then comes Which AI do you want to test? with three options, each described in one sentence:

Fabric data agent. A data agent you already have on this semantic model answers the questions.
Power BI Copilot. BI Pixie sends each question to the same query engine behind Copilot in Power BI.
A generic AI agent. BI Pixie emulates an AI agent reading your semantic model's metadata, the way most AI assistants and custom agents do, including Microsoft 365 Copilot.

All three are always listed, in that order. The Workload leads with the Fabric data agent, because inside Fabric that is the real thing; the portal leads with Power BI Copilot. An option that is not ready to run stays on the list and states its prerequisite in place. A Recommended badge marks the first option that is ready to run. Under each option, Under the hood opens three short blocks, What BI Pixie uses, What BI Pixie sends, and Requirements, naming the Microsoft services involved so you can look them up. The guidance line under the list states the recommended sequence: diagnose with a generic AI agent first, then confirm with Power BI Copilot.

Select Test with this AI to bind your choice, or Continue for the Fabric data agent, which leads into choosing the agent. Afterwards the card collapses to Measured through: Fabric data agent with a Test a different AI control. Your choice is remembered per semantic model, and changing it on a semantic model that already has runs asks you to confirm, because runs measured through different AIs are never compared as one trend.

One permission prompt, on the button. For Power BI Copilot and the generic AI agent, selecting Test with this AI asks Power BI for read-only access to your semantic models, so BI Pixie can put the questions to it, and for Copilot also for permission to use Copilot's query engine, which Microsoft's prompt words as Execute ML models. You see Microsoft's prompt once. Reading the three options asks for nothing, and if you dismiss the prompt, the option says so and Test with this AI asks once more. The portal attaches this access silently and never shows the prompt.

One tenant setting is required whichever AI you test: Dataset Execute Queries REST API, under Tenant settings > Integration in the Fabric admin portal. BI Pixie computes the correct answers the same way for every option, by running a direct query against your own data, and that query needs this setting. When it is off, the run stops at the run gate and says so, whichever option you chose.

The Three Options

Fabric data agent

BI Pixie sends each question to your data agent in plain language and records the answer it gives back. This tests the agent your team built and published, not an emulation. The data agent writes and runs its own query, so BI Pixie does not see the query behind an answer, and the result cannot show one. These prerequisites apply to this option only:

A workspace on a paid Fabric capacity, F2 or higher, or P1. This is verified when the run starts.
A published data agent on the semantic model, or a test agent BI Pixie creates for you.
BI Pixie installed in the workspace: testing a data agent runs through the BI Pixie Dashboard in your Fabric workspace. Without it, this option offers Install the dashboard and the other two options continue to work.
Your tenant's cross-geo AI settings allowing AI processing for your capacity's region, where that region requires them.

After Continue, BI Pixie lists the data agents that use the semantic model with a Published or Not published badge. Select one, or select Create test agent to have BI Pixie create an agent grounded on the semantic model. An unpublished agent cannot answer questions: select Publish it for me, or use Open in Fabric, which opens the agent in a browser tab, followed by I published it, check again.

Trial capacities cannot run data agents. Microsoft does not permit data agents on trial Fabric capacities. This is a platform limitation rather than a BI Pixie plan gate, and no BI Pixie plan can lift it. BI Pixie blocks the run and states the two ways forward: move the workspace to a paid capacity, or point the benchmark at a test agent in a workspace that already has one. The generic AI agent has no capacity requirement at all; Power BI Copilot's capacity requirement is described under its own option.

Power BI Copilot

This is the confirmation option. BI Pixie passes each question to Microsoft's Generate Query tool, which generates DAX from natural language using the same engine as Copilot in Power BI, then runs the query that comes back and reads the result. BI Pixie scores the query Copilot writes, not the answer Copilot presents to a user, so this is a close approximation of Copilot rather than the full Copilot experience, and the closest measure of what your users get from it. Microsoft assembles Copilot's view of the semantic model, so a wrong answer here cannot be traced the way the generic agent's can.

Requires a Copilot license in your organization and the tenant setting described below.
The workspace that holds the semantic model must be on a Fabric capacity: the query engine behind Copilot in Power BI refuses to answer without one, even though Copilot itself reaches semantic models in Power BI Pro workspaces. BI Pixie does not check the capacity in advance. The option stays selectable, a run on a workspace without a capacity completes with its questions not scored, and BI Pixie shows what the engine said for each one.

A generic AI agent

This is the diagnostic option. BI Pixie assembles exactly what the agent is shown, your schema, descriptions, AI Instructions, and synonyms, with every field you excluded from AI withheld, and sends that view and one question to your AI provider. For a question that needs a query, the provider writes DAX, and BI Pixie runs it and reads the result. Because BI Pixie assembled everything the agent saw, every wrong answer can be traced and fixed.

Requires a bring-your-own AI provider, which plays the emulated agent. The BI Pixie data agent cannot play that role, so an account whose only provider is the data agent is pointed to Connect an AI provider. Also requires the tenant setting described below.
Uses no Fabric capacity. It works on semantic models whose workspace has none, including Power BI Pro and Premium Per User workspaces.
It is not Power BI Copilot, and its result does not predict what Copilot returns.

Power BI Copilot and the generic AI agent both require one tenant setting: "Users can use the Power BI Model Context Protocol server endpoint (preview)", under Tenant settings > Integration in the Fabric admin portal. When it is off, the option shows a sentence you can forward to your Fabric administrator, with a Copy for your admin button. The Fabric data agent option is unaffected by this setting.

What Runs Where

AI tested	What answers	What checks the answers	Fabric capacity
Fabric data agent	Your data agent in your Fabric workspace	A direct comparison against your data, supported by an AI check when an AI provider is connected	Required. Your data agent answers inside your Fabric tenant, and its answers execute on your capacity.
Power BI Copilot	Copilot's own query engine	A direct comparison against your data, supported by an AI check when an AI provider is connected	Required. The query engine behind Copilot in Power BI refuses to answer for a workspace without one, and the run's queries execute on that capacity.
A generic AI agent	Your bring-your-own AI provider, shown your semantic model's metadata	A direct comparison against your data, supported by an AI check when an AI provider is connected	Not required. A semantic model on a capacity workspace uses that capacity for its queries, as any report would.

The correct answers are computed the same way for every option: a direct query against your own data. Your plan's allowance counts a run identically whichever AI measures it.

What's Next

Strategy, the next card on the screen.
AI Provider, required for the generic AI agent.
AI Assistants, to install the BI Pixie data agent.