AI to Test

The first choice on the benchmark screen is which AI answers the questions. Two AI layers are involved, and the screen says so: BI Pixie creates the questions and computes the correct answers from your data on its own, and the AI you choose here only has to answer them. Your AI provider is never the thing being tested, unless you choose the generic AI agent, which it plays.

Choose the AI

When you start from New benchmark in the page header, the screen opens on What gets tested, where you choose the workspace and the semantic model; both lists can be typed into, which matters in a tenant holding hundreds of workspaces. Starting from a semantic model's row skips that card. Then comes The AI that answers the questions, asking Which AI do you want to test? with three options, each described in one sentence:

Power BI Copilot. BI Pixie sends each question to the same query engine behind Copilot in Power BI.
Fabric data agent. A data agent you already have on this semantic model answers the questions.
A generic AI agent. BI Pixie emulates an AI agent reading your semantic model's metadata, the way most AI assistants and custom agents do, including Microsoft 365 Copilot.

All three are always listed, in that order. The portal leads with Power BI Copilot, because for a portal account that is the AI your users meet; the Workload leads with the Fabric data agent. An option that is not ready to run stays on the list and states its prerequisite in place. A Recommended badge marks the first option that is ready to run. Under each option, Under the hood opens three short blocks, What BI Pixie uses, What BI Pixie sends, and Requirements, naming the Microsoft services involved so you can look them up. The guidance line under the list states the recommended sequence: diagnose with a generic AI agent first, then confirm with Power BI Copilot.

Select Test with this AI to bind your choice, or Continue for the Fabric data agent, which leads into choosing the agent. Afterwards the card collapses to Measured through: Power BI Copilot with a Test a different AI control. Your choice is remembered per semantic model, and changing it on a semantic model that already has runs asks you to confirm, because runs measured through different AIs are never compared as one trend. The portal attaches the Power BI access these options need silently, so no permission prompt appears here.

One tenant setting is required whichever AI you test: Dataset Execute Queries REST API, under Tenant settings > Integration in the Fabric admin portal. BI Pixie computes the correct answers the same way for every option, by running a direct query against your own data, and that query needs this setting. When it is off, the run stops at the run gate and says so, whichever option you chose.

The Three Options

Power BI Copilot

This is the confirmation option. BI Pixie passes each question to Microsoft's Generate Query tool, which generates DAX from natural language using the same engine as Copilot in Power BI, then runs the query that comes back and reads the result. BI Pixie scores the query Copilot writes, not the answer Copilot presents to a user, so this is a close approximation of Copilot rather than the full Copilot experience, and the closest measure of what your users get from it. Microsoft assembles Copilot's view of the semantic model, so a wrong answer here cannot be traced the way the generic agent's can.

Requires a Copilot license in your organization and the tenant setting described below.
The workspace that holds the semantic model must be on a Fabric capacity: the query engine behind Copilot in Power BI refuses to answer without one, even though Copilot itself reaches semantic models in Power BI Pro workspaces. BI Pixie does not check the capacity in advance. The option stays selectable, a run on a workspace without a capacity completes with its questions not scored, and BI Pixie shows what the engine said for each one.

Fabric data agent

BI Pixie sends each question to your data agent in plain language and records the answer it gives back. This tests the agent your team built and published, not an emulation. The data agent writes and runs its own query, so BI Pixie does not see the query behind an answer, and the result cannot show one. These prerequisites apply to this option only:

A workspace on a paid Fabric capacity, F2 or higher, or P1. This is verified when the run starts.
A published data agent on the semantic model, or a test agent BI Pixie creates for you.
BI Pixie installed in the workspace: testing a data agent runs through the BI Pixie Dashboard in your Fabric workspace. Without it, this option offers Install the dashboard and the other two options continue to work.
Your tenant's cross-geo AI settings allowing AI processing for your capacity's region, where that region requires them.

After Continue, BI Pixie lists the data agents that use the semantic model with a Published or Not published badge. Select one, or select Create test agent to have BI Pixie create an agent grounded on the semantic model. An unpublished agent cannot answer questions: select Publish it for me, or use Open in Fabric followed by I published it, check again.

Trial capacities cannot run data agents. Microsoft does not permit data agents on trial Fabric capacities. This is a platform limitation rather than a BI Pixie plan gate, and no BI Pixie plan can lift it. BI Pixie blocks the run and states the two ways forward: move the workspace to a paid capacity, or point the benchmark at a test agent in a workspace that already has one. The generic AI agent has no capacity requirement at all; Power BI Copilot's capacity requirement is described under its own option.

A generic AI agent

This is the diagnostic option. BI Pixie assembles exactly what the agent is shown, your schema, descriptions, AI Instructions, and synonyms, with every field you excluded from AI withheld, and sends that view and one question to your AI provider. For a question that needs a query, the provider writes DAX, and BI Pixie runs it and reads the result. Because BI Pixie assembled everything the agent saw, every wrong answer can be traced and fixed.

Requires a connected AI provider, which plays the emulated agent, and the tenant setting described below.
Uses no Fabric capacity. It works on semantic models whose workspace has none, including Power BI Pro and Premium Per User workspaces.
It is not Power BI Copilot, and its result does not predict what Copilot returns.

Power BI Copilot and the generic AI agent both require one tenant setting: "Users can use the Power BI Model Context Protocol server endpoint (preview)", under Tenant settings > Integration in the Fabric admin portal. When it is off, the portal shows a sentence you can forward to your Fabric administrator, with a Copy for your admin button. The Fabric data agent option is unaffected by this setting.

What Runs Where

AI tested	What answers	What checks the answers	Fabric capacity
Power BI Copilot	Copilot's own query engine	A direct comparison against your data, supported by an AI check when an AI provider is connected	Required. The query engine behind Copilot in Power BI refuses to answer for a workspace without one, and the run's queries execute on that capacity.
Fabric data agent	Your data agent in your Fabric workspace	A direct comparison against your data, supported by an AI check when an AI provider is connected	Required. Your data agent answers inside your Fabric tenant, and its answers execute on your capacity.
A generic AI agent	Your AI provider, shown your semantic model's metadata	A direct comparison against your data, supported by an AI check when an AI provider is connected	Not required. A semantic model on a capacity workspace uses that capacity for its queries, as any report would.

The correct answers are computed the same way for every option: a direct query against your own data. Your plan's allowance counts a run identically whichever AI measures it.

What's Next

Strategy, the next card on the screen.
AI Provider, required for the generic AI agent.
AI Assistants, to create the BI Pixie data agent.