Skip to content

Benchmarks reference

A semantic model's benchmark questions and its scored runs, as the portal's benchmark tab shows them. Reading spends no allowance.

Every route below follows the conventions in Work with the BI Pixie API. You can also download the OpenAPI document.

List a semantic model's benchmark runs #

GET /v1/workspaces/{workspaceId}/semanticModels/{semanticModelId}/benchmarks/runs

The semantic model's scored benchmark runs, newest first, as the portal's past-runs list shows them: what answered the questions, the pass rate the portal renders, the verdict counts, the response-time statistics and why unscored questions were set aside. The fifty most recent runs are listed, in pages. A run stored before runs had ids cannot be opened by one and is left out.

The pass rate is correct out of every question asked, the rule the portal's headline uses. Reading the list spends nothing and asks the store nothing.

Scope
BIPixie.Read.All
Plan
Every plan
Counts against
Calls that read, 300 a minute

Parameters

Parameters for List a semantic model's benchmark runs
Name In Type Required Description
workspaceId path string (uuid) Yes The id of the Fabric workspace.
semanticModelId path string (uuid) Yes The id of the semantic model in that workspace.
maxResults query integer No The largest number of rows to return in one page.
continuationToken query string No The token the previous page answered with.

Responses

200. The runs, newest first.

Headers of the 200 response to List a semantic model's benchmark runs
Header Description
x-ms-request-id The id of this request. Quote it to support.
The 200 response to List a semantic model's benchmark runs, BenchmarkRunList
Field Type Always present Description
value array of BenchmarkRunSummary Yes
runId string Yes
storedAtUtc string or null (date-time) No When the scored result was stored.
status string Yes Completed, or CompletedPartial for a run that ended before every question was asked. Cancelled arrives with the cancel route. One of Completed, CompletedPartial.
answeredBy BenchmarkAnsweredBy Yes What answered the questions, as the run recorded it.
kind string or null Yes fabric_data_agent, pbi_mcp_agent or pbi_mcp_copilot.
dataAgentId string or null No The data agent that answered. Null when a generic AI agent or Power BI Copilot did.
dataAgentName string or null No
providerId string or null No The saved AI provider that answered, when a generic AI agent did.
providerName string or null No The saved AI provider's name, when one was recorded.
passRate number or null Yes The share of questions answered correctly in every repeat, out of every question asked, from 0 to 1: the number the portal's headline shows. Null when nothing reached a verdict. A question that was not scored counts as not answered correctly, so a run that could not check most of its answers reads low. Read it with scored, notScored and passRateSummary, which say how many questions it could check.
scored integer or null No How many questions reached a verdict: stable, flaky and failed together. Null for a run stored before its counts were recorded.
notScored integer or null No How many questions did not reach a verdict, the portal's "not scored". Each one counts against passRate, and notScoredReasons says why they were set aside.
passRateSummary string or null No What passRate stands for, in a sentence: how many questions were answered correctly out of how many, how many were not scored, and, when most were not scored, that few answers could be checked. Null for a run with no questions or no recorded counts.
stable integer or null No Answered correctly every time.
flaky integer or null No Answered correctly at least once and wrongly at least once.
failed integer or null No Answered, and wrong.
inconclusive integer or null No Never reached a verdict.
total integer or null Yes
questionsCount integer or null No How many questions the run was given.
questionSetHash string or null No The hash of the question set the run used, which the saved questions carry too.
responseTime one of BenchmarkResponseTime, null No
notScoredReasons object or null No Why unscored questions were set aside, counted by reason. Null when every question was scored. A run scored before 2026-09-26, when BI Pixie began recording the reasons, can show null in this list; reading that run on its own gives them, rebuilt from its stored verdicts.
continuationToken string No
continuationUri string (uri) No

Errors

Every refusal is an ErrorResponse with a stable errorCode. See Errors for what each HTTP status means and every code the API can return.

This is an example of a refusal. Every refusal has the same fields.

Example refusal 
HTTP/1.1 404 Not Found
Content-Type: application/json
x-ms-request-id: cbf21b6d-b661-4b52-88cb-d79712e504fc

{
  "errorCode": "ItemNotFound",
  "message": "No item with this id is visible to the signed-in person in this workspace, or it is not of the type the route takes.",
  "isRetriable": false,
  "moreDetails": [],
  "relatedResource": {
    "resourceId": "d972b3f5-a2fa-47ec-94ff-0403a4d159fa",
    "resourceType": "SemanticModel"
  },
  "parameters": [
    {
      "name": "itemId",
      "value": "d972b3f5-a2fa-47ec-94ff-0403a4d159fa"
    }
  ],
  "requestId": "cbf21b6d-b661-4b52-88cb-d79712e504fc"
}

Example

Request 
curl -i -X GET \
  -H "Authorization: Bearer $TOKEN" \
  "https://api.bipixie.com/v1/workspaces/$WORKSPACE_ID/semanticModels/$SEMANTIC_MODEL_ID/benchmarks/runs"

An example of what the call answers. The values are fictional.

Response, 200 
HTTP/1.1 200 OK
Content-Type: application/json
x-ms-request-id: cbf21b6d-b661-4b52-88cb-d79712e504fc

{
  "value": [
    {
      "runId": "f2e14a95-8025-41de-a48f-860f8cfd2e68",
      "storedAtUtc": "2026-09-14T09:42:17Z",
      "status": "Completed",
      "answeredBy": {
        "kind": "fabric_data_agent",
        "dataAgentId": "05dd87d1-624f-4ffa-ab88-ee231a75a904",
        "dataAgentName": "Contoso Sales Agent",
        "providerId": "b2df6be3-7769-4222-90ea-9fd98f4592e0",
        "providerName": "Azure OpenAI"
      },
      "passRate": 0.75,
      "scored": 12,
      "notScored": 78,
      "passRateSummary": "Contoso Sales",
      "stable": 3,
      "flaky": 3,
      "failed": 3,
      "inconclusive": 3,
      "total": 12,
      "questionsCount": 12,
      "questionSetHash": "12056ac56f6dd1de",
      "responseTime": null,
      "notScoredReasons": {}
    }
  ],
  "continuationToken": "eyJza2lwIjoxMDB9",
  "continuationUri": "https://api.bipixie.com/v1/workspaces/e17c1b59-7a2e-491e-acdd-11434854bbb0/semanticModels/f1fdc461-3ca1-4702-ae48-afc3522409fc/benchmarks/runs?continuationToken=eyJza2lwIjoxMDB9"
}

Get one benchmark run #

GET /v1/workspaces/{workspaceId}/semanticModels/{semanticModelId}/benchmarks/runs/{runId}

One scored run, whole: the headline fields the portal's result card is computed from, and under document the scored artifact as BI Pixie stores it, with every question, the answers it got and each verdict, relayed from the account's store and never reshaped. The run is named by the runId the list gives; an id that names no run of this semantic model in this workspace, or a run since deleted, answers 404 RunNotFound. A store that could not answer is 502 StoreUnavailable, and one behind a paused capacity is 409 CapacityPaused.

Scope
BIPixie.Read.All
Plan
Every plan
Counts against
Calls that read, 300 a minute

Parameters

Parameters for Get one benchmark run
Name In Type Required Description
workspaceId path string (uuid) Yes The id of the Fabric workspace.
semanticModelId path string (uuid) Yes The id of the semantic model in that workspace.
runId path string (uuid) Yes The run's id, as the runs list gives it.

Responses

200. The run.

Headers of the 200 response to Get one benchmark run
Header Description
x-ms-request-id The id of this request. Quote it to support.
The 200 response to Get one benchmark run, BenchmarkRun
Field Type Always present Description
runId string Yes
workspaceId string or null No
semanticModelId string or null Yes
semanticModelName string or null No
storedAtUtc string or null (date-time) No
status string Yes One of Completed, CompletedPartial.
answeredBy BenchmarkAnsweredBy Yes What answered the questions, as the run recorded it.
kind string or null Yes fabric_data_agent, pbi_mcp_agent or pbi_mcp_copilot.
dataAgentId string or null No The data agent that answered. Null when a generic AI agent or Power BI Copilot did.
dataAgentName string or null No
providerId string or null No The saved AI provider that answered, when a generic AI agent did.
providerName string or null No The saved AI provider's name, when one was recorded.
passRate number or null Yes As in the runs list, out of every question asked; read it with scored, notScored and passRateSummary.
scored integer or null No
notScored integer or null No
passRateSummary string or null No
stable integer or null No
flaky integer or null No
failed integer or null No
inconclusive integer or null No
total integer or null Yes
questionsCount integer or null No
questionSetHash string or null No
responseTime one of BenchmarkResponseTime, null No
notScoredReasons object or null No Why unscored questions were set aside, counted by reason. For a run scored before 2026-09-26 the counts are rebuilt from its stored verdicts by the rule BI Pixie applies to every run since, and document is still the run as stored.
document object Yes The scored artifact as stored: questions[] with each question, the answers it got and its verdict, suite_snapshot with the questions the run was given, and the run's own schema_version. When a generic AI agent or Power BI Copilot answered, agent_ref.data_agent_id is null, as dataAgentId is, because no data agent answered.

Errors

Every refusal is an ErrorResponse with a stable errorCode. See Errors for what each HTTP status means and every code the API can return.

This is an example of a refusal. Every refusal has the same fields.

Example refusal 
HTTP/1.1 404 Not Found
Content-Type: application/json
x-ms-request-id: cbf21b6d-b661-4b52-88cb-d79712e504fc

{
  "errorCode": "ItemNotFound",
  "message": "No item with this id is visible to the signed-in person in this workspace, or it is not of the type the route takes.",
  "isRetriable": false,
  "moreDetails": [],
  "relatedResource": {
    "resourceId": "d972b3f5-a2fa-47ec-94ff-0403a4d159fa",
    "resourceType": "SemanticModel"
  },
  "parameters": [
    {
      "name": "itemId",
      "value": "d972b3f5-a2fa-47ec-94ff-0403a4d159fa"
    }
  ],
  "requestId": "cbf21b6d-b661-4b52-88cb-d79712e504fc"
}

Example

Request 
curl -i -X GET \
  -H "Authorization: Bearer $TOKEN" \
  "https://api.bipixie.com/v1/workspaces/$WORKSPACE_ID/semanticModels/$SEMANTIC_MODEL_ID/benchmarks/runs/$RUN_ID"

An example of what the call answers. The values are fictional.

Response, 200 
HTTP/1.1 200 OK
Content-Type: application/json
x-ms-request-id: cbf21b6d-b661-4b52-88cb-d79712e504fc

{
  "runId": "f2e14a95-8025-41de-a48f-860f8cfd2e68",
  "workspaceId": "e17c1b59-7a2e-491e-acdd-11434854bbb0",
  "semanticModelId": "f1fdc461-3ca1-4702-ae48-afc3522409fc",
  "semanticModelName": "Contoso Sales",
  "storedAtUtc": "2026-09-14T09:42:17Z",
  "status": "Completed",
  "answeredBy": {
    "kind": "fabric_data_agent",
    "dataAgentId": "05dd87d1-624f-4ffa-ab88-ee231a75a904",
    "dataAgentName": "Contoso Sales Agent",
    "providerId": "b2df6be3-7769-4222-90ea-9fd98f4592e0",
    "providerName": "Azure OpenAI"
  },
  "passRate": 0.75,
  "scored": 12,
  "notScored": 78,
  "passRateSummary": "Contoso Sales",
  "stable": 3,
  "flaky": 3,
  "failed": 3,
  "inconclusive": 3,
  "total": 12,
  "questionsCount": 12,
  "questionSetHash": "12056ac56f6dd1de",
  "responseTime": null,
  "notScoredReasons": {},
  "document": {}
}

Get a semantic model's saved benchmark questions #

GET /v1/workspaces/{workspaceId}/semanticModels/{semanticModelId}/benchmarks/questions

The benchmark questions saved for this semantic model, as the question set holds them under document, with the saved strategy and its answer-checking mode. A semantic model with no saved questions answers saved: false with no document, which is what a run would refuse with 400 BenchmarkQuestionsMissing. Available on every plan.

Scope
BIPixie.Read.All
Plan
Every plan
Counts against
Calls that read, 300 a minute

Parameters

Parameters for Get a semantic model's saved benchmark questions
Name In Type Required Description
workspaceId path string (uuid) Yes The id of the Fabric workspace.
semanticModelId path string (uuid) Yes The id of the semantic model in that workspace.

Responses

200. The saved questions, or that there are none.

Headers of the 200 response to Get a semantic model's saved benchmark questions
Header Description
x-ms-request-id The id of this request. Quote it to support.
The 200 response to Get a semantic model's saved benchmark questions, BenchmarkQuestions
Field Type Always present Description
saved boolean Yes Whether a question set is saved for this semantic model.
questionSetHash string or null No The hash the saved document carries, which only a question set saved straight from a generated proposal has. Null for questions saved in the BI Pixie portal. To match the saved questions with runs, use runQuestionSetHash.
runQuestionSetHash string or null No The questionSetHash a run of the saved questions, as they stand now, carries: the questions that would run, without the excluded ones. Null when nothing saved would run.
runCount integer or null No How many of the semantic model's most recent fifty runs asked exactly these saved questions. 0 means they have not been run as they stand now, including a set changed since its last run. Null when the runs could not be read or nothing saved would run.
lastRunAtUtc string or null (date-time) No When the newest of those runs was stored. Null when there is none.
questionsCount integer Yes
questionsOutdated integer No How many of the questions BI Pixie wrote predate the current question rules, so a preview of the questions would rewrite them.
questionsOutdatedReason string or null No Why the outdated questions are outdated, in sentences: how many there are, what BI Pixie changed in how it writes questions since they were written, and the step that replaces them. Null when no question is outdated.
questionsOutdatedChanges array of string No One sentence for each change to how BI Pixie writes questions that the oldest outdated question predates, oldest first. The same sentences appear in questionsOutdatedReason. Empty when no question is outdated.
periodNote string or null No Which period the questions cover and why BI Pixie chose it, in a sentence. Null when the set records no period.
shortfalls array of string No One sentence for each way the saved questions fall short of the saved strategy (fewer questions than it asks for, a level of question it asks for and the set lacks, or one wording where it asks for different words), with the cause where BI Pixie recorded one. Empty when the set matches its strategy.
answerCheckingMode string or null No The saved strategy's mode for checking answers.
strategy object or null No The saved strategy as stored.
document object or null Yes The saved question set as stored, questions[] included.

Errors

Every refusal is an ErrorResponse with a stable errorCode. See Errors for what each HTTP status means and every code the API can return.

This is an example of a refusal. Every refusal has the same fields.

Example refusal 
HTTP/1.1 404 Not Found
Content-Type: application/json
x-ms-request-id: cbf21b6d-b661-4b52-88cb-d79712e504fc

{
  "errorCode": "ItemNotFound",
  "message": "No item with this id is visible to the signed-in person in this workspace, or it is not of the type the route takes.",
  "isRetriable": false,
  "moreDetails": [],
  "relatedResource": {
    "resourceId": "d972b3f5-a2fa-47ec-94ff-0403a4d159fa",
    "resourceType": "SemanticModel"
  },
  "parameters": [
    {
      "name": "itemId",
      "value": "d972b3f5-a2fa-47ec-94ff-0403a4d159fa"
    }
  ],
  "requestId": "cbf21b6d-b661-4b52-88cb-d79712e504fc"
}

Example

Request 
curl -i -X GET \
  -H "Authorization: Bearer $TOKEN" \
  "https://api.bipixie.com/v1/workspaces/$WORKSPACE_ID/semanticModels/$SEMANTIC_MODEL_ID/benchmarks/questions"

An example of what the call answers. The values are fictional.

Response, 200 
HTTP/1.1 200 OK
Content-Type: application/json
x-ms-request-id: cbf21b6d-b661-4b52-88cb-d79712e504fc

{
  "saved": true,
  "questionSetHash": "12056ac56f6dd1de",
  "runQuestionSetHash": "41d8195ced5b16a9",
  "runCount": 12,
  "lastRunAtUtc": "2026-09-14T09:42:17Z",
  "questionsCount": 12,
  "questionsOutdated": 3,
  "questionsOutdatedReason": "Contoso Sales",
  "questionsOutdatedChanges": [
    "Contoso Sales"
  ],
  "periodNote": "Contoso Sales",
  "shortfalls": [
    "Contoso Sales"
  ],
  "answerCheckingMode": "preserve",
  "strategy": {},
  "document": {}
}

Save a semantic model's benchmark questions #

PUT /v1/workspaces/{workspaceId}/semanticModels/{semanticModelId}/benchmarks/questions

Saves the benchmark questions in document as this semantic model's question set, replacing the one saved before. document has the shape the read of the saved benchmark questions returns, and the shape a proposal from generateBenchmarkQuestions returns, so a proposal can be reviewed, edited and saved as it is. BI Pixie stores the document as sent. The answer is an operation that has already succeeded; its result is the saved questions as the read returns them. Saving counts toward no allowance.

Scope
BIPixie.ReadWrite.All
Plan
Every plan
Counts against
Calls that start work, 30 a minute

Parameters

Parameters for Save a semantic model's benchmark questions
Name In Type Required Description
workspaceId path string (uuid) Yes The id of the Fabric workspace.
semanticModelId path string (uuid) Yes The id of the semantic model in that workspace.

Request body

JSON, SaveBenchmarkQuestionsRequest. The body is required.

Request body for Save a semantic model's benchmark questions
Field Type Required Description
document object Yes The benchmark questions to save, in the shape BenchmarkQuestions returns under document.

Responses

202. The questions were saved. The body carries the operation's id and address.

Headers of the 202 response to Save a semantic model's benchmark questions
Header Description
x-ms-request-id The id of this request. Quote it to support.
x-ms-operation-id The id of the operation.
Location The address of the operation's state.
Retry-After Seconds to wait before asking for the operation's state.

To follow the work this call started, see Follow a long-running operation.

The 202 response to Save a semantic model's benchmark questions, Accepted
Field Type Always present Description
operationId string (uuid) Yes The operation's id, which the operation's state and result take.
location string (uri) Yes Where to ask about the operation.

400. The request is not well formed, and nothing was started, changed or counted. errorCode is InvalidRequest, or InvalidPreview for a preview that is not available. A body member the operation does not document is refused this way, and the message names it and the documented member closest to it.

Headers of the 400 response to Save a semantic model's benchmark questions
Header Description
x-ms-request-id The id of this request. Quote it to support.
The 400 response to Save a semantic model's benchmark questions, ErrorResponse
Field Type Always present Description
errorCode string Yes Each code keeps its meaning once published. New codes can be added over time, and a code is never reused. One of InvalidRequest, InvalidPreview, PayloadTooLarge, BenchmarkQuestionsMissing, OperationNotSupportedForItem, Unauthorized, TokenExpired, InsufficientScopes, AppOnlyTokenNotSupported, UserNotRegistered, AccountNotActive, AdminRoleRequired, FeatureNotInPlan, FeatureTurnedOff, LicenseNotYours, TrackedItemLimitReached, LicenseTrackedItemLimitReached, FabricConsentRequired, FabricAccessDenied, WorkspaceNotFound, ItemNotFound, AssessmentNotFound, RunNotFound, AbTestNotFound, OperationNotFound, ScheduleNotFound, AiProviderNotFound, AiCredentialNotFound, OperationInProgress, OperationHasNoResult, ResultNotReady, ScheduleConflict, AiProviderConflict, ItemChanged, CapacityPaused, OperationExpired, RegionNotReady, CapacityRequired, CapacityUnpaid, StoreNotConnected, RequestBlocked, AssessmentDailyLimitReached, BenchmarkRunsExhausted, OptimizationApplyDailyLimitReached, FirstLookLimitReached, BenchmarkBuildLimitReached, FabricUnavailable, StoreUnavailable, ThrottleStoreUnavailable, UnknownError.
message string Yes
isRetriable boolean No
moreDetails array of object No
relatedResource object or null No
resourceId string No
resourceType string No
parameters array of object No
name string Yes
value string Yes
requestId string (uuid) Yes

Errors

Every refusal is an ErrorResponse with a stable errorCode. See Errors for what each HTTP status means and every code the API can return.

This is an example of a refusal. Every refusal has the same fields.

Example refusal 
HTTP/1.1 404 Not Found
Content-Type: application/json
x-ms-request-id: cbf21b6d-b661-4b52-88cb-d79712e504fc

{
  "errorCode": "ItemNotFound",
  "message": "No item with this id is visible to the signed-in person in this workspace, or it is not of the type the route takes.",
  "isRetriable": false,
  "moreDetails": [],
  "relatedResource": {
    "resourceId": "d972b3f5-a2fa-47ec-94ff-0403a4d159fa",
    "resourceType": "SemanticModel"
  },
  "parameters": [
    {
      "name": "itemId",
      "value": "d972b3f5-a2fa-47ec-94ff-0403a4d159fa"
    }
  ],
  "requestId": "cbf21b6d-b661-4b52-88cb-d79712e504fc"
}

Example

Request 
curl -i -X PUT \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "document": {}
}' \
  "https://api.bipixie.com/v1/workspaces/$WORKSPACE_ID/semanticModels/$SEMANTIC_MODEL_ID/benchmarks/questions"

An example of what the call answers. The values are fictional.

Response, 202 
HTTP/1.1 202 Accepted
Content-Type: application/json
x-ms-request-id: cbf21b6d-b661-4b52-88cb-d79712e504fc
x-ms-operation-id: cd57b041-e44a-4dda-a4ff-02935ffb8c04
Location: https://api.bipixie.com/v1/operations/cd57b041-e44a-4dda-a4ff-02935ffb8c04
Retry-After: 5

{
  "operationId": "cd57b041-e44a-4dda-a4ff-02935ffb8c04",
  "location": "https://api.bipixie.com/v1/workspaces/e17c1b59-7a2e-491e-acdd-11434854bbb0/semanticModels/f1fdc461-3ca1-4702-ae48-afc3522409fc/benchmarks/questions"
}

Write benchmark questions for a semantic model #

POST /v1/workspaces/{workspaceId}/semanticModels/{semanticModelId}/benchmarks/questions/generate

Writes benchmark questions from the semantic model and how it is used, with the account's AI provider, as the BI Pixie portal's Generate button does, and calculates each question's correct answer by querying the semantic model. It counts toward the benchmark builds the account has each day, and past the limit answers 429 BenchmarkBuildLimitReached.

approval has no default. With required, nothing is saved: the operation's result is the proposal, with saved: false, and its document is what saveBenchmarkQuestions takes to save it once reviewed. With none, the proposal is saved as this semantic model's benchmark questions in the same operation, replacing the ones saved before, and the result is the saved questions.

strategy is optional when a strategy is saved for this semantic model, and is then the saved one; count overrides how many questions it asks for. Querying the semantic model needs permission on the signed-in person's behalf; without it the request answers 403 FabricConsentRequired with the page where it is granted, before anything is counted.

Scope
BIPixie.ReadWrite.All
Plan
Every plan
Counts against
Calls that start work, 30 a minute

Parameters

Parameters for Write benchmark questions for a semantic model
Name In Type Required Description
workspaceId path string (uuid) Yes The id of the Fabric workspace.
semanticModelId path string (uuid) Yes The id of the semantic model in that workspace.

Request body

JSON, GenerateBenchmarkQuestionsRequest. The body is required.

Request body for Write benchmark questions for a semantic model
Field Type Required Description
approval string Yes required returns the proposed questions without saving them; none saves them in the same operation. There is no default. One of required, none.
count integer No How many questions to ask for. Defaults to the strategy's own count.
strategy object No The strategy to write the questions with, in the shape BenchmarkQuestions returns under strategy.payload.strategy. Defaults to the saved strategy; with neither the request answers 400 InvalidRequest.

Responses

202. Writing the questions was started. The body carries the operation's id and address.

Headers of the 202 response to Write benchmark questions for a semantic model
Header Description
x-ms-request-id The id of this request. Quote it to support.
x-ms-operation-id The id of the operation.
Location The address of the operation's state.
Retry-After Seconds to wait before asking for the operation's state.

To follow the work this call started, see Follow a long-running operation.

The 202 response to Write benchmark questions for a semantic model, Accepted
Field Type Always present Description
operationId string (uuid) Yes The operation's id, which the operation's state and result take.
location string (uri) Yes Where to ask about the operation.

400. The request is not well formed, and nothing was started, changed or counted. errorCode is InvalidRequest, or InvalidPreview for a preview that is not available. A body member the operation does not document is refused this way, and the message names it and the documented member closest to it.

Headers of the 400 response to Write benchmark questions for a semantic model
Header Description
x-ms-request-id The id of this request. Quote it to support.
The 400 response to Write benchmark questions for a semantic model, ErrorResponse
Field Type Always present Description
errorCode string Yes Each code keeps its meaning once published. New codes can be added over time, and a code is never reused. One of InvalidRequest, InvalidPreview, PayloadTooLarge, BenchmarkQuestionsMissing, OperationNotSupportedForItem, Unauthorized, TokenExpired, InsufficientScopes, AppOnlyTokenNotSupported, UserNotRegistered, AccountNotActive, AdminRoleRequired, FeatureNotInPlan, FeatureTurnedOff, LicenseNotYours, TrackedItemLimitReached, LicenseTrackedItemLimitReached, FabricConsentRequired, FabricAccessDenied, WorkspaceNotFound, ItemNotFound, AssessmentNotFound, RunNotFound, AbTestNotFound, OperationNotFound, ScheduleNotFound, AiProviderNotFound, AiCredentialNotFound, OperationInProgress, OperationHasNoResult, ResultNotReady, ScheduleConflict, AiProviderConflict, ItemChanged, CapacityPaused, OperationExpired, RegionNotReady, CapacityRequired, CapacityUnpaid, StoreNotConnected, RequestBlocked, AssessmentDailyLimitReached, BenchmarkRunsExhausted, OptimizationApplyDailyLimitReached, FirstLookLimitReached, BenchmarkBuildLimitReached, FabricUnavailable, StoreUnavailable, ThrottleStoreUnavailable, UnknownError.
message string Yes
isRetriable boolean No
moreDetails array of object No
relatedResource object or null No
resourceId string No
resourceType string No
parameters array of object No
name string Yes
value string Yes
requestId string (uuid) Yes

Errors

Every refusal is an ErrorResponse with a stable errorCode. See Errors for what each HTTP status means and every code the API can return.

This is an example of a refusal. Every refusal has the same fields.

Example refusal 
HTTP/1.1 404 Not Found
Content-Type: application/json
x-ms-request-id: cbf21b6d-b661-4b52-88cb-d79712e504fc

{
  "errorCode": "ItemNotFound",
  "message": "No item with this id is visible to the signed-in person in this workspace, or it is not of the type the route takes.",
  "isRetriable": false,
  "moreDetails": [],
  "relatedResource": {
    "resourceId": "d972b3f5-a2fa-47ec-94ff-0403a4d159fa",
    "resourceType": "SemanticModel"
  },
  "parameters": [
    {
      "name": "itemId",
      "value": "d972b3f5-a2fa-47ec-94ff-0403a4d159fa"
    }
  ],
  "requestId": "cbf21b6d-b661-4b52-88cb-d79712e504fc"
}

Example

Request 
curl -i -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "approval": "required"
}' \
  "https://api.bipixie.com/v1/workspaces/$WORKSPACE_ID/semanticModels/$SEMANTIC_MODEL_ID/benchmarks/questions/generate"

An example of what the call answers. The values are fictional.

Response, 202 
HTTP/1.1 202 Accepted
Content-Type: application/json
x-ms-request-id: cbf21b6d-b661-4b52-88cb-d79712e504fc
x-ms-operation-id: cd57b041-e44a-4dda-a4ff-02935ffb8c04
Location: https://api.bipixie.com/v1/operations/cd57b041-e44a-4dda-a4ff-02935ffb8c04
Retry-After: 5

{
  "operationId": "cd57b041-e44a-4dda-a4ff-02935ffb8c04",
  "location": "https://api.bipixie.com/v1/workspaces/e17c1b59-7a2e-491e-acdd-11434854bbb0/semanticModels/f1fdc461-3ca1-4702-ae48-afc3522409fc/benchmarks/questions/generate"
}

Check which saved benchmark questions BI Pixie can score #

POST /v1/workspaces/{workspaceId}/semanticModels/{semanticModelId}/benchmarks/questions/checkCoverage

For each saved benchmark question, whether BI Pixie can calculate its correct answer by querying the semantic model, and so score the answer it gets, with the reason when it cannot. The answer comes one page at a time, with continuationToken and continuationUri while more are left. The questions themselves are never returned; each is named by its questionId. A semantic model with no saved benchmark questions answers 400 BenchmarkQuestionsMissing. Checking saves nothing and counts toward no allowance. Querying the semantic model needs permission on the signed-in person's behalf; without it the request answers 403 FabricConsentRequired.

Scope
BIPixie.Read.All
Plan
Every plan
Counts against
Calls that read, 300 a minute

Parameters

Parameters for Check which saved benchmark questions BI Pixie can score
Name In Type Required Description
workspaceId path string (uuid) Yes The id of the Fabric workspace.
semanticModelId path string (uuid) Yes The id of the semantic model in that workspace.
continuationToken query string No The token the previous page answered with.

Responses

200. One page of verdicts.

Headers of the 200 response to Check which saved benchmark questions BI Pixie can score
Header Description
x-ms-request-id The id of this request. Quote it to support.
The 200 response to Check which saved benchmark questions BI Pixie can score, BenchmarkCoverage
Field Type Always present Description
value array of object Yes One verdict for each saved benchmark question on this page, in saved order.
questionId string Yes The question's id in the saved benchmark questions.
scorable boolean or null Yes Whether BI Pixie can calculate the question's correct answer and so score it. null when there was nothing BI Pixie could check, which a run treats as scorable.
reason string or null No Why the question cannot be scored, when scorable is false.
total integer Yes How many saved benchmark questions can be run, across every page.
continuationToken string No Present while more verdicts are left.
continuationUri string (uri) No The address of the next page, present while more verdicts are left.

Errors

Every refusal is an ErrorResponse with a stable errorCode. See Errors for what each HTTP status means and every code the API can return.

This is an example of a refusal. Every refusal has the same fields.

Example refusal 
HTTP/1.1 404 Not Found
Content-Type: application/json
x-ms-request-id: cbf21b6d-b661-4b52-88cb-d79712e504fc

{
  "errorCode": "ItemNotFound",
  "message": "No item with this id is visible to the signed-in person in this workspace, or it is not of the type the route takes.",
  "isRetriable": false,
  "moreDetails": [],
  "relatedResource": {
    "resourceId": "d972b3f5-a2fa-47ec-94ff-0403a4d159fa",
    "resourceType": "SemanticModel"
  },
  "parameters": [
    {
      "name": "itemId",
      "value": "d972b3f5-a2fa-47ec-94ff-0403a4d159fa"
    }
  ],
  "requestId": "cbf21b6d-b661-4b52-88cb-d79712e504fc"
}

Example

Request 
curl -i -X POST \
  -H "Authorization: Bearer $TOKEN" \
  "https://api.bipixie.com/v1/workspaces/$WORKSPACE_ID/semanticModels/$SEMANTIC_MODEL_ID/benchmarks/questions/checkCoverage"

An example of what the call answers. The values are fictional.

Response, 200 
HTTP/1.1 200 OK
Content-Type: application/json
x-ms-request-id: cbf21b6d-b661-4b52-88cb-d79712e504fc

{
  "value": [
    {
      "questionId": "6dcca226-e149-4087-be6c-b1385f68e4b9",
      "scorable": true,
      "reason": "Not available for this item."
    }
  ],
  "total": 12,
  "continuationToken": "eyJza2lwIjoxMDB9",
  "continuationUri": "https://api.bipixie.com/v1/workspaces/e17c1b59-7a2e-491e-acdd-11434854bbb0/semanticModels/f1fdc461-3ca1-4702-ae48-afc3522409fc/benchmarks/questions/checkCoverage?continuationToken=eyJza2lwIjoxMDB9"
}

Run a semantic model's saved benchmark questions #

POST /v1/workspaces/{workspaceId}/semanticModels/{semanticModelId}/benchmarks/run

Asks the saved benchmark questions of the data agent, Copilot or AI provider named in the request, and scores each answer against the correct answer, as the BI Pixie portal's Run button does. The run counts toward the benchmark runs of the account's plan. The questions are the ones saved for this semantic model; they cannot be sent with the request. The operation's result is the scored run, exactly as getBenchmarkRun answers it, and the run joins the semantic model's benchmark runs. A run can be stopped with cancelOperation.

agent.kind names what answers the questions: DataAgent or BIPixieTestAgent with a dataAgentId, and a workspaceId when the data agent is in another workspace; PowerBICopilot; or GenericAgent, optionally with the providerId of a saved AI provider. repeats overrides how many times each question is asked; without it the saved strategy decides. A semantic model with no saved benchmark questions that can be run answers 400 BenchmarkQuestionsMissing, before anything is counted.

Scope
BIPixie.ReadWrite.All
Plan
Every plan
Counts against
Calls that start work, 30 a minute

Parameters

Parameters for Run a semantic model's saved benchmark questions
Name In Type Required Description
workspaceId path string (uuid) Yes The id of the Fabric workspace.
semanticModelId path string (uuid) Yes The id of the semantic model in that workspace.

Request body

JSON, RunBenchmarkRequest. The body is required.

Request body for Run a semantic model's saved benchmark questions
Field Type Required Description
agent object Yes What answers the benchmark questions.
kind string Yes One of DataAgent, BIPixieTestAgent, PowerBICopilot, GenericAgent.
dataAgentId string (uuid) No The data agent to ask. Required for DataAgent and BIPixieTestAgent.
workspaceId string (uuid) No The data agent's workspace, when it is not the semantic model's. Defaults to the semantic model's workspace.
providerId string No For GenericAgent, the id of a saved AI provider to answer with.
repeats integer No How many times each question is asked. Defaults to the saved strategy's.

Responses

202. The run was started. The body carries the operation's id and address.

Headers of the 202 response to Run a semantic model's saved benchmark questions
Header Description
x-ms-request-id The id of this request. Quote it to support.
x-ms-operation-id The id of the operation.
Location The address of the operation's state.
Retry-After Seconds to wait before asking for the operation's state.

To follow the work this call started, see Follow a long-running operation.

The 202 response to Run a semantic model's saved benchmark questions, Accepted
Field Type Always present Description
operationId string (uuid) Yes The operation's id, which the operation's state and result take.
location string (uri) Yes Where to ask about the operation.

400. The request is not well formed, and nothing was started, changed or counted. errorCode is InvalidRequest, or InvalidPreview for a preview that is not available. A body member the operation does not document is refused this way, and the message names it and the documented member closest to it.

Headers of the 400 response to Run a semantic model's saved benchmark questions
Header Description
x-ms-request-id The id of this request. Quote it to support.
The 400 response to Run a semantic model's saved benchmark questions, ErrorResponse
Field Type Always present Description
errorCode string Yes Each code keeps its meaning once published. New codes can be added over time, and a code is never reused. One of InvalidRequest, InvalidPreview, PayloadTooLarge, BenchmarkQuestionsMissing, OperationNotSupportedForItem, Unauthorized, TokenExpired, InsufficientScopes, AppOnlyTokenNotSupported, UserNotRegistered, AccountNotActive, AdminRoleRequired, FeatureNotInPlan, FeatureTurnedOff, LicenseNotYours, TrackedItemLimitReached, LicenseTrackedItemLimitReached, FabricConsentRequired, FabricAccessDenied, WorkspaceNotFound, ItemNotFound, AssessmentNotFound, RunNotFound, AbTestNotFound, OperationNotFound, ScheduleNotFound, AiProviderNotFound, AiCredentialNotFound, OperationInProgress, OperationHasNoResult, ResultNotReady, ScheduleConflict, AiProviderConflict, ItemChanged, CapacityPaused, OperationExpired, RegionNotReady, CapacityRequired, CapacityUnpaid, StoreNotConnected, RequestBlocked, AssessmentDailyLimitReached, BenchmarkRunsExhausted, OptimizationApplyDailyLimitReached, FirstLookLimitReached, BenchmarkBuildLimitReached, FabricUnavailable, StoreUnavailable, ThrottleStoreUnavailable, UnknownError.
message string Yes
isRetriable boolean No
moreDetails array of object No
relatedResource object or null No
resourceId string No
resourceType string No
parameters array of object No
name string Yes
value string Yes
requestId string (uuid) Yes

Errors

Every refusal is an ErrorResponse with a stable errorCode. See Errors for what each HTTP status means and every code the API can return.

This is an example of a refusal. Every refusal has the same fields.

Example refusal 
HTTP/1.1 404 Not Found
Content-Type: application/json
x-ms-request-id: cbf21b6d-b661-4b52-88cb-d79712e504fc

{
  "errorCode": "ItemNotFound",
  "message": "No item with this id is visible to the signed-in person in this workspace, or it is not of the type the route takes.",
  "isRetriable": false,
  "moreDetails": [],
  "relatedResource": {
    "resourceId": "d972b3f5-a2fa-47ec-94ff-0403a4d159fa",
    "resourceType": "SemanticModel"
  },
  "parameters": [
    {
      "name": "itemId",
      "value": "d972b3f5-a2fa-47ec-94ff-0403a4d159fa"
    }
  ],
  "requestId": "cbf21b6d-b661-4b52-88cb-d79712e504fc"
}

Example

Request 
curl -i -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "agent": {
    "kind": "DataAgent"
  }
}' \
  "https://api.bipixie.com/v1/workspaces/$WORKSPACE_ID/semanticModels/$SEMANTIC_MODEL_ID/benchmarks/run"

An example of what the call answers. The values are fictional.

Response, 202 
HTTP/1.1 202 Accepted
Content-Type: application/json
x-ms-request-id: cbf21b6d-b661-4b52-88cb-d79712e504fc
x-ms-operation-id: cd57b041-e44a-4dda-a4ff-02935ffb8c04
Location: https://api.bipixie.com/v1/operations/cd57b041-e44a-4dda-a4ff-02935ffb8c04
Retry-After: 5

{
  "operationId": "cd57b041-e44a-4dda-a4ff-02935ffb8c04",
  "location": "https://api.bipixie.com/v1/workspaces/e17c1b59-7a2e-491e-acdd-11434854bbb0/semanticModels/f1fdc461-3ca1-4702-ae48-afc3522409fc/benchmarks/run"
}

What's next