Export to OpenTelemetry

A self-hosted or bring-your-own-cloud deployment puts the BI Pixie Event Hub in your own Azure subscription. You can read that hub with an OpenTelemetry collector and land BI Pixie engagement events in the observability platform you already run, alongside the rest of your telemetry.

The transport is yours. BI Pixie publishes the semantics: a stable mapping from the BI Pixie event model to OTLP, and a collector configuration that has been run against real events. Nothing leaves your tenant, and BI Pixie opens no new outbound path.

What you need

• A self-hosted or bring-your-own-cloud deployment, so the Event Hub is in your subscription.
• OpenTelemetry Collector Contrib version 0.160.0 or later. The Contrib distribution is required, because the Azure Event Hub receiver is not in the core build.
• A consumer group on the hub that belongs to your collector, and a Listen-only shared access policy.

Give the collector its own consumer group

Create a new consumer group on the BI Pixie hub, and point the collector at that one. Do not use $Default. BI Pixie's own consumer reads $Default, and two readers on one consumer group take each other's partition leases, which shows up as both of them missing events.

Create a Listen-only shared access policy on the hub for the collector as well, rather than reusing the namespace's RootManageSharedAccessKey. The collector only ever reads.

Set the receiver format to raw

This is the one setting that will cost you an afternoon if it is wrong, so it is worth stating plainly. The Azure Event Hub receiver defaults to format: azure, which expects Azure diagnostic-log JSON. A BI Pixie event is a flat JSON object, so with the default the receiver reads every event and produces nothing from it. There is no error and no warning. The collector looks healthy and your backend stays empty, which is indistinguishable from a hub with no traffic.

Set format: raw, and add the transform below to parse the event.

Pin the timezone

Each event carries its own timestamp for when the interaction happened. That timestamp has no timezone offset in it, and the collector's transform will otherwise read it in whatever timezone the collector host happens to run in. A collector in Chicago will record every event five hours late, and the records will look perfectly well formed.

The configuration below passes "UTC" to the Time function for this reason. Keep it. The result is that Timestamp is when the person interacted with the report, and ObservedTimestamp is when your collector read the record, which is what those two fields are for.

What the configuration does with identity

The configuration below is private by default, matching the posture of the rest of BI Pixie:

The collector configuration

Fill in your connection string and replace the debug exporter with your own backend. The full annotated version, including the attribute reference and the reasoning behind each decision, is in the BI Pixie repository under project/otel/.

receivers:
  azure_event_hub:
    connection: "${env:BIPIXIE_EVENTHUB_CONNECTION}"
    group: otel          # your own consumer group, never $Default
    format: raw          # required: the default drops every event silently

processors:
  transform/bipixie:
    error_mode: propagate
    log_statements:
      - context: log
        statements:
          - set(cache["ev"], ParseJSON(Decode(body, "utf-8")))

          - set(resource.attributes["service.name"], "bipixie")
          - set(resource.attributes["bipixie.license.key"], cache["ev"]["k"])

          # "UTC" is required, see Pin the timezone above
          - set(time, Time(cache["ev"]["ts"], "%Y-%m-%d %H:%M:%S.%f", "UTC"))

          - set(attributes["event.name"], Concat(["bipixie", cache["ev"]["e"]], "."))
          - set(attributes["bipixie.project"], cache["ev"]["a"])
          - set(attributes["bipixie.report"], cache["ev"]["r"])
          - set(attributes["bipixie.page"], cache["ev"]["p"])

          - set(attributes["user.name"], SHA256(cache["ev"]["u"])) where cache["ev"]["u"] != nil and cache["ev"]["u"] != ""

          - set(attributes["bipixie.bookmark"], cache["ev"]["b"]) where cache["ev"]["b"] != ""
          - set(attributes["bipixie.link"], cache["ev"]["ur"]) where cache["ev"]["ur"] != ""
          - set(attributes["bipixie.visual.id"], Split(cache["ev"]["d"], ";")[0]) where cache["ev"]["d"] != ""
          - set(attributes["bipixie.visual.name"], Split(cache["ev"]["d"], ";")[1]) where cache["ev"]["d"] != ""
          - set(attributes["bipixie.visual.type"], cache["ev"]["t"]) where cache["ev"]["t"] != ""
          - set(attributes["bipixie.filter.column"], cache["ev"]["fc"]) where cache["ev"]["fc"] != ""
          - set(attributes["bipixie.filter.distinct_count"], Int(cache["ev"]["fd"])) where cache["ev"]["fd"] != nil
          - set(attributes["bipixie.page.index"], Int(cache["ev"]["pi"])) where cache["ev"]["pi"] != nil
          - set(attributes["bipixie.workspace"], cache["ev"]["w"]) where cache["ev"]["w"] != nil
          - set(attributes["user_agent.original"], cache["ev"]["user_agent"]) where cache["ev"]["user_agent"] != nil

          - set(body, "")

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [azure_event_hub]
      processors: [transform/bipixie]
      exporters: [debug]

Check that it works

Run the collector with the debug exporter first and open a report that BI Pixie tracks. One record should appear carrying event.name, the report and page names, a hashed viewer name, and a Timestamp that matches when you opened the report rather than when the collector read it.

If the collector reports zero log records, the receiver format is still azure. That is the trap described above, and it is not an idle hub.

What this covers

This page covers the Event Hub tap, which is the one that is natively a stream. Reading landed files from Azure Data Lake Storage or from OneLake is possible with the same attribute names and the same privacy defaults, and differs only in the parsing step, since those files are tab separated with no header row. Those recipes are not published yet.

Events are exported as OTLP log records carrying event.name, which is the OpenTelemetry shape for discrete events. They are not spans. A report interaction has no duration and no parent, so a span would require inventing trace identifiers that mean nothing and would render as a misleading waterfall in your backend.