API Reference

This page is generated from the docstrings in the source, so it always matches the installed version. For task-oriented walkthroughs, start with the Core API, Evals & Experiments, and the other guides in the navigation; this reference documents the same public surface symbol by symbol.

Everything below is importable from the top-level bir package unless a section names a submodule (bir.evals, bir.testing, or bir.logging).

Tracing

The decorator and context managers that record traces, spans, generations, tool calls, retrievals, scores, and prompt metadata.

bir.observe(name=None, *, capture_inputs=None, capture_outputs=None, metadata=None)

Decorate a sync or async function and record one trace event for each call.

Generator and async-generator functions are also supported and are traced for their full iteration lifetime rather than only their creation: the wrapper stays lazy (no body runs and nothing is written until the first iteration), the trace stays open across every next/send/throw (or asend/athrow) so child spans and generations created in the body attach to it, and it is finalized when the generator is exhausted (recorded as a successful trace), raises (recorded as a redacted error and re-raised), or is closed/cancelled early (recorded as a successful trace whose metadata.generator.outcome is "closed"). Yielded values are never buffered; with output capture enabled only a bounded yielded-item count is recorded under metadata.generator.items.

metadata is an optional static mapping recorded on the produced trace ROOT event, redacted with the same rules as captured input/output. It is the decorator-side counterpart to trace(metadata=...) for tagging an entry point (route, tenant, feature flag) without rewriting it as a manual with trace(...) block. It is attached only when the decorated call opens a new trace root; a nested @observe() call records a span and never carries this trace-level metadata. For observed generators it composes with the recorded metadata.generator.* outcome (the generator keys win on conflict).

bir.trace(name, *, metadata=None)

Create a trace root with a context manager.

bir.span(name)

Create a nested span inside the current trace.

bir.generation(name, *, model=None, input=None, metadata=None, prompt=None, capture_input=None, capture_output=None)

Create a generation event for an LLM call inside the current trace.

bir.tool_call(name, *, input=None, metadata=None, capture_input=None, capture_output=None)

Create a tool call event inside the current trace.

bir.retrieval(name, *, query, metadata=None, capture_input=None, capture_output=None)

Create a retrieval tool call using Bir's documented RAG event shape.

bir.score(name, value, *, metadata=None)

Attach a score event to the current trace.

Optional metadata (for example an evaluator's reasoning or threshold) is redacted with the same rules as captured input/output and stored on the score event so it can be inspected in the dashboard later.

bir.prompt(name, *, version=None, template=None, variables=None, rendered=None, metadata=None, capture_template=False, capture_variables=False, capture_rendered=False)

Describe the prompt version used by a generation.

Configuration

bir.configure(*, trace_path=None, capture_inputs=None, capture_outputs=None, service_name=None, environment=None, source=None, enabled=None, sample_rate=None, sample_rules=None, max_bytes=None, backup_count=None, additional_secret_keys=None, additional_redaction_patterns=None, model_prices=None, max_value_length=None, max_collection_items=None)

Configure local SDK behavior.

service_name and environment are recorded on trace root events under metadata.service so traces can be filtered by deployment later.

source tags every trace root with metadata.source so traces can be filtered by where they originated. It is the SDK-side counterpart to the source the Bir server/dashboard already filter on (the product's Playground records "playground"); the server matches it by exact, trimmed value, so pick a stable label such as "checkout-api". Like the trace metadata argument, an explicit source in a trace(metadata=...) block still wins over this configured default. Defaults to None (no source recorded).

enabled is the master on/off switch for all recording. The default True keeps every primitive recording as configured. Set it to False for an explicit, intent-revealing kill switch (a feature flag, an incident toggle, a test): @observe, trace/span/generation/ tool_call/retrieval, and score all still run the wrapped code and still propagate exceptions, but nothing is ever written, making Bir a true no-op without touching call sites. It is enforced through the same path as sampling, so a trace already in flight when recording is disabled stops writing immediately, and configure(enabled=True) restores full recording for traces started afterward. get_current_trace_id() / get_current_span_id() still return the live in-process ids inside a trace while disabled (matching a sampled-out trace), so log correlation keeps working even though nothing is persisted.

sample_rate is the probability (0.0 to 1.0) that a trace is recorded. It is decided once per trace root; when a trace is sampled out the function still runs and still raises, but the trace and every event under it write nothing. The default 1.0 records every trace. sample_rate only applies while enabled is True; enabled=False turns everything off regardless of the rate.

sample_rules is an opt-in mapping of exact trace root name to a sampling rate for that root. A matching rule overrides the global sample_rate; an unmatched root uses the global rate. Rules are validated once here and stored immutably, and the decision is still made once per trace root and inherited by every descendant event. Passing sample_rules replaces the prior rule table (an empty mapping clears it); omitting it leaves the current rules unchanged.

max_bytes enables opt-in size-based rotation of the local trace file. It defaults to None (unlimited), which keeps the historical single-file behavior. When set to a non-negative integer, the active file is rotated before any write that would push it past the cap: traces.jsonl becomes traces.jsonl.1, the previous .1 becomes .2, and so on, keeping at most backup_count rotated files and dropping the oldest. Rotation always happens on whole-line boundaries, so every file stays valid JSONL and a JSON object is never split across files (a single line larger than max_bytes is still written whole). backup_count defaults to 3; 0 keeps no rotated files and simply drops the active file when it fills.

Note that a single logical trace may be split across rotated files when rotation happens mid-trace, so reading with include_rotated=True can surface incomplete traces near a rotation boundary.

additional_secret_keys and additional_redaction_patterns add to the built-in redaction rules; they can only ever widen what is redacted and can never disable, replace, reorder, or change the [redacted] marker of the built-in rules. additional_secret_keys is an iterable of extra mapping-key names: a captured mapping key is redacted when it matches one of them exactly and case-insensitively, treating - and _ as equivalent (this is whole-name exact matching, unlike the built-in substring rules). additional_redaction_patterns is an iterable of regex strings and/or already-compiled re.Pattern objects; every match of each pattern in any captured string, repr fallback, prompt text, eval metadata, or error message is replaced with [redacted], running after all built-in text patterns. Both are validated and compiled once here, so an empty key, empty pattern, invalid regex, non-string entry, bytes pattern, or an over-large list raises ValueError/TypeError immediately. Passing either argument replaces the previously configured additional rules of that kind (passing an empty iterable clears them); omitting it leaves the current additional rules unchanged. The built-in rules always remain in force either way.

model_prices is an opt-in, local-only price table that auto-fills a generation's cost from its token usage. It is a mapping of model name to a rates mapping holding a non-negative, finite input and/or output per-token rate plus an optional currency (default "USD"). Bir bundles no prices, so the rates — and keeping them current — are yours to supply. When a generation has usage and a model matching a configured entry but no explicitly set cost, its input_cost/output_cost/total_cost are derived from the matching rates at the configured currency exactly as a manual set_cost(...) would record them (input rate times input tokens, output rate times output tokens, total summed when both sides are priced). An explicit set_cost(...) always wins and is never overwritten, and a generation whose usage lacks the needed token split is left without a derived cost. The table is validated once here, so a non-mapping table, a non-string or empty model name, a non-mapping or empty rate entry, an unknown rate key, a boolean, negative, or non-finite rate, an invalid currency, or an over-large table raises ValueError/TypeError immediately. Passing model_prices replaces the previously configured table (an empty mapping clears it); omitting it leaves the current table unchanged. With no table configured, generation cost behavior is unchanged.

max_value_length and max_collection_items are opt-in capture-size limits that bound a single captured value so one huge payload (a base64 image, a megabyte of model output) cannot bloat the local store. Both default to None (unlimited), which keeps captured output byte-for-byte unchanged. When max_value_length is a non-negative integer, a captured string longer than it is truncated to that many characters with a visible …[truncated] marker appended; truncation always runs after redaction, so a secret is replaced before any cut and can never be split in a way that defeats the redactor. When max_collection_items is a non-negative integer, a captured list, tuple, set, or mapping larger than it keeps only the first that-many items and records a single …[truncated] marker for the remainder, leaving the output valid JSON. Both apply uniformly to every capture path (inputs, outputs, metadata, repr fallbacks, and dataset and experiment capture) and compose with the existing capture-depth cap. They only bound captured values, never event names, models, ids, or the schema. A non-integer, boolean, or negative limit raises TypeError/ValueError here.

Any field left unset falls back to the value supplied by the matching environment variable (BIR_TRACE_PATH, BIR_CAPTURE_INPUTS, BIR_CAPTURE_OUTPUTS, BIR_DISABLED, BIR_SAMPLE_RATE, BIR_SERVICE_NAME, BIR_ENVIRONMENT, BIR_SOURCE, BIR_MAX_VALUE_LENGTH, BIR_MAX_COLLECTION_ITEMS), which is read once at import time, and otherwise to the hardcoded default. A truthy BIR_DISABLED sets enabled=False (it is the inverse of the enabled field, so the common "turn it off in production" case is a single boolean variable). Explicit arguments to this function take precedence over the environment, so configure(enabled=True) re-enables recording even when BIR_DISABLED is set.

Loading and sending events

Read locally recorded traces back into memory and post them to a Bir server, and read the active trace and span ids for correlating your own logs and metrics.

bir.load_events(path=None, *, include_rotated=False)

Load local JSONL trace events.

By default only the active trace file is read. Pass include_rotated=True to also read size-rotated siblings (traces.jsonl.1 ..) created by configure(max_bytes=...). Rotated files are read oldest-first so the returned events stay in the same chronological order they were written, matching how a single never-rotated file would read. Because rotation can occur mid-trace, a single logical trace may be split across files.

bir.load_traces(path=None, *, include_rotated=False)

Load local traces grouped by trace_id.

include_rotated is forwarded to :func:load_events; see its note about traces possibly being split across rotated files.

bir.send_events(server_url='http://127.0.0.1:8000', *, path=None, timeout=10.0, retries=2, backoff=0.5, mark_sent=False, include_rotated=False)

Send local JSONL trace events to a Bir ingestion server.

Transient failures are retried with exponential backoff: a network error, timeout, or HTTP 5xx is retried up to retries times (default 2), sleeping backoff * 2**attempt seconds between tries (backoff defaults to 0.5). A 4xx response is a permanent rejection and is raised immediately without retry, matching the un-retried behavior. A healthy send still makes a single attempt, so the default behavior is unchanged.

mark_sent is opt-in bookkeeping for cheap re-sends. When True, the IDs the server accepts are recorded in a sidecar file next to the trace file (<trace_path>.sent) and skipped on later sends, so attempted reflects only events not yet recorded as sent. The sidecar is SDK-local: it never modifies the trace JSONL or the event schema, and a missing or corrupt sidecar is treated as empty so it can never block a send. With the default mark_sent=False nothing is recorded and re-sending the whole file stays safe because the server is idempotent on event IDs.

include_rotated is opt-in upload of size-rotated trace files. The default False uploads only the active trace file, matching the historical behavior. When True, retained rotated siblings (traces.jsonl.1 ..) created by configure(max_bytes=...) are uploaded oldest-first followed by the active file, so rotation can no longer strand unsent events. Events are deduplicated by ID when a rotated file overlaps the active file, and the mark_sent sidecar still anchors to the active trace path so recorded IDs are skipped across the whole selected file set.

bir.get_current_trace_id()

Return the active trace's id, or None outside any trace.

The value is the same id written to the trace_id field of every event recorded while this trace is active, so an application log stamped with it can later be correlated with the trace. Read from a task-local context, so each asyncio task and thread observes its own active trace and never another's. While recording is disabled (configure(enabled=False)) or the trace was sampled out, this still returns the live id inside the trace so log correlation keeps working even though nothing is persisted.

bir.get_current_span_id()

Return the innermost active span's id, or None outside any trace.

Inside a nested span()/generation()/tool_call() this is the innermost open node's id; directly inside a trace with no open child it is the trace root's id. The value is the same id written to the parent_id field of any child event created at this point, so it names what an event recorded now would attach to. Read from a task-local context, so each asyncio task and thread observes its own ids and never another's. Like :func:get_current_trace_id, this still returns the live id while recording is disabled or the trace was sampled out, so log correlation is unaffected.

Data types

The frozen dataclasses returned by the loaders and helpers above.

bir.TraceEvent dataclass

A single trace, span, generation, tool call, or score loaded from storage.

duration_ms property

Return the event duration in milliseconds.

bir.LoadedTrace dataclass

A trace root event with all events that share its trace ID.

duration_ms property

Return the root trace duration in milliseconds.

bir.SendEventsResult dataclass

Result returned after sending local events to a Bir server.

skipped property

Return events the server did not newly accept, usually duplicates.

bir.PromptRecord dataclass

Prompt metadata attached to a generation event.

render()

Render the prompt template with variables when no explicit rendered value exists.

to_metadata()

Return the redacted metadata representation stored on a generation event.

Evals and experiments

Deterministic evaluators, datasets, and experiment runners from bir.evals.

bir.evals

Deterministic evaluation, dataset, and experiment helpers for Bir.

Dataset dataclass

A collection of uniquely identified examples for experiment runs.

__iter__()

Iterate over dataset examples.

__len__()

Return the number of examples in the dataset.

from_jsonl(path) classmethod

Load dataset examples from a JSONL file.

to_jsonl(path, *, redact=True)

Write dataset examples to a JSONL file.

Redaction is enabled by default so exported datasets use the same safe capture behavior as trace and experiment artifacts. Pass redact=False only when you intentionally want to preserve raw example payloads.

DatasetExample dataclass

One input, expected output, and metadata row in an evaluation dataset.

to_dict(*, redact=True)

Return a JSON-serializable dataset row, redacted by default.

DeterministicEvaluator dataclass

Callable evaluator wrapper used by experiment runs.

evaluate(output, *, expected=None, context=None)

Evaluate a task output and return an EvalResult.

EvalResult dataclass

A numeric evaluator score with optional JSON-safe metadata.

to_dict()

Return a JSON-serializable representation of the score.

EvaluationContext dataclass

Runtime context passed to evaluators that need experiment metadata.

ExperimentDiff dataclass

Aggregate-score differences between two experiment runs.

tolerance is the global tolerance, while effective_tolerances records the tolerance actually applied to each shared evaluator after per-evaluator overrides. missing_score is the configured policy for evaluators present only in the baseline, and regression_reasons maps every evaluator that fails the gate to a machine-readable reason. example_deltas is the opt-in per-example detail: for each shared evaluator it maps an example_id present in both runs to the candidate-minus-baseline delta for that example, and is empty unless :func:compare_experiments was called with per_example=True. All mappings are ordered by key so the diff serializes deterministically.

has_regressions property

Return whether the configured policy reports any regression.

A shared evaluator that dropped beyond its effective tolerance always counts. When the missing-score policy is regress, evaluators present only in the baseline also count: a removed evaluator drops coverage even though no aggregate delta can be computed.

to_dict()

Return a deterministic JSON-serializable representation of the diff.

example_deltas is included only when populated (it is empty unless per-example detail was requested), so the default aggregate-only output is byte-for-byte unchanged from before the field existed.

ExperimentExampleResult dataclass

The task output and evaluator scores for one dataset example.

duration_ms property

Return the example runtime in milliseconds.

to_dict()

Return a JSON-serializable experiment result row.

ExperimentResult dataclass

All example results and aggregate scores for one experiment run.

aggregate_scores property

Return the mean score for each evaluator name.

to_dict()

Return a JSON-serializable experiment payload.

ExperimentSummary dataclass

Compact metadata persisted next to an experiment result file.

to_dict()

Return a JSON-serializable experiment summary.

SendExperimentResult dataclass

Result returned after sending an experiment to a Bir server.

answer_contains_citation(*, pattern=None, name='answer_contains_citation')

Create an evaluator that checks whether an answer includes a citation marker.

This is a deterministic format check, not proof of grounding or relevance: it only confirms that a citation marker is present in the answer text, not that the citation is correct or that the cited source supports the answer.

The task output may be a plain answer string or a structured RAG mapping with an answer string:

"Paris is the capital of France [1]." {"answer": "Paris is the capital of France [1].", "contexts": [...]}

By default any bracketed marker such as [1] or [doc-1] counts as a citation. Pass pattern to override the citation regex, for example r"\(\d+\)" to require parenthetical markers like (1).

answer_context_overlap(min_ratio, *, name='answer_context_overlap')

Create an evaluator that checks how much of an answer is supported by retrieved context.

The overlap ratio is the fraction of answer word tokens that also appear in the retrieved context texts. It is a deterministic heuristic for spotting unsupported answers, not proof of faithfulness: paraphrased but faithful answers can score low, and unfaithful answers that reuse context words can score high.

The task output must be a mapping with an answer string and a contexts list of retrieved text strings:

{"answer": "...", "contexts": ["doc text", "other doc text"]}

compare_experiments(baseline, candidate, *, tolerance=0.0, score_tolerances=None, missing_score=_MISSING_SCORE_IGNORE, per_example=False)

Compare shared aggregate evaluator scores from two experiment runs.

A shared evaluator regresses when candidate - baseline is strictly less than -tolerance. score_tolerances maps an evaluator name to a non-negative, finite tolerance that overrides the global tolerance for that evaluator only; the strict math.isclose boundary is preserved per evaluator. Every override name must be a shared evaluator present in both runs, so a typo or a tolerance for a non-comparable evaluator raises a clear error instead of being silently ignored.

missing_score selects how evaluators present only in the baseline are treated. "ignore" (the default) reports them without failing the gate, matching the historical behavior. "regress" treats each baseline-only evaluator as a regression, because a removed evaluator silently drops coverage even though no aggregate delta can be computed. Evaluators found only in the candidate are always reported but never counted as regressions.

per_example is opt-in reporting detail and never changes the aggregate comparison or the gate decision. When True, the returned diff's example_deltas records, for each shared evaluator, the candidate-minus-baseline score delta of every example_id that both runs scored with that evaluator; examples present in only one run (or not scored by the evaluator, such as an errored example) are skipped. When False (the default) example_deltas is empty and the diff is identical to before.

contains(expected=_USE_EXAMPLE_EXPECTED, *, case_sensitive=True, name='contains')

Create an evaluator that scores 1.0 when output text contains a string.

cost_under(max_cost, *, field='total_cost', name='cost_under')

Create an evaluator that scores 1.0 when a reported cost is under a threshold.

custom_evaluator(name, evaluate, *, uses_context=False)

Wrap a user-provided callable as a deterministic evaluator.

exact_match(expected=_USE_EXAMPLE_EXPECTED, *, name='exact_match')

Create an evaluator that scores 1.0 when output equals the expected value.

field_contains(path, expected=_USE_EXAMPLE_EXPECTED, *, case_sensitive=True, name='field_contains')

Create an evaluator that checks whether a nested string field contains text.

field_equals(path, expected=_USE_EXAMPLE_EXPECTED, *, name='field_equals')

Create an evaluator that compares a nested output field to an expected value.

json_valid(*, name='json_valid')

Create an evaluator that scores 1.0 for JSON-compatible output.

latency_under(max_ms, *, name='latency_under')

Create an evaluator that scores 1.0 when task latency is under a threshold.

list_experiments(directory=Path('.bir') / 'experiments')

List experiment summaries in newest-first order.

load_experiment(path)

Load an experiment result JSONL file.

load_experiment_summary(path)

Load an experiment summary JSON file.

numeric_between(min_value=None, max_value=None, *, field=None, name='numeric_between')

Create an evaluator that checks a numeric output or field against bounds.

regex_match(pattern, *, flags=0, name='regex_match')

Create an evaluator that scores 1.0 when output text matches a regex.

render_experiment_report(result, *, format='html')

Render one experiment to a self-contained report string.

The report bundles the run summary, the per-evaluator aggregate means, and a per-example table of statuses and scores into a single string with no external assets, so a local-first user can share or archive an experiment result without standing up the server or dashboard. format selects "html" (the default, a complete standalone HTML document with inline styles) or "markdown".

Output is deterministic for a given experiment: evaluators are ordered by name and examples follow their persisted dataset order, so re-rendering the same result yields a byte-identical string. Only already-persisted (already redacted) values are rendered, and every experiment-derived string is escaped for the chosen format, so example data cannot inject markup. Built with the standard library only.

retrieved_context_contains(expected, *, case_sensitive=True, name='retrieved_context_contains')

Create an evaluator that checks whether retrieved context contains a string.

This is a deterministic retrieval check, not proof of relevance or faithfulness: it only confirms that expected appears verbatim in one of the retrieved context strings, not that the answer relied on it.

The task output must be a mapping with a contexts list of retrieved text strings:

{"answer": "...", "contexts": ["doc text", "other doc text"]}

run_experiment(name, *, dataset, task, evaluators, path=None, raise_on_error=True, record_traces=False, max_workers=1, timeout=None)

Run a task over a dataset and persist per-example evaluator results.

When max_workers is greater than 1, examples run concurrently inside a :class:concurrent.futures.ThreadPoolExecutor with up to max_workers threads. Results, JSONL rows, and summary aggregates are always written in dataset order regardless of completion order. Every other behavior matches the sequential path: raise_on_error persists through the first failing example in dataset order and re-raises that exception; record_traces isolation is preserved because each worker thread inherits its own copy of the context-var state, so trace trees never bleed across examples.

timeout is an optional per-example limit in seconds (a positive, finite number; default None means unlimited). When set, an example whose task runs longer than timeout is recorded as an "error"-status result with a "task timed out after Ns" message — the same shape as any other failed example, so raise_on_error is honored — and the run continues with the remaining examples. The limit applies to each example's own runtime, measured from the moment its task starts executing: with max_workers greater than 1, an example queued behind others waiting for a free worker thread accrues no timeout while it waits (the serial max_workers=1 path uses a dedicated single-worker executor per example, so its task starts immediately). Python cannot force a thread to stop, so a timed-out task keeps running in the background until it returns on its own; its result is discarded, and it occupies its worker slot until then, which can delay — but never time out — examples still waiting in the queue. timeout=None is byte-for-byte identical to the previous behavior.

run_experiment_async(name, *, dataset, task, evaluators, path=None, raise_on_error=True, record_traces=False, max_concurrency=1, timeout=None) async

Run a task over a dataset with bounded concurrency and persist results.

This is the asynchronous counterpart to :func:run_experiment. The task may be a coroutine function, a plain sync callable, or a sync callable that returns an awaitable; the return value is awaited only when :func:inspect.isawaitable reports it is awaitable, so the callable itself is never inspected. Up to max_concurrency examples run concurrently, but the returned results, the persisted JSONL rows, and the summary aggregates always follow dataset order regardless of completion order.

Every other behavior matches :func:run_experiment: evaluator execution, task input binding, redaction, raise_on_error semantics, and the persisted JSONL/summary schema are identical. Each example runs in its own asyncio task, whose copied context isolates the trace contextvars, so record_traces=True produces a separate trace tree per example even while they run concurrently.

Like :func:run_experiment, raise_on_error=True persists results through the first failing example in dataset order, writes the matching error summary, and re-raises that example's exception. Because examples run concurrently, later examples may already have executed when the failure is detected; their results are simply not persisted. If the surrounding coroutine is cancelled, the in-flight example tasks are cancelled and awaited and CancelledError propagates without writing a misleading success summary.

timeout is an optional per-example limit in seconds (a positive, finite number; default None means unlimited). When set, each example coroutine is wrapped in :func:asyncio.wait_for; an example that runs longer than timeout has its task cancelled and awaited (so no pending-task warning leaks) and is recorded as an "error"-status result with a "task timed out after Ns" message — the same shape as any other failed example, so raise_on_error is honored and dataset order is preserved. With record_traces=True, a timed-out example's trace root is written with error status before the cancellation unwinds — its already-recorded child events stay attached and loadable — and the error result's trace_id links that trace. timeout=None is byte-for-byte identical to the previous behavior.

send_experiment(path, server_url='http://127.0.0.1:8000', *, timeout=10.0, retries=2, backoff=0.5)

Send a persisted experiment and its summary to a Bir server.

Transient failures are retried with exponential backoff, matching send_events: a network error, timeout, or HTTP 5xx is retried up to retries times (default 2), sleeping backoff * 2**attempt seconds between tries (backoff defaults to 0.5). A 4xx response is a permanent rejection raised immediately without retry, as are a missing experiment or summary file and an invalid success response body. A healthy send still makes a single attempt with no sleep, so the default behavior is unchanged.

similarity_above(threshold, expected=_USE_EXAMPLE_EXPECTED, *, case_sensitive=True, name='similarity_above')

Create an evaluator that scores 1.0 when output text is similar enough to expected.

Similarity is the normalized :class:difflib.SequenceMatcher ratio between the output text and the expected text, a deterministic, dependency-free fuzzy check that sits between exact equality (:func:exact_match) and substring presence (:func:contains). It tolerates typos, reordering, and minor wording differences without an embedding model or any new dependency. The score is 1.0 when the achieved ratio is at or above threshold (the boundary is inclusive) and 0.0 otherwise. Pass case_sensitive=False to lowercase both sides before comparing. The achieved ratio and threshold are recorded in EvalResult.metadata so failures are inspectable.

Testing helpers

Assert on your own instrumentation from bir.testing.

bir.testing

Test helpers for asserting on your Bir instrumentation.

The public :func:capture_traces context manager redirects trace writes to a private temporary file for the duration of a with block and yields a :class:CapturedTraces handle that reads the captured events and traces back in memory through the SDK's own public loaders. It lets application authors assert on the traces their instrumentation produces without pointing configure(trace_path=...) at a scratch file by hand or touching their real .bir/ directory.

Like :func:bir.configure, this mutates process-global SDK configuration for the duration of the block: on entry it swaps the active trace_path to the temporary file, and on exit it restores the previous configuration in full (including a user-set trace_path), even when the body raises. It only redirects where events are written — capture opt-in, sampling, and redaction are left exactly as configured — so a captured event is identical to what a real .bir/traces.jsonl write would contain. Because it mutates global state it is not safe to run concurrently from multiple threads, the same caveat that applies to configure.

CapturedTraces

Read-back handle for the events recorded inside a capture_traces block.

While the block is open, :meth:events and :meth:traces read live from the temporary trace file through the public :func:bir.load_events / :func:bir.load_traces loaders, so a test body can assert on progress as it goes. When the block exits, the final state is snapshotted in memory just before the temporary file is removed, so the same methods keep returning the captured data after the with block has closed.

trace_path property

Temporary file that events are captured to for the block's duration.

events()

Return the events captured in the block, in load order.

Inside the block this re-reads the temporary file on each call; after the block it returns the snapshot taken just before cleanup. Reads include any size-rotated siblings, so capture stays complete even if the inherited configuration rotates the file mid-block.

traces()

Return the captured events grouped into traces by trace_id.

Inside the block this re-reads the temporary file on each call; after the block it returns the snapshot taken just before cleanup.

capture_traces()

Redirect trace writes to a temporary file and yield a read-back handle.

Use this in tests to assert on the traces your own instrumentation produces without writing to (or reading from) your real .bir/ directory::

from bir.testing import capture_traces

with capture_traces() as captured:
    answer_question("hello")

events = captured.events()
assert [event.type for event in events] == ["trace", "generation"]

Only the active trace_path is swapped; every other configured setting (capture opt-in, sampling, redaction, the price table, and service metadata) is kept, so events are captured exactly as they would be in production — just into an isolated temporary file. The previous configuration, including a user-set trace_path, is restored when the block exits, even if the body raises, and the temporary file and directory are removed.

Like :func:bir.configure, this mutates process-global configuration for the duration of the block and is therefore not safe to run concurrently across threads. Nested capture_traces() blocks are fine: each restores the configuration that was active when it was entered.

Logging integration

Stamp standard-library log records with the active trace and span ids using bir.logging.

bir.logging

Stamp standard-library log records with the active Bir trace and span ids.

Correlating your application logs with Bir traces is the primary documented use of :func:bir.get_current_trace_id / :func:bir.get_current_span_id. Doing it by hand means threading extra={"trace_id": ...} through every logging call. This module removes that plumbing: attach :class:BirTraceIdFilter once and every :class:logging.LogRecord that flows through the logger or handler gains two attributes, ready for any formatter to render::

import logging

from bir.logging import install_trace_id_filter

install_trace_id_filter()
logging.basicConfig(
    format="%(asctime)s %(levelname)s [trace=%(bir_trace_id)s span=%(bir_span_id)s] %(message)s"
)

The stamped attributes mirror the accessors exactly: inside a trace they equal :func:bir.get_current_trace_id / :func:bir.get_current_span_id, and outside any trace they are None. The ids are read from the same task-local context as the accessors, so each asyncio task and thread sees its own ids and never another's.

Like the accessors, this is read-only: the filter only reads the active ids onto the record. There is no setter and no cross-process propagation, consistent with the accessors' design. The filter never raises, so it is safe to leave attached on every log call, inside or outside a trace.

BirTraceIdFilter

Bases: Filter

A :class:logging.Filter that stamps the active Bir ids onto every record.

On each record it sets two attributes from the current task-local context:

  • record.bir_trace_id — the active trace's id (see :func:bir.get_current_trace_id), or None outside any trace.
  • record.bir_span_id — the innermost open span/generation/tool-call id, or the trace root when none is open (see :func:bir.get_current_span_id), or None outside any trace.

The names match :data:TRACE_ID_FIELD and :data:SPAN_ID_FIELD and are safe to render with %(bir_trace_id)s / %(bir_span_id)s. The values equal the trace_id / parent_id later written to the JSONL, so a stamped log lines up with the trace. :meth:filter always returns True (it is used purely to annotate, never to drop records) and never raises.

Despite the :class:logging.Filter base, this does not filter by logger name — pass it to addFilter on a logger or a handler. Attaching it to a logger stamps records created by that logger; attaching it to a handler stamps every record the handler emits (including those propagated from child loggers).

install_trace_id_filter(target=None)

Attach a :class:BirTraceIdFilter to a logger or handler and return it.

With no argument the filter is added to the root logger, which is enough for the common case where application loggers propagate to root. Pass a specific :class:logging.Logger or :class:logging.Handler to scope it; attaching to a handler is the most reliable way to stamp every record that handler emits, including ones propagated from child loggers.

Returns the created filter so it can later be removed with target.removeFilter(returned_filter). Calling this more than once on the same target attaches independent filters; each stamps the same attributes, so the duplication is harmless but you can avoid it by reusing the returned instance.