API Reference
This page is generated from the docstrings in the source, so it always matches the installed version. For task-oriented walkthroughs, start with the Core API, Evals & Experiments, and the other guides in the navigation; this reference documents the same public surface symbol by symbol.
Everything below is importable from the top-level bir package unless a section
names a submodule (bir.evals, bir.testing, or bir.logging).
Tracing
The decorator and context managers that record traces, spans, generations, tool calls, retrievals, scores, and prompt metadata.
bir.observe(name=None, *, capture_inputs=None, capture_outputs=None, metadata=None)
Decorate a sync or async function and record one trace event for each call.
Generator and async-generator functions are also supported and are traced for
their full iteration lifetime rather than only their creation: the wrapper
stays lazy (no body runs and nothing is written until the first iteration),
the trace stays open across every next/send/throw (or
asend/athrow) so child spans and generations created in the body
attach to it, and it is finalized when the generator is exhausted (recorded as
a successful trace), raises (recorded as a redacted error and re-raised), or is
closed/cancelled early (recorded as a successful trace whose
metadata.generator.outcome is "closed"). Yielded values are never
buffered; with output capture enabled only a bounded yielded-item count is
recorded under metadata.generator.items.
metadata is an optional static mapping recorded on the produced trace ROOT
event, redacted with the same rules as captured input/output. It is the
decorator-side counterpart to trace(metadata=...) for tagging an entry
point (route, tenant, feature flag) without rewriting it as a manual
with trace(...) block. It is attached only when the decorated call opens a
new trace root; a nested @observe() call records a span and never carries
this trace-level metadata. For observed generators it composes with the
recorded metadata.generator.* outcome (the generator keys win on conflict).
bir.trace(name, *, metadata=None)
Create a trace root with a context manager.
bir.span(name)
Create a nested span inside the current trace.
bir.generation(name, *, model=None, input=None, metadata=None, prompt=None, capture_input=None, capture_output=None)
Create a generation event for an LLM call inside the current trace.
bir.tool_call(name, *, input=None, metadata=None, capture_input=None, capture_output=None)
Create a tool call event inside the current trace.
bir.retrieval(name, *, query, metadata=None, capture_input=None, capture_output=None)
Create a retrieval tool call using Bir's documented RAG event shape.
bir.score(name, value, *, metadata=None)
Attach a score event to the current trace.
Optional metadata (for example an evaluator's reasoning or threshold) is
redacted with the same rules as captured input/output and stored on the
score event so it can be inspected in the dashboard later.
bir.prompt(name, *, version=None, template=None, variables=None, rendered=None, metadata=None, capture_template=False, capture_variables=False, capture_rendered=False)
Describe the prompt version used by a generation.
Configuration
bir.configure(*, trace_path=None, capture_inputs=None, capture_outputs=None, service_name=None, environment=None, source=None, enabled=None, sample_rate=None, sample_rules=None, max_bytes=None, backup_count=None, additional_secret_keys=None, additional_redaction_patterns=None, model_prices=None, max_value_length=None, max_collection_items=None)
Configure local SDK behavior.
service_name and environment are recorded on trace root events
under metadata.service so traces can be filtered by deployment later.
source tags every trace root with metadata.source so traces can be
filtered by where they originated. It is the SDK-side counterpart to the
source the Bir server/dashboard already filter on (the product's
Playground records "playground"); the server matches it by exact,
trimmed value, so pick a stable label such as "checkout-api". Like the
trace metadata argument, an explicit source in a trace(metadata=...)
block still wins over this configured default. Defaults to None (no source
recorded).
enabled is the master on/off switch for all recording. The default
True keeps every primitive recording as configured. Set it to False
for an explicit, intent-revealing kill switch (a feature flag, an incident
toggle, a test): @observe, trace/span/generation/
tool_call/retrieval, and score all still run the wrapped code and
still propagate exceptions, but nothing is ever written, making Bir a true
no-op without touching call sites. It is enforced through the same path as
sampling, so a trace already in flight when recording is disabled stops
writing immediately, and configure(enabled=True) restores full recording
for traces started afterward. get_current_trace_id() /
get_current_span_id() still return the live in-process ids inside a trace
while disabled (matching a sampled-out trace), so log correlation keeps
working even though nothing is persisted.
sample_rate is the probability (0.0 to 1.0) that a trace is
recorded. It is decided once per trace root; when a trace is sampled out the
function still runs and still raises, but the trace and every event under it
write nothing. The default 1.0 records every trace. sample_rate only
applies while enabled is True; enabled=False turns everything off
regardless of the rate.
sample_rules is an opt-in mapping of exact trace root name to a sampling
rate for that root. A matching rule overrides the global sample_rate; an
unmatched root uses the global rate. Rules are validated once here and stored
immutably, and the decision is still made once per trace root and inherited by
every descendant event. Passing sample_rules replaces the prior rule table
(an empty mapping clears it); omitting it leaves the current rules unchanged.
max_bytes enables opt-in size-based rotation of the local trace file. It
defaults to None (unlimited), which keeps the historical single-file
behavior. When set to a non-negative integer, the active file is rotated
before any write that would push it past the cap: traces.jsonl becomes
traces.jsonl.1, the previous .1 becomes .2, and so on, keeping at
most backup_count rotated files and dropping the oldest. Rotation always
happens on whole-line boundaries, so every file stays valid JSONL and a JSON
object is never split across files (a single line larger than max_bytes
is still written whole). backup_count defaults to 3; 0 keeps no
rotated files and simply drops the active file when it fills.
Note that a single logical trace may be split across rotated files when
rotation happens mid-trace, so reading with include_rotated=True can
surface incomplete traces near a rotation boundary.
additional_secret_keys and additional_redaction_patterns add to the
built-in redaction rules; they can only ever widen what is redacted and can
never disable, replace, reorder, or change the [redacted] marker of the
built-in rules. additional_secret_keys is an iterable of extra
mapping-key names: a captured mapping key is redacted when it matches one of
them exactly and case-insensitively, treating - and _ as equivalent
(this is whole-name exact matching, unlike the built-in substring rules).
additional_redaction_patterns is an iterable of regex strings and/or
already-compiled re.Pattern objects; every match of each pattern in any
captured string, repr fallback, prompt text, eval metadata, or error message
is replaced with [redacted], running after all built-in text patterns.
Both are validated and compiled once here, so an empty key, empty pattern,
invalid regex, non-string entry, bytes pattern, or an over-large list raises
ValueError/TypeError immediately. Passing either argument replaces the
previously configured additional rules of that kind (passing an empty
iterable clears them); omitting it leaves the current additional rules
unchanged. The built-in rules always remain in force either way.
model_prices is an opt-in, local-only price table that auto-fills a
generation's cost from its token usage. It is a mapping of model name to a
rates mapping holding a non-negative, finite input and/or output
per-token rate plus an optional currency (default "USD"). Bir bundles
no prices, so the rates — and keeping them current — are yours to supply. When
a generation has usage and a model matching a configured entry but no
explicitly set cost, its input_cost/output_cost/total_cost are
derived from the matching rates at the configured currency exactly as a manual
set_cost(...) would record them (input rate times input tokens, output rate
times output tokens, total summed when both sides are priced). An explicit
set_cost(...) always wins and is never overwritten, and a generation whose
usage lacks the needed token split is left without a derived cost. The table is
validated once here, so a non-mapping table, a non-string or empty model name,
a non-mapping or empty rate entry, an unknown rate key, a boolean, negative, or
non-finite rate, an invalid currency, or an over-large table raises
ValueError/TypeError immediately. Passing model_prices replaces the
previously configured table (an empty mapping clears it); omitting it leaves
the current table unchanged. With no table configured, generation cost behavior
is unchanged.
max_value_length and max_collection_items are opt-in capture-size
limits that bound a single captured value so one huge payload (a base64
image, a megabyte of model output) cannot bloat the local store. Both
default to None (unlimited), which keeps captured output byte-for-byte
unchanged. When max_value_length is a non-negative integer, a captured
string longer than it is truncated to that many characters with a visible
…[truncated] marker appended; truncation always runs after redaction,
so a secret is replaced before any cut and can never be split in a way that
defeats the redactor. When max_collection_items is a non-negative
integer, a captured list, tuple, set, or mapping larger than it keeps only
the first that-many items and records a single …[truncated] marker for
the remainder, leaving the output valid JSON. Both apply uniformly to every
capture path (inputs, outputs, metadata, repr fallbacks, and dataset and
experiment capture) and compose with the existing capture-depth cap. They
only bound captured values, never event names, models, ids, or the schema. A
non-integer, boolean, or negative limit raises TypeError/ValueError
here.
Any field left unset falls back to the value supplied by the matching
environment variable (BIR_TRACE_PATH, BIR_CAPTURE_INPUTS,
BIR_CAPTURE_OUTPUTS, BIR_DISABLED, BIR_SAMPLE_RATE,
BIR_SERVICE_NAME, BIR_ENVIRONMENT, BIR_SOURCE,
BIR_MAX_VALUE_LENGTH, BIR_MAX_COLLECTION_ITEMS), which is read once at
import time, and otherwise to the hardcoded default. A truthy BIR_DISABLED
sets enabled=False (it is the inverse of the enabled field, so the
common "turn it off in production" case is a single boolean variable).
Explicit arguments to this function take precedence over the environment, so
configure(enabled=True) re-enables recording even when BIR_DISABLED is
set.
Loading and sending events
Read locally recorded traces back into memory and post them to a Bir server, and read the active trace and span ids for correlating your own logs and metrics.
bir.load_events(path=None, *, include_rotated=False)
Load local JSONL trace events.
By default only the active trace file is read. Pass include_rotated=True
to also read size-rotated siblings (traces.jsonl.1 ..) created by
configure(max_bytes=...). Rotated files are read oldest-first so the
returned events stay in the same chronological order they were written,
matching how a single never-rotated file would read. Because rotation can
occur mid-trace, a single logical trace may be split across files.
bir.load_traces(path=None, *, include_rotated=False)
Load local traces grouped by trace_id.
include_rotated is forwarded to :func:load_events; see its note about
traces possibly being split across rotated files.
bir.send_events(server_url='http://127.0.0.1:8000', *, path=None, timeout=10.0, retries=2, backoff=0.5, mark_sent=False, include_rotated=False)
Send local JSONL trace events to a Bir ingestion server.
Transient failures are retried with exponential backoff: a network error,
timeout, or HTTP 5xx is retried up to retries times (default 2),
sleeping backoff * 2**attempt seconds between tries (backoff defaults to
0.5). A 4xx response is a permanent rejection and is raised immediately
without retry, matching the un-retried behavior. A healthy send still makes a
single attempt, so the default behavior is unchanged.
mark_sent is opt-in bookkeeping for cheap re-sends. When True, the IDs
the server accepts are recorded in a sidecar file next to the trace file
(<trace_path>.sent) and skipped on later sends, so attempted reflects
only events not yet recorded as sent. The sidecar is SDK-local: it never
modifies the trace JSONL or the event schema, and a missing or corrupt sidecar
is treated as empty so it can never block a send. With the default
mark_sent=False nothing is recorded and re-sending the whole file stays
safe because the server is idempotent on event IDs.
include_rotated is opt-in upload of size-rotated trace files. The default
False uploads only the active trace file, matching the historical
behavior. When True, retained rotated siblings (traces.jsonl.1 ..)
created by configure(max_bytes=...) are uploaded oldest-first followed by
the active file, so rotation can no longer strand unsent events. Events are
deduplicated by ID when a rotated file overlaps the active file, and the
mark_sent sidecar still anchors to the active trace path so recorded IDs
are skipped across the whole selected file set.
bir.get_current_trace_id()
Return the active trace's id, or None outside any trace.
The value is the same id written to the trace_id field of every event
recorded while this trace is active, so an application log stamped with it can
later be correlated with the trace. Read from a task-local context, so each
asyncio task and thread observes its own active trace and never another's.
While recording is disabled (configure(enabled=False)) or the trace was
sampled out, this still returns the live id inside the trace so log
correlation keeps working even though nothing is persisted.
bir.get_current_span_id()
Return the innermost active span's id, or None outside any trace.
Inside a nested span()/generation()/tool_call() this is the
innermost open node's id; directly inside a trace with no open child it is the
trace root's id. The value is the same id written to the parent_id field
of any child event created at this point, so it names what an event recorded
now would attach to. Read from a task-local context, so each asyncio task and
thread observes its own ids and never another's. Like
:func:get_current_trace_id, this still returns the live id while recording
is disabled or the trace was sampled out, so log correlation is unaffected.
Data types
The frozen dataclasses returned by the loaders and helpers above.
bir.TraceEvent
dataclass
A single trace, span, generation, tool call, or score loaded from storage.
duration_ms
property
Return the event duration in milliseconds.
bir.LoadedTrace
dataclass
A trace root event with all events that share its trace ID.
duration_ms
property
Return the root trace duration in milliseconds.
bir.SendEventsResult
dataclass
Result returned after sending local events to a Bir server.
skipped
property
Return events the server did not newly accept, usually duplicates.
bir.PromptRecord
dataclass
Prompt metadata attached to a generation event.
render()
Render the prompt template with variables when no explicit rendered value exists.
to_metadata()
Return the redacted metadata representation stored on a generation event.
Evals and experiments
Deterministic evaluators, datasets, and experiment runners from bir.evals.
bir.evals
Deterministic evaluation, dataset, and experiment helpers for Bir.
Dataset
dataclass
A collection of uniquely identified examples for experiment runs.
__iter__()
Iterate over dataset examples.
__len__()
Return the number of examples in the dataset.
from_jsonl(path)
classmethod
Load dataset examples from a JSONL file.
to_jsonl(path, *, redact=True)
Write dataset examples to a JSONL file.
Redaction is enabled by default so exported datasets use the same safe
capture behavior as trace and experiment artifacts. Pass
redact=False only when you intentionally want to preserve raw
example payloads.
DatasetExample
dataclass
One input, expected output, and metadata row in an evaluation dataset.
to_dict(*, redact=True)
Return a JSON-serializable dataset row, redacted by default.
DeterministicEvaluator
dataclass
Callable evaluator wrapper used by experiment runs.
evaluate(output, *, expected=None, context=None)
Evaluate a task output and return an EvalResult.
EvalResult
dataclass
A numeric evaluator score with optional JSON-safe metadata.
to_dict()
Return a JSON-serializable representation of the score.
EvaluationContext
dataclass
Runtime context passed to evaluators that need experiment metadata.
ExperimentDiff
dataclass
Aggregate-score differences between two experiment runs.
tolerance is the global tolerance, while effective_tolerances records
the tolerance actually applied to each shared evaluator after per-evaluator
overrides. missing_score is the configured policy for evaluators present
only in the baseline, and regression_reasons maps every evaluator that
fails the gate to a machine-readable reason. example_deltas is the opt-in
per-example detail: for each shared evaluator it maps an example_id present in
both runs to the candidate-minus-baseline delta for that example, and is empty
unless :func:compare_experiments was called with per_example=True. All
mappings are ordered by key so the diff serializes deterministically.
has_regressions
property
Return whether the configured policy reports any regression.
A shared evaluator that dropped beyond its effective tolerance always
counts. When the missing-score policy is regress, evaluators present
only in the baseline also count: a removed evaluator drops coverage even
though no aggregate delta can be computed.
to_dict()
Return a deterministic JSON-serializable representation of the diff.
example_deltas is included only when populated (it is empty unless
per-example detail was requested), so the default aggregate-only output is
byte-for-byte unchanged from before the field existed.
ExperimentExampleResult
dataclass
The task output and evaluator scores for one dataset example.
duration_ms
property
Return the example runtime in milliseconds.
to_dict()
Return a JSON-serializable experiment result row.
ExperimentResult
dataclass
All example results and aggregate scores for one experiment run.
aggregate_scores
property
Return the mean score for each evaluator name.
to_dict()
Return a JSON-serializable experiment payload.
ExperimentSummary
dataclass
Compact metadata persisted next to an experiment result file.
to_dict()
Return a JSON-serializable experiment summary.
SendExperimentResult
dataclass
Result returned after sending an experiment to a Bir server.
answer_contains_citation(*, pattern=None, name='answer_contains_citation')
Create an evaluator that checks whether an answer includes a citation marker.
This is a deterministic format check, not proof of grounding or relevance: it only confirms that a citation marker is present in the answer text, not that the citation is correct or that the cited source supports the answer.
The task output may be a plain answer string or a structured RAG mapping
with an answer string:
"Paris is the capital of France [1]."
{"answer": "Paris is the capital of France [1].", "contexts": [...]}
By default any bracketed marker such as [1] or [doc-1] counts as a
citation. Pass pattern to override the citation regex, for example
r"\(\d+\)" to require parenthetical markers like (1).
answer_context_overlap(min_ratio, *, name='answer_context_overlap')
Create an evaluator that checks how much of an answer is supported by retrieved context.
The overlap ratio is the fraction of answer word tokens that also appear in the retrieved context texts. It is a deterministic heuristic for spotting unsupported answers, not proof of faithfulness: paraphrased but faithful answers can score low, and unfaithful answers that reuse context words can score high.
The task output must be a mapping with an answer string and a
contexts list of retrieved text strings:
{"answer": "...", "contexts": ["doc text", "other doc text"]}
compare_experiments(baseline, candidate, *, tolerance=0.0, score_tolerances=None, missing_score=_MISSING_SCORE_IGNORE, per_example=False)
Compare shared aggregate evaluator scores from two experiment runs.
A shared evaluator regresses when candidate - baseline is strictly less
than -tolerance. score_tolerances maps an evaluator name to a
non-negative, finite tolerance that overrides the global tolerance for
that evaluator only; the strict math.isclose boundary is preserved per
evaluator. Every override name must be a shared evaluator present in both
runs, so a typo or a tolerance for a non-comparable evaluator raises a clear
error instead of being silently ignored.
missing_score selects how evaluators present only in the baseline are
treated. "ignore" (the default) reports them without failing the gate,
matching the historical behavior. "regress" treats each baseline-only
evaluator as a regression, because a removed evaluator silently drops
coverage even though no aggregate delta can be computed. Evaluators found
only in the candidate are always reported but never counted as regressions.
per_example is opt-in reporting detail and never changes the aggregate
comparison or the gate decision. When True, the returned diff's
example_deltas records, for each shared evaluator, the
candidate-minus-baseline score delta of every example_id that both runs
scored with that evaluator; examples present in only one run (or not scored by
the evaluator, such as an errored example) are skipped. When False (the
default) example_deltas is empty and the diff is identical to before.
contains(expected=_USE_EXAMPLE_EXPECTED, *, case_sensitive=True, name='contains')
Create an evaluator that scores 1.0 when output text contains a string.
cost_under(max_cost, *, field='total_cost', name='cost_under')
Create an evaluator that scores 1.0 when a reported cost is under a threshold.
custom_evaluator(name, evaluate, *, uses_context=False)
Wrap a user-provided callable as a deterministic evaluator.
exact_match(expected=_USE_EXAMPLE_EXPECTED, *, name='exact_match')
Create an evaluator that scores 1.0 when output equals the expected value.
field_contains(path, expected=_USE_EXAMPLE_EXPECTED, *, case_sensitive=True, name='field_contains')
Create an evaluator that checks whether a nested string field contains text.
field_equals(path, expected=_USE_EXAMPLE_EXPECTED, *, name='field_equals')
Create an evaluator that compares a nested output field to an expected value.
json_valid(*, name='json_valid')
Create an evaluator that scores 1.0 for JSON-compatible output.
latency_under(max_ms, *, name='latency_under')
Create an evaluator that scores 1.0 when task latency is under a threshold.
list_experiments(directory=Path('.bir') / 'experiments')
List experiment summaries in newest-first order.
load_experiment(path)
Load an experiment result JSONL file.
load_experiment_summary(path)
Load an experiment summary JSON file.
numeric_between(min_value=None, max_value=None, *, field=None, name='numeric_between')
Create an evaluator that checks a numeric output or field against bounds.
regex_match(pattern, *, flags=0, name='regex_match')
Create an evaluator that scores 1.0 when output text matches a regex.
render_experiment_report(result, *, format='html')
Render one experiment to a self-contained report string.
The report bundles the run summary, the per-evaluator aggregate means, and a
per-example table of statuses and scores into a single string with no external
assets, so a local-first user can share or archive an experiment result
without standing up the server or dashboard. format selects "html"
(the default, a complete standalone HTML document with inline styles) or
"markdown".
Output is deterministic for a given experiment: evaluators are ordered by name and examples follow their persisted dataset order, so re-rendering the same result yields a byte-identical string. Only already-persisted (already redacted) values are rendered, and every experiment-derived string is escaped for the chosen format, so example data cannot inject markup. Built with the standard library only.
retrieved_context_contains(expected, *, case_sensitive=True, name='retrieved_context_contains')
Create an evaluator that checks whether retrieved context contains a string.
This is a deterministic retrieval check, not proof of relevance or
faithfulness: it only confirms that expected appears verbatim in one of
the retrieved context strings, not that the answer relied on it.
The task output must be a mapping with a contexts list of retrieved text
strings:
{"answer": "...", "contexts": ["doc text", "other doc text"]}
run_experiment(name, *, dataset, task, evaluators, path=None, raise_on_error=True, record_traces=False, max_workers=1, timeout=None)
Run a task over a dataset and persist per-example evaluator results.
When max_workers is greater than 1, examples run concurrently inside a
:class:concurrent.futures.ThreadPoolExecutor with up to max_workers
threads. Results, JSONL rows, and summary aggregates are always written in
dataset order regardless of completion order. Every other behavior matches
the sequential path: raise_on_error persists through the first failing
example in dataset order and re-raises that exception; record_traces
isolation is preserved because each worker thread inherits its own copy of
the context-var state, so trace trees never bleed across examples.
timeout is an optional per-example limit in seconds (a positive, finite
number; default None means unlimited). When set, an example whose task
runs longer than timeout is recorded as an "error"-status result with
a "task timed out after Ns" message — the same shape as any other failed
example, so raise_on_error is honored — and the run continues with the
remaining examples. The limit applies to each example's own runtime, measured
from the moment its task starts executing: with max_workers greater than
1, an example queued behind others waiting for a free worker thread accrues
no timeout while it waits (the serial max_workers=1 path uses a dedicated
single-worker executor per example, so its task starts immediately). Python
cannot force a thread to stop, so a timed-out task keeps running in the
background until it returns on its own; its result is discarded, and it
occupies its worker slot until then, which can delay — but never time out —
examples still waiting in the queue. timeout=None is byte-for-byte
identical to the previous behavior.
run_experiment_async(name, *, dataset, task, evaluators, path=None, raise_on_error=True, record_traces=False, max_concurrency=1, timeout=None)
async
Run a task over a dataset with bounded concurrency and persist results.
This is the asynchronous counterpart to :func:run_experiment. The task
may be a coroutine function, a plain sync callable, or a sync callable that
returns an awaitable; the return value is awaited only when
:func:inspect.isawaitable reports it is awaitable, so the callable itself is
never inspected. Up to max_concurrency examples run concurrently, but the
returned results, the persisted JSONL rows, and the summary aggregates always
follow dataset order regardless of completion order.
Every other behavior matches :func:run_experiment: evaluator execution,
task input binding, redaction, raise_on_error semantics, and the
persisted JSONL/summary schema are identical. Each example runs in its own
asyncio task, whose copied context isolates the trace contextvars, so
record_traces=True produces a separate trace tree per example even while
they run concurrently.
Like :func:run_experiment, raise_on_error=True persists results through
the first failing example in dataset order, writes the matching error
summary, and re-raises that example's exception. Because examples run
concurrently, later examples may already have executed when the failure is
detected; their results are simply not persisted. If the surrounding
coroutine is cancelled, the in-flight example tasks are cancelled and awaited
and CancelledError propagates without writing a misleading success
summary.
timeout is an optional per-example limit in seconds (a positive, finite
number; default None means unlimited). When set, each example coroutine
is wrapped in :func:asyncio.wait_for; an example that runs longer than
timeout has its task cancelled and awaited (so no pending-task warning
leaks) and is recorded as an "error"-status result with a "task timed
out after Ns" message — the same shape as any other failed example, so
raise_on_error is honored and dataset order is preserved. With
record_traces=True, a timed-out example's trace root is written with
error status before the cancellation unwinds — its already-recorded child
events stay attached and loadable — and the error result's trace_id
links that trace. timeout=None is byte-for-byte identical to the
previous behavior.
send_experiment(path, server_url='http://127.0.0.1:8000', *, timeout=10.0, retries=2, backoff=0.5)
Send a persisted experiment and its summary to a Bir server.
Transient failures are retried with exponential backoff, matching
send_events: a network error, timeout, or HTTP 5xx is retried up to
retries times (default 2), sleeping backoff * 2**attempt seconds
between tries (backoff defaults to 0.5). A 4xx response is a permanent
rejection raised immediately without retry, as are a missing experiment or
summary file and an invalid success response body. A healthy send still makes
a single attempt with no sleep, so the default behavior is unchanged.
similarity_above(threshold, expected=_USE_EXAMPLE_EXPECTED, *, case_sensitive=True, name='similarity_above')
Create an evaluator that scores 1.0 when output text is similar enough to expected.
Similarity is the normalized :class:difflib.SequenceMatcher ratio between
the output text and the expected text, a deterministic, dependency-free fuzzy
check that sits between exact equality (:func:exact_match) and substring
presence (:func:contains). It tolerates typos, reordering, and minor
wording differences without an embedding model or any new dependency. The
score is 1.0 when the achieved ratio is at or above threshold (the
boundary is inclusive) and 0.0 otherwise. Pass case_sensitive=False to
lowercase both sides before comparing. The achieved ratio and threshold are
recorded in EvalResult.metadata so failures are inspectable.
Testing helpers
Assert on your own instrumentation from bir.testing.
bir.testing
Test helpers for asserting on your Bir instrumentation.
The public :func:capture_traces context manager redirects trace writes to a
private temporary file for the duration of a with block and yields a
:class:CapturedTraces handle that reads the captured events and traces back in
memory through the SDK's own public loaders. It lets application authors assert
on the traces their instrumentation produces without pointing
configure(trace_path=...) at a scratch file by hand or touching their real
.bir/ directory.
Like :func:bir.configure, this mutates process-global SDK configuration for the
duration of the block: on entry it swaps the active trace_path to the
temporary file, and on exit it restores the previous configuration in full
(including a user-set trace_path), even when the body raises. It only
redirects where events are written — capture opt-in, sampling, and redaction are
left exactly as configured — so a captured event is identical to what a real
.bir/traces.jsonl write would contain. Because it mutates global state it is
not safe to run concurrently from multiple threads, the same caveat that applies
to configure.
CapturedTraces
Read-back handle for the events recorded inside a capture_traces block.
While the block is open, :meth:events and :meth:traces read live from the
temporary trace file through the public :func:bir.load_events /
:func:bir.load_traces loaders, so a test body can assert on progress as it
goes. When the block exits, the final state is snapshotted in memory just
before the temporary file is removed, so the same methods keep returning the
captured data after the with block has closed.
trace_path
property
Temporary file that events are captured to for the block's duration.
events()
Return the events captured in the block, in load order.
Inside the block this re-reads the temporary file on each call; after the block it returns the snapshot taken just before cleanup. Reads include any size-rotated siblings, so capture stays complete even if the inherited configuration rotates the file mid-block.
traces()
Return the captured events grouped into traces by trace_id.
Inside the block this re-reads the temporary file on each call; after the block it returns the snapshot taken just before cleanup.
capture_traces()
Redirect trace writes to a temporary file and yield a read-back handle.
Use this in tests to assert on the traces your own instrumentation produces
without writing to (or reading from) your real .bir/ directory::
from bir.testing import capture_traces
with capture_traces() as captured:
answer_question("hello")
events = captured.events()
assert [event.type for event in events] == ["trace", "generation"]
Only the active trace_path is swapped; every other configured setting
(capture opt-in, sampling, redaction, the price table, and service metadata)
is kept, so events are captured exactly as they would be in production — just
into an isolated temporary file. The previous configuration, including a
user-set trace_path, is restored when the block exits, even if the body
raises, and the temporary file and directory are removed.
Like :func:bir.configure, this mutates process-global configuration for the
duration of the block and is therefore not safe to run concurrently across
threads. Nested capture_traces() blocks are fine: each restores the
configuration that was active when it was entered.
Logging integration
Stamp standard-library log records with the active trace and span ids using
bir.logging.
bir.logging
Stamp standard-library log records with the active Bir trace and span ids.
Correlating your application logs with Bir traces is the primary documented use of
:func:bir.get_current_trace_id / :func:bir.get_current_span_id. Doing it by hand
means threading extra={"trace_id": ...} through every logging call. This
module removes that plumbing: attach :class:BirTraceIdFilter once and every
:class:logging.LogRecord that flows through the logger or handler gains two
attributes, ready for any formatter to render::
import logging
from bir.logging import install_trace_id_filter
install_trace_id_filter()
logging.basicConfig(
format="%(asctime)s %(levelname)s [trace=%(bir_trace_id)s span=%(bir_span_id)s] %(message)s"
)
The stamped attributes mirror the accessors exactly: inside a trace they equal
:func:bir.get_current_trace_id / :func:bir.get_current_span_id, and outside any
trace they are None. The ids are read from the same task-local context as the
accessors, so each asyncio task and thread sees its own ids and never another's.
Like the accessors, this is read-only: the filter only reads the active ids onto the record. There is no setter and no cross-process propagation, consistent with the accessors' design. The filter never raises, so it is safe to leave attached on every log call, inside or outside a trace.
BirTraceIdFilter
Bases: Filter
A :class:logging.Filter that stamps the active Bir ids onto every record.
On each record it sets two attributes from the current task-local context:
record.bir_trace_id— the active trace's id (see :func:bir.get_current_trace_id), orNoneoutside any trace.record.bir_span_id— the innermost open span/generation/tool-call id, or the trace root when none is open (see :func:bir.get_current_span_id), orNoneoutside any trace.
The names match :data:TRACE_ID_FIELD and :data:SPAN_ID_FIELD and are safe to
render with %(bir_trace_id)s / %(bir_span_id)s. The values equal the
trace_id / parent_id later written to the JSONL, so a stamped log lines
up with the trace. :meth:filter always returns True (it is used purely to
annotate, never to drop records) and never raises.
Despite the :class:logging.Filter base, this does not filter by logger name —
pass it to addFilter on a logger or a handler. Attaching it to a logger
stamps records created by that logger; attaching it to a handler stamps every
record the handler emits (including those propagated from child loggers).
install_trace_id_filter(target=None)
Attach a :class:BirTraceIdFilter to a logger or handler and return it.
With no argument the filter is added to the root logger, which is enough for the
common case where application loggers propagate to root. Pass a specific
:class:logging.Logger or :class:logging.Handler to scope it; attaching to a
handler is the most reliable way to stamp every record that handler emits,
including ones propagated from child loggers.
Returns the created filter so it can later be removed with
target.removeFilter(returned_filter). Calling this more than once on the same
target attaches independent filters; each stamps the same attributes, so the
duplication is harmless but you can avoid it by reusing the returned instance.