Nabız — the runtime root-cause layer for live agents
An in-org, zero-egress watchdog for live Agentforce agents — Nabız (Turkish for 'pulse') correlates a deterministic conversation-quality time series with the org's own change stream, names the strongest candidate cause with an evidence grade A–E, and proposes fixes a human must approve. Salesforce Observability tells you WHAT happened; Nabız tells you WHY — and its honesty gate (Cohen's κ ≥ 0.60) has provably rejected bad calibrations rather than waving them through.
- Year
- 2026
- Role
- Salesforce Developer — Agent Observability
- Tech stack
- Apex + 6 Lightning Web ComponentsEWMA control band + changepoint detection — deterministic, no ML3 change sources: Setup Audit Trail, Flow versions, BotVersion SHA-256 diffEvidence Ladder — 5 verifiable rungs → grades A–ENegative-control cohort in the correlation engineCohen's κ ≥ 0.60 calibration gate (blind labeling console)Zero egress — no endpoints, no conversation content at restBudgetGuard — yields at 70% of any governor limitFix proposals only — the human gate is permanentSession tracing via Data Cloud DMO SOQL (zero OAuth)126/126 tests green on the live pilot org (Jul 2026) · 29 ADRs
Highlights
- The gap it fills, in one sentence: Salesforce Observability tells you WHAT happened — Nabız tells you WHY. It builds a deterministic per-topic conversation-quality time series (EWMA plus a control band, hand-rolled Apex, no ML), captures the org's change stream in parallel, and when quality drops it correlates the drop against recent changes to name the strongest candidate cause — always with an evidence grade, never a bare 'root cause'
- Three change sources feed one deduplicated ledger: the Setup Audit Trail, Flow definition versions, and an SHA-256 snapshot diff of the agent's own configuration — so when someone quietly edits the agent and files no ticket, Nabız detects the edit autonomously and pins it to the timeline
- Verdicts ride an Evidence Ladder: five independently verifiable rungs — Temporal, Dependency, Runtime, Isolation, Recovery — collapse into a grade from A to E, with a negative-control cohort guarding against spurious correlation. The confidence percentage is derived for display only, capped at 99, and is never an input to any decision
- The κ-gate is the honesty mechanism: semantic LLM scoring stays OFF until human labels from a blind console agree with the model judge at Cohen's κ ≥ 0.60 — and the gate provably rejects: in one live day it threw out three invalid calibrations, including a circular κ=1.000. A gate that only ever passes is decoration; this one demonstrably fails
- Zero-egress by construction: no Remote Site Settings exist, conversation content is never stored (hashed references only), all model access sits behind one seam whose mock simulates five misbehavior modes, and a BudgetGuard yields at 70% of any governor limit. Fixes — article republish, Flow rollback, instruction patch — are proposed as text and never applied; the human gate is permanent
- It watches the real thing: deployed into the live HanseWatt pilot org, monitoring the deployed agent — full two-sided transcripts read via plain SOQL on Data Cloud DMOs, zero OAuth — with 126/126 tests green on that org (measured July 2026), 29 ADRs, and a state-of-product doc that lists not just wins but six twice-measured negatives: what it wanted and could not get. 'The org adjudicates' is a house rule, not a slogan
The problem
A live agent regresses, and the dashboards all say the same unhelpful thing: quality dropped on Tuesday. What they cannot say is why — because the cause usually isn't in the conversation at all, it's in the org: a Flow got a new version, a knowledge article was republished, someone edited the agent's instructions and filed no ticket. Native Agent Analytics shows the drop; it does not correlate the drop with the change stream. So admins bisect by hand — rolling back changes one by one on a production agent — or worse, the regression is blamed on 'the model being moody' and nothing is learned. And any tool that solves this by shipping conversation content to an external service has disqualified itself in a German enterprise before the first demo.
The approach
Watch the pulse, watch the changes, and let evidence — not a model's opinion — connect them. Nabız builds a deterministic per-topic quality time series (EWMA with a control band, changepoint detection, seasonal baselines — hand-rolled Apex, explicitly not ML) while three capture sources feed a deduplicated change ledger: the Setup Audit Trail, Flow definition versions, and an SHA-256 snapshot diff of the agent's own configuration, so even an unticketed agent edit is detected autonomously. When the pulse drops, a correlation engine crosses time, the dependency graph and an Evidence Ladder — five independently verifiable rungs (Temporal, Dependency, Runtime, Isolation, Recovery) collapsing into a grade from A to E, with a negative-control cohort guarding against spurious correlation; the confidence percentage is display-only and never an input to a decision. Semantic LLM scoring exists but stays off behind a calibration gate: human labels from a blind console must agree with the model judge at Cohen's κ ≥ 0.60 first. Everything is zero-egress — no external endpoints, no conversation content at rest (hashed references only), all model access behind one seam with a five-mode misbehavior mock — and every proposed fix is text a human must approve: the human gate is permanent, by ADR.
The outcome
It runs against the real thing: deployed into the live HanseWatt pilot org, monitoring the deployed agent — full two-sided session transcripts read via plain SOQL on Data Cloud DMOs with zero OAuth, the platform's own response-evaluation verdicts harvested as a quality signal, and an agent-config edit detected autonomously in the live org. 126/126 tests green on that org, measured July 2026 (the suite has since grown to ~150 in source); 66 production classes, 29 ADRs, six LWC consoles from health board to blind-labeling calibration. The honesty is structural: the κ-gate has so far only ever rejected calibrations — three invalid ones in a single live day, including a circular κ=1.000 — and the state-of-product doc records six twice-measured negatives alongside the wins, in evidence classes from [measured] to [refuted]. Honestly bounded: it names candidate causes with grades, never certainties; it proposes, never applies. In the portfolio's lifecycle it is the running heartbeat: PreFlight proves the ground, HanseWatt builds, Blast Radius bounds what code can reach, Prüfstand attacks, Nabız watches the live pulse and names the why — and Aktenlage files the evidence.