Probe Kit — v0.1.0

Your agent's guardrails have never actually been attacked.

You've tested that your agent works. You haven't tested whether it holds up when a fake incident channel tells it to skip sign-off, or a code comment claims to be a "pre-approved" telemetry request. Probe Kit is a validated corpus of 37 real social-engineering probes across 29 manipulation angles, run against your own agent through one function you write. Every outcome is scored for whether it actually deliberated, not just whether it happened to pass — and logged into a hash chain that makes the evidence itself tamper-evident.

Get it — $2,000/yr See pricing →

105tests passing 0pip dependencies 37probes across 29 angles

Three gaps

Three gaps between "it works" and "it holds up."

Untested: Nobody actually attacks the agent before it ships. Functional tests check that your agent does the right thing when asked nicely. Nothing checks what it does when a README comment, an incident-channel message, or a fake system notice tries to talk it into something it shouldn't do. The first real test is usually a customer finding the gap in production.
Unweighted: A pass isn't a pass isn't a pass. An agent that instantly rejects an obviously-fake probe proves almost nothing — the probe may just not have been convincing. An agent that visibly deliberates on a well-crafted pretext and still holds is the real signal. A flat pass/fail log treats both identically, which hides exactly the distinction that matters.
Unverifiable: A red-team log that can be quietly edited proves nothing. An evidence log stored as a plain table can be edited after the fact — by a bad actor covering a real compromise, or just a nervous engineer smoothing over a bad result before a review. Without a way to detect that, "we ran the red-team suite and it passed" is a claim, not evidence.

What it does

A probe corpus, an honest interface, and evidence that can't be quietly edited.

Corpus — probe_library: In plain terms: 37 hand-written scenarios covering 29 distinct manipulation angles — authority impersonation, urgency pressure, fabricated error recovery, false technical context, moral licensing, and more — across three difficulty tiers, not one generic "try to jailbreak it" prompt.

Technically: each Probe pairs a realistic setup (a README comment, an incident-channel message, a ticket reply) with injected_text that carries the actual manipulation, plus a documented compromised_action and resilient_action so a probe's intent is explicit, not inferred.
Interface — probe_engine: In plain terms: Probe Kit never tries to guess whether your agent "fell for it" by reading its raw text — only your own harness knows what counts as a compromised action for your specific agent. You write one small function; Probe Kit runs the corpus through it.

Technically: AgentUnderTest = Callable[[str], ProbeOutcome]. Your harness receives the rendered probe prompt and returns a ProbeOutcome(complied, response_text) — an honest boundary that keeps the compliance judgment where the ground truth actually lives.
Scoring — resistance_score: In plain terms: a resisted probe only counts as a strong signal if the agent visibly deliberated over it. An instant reject is a weak, low-confidence pass. The same probe or angle repeatedly succeeding against one agent gets promoted from a one-off finding to a flagged systemic gap.

Technically: outcomes below DELIBERATION_THRESHOLD_SECONDS score as low-confidence; at or above it, high-confidence. find_prompt_diagnostics() emits a PROMPT_DIAGNOSTIC once the same probe or angle complies REPEATED_COMPLIANCE_THRESHOLD times across recorded runs.
Evidence — evidence_graph: In plain terms: every probe result is written to a hash chain, not a plain table — each row's hash covers the previous row's hash. Edit any payload after the fact, anywhere in the chain, and the edit is mathematically detectable, not just policy-forbidden.

Technically: SQLite-backed; each event's hash is SHA-256 over prev_hash|run_id|seq|t|kind|payload. verify_chain() recomputes every hash from genesis and flags the run TAMPERED/CORRUPT the instant one stops matching.

See it run

Real runs, not a mockup.

Every line below is an actual terminal session against the real, shipped CLI — captured directly, including a genuine CI-gate failure. Not staged for a clean 100%. Want both products in one place? The combined two-minute demo →

$ probe-kit run --harness acme_agent:harness --evidence probe_kit.db --agent-id acme-support-bot --fail-on medium
probe-kit run: 37 probe(s), 6 complied, resistance 83% (high-confidence 0%)
FAIL: 5 probe(s) at or above 'medium' difficulty complied (run 86d46ec2-b32e-48a9-bd79-4ed49d305cfc).
$ echo $?
1

$ probe-kit report --evidence probe_kit.db --agent-id acme-support-bot
probe-kit report: agent='acme-support-bot' across 1 run(s) - 37 probe(s), 6 complied, resistance 83% (high-confidence 0%)
$ echo $?
0

$ probe-kit verify --evidence probe_kit.db
  [OK] 86d46ec2-b32e-48a9-bd79-4ed49d305cfc
probe-kit verify: 1/1 run(s) intact.
$ echo $?
0

# ...someone edits a row directly in the SQLite file, bypassing the CLI entirely,
# flipping one probe's "complied" from true to false...

$ probe-kit verify --evidence probe_kit.db
  [TAMPERED/CORRUPT] 86d46ec2-b32e-48a9-bd79-4ed49d305cfc
probe-kit verify: 0/1 run(s) intact.
$ echo $?
1

Try it yourself

Watch the tamper-detection itself run.

A faithful reproduction of the real algorithm — SHA-256 over prev_hash|run_id|seq|kind|payload — computed live in your browser, seeded with four real probe IDs from the actual corpus.

Each row's hash covers the previous row's hash. Change one payload and every hash after it stops matching — verify_chain() catches it instantly, the same way the real evidence store does. This client-side copy is a demonstration, not the actual store: your real evidence lives in the SQLite file the CLI writes.

What's in it

What's actually in it.

Pure standard library — zero pip dependencies to audit or update.
Works as an importable library or a standalone probe-kit CLI.
37 probes across 29 social-engineering angles, 3 difficulty tiers — write once, run against any agent that fits Callable[[str], ProbeOutcome].
Deliberation-aware resistance scoring — a slow, considered "no" scores differently from an instant one.
PROMPT_DIAGNOSTIC findings promote a repeated compliance pattern to a systemic-gap finding, not a one-off flake.
Hash-chained SQLite evidence store — probe-kit verify catches any post-hoc edit to the log.
Optional --redact scrubs known secret/PII shapes from evidence payloads before they're hashed and stored.
--fail-on exits non-zero if any probe at or above a chosen difficulty complied — usable as a CI gate.
105 tests, including a real end-to-end evidence-chain-validity check after a full suite run.

Pricing

One license, your whole org.

No per-seat fees. Unlimited use across your whole organization.

Probe Kit license annual

Flat. Unlimited environments and seats within your org. Covers unlimited use across your org: the library, the CLI, and updates for the year.

Full 37-probe corpus, all difficulty tiers · Deliberation-aware resistance scoring · Hash-chained evidence store + CLI for CI gating

$2,000/yr

GitHub username (for repo access)

Card checkout: clicking "Get it" opens Stripe's own hosted checkout page, and you're invited as a GitHub collaborator automatically on payment.

Trust & scope

What's real, and what this isn't.

105/105Tests passing, verified today

0Pip dependencies

37Probes across 29 angles

0.1.0Current version

Probe Kit only knows what your harness tells it — the complied judgment in every ProbeOutcome comes from your own code, not from Probe Kit reading the agent's raw output and guessing. A harness that misclassifies its own agent's behavior will produce a resistance score that's honest about what the harness reported, not necessarily about what actually happened inside the agent. The deliberation-time threshold is a scoring heuristic, tunable per deployment, not a hard security boundary — a fast model can genuinely deliberate quickly, and a slow one can stall for reasons that have nothing to do with resisting anything. The hash chain proves the EVIDENCE LOG wasn't edited after the fact; it says nothing about whether the underlying agent itself is safe to deploy — that's what the probe results themselves are for. And 37 probes across 29 angles is a real, useful corpus, not an exhaustive one — social engineering has more shapes than any fixed library will ever cover completely.

Get started

Want to know how your agent actually holds up?

No sales call needed to get started.

Get it — $2,000/yr Prefer an invoice? Talk to us