We Pointed an Autonomous AI Engineer at Our Own Repo - It Found a Real Bug

We build a scanner that grades how ready a website is for AI agents. So we did the obvious thing to our own codebase: we pointed an autonomous AI engineer — Devin Cloud, on its free SWE-2 tier — at our public repository and told it to find something worth fixing. It came back with four findings. Then we did the part most "AI found a bug" posts skip: we verified every single claim by hand, file and line, before believing it. Result: zero false positives, one real security bug, and a fix PR the agent opened itself.

This is a field report, not a review of Devin. The interesting question for us is not "is the tool good" — it is "can an agent's audit output be trusted, and how do you check it cheaply". That question is our product thesis applied to ourselves.

What exactly did we ask the agent to do?

Short answer: one cloud session, one prompt — clone the repo, review the codebase, report concrete improvement suggestions. No hints, no file pointers, no checklist. The run cost nothing: SWE-2 sessions are free (2 concurrent per user on the Teams plan).

The session ran on Devin's VM against spreadzp/agentbadge at commit 059eea2f. Under the hood the agent cloned the repo, tried bun install (which itself became finding #3), ran TypeScript and the test suite under Node, and read the routes, stores and CI configuration. Total wall time: under an hour, unattended. The full prompt and transcript live in our docs (docs/DEVIN/LOGS/cloud/step3-*.log).

What did it find?

Short answer: four findings, ordered by value — one genuine security bug, one missing CI gate, one broken contributor experience, one hygiene issue. Every one of them verified against the code. Zero false positives.

F-1, the real bug: GET /api/trust/:domain/challenge issues domain-ownership challenge tokens. It requires no authentication, and every call overwrites the stored token for that domain (challenge-store.ts literally documents "Overwrites any existing challenge"). Any crawler, link prefetcher or curious visitor hitting that URL invalidates the DNS TXT record a domain owner already published — their /verify-ownership then fails with 403. The tokens also live in a process-memory Map with no size limit, die on every deploy despite a documented 90-day lifetime, and are compared with plain !== instead of a constant-time comparison (trust.ts:281). Four sub-problems, one endpoint.

F-2: no CI job runs our own bun run gate script. Three workflows exist — scanner smoke test, architecture-graph check, weekly board conformance — and none of them run tsc, eslint or vitest on a pull request.

F-3: a fresh clone does not build. package.json depends on @agentbadge/payments via file:../../packages/payments — a path outside the repo. bun install fails on it, ~620 test files then fail on required env vars that .env.example leaves blank, and 9–11 spec tests read ../docs/EPICS/… which is also outside the repo. Bonus: our own blog links to packages/payments/src on GitHub — a dead link, the directory does not exist.

F-4: committed build artifacts: an 854 KB agentbadge-report.json, a coverage-summary.json, and about ten hackathon-era planning docs sitting in the repo root.

How did we verify the audit before believing it?

Short answer: the same way we verify any agent's claim — check it against ground truth. Each finding was re-derived from source: route definitions, the store implementation, workflow files, package.json, git ls-files. Every claim below carries the file:line evidence we checked; the verification took about fifteen minutes of human attention.

This is the part the demo transcripts do not show you. Agent audit output is a claim set, not a verdict — exactly the same epistemics as our own scanner's evidence model. A claim that cannot name its evidence is worth nothing; a claim that names trust.ts:281 costs you thirty seconds to confirm. Devin's report named its evidence well enough that all four findings confirmed on first inspection. That is not a small thing — it is the difference between a useful tool and a confidence generator.

We also checked what it got wrong, because honest field reports include the caveats. Two numbers in the report are overstated by methodology: the TypeScript error count (310) and the test failure count (272 of 830 files) come from a Node.js run, while our suite is Bun-native — the agent itself flagged that Bun's runtime breaks vi.mocked and gives misleading results. The direction is right (there is no CI running the gate), the precise counts are not reproducible as stated. We record that as confirmed-with-asterisk, not confirmed.

What happened after the audit?

Short answer: we resumed the same cloud session and told it to fix F-1 — replace the stateful token map with a stateless HMAC challenge (agentbadge-verify= + HMAC(secret, domain+epoch)), extend the trust tests, and open a PR. Stateless derivation removes all four sub-problems at once: nothing to store, nothing to lose on restart, nothing to grow unbounded, constant-time compare.

A second session audited agentbadge.xyz itself (~1,970 URLs, read-only): it found five placeholder pages sitting in the sitemap, 252 broken agent-guide links, JSON-LD emitting "price": "undefined" on ~170 pages, and a robots.txt where per-bot groups silently cancel the global Disallows for every listed AI crawler. We ran the same claim-checking pass — five out of five P0 findings verified against source (two already fixed in HEAD, pending deploy); the full verdict list lives in docs/DEVIN/REVIEW/step5-site-audit-review-*.md alongside the repo audit review. F-2 and F-3 are ours to fix — the agent found them, but CI wiring and dependency vendoring are decisions, not diffs.

What does this prove about agent-native workflows?

Short answer: that the bottleneck moved — an autonomous agent produced a competent audit for free in under an hour, and the expensive part became verification and decision-making, exactly the shape our own stack assumes.

The workflow that worked: agent produces claims → human verifies claims against ground truth → agent executes the approved fix → human reviews the diff. That is the same loop our supply-chain bricks formalize — self-declared claims with provenance, verification as a separate step, write paths gated by verifiable state. We did not design this session to be dogfooding, but it turned into it: an agent audited the agent-infrastructure, produced evidence-backed claims, and the claim-checking took a human fifteen minutes because the evidence was real.

The honest limit, stated plainly: one session, one repo, one audit. We are not claiming autonomous engineers replace reviewers — the security bug it found sat in the code for months precisely because it is the kind of thing neither a test suite nor a casual review catches. What it replaces is the blank page: the audit you never ran because nobody had a free day.