Your AI agent fixed the bug, but what else did it change?

AI‑Assisted Patching: Beyond the Bug Fix

When an AI agent patches a vulnerability, the headline is the bug that disappears. The deeper story is how that fix is validated, documented, and kept from breaking existing functionality. Trail of Bits’ patching‑bugs and regression‑hunt plugins embody a systematic approach that turns a single AI‑generated fix into a repeatable, auditable workflow.

---

1. A Structured Patch Lifecycle

Both plugins follow the same high‑level pattern:

1. Reproduce the bug on a live deployment, ensuring the failure is observable through the same interfaces a real user would use.
2. Generate a fix with an AI agent, guided by a minimal set of opinionated instructions that capture Trail of Bits’ definition of a “good” patch.
3. Validate the fix with a three‑arm proof: the original code must fail, the patched code must pass, and a clean revert must fail again. This isolates the patch’s effect from environmental noise.
4. Run regression‑hunt to trace the diff through callers, replay candidate regressions on a fresh instance, and adjudicate each finding against a strict evidence checklist.
5. Produce a schema‑based report that includes the claim, code revision, raw evidence, and any open questions for human review.

The modular design means teams can run either step in isolation—use regression‑hunt on an existing PR, or apply patching‑bugs without the regression check if they already have a trusted test suite.

---

2. Reducing False Positives

In a preliminary study of 130 merged PRs, 21 regressions were confirmed (16 functional, 1 performance, 4 build/test). Regression‑hunt identified 58 candidates, achieving roughly 36 % precision. Compared with a baseline Codex agent prompted simply to “review the PR for regressions,” regression‑hunt cut false‑positive findings by 25 % (95 % CI 14‑38 %).

The improvement stems from two design choices:

• Explicit regression definition – the plugin only reports a regression after the agent proves the issue is not a false positive, pre‑existing bug, or incomplete fix.
• Threat‑model integration – a project‑specific `THREAT_MODEL.md` is injected into every agent step, forcing the AI to align its reasoning with the maintainer’s security assumptions.

The trade‑off is a modest increase in borderline cases where evidence is insufficient for a definitive classification. Ongoing work focuses on enriching evidence‑gathering strategies to lift those cases into confirmed regressions.

---

3. Codifying Project Knowledge

Both plugins rely on two optional but highly recommended artifacts:

• `THREAT_MODEL.md` – enumerates assets, trust boundaries, and the definition of a vulnerability or regression for the project.
• `PATCHING_GUIDELINES.md` (or `AGENTS.md`) – describes build/test procedures, patch size limits, preferred locations for changes, and the criteria a regression test must satisfy.

When these files are present, the agents automatically incorporate the project’s conventions, producing patches that respect existing style and security expectations. Early adopters such as GPAC, liboqs, and validator.js have reported smoother review cycles because the AI’s output already matches their internal standards.

---

4. Looking Ahead

The plugins are still evolving. Planned enhancements include:

• Automated evaluation metrics that measure whether a patch fixes the original bug and its variants without introducing regressions.
• Trace‑level learning from engineer‑agent sessions to identify steps the AI skips or where its evidence is weakest.
• Configurable aggressiveness – maintainers can choose a tighter regression filter for low‑risk projects or loosen it to surface more candidates when broader coverage is valuable.

By continuously feeding back real‑world usage data, Trail of Bits aims to refine the plugins into a robust, community‑driven toolkit for AI‑assisted security maintenance.

---

Practical Takeaway

If your project already struggles with the speed of bug disclosure, consider integrating these plugins:

1. Add `THREATMODEL.md` and `PATCHINGGUIDELINES.md` to your repository.
2. Install the `patching-bugs` and `regression-hunt` plugins (available via the Trail of Bits plugin registry).
3. Run the plugins on a new bug report and let the AI produce a three‑arm proof and regression analysis.
4. Review the schema‑based report, address any open questions, and merge the vetted patch.

Standardizing the patching workflow not only speeds up remediation but also builds confidence that AI‑generated fixes won’t silently introduce new problems. In a world where vulnerabilities surface faster than ever, that confidence is the missing piece.