Most agent evaluation tools measure capability (latency, token use, prompt-injection surface). That leaves a blind spot: an agent can “do the job” while lying, overriding humans, leaking data, or invoking tools it shouldn’t. iFixAi treats auditing as a governance and assurance problem rather than a raw capability benchmark, surfacing those business-level gaps quickly.
What Sets It Apart
- Focus on governance and operational assurance: a fixed, reproducible suite of inspections (60 total; 32 core + 28 extended) grouped into five graded pillars — Manipulation, Fabrication, Deception, Unpredictability and Opacity — so scores are comparable across runs.
- Citable, dual-role judging model: runs separate SUT (system under test) and Judge roles; a run is citable only when an independent judge grades the agent, avoiding self-assessment bias.
- Multi-mode runtime: CLI wizard, scriptable flags for CI, and plugins/skills for agents (Claude Code, Codex, Cursor, VS Code, etc.), plus HTTP adapter to probe real deployed endpoints without instrumenting the target.
- Practical outputs for ops and risk teams: JSON + Markdown reports, terminal scorecard, seeded fixtures for fast demos, and an optional Pro service that expands inspections and adds business-friendly Operational Assurance and Regulatory Compliance reports.
Who It's For and Trade-offs
Great fit if you need to know whether an agent behaves within your organization’s business rules (tool authorizations, escalation, data boundaries), want reproducible evidence for risk teams, or need a cross-vendor judge setup for a citable audit. The tool maps directly to questions like “Will this agent spend money, change accounts, or override a manager’s decision?”
Look elsewhere if you only need low-level capability benchmarks (throughput, latency, token efficiency) or a lightweight liveness probe; iFixAi is a diagnostic, not a certification, and full-suite runs are resource-intensive (many judge calls). Some inspections require exposing agent internals (tools, audit trails); otherwise they return "insufficient_evidence" and are reported but not scored. A citable grade requires separate provider keys for SUT and judge(s).
Where It Fits
Compared with eval harnesses and red-teaming tools, iFixAi sits at the intersection of adversarial testing and operational assurance: it reuses red-team techniques (prompt-injection, tool governance) but emphasizes principal fidelity, oversight pathways, and sociotechnical failure modes that matter to business risk and compliance teams.
Practical notes
- Default quick runs use a 10-inspection “essential” suite;
--full-runexecutes all 60 inspections. Reports are saved under ./ifixai-results/ in JSON and Markdown. - Open-source engine (Apache-2.0) for local runs; iFixAi Pro is a paid independent auditing service and dashboard with expanded inspections and an "Audited by iFixAi" badge.
- Telemetry is pseudonymous and opt-outable; many checks still require provider/model keys for both SUT and judge roles to avoid self-judged bias.