AI Configuration & BYOK
The AI Quality Orchestrator is a suite of seven opt-in AI features. This page explains how to enable features and configure your LLM provider, including how to supply your own API keys.
Enabling AI Features
All AI features are off by default (opt-in model). Enable them individually in Settings → Features under the "AI Features" section.
| Feature | Feature Flag | Where to Use |
|---|---|---|
| Auto-Bug Generator | autoBugGeneration | Investigation Hub drawer |
| Flakiness Detective | flakinessDetective | Stability page (sidebar) |
| Smart Test Optimizer | testOptimizer | Bulk Actions bar in Test Cases |
| Smart PR Routing | prRouting | Run Settings toggle |
| Quality Chatbot | qualityChatbot | Ask AI (sidebar) |
| Spec-to-Test Generation | specToTest | "Import from Spec" button in Test Cases |
| Automation Planner | automationPlanner | Automation Planner (sidebar) |
Configuring AI — BYOK (Settings → AI Models)
Agnox ships with platform-managed LLM keys as a convenience. To use your own cloud account keys (zero extra cost to your Agnox plan), configure Bring Your Own Key (BYOK):
- Go to Settings → AI Models and scroll to the AI Configuration section.
- Under Default AI Model, select the model all AI features will use by default:
gemini-2.5-flash(default — fastest, best for most workloads)gpt-4oclaude-3-5-sonnet
- For each provider you want to supply your own key for, locate its row in the Bring Your Own Key table:
- Status shows "Using Platform Default" (grey) until a key is provided, or "Configured" (green) when your key is active.
- Paste your API key into the masked input field and click Save Key.
- To rotate or remove a key, click Remove — the platform fallback key is used immediately.
- Click Save Settings.
BYOK Provider Reference
| Setting | Options | Description |
|---|---|---|
| Default AI Model | gemini-2.5-flash, gpt-4o, claude-3-5-sonnet | Applied to all AI features unless overridden |
| BYOK — Gemini | Optional | Your Google AI Studio or Vertex AI key |
| BYOK — OpenAI | Optional | Your OpenAI platform key |
| BYOK — Anthropic | Optional | Your Anthropic Console key |
Security: Keys are encrypted at rest using AES-256-GCM before being persisted in MongoDB. Plaintext keys are never stored, logged, or returned by any API response. The
resolveLlmConfig()utility on the server is the only code path that decrypts a BYOK key, and only at the moment of an LLM call.
Dual-Agent (Actor-Critic) Architecture
Several AI features use a Dual-Agent pipeline to deliver high-quality, hallucination-resistant output:
┌──────────────────────────────────────────────────────────────────┐
│ STEP 1: Analyzer (Actor) │
│ Temperature: 0.4 — creative, generates suggestions │
│ Output: Structured JSON │
└───────────────────────────┬──────────────────────────────────────┘
│ structured output
▼
┌──────────────────────────────────────────────────────────────────┐
│ STEP 2: Critic (Evaluator) │
│ Temperature: 0.0 — deterministic, no creativity │
│ Input: Raw evidence + Analyzer output │
│ Task: Validate every claim. Override hallucinated suggestions. │
│ Output: Final developer-facing Markdown │
└──────────────────────────────────────────────────────────────────┘
Why two passes?
- The Analyzer (0.4 temperature) generates creative, detailed suggestions but can occasionally hallucinate file names or APIs not present in the logs.
- The Critic (0.0 temperature, fully deterministic) cross-checks every claim against raw evidence. Any suggestion not grounded in the provided data is overridden before it reaches you.
- This pattern prevents the most common failure mode of single-pass LLM analysis: confident but wrong answers.
Features using this pattern: Root Cause Analysis, Auto-Bug Generator, Smart Test Optimizer, Spec-to-Test Generation (4-stage pipeline).
The Automation Planner uses a different two-stage architecture: Stage 1 sends only metadata (titles, suites, tags) for risk stratification; Stage 2 sends full step data only for the tests the user explicitly selected. This bounds token cost regardless of corpus size.