Provider Resolution & Eval Harness
Axon is built to support native provider SDKs alongside any OpenAI-compatible endpoint. It also ships with a built-in offline evaluation harness to measure routing accuracy against human-audited prompt datasets.
Provider Resolution Sequence
Model-name string matching is fragile because open-source and third-party hosted models use inconsistent naming across different hosting providers. Axon resolves providers in a strict 3-step sequence:
- 1. Explicit
config.providerOverride:If specified (e.g.
provider: "openai"), Axon uses that adapter directly and skips name inference. - 2.
config.baseURLCompatibility Routing:If a
baseURLis provided without an explicit provider, Axon routes the model through the generic OpenAI-compatible adapter. This enables instant integration with Groq, Together AI, Fireworks, vLLM, DeepSeek, Qwen, and Ollama. - 3. Name-Pattern Inference:
Fallback for recognized naming standards:
gpt-*(OpenAI),claude-*(Anthropic), andgemini-*(Google).
// Example: Hosting Qwen on Groq via OpenAI compatibility
const axon = new Axon({
tiers: {
frontier: {
model: "gpt-4o",
apiKey: process.env.OPENAI_API_KEY!,
},
balanced: {
model: "qwen/qwen-2.5-coder-32b",
apiKey: process.env.GROQ_API_KEY!,
baseURL: "https://api.groq.com/openai/v1", // Auto-routes via OpenAI-compat adapter
},
fast: {
model: "claude-3-5-haiku-20241022",
apiKey: process.env.ANTHROPIC_API_KEY!,
},
},
fallbackTier: "balanced",
});Dev-Owned Model Selection
Axon maintains a strict architectural posture: Axon does not have opinions on your end-user UI.
- Axon never presents model selection dialogs or end-user dashboards.
- The developer's app resolves model names, keys, and base URLs before passing them into the
Axonconstructor. - If your application allows end-users to Bring Your Own Key, we recommend using pre-configured dropdowns where valid
baseURLand provider options are paired with each model option to prevent invalid routing.
The Eval Harness
How do you prove that Axon's Gate and Judge pipeline routes prompts correctly? Axon includes an offline evaluation harness designed to test 100–200 hand-labeled prompts against the live classifier.
Measures the tier-match rate between Axon's output and human ground truth across all 18 rubric condition scenarios.
Results write row-by-row to disk. If an eval run is interrupted, subsequent runs skip already-completed rows.
Running the Benchmark
# 1. Provide Judge credentials in your environment
export AXON_JUDGE_MODEL="gemini-1.5-flash"
export AXON_JUDGE_API_KEY="AIzaSy..."
# 2. Build the package distribution
npm run build
# 3. Run the evaluation against your labeled prompt dataset
npm run eval -- --input eval/prompts.csvThe harness invokes the public classify() method on the built distribution and outputs:
- Overall tier agreement percentage (exits with code
2if match rate is under 80%). - Split metrics showing how many prompts were routed by the Gate vs. the Judge.
- Per-axis alignment diagnostics on human-audited rows.
- A sidecar
.meta.jsonfile recording model parameters and timestamp without leaking API keys.