Reliability & Internals

Provider Resolution & Eval Harness

Axon is built to support native provider SDKs alongside any OpenAI-compatible endpoint. It also ships with a built-in offline evaluation harness to measure routing accuracy against human-audited prompt datasets.

Provider Resolution Sequence

Model-name string matching is fragile because open-source and third-party hosted models use inconsistent naming across different hosting providers. Axon resolves providers in a strict 3-step sequence:

  1. 1. Explicit config.provider Override:

    If specified (e.g. provider: "openai"), Axon uses that adapter directly and skips name inference.

  2. 2. config.baseURL Compatibility Routing:

    If a baseURL is provided without an explicit provider, Axon routes the model through the generic OpenAI-compatible adapter. This enables instant integration with Groq, Together AI, Fireworks, vLLM, DeepSeek, Qwen, and Ollama.

  3. 3. Name-Pattern Inference:

    Fallback for recognized naming standards: gpt-* (OpenAI), claude-* (Anthropic), and gemini-* (Google).

// Example: Hosting Qwen on Groq via OpenAI compatibility
const axon = new Axon({
  tiers: {
    frontier: {
      model: "gpt-4o",
      apiKey: process.env.OPENAI_API_KEY!,
    },
    balanced: {
      model: "qwen/qwen-2.5-coder-32b",
      apiKey: process.env.GROQ_API_KEY!,
      baseURL: "https://api.groq.com/openai/v1", // Auto-routes via OpenAI-compat adapter
    },
    fast: {
      model: "claude-3-5-haiku-20241022",
      apiKey: process.env.ANTHROPIC_API_KEY!,
    },
  },
  fallbackTier: "balanced",
});

Dev-Owned Model Selection

Axon maintains a strict architectural posture: Axon does not have opinions on your end-user UI.

  • Axon never presents model selection dialogs or end-user dashboards.
  • The developer's app resolves model names, keys, and base URLs before passing them into the Axon constructor.
  • If your application allows end-users to Bring Your Own Key, we recommend using pre-configured dropdowns where valid baseURL and provider options are paired with each model option to prevent invalid routing.

The Eval Harness

How do you prove that Axon's Gate and Judge pipeline routes prompts correctly? Axon includes an offline evaluation harness designed to test 100–200 hand-labeled prompts against the live classifier.

Primary Metric≥ 80% Match

Measures the tier-match rate between Axon's output and human ground truth across all 18 rubric condition scenarios.

Crash-ResistantResumable CSV

Results write row-by-row to disk. If an eval run is interrupted, subsequent runs skip already-completed rows.

Running the Benchmark

# 1. Provide Judge credentials in your environment
export AXON_JUDGE_MODEL="gemini-1.5-flash"
export AXON_JUDGE_API_KEY="AIzaSy..."

# 2. Build the package distribution
npm run build

# 3. Run the evaluation against your labeled prompt dataset
npm run eval -- --input eval/prompts.csv

The harness invokes the public classify() method on the built distribution and outputs:

  • Overall tier agreement percentage (exits with code 2 if match rate is under 80%).
  • Split metrics showing how many prompts were routed by the Gate vs. the Judge.
  • Per-axis alignment diagnostics on human-audited rows.
  • A sidecar .meta.json file recording model parameters and timestamp without leaking API keys.