Axon LLM Router
An in-process, SDK-based multi-tier inference LLM router. Get frontier-level quality on critical requests and fast, cost-effective execution on everyday tasks without manual model switching.
Installation
Install the Axon SDK into your project using your package manager of choice:
npm install axon-llmrouter60-Second Quickstart
Configure your model tiers, set a fallback tier, and run your first prompt through the dual-stage inference pipeline:
import { Axon } from "axon-llmrouter";
// 1. Initialize Axon with model tiers
const axon = new Axon({
tiers: {
frontier: { model: "gpt-4o", apiKey: process.env.OPENAI_API_KEY! },
balanced: { model: "claude-3-5-sonnet-20241022", apiKey: process.env.ANTHROPIC_API_KEY! },
fast: { model: "claude-3-5-haiku-20241022", apiKey: process.env.ANTHROPIC_API_KEY! },
},
// Optional: Judge model evaluates prompts that pass heuristics.
// Defaults to the fast tier model if omitted.
judge: { model: "gemini-1.5-flash", apiKey: process.env.GEMINI_API_KEY! },
fallbackTier: "balanced",
});
// 2. Dispatch prompt through the router
const result = await axon.infer("Summarize the main changes in this pull request", {
context: {
codeContext: "git diff...",
},
});
console.log("Response:", result.response);
console.log("Allocated Tier:", result.tier); // "frontier" | "balanced" | "fast"
console.log("Cost Saved:", result.costSaved); // e.g. 0.012 ($)
console.log("Latency Saved (ms):", result.latencySaved);High-Level Architecture Flow
Axon uses a two-stage routing pipeline designed for maximum efficiency: Gate (deterministic heuristics allowlist) and Judge (rubric-based LLM classification).
User prompt enters axon.infer() with optional conversation context, code context, and metadata.
Narrow allowlist & broad denylist. Confidently trivial prompts (typos, lookups, ≤20 words) bypass LLM evaluation and route directly to Fast tier.
Escalated prompts are rated along 4 independent categorical axes: Blast Radius, Irreversibility, Reasoning Depth, and Ambiguity.
The prompt is dispatched to the resolved tier's model. Returns model completion, tier metadata, and computed cost/latency savings.
Architecture & System Overview
Axon is built from the ground up for production engineering workflows. Here is a summary of the core pillars documented throughout this site:
3-Tier Model System
Frontier, Balanced, and Fast tiers. Severity is a strict hard-floor; cost and latency savings are maximized without risking output quality on complex prompts.
Gate Layer (Heuristics)
Sub-millisecond keyword and length rules. Catches trivial tasks instantly while scope denylist words trigger immediate escalation to the Judge layer.
Judge Layer (Rubrics)
Rates Blast Radius, Irreversibility, Reasoning Depth, and Ambiguity. Irreversible tasks automatically trigger Frontier routing via a veto rule.
Failure Resilience & BYOK
Engineered for Bring-Your-Own-Key setups. Proactive and live health checks, automated fallback tier routing, and strict confirmation stops for irreversible requests.
Cost & Latency Baselines
Computes real savings against your chosen fallback baseline using a merged pricing map. Unknown models safely return null rather than throwing errors.
Provider Resolution & Eval
Native support for OpenAI, Anthropic, and Gemini, plus any OpenAI-compatible endpoint (Groq, Together, vLLM). Includes a 100–200 prompt CSV benchmark harness.