v1.0.0TypeScript SDKProduction Ready

Axon LLM Router

An in-process, SDK-based multi-tier inference LLM router. Get frontier-level quality on critical requests and fast, cost-effective execution on everyday tasks without manual model switching.

npm package: axon-llmrouter

Installation

Install the Axon SDK into your project using your package manager of choice:

npm install axon-llmrouter

60-Second Quickstart

Configure your model tiers, set a fallback tier, and run your first prompt through the dual-stage inference pipeline:

import { Axon } from "axon-llmrouter";

// 1. Initialize Axon with model tiers
const axon = new Axon({
  tiers: {
    frontier: { model: "gpt-4o", apiKey: process.env.OPENAI_API_KEY! },
    balanced: { model: "claude-3-5-sonnet-20241022", apiKey: process.env.ANTHROPIC_API_KEY! },
    fast: { model: "claude-3-5-haiku-20241022", apiKey: process.env.ANTHROPIC_API_KEY! },
  },
  // Optional: Judge model evaluates prompts that pass heuristics.
  // Defaults to the fast tier model if omitted.
  judge: { model: "gemini-1.5-flash", apiKey: process.env.GEMINI_API_KEY! },
  fallbackTier: "balanced",
});

// 2. Dispatch prompt through the router
const result = await axon.infer("Summarize the main changes in this pull request", {
  context: {
    codeContext: "git diff...",
  },
});

console.log("Response:", result.response);
console.log("Allocated Tier:", result.tier); // "frontier" | "balanced" | "fast"
console.log("Cost Saved:", result.costSaved); // e.g. 0.012 ($)
console.log("Latency Saved (ms):", result.latencySaved);

High-Level Architecture Flow

Axon uses a two-stage routing pipeline designed for maximum efficiency: Gate (deterministic heuristics allowlist) and Judge (rubric-based LLM classification).

Prompt Execution PipelineIn-Process Routing
1 Incoming Prompt

User prompt enters axon.infer() with optional conversation context, code context, and metadata.

2 Stage 1: Gate0ms LLM Cost

Narrow allowlist & broad denylist. Confidently trivial prompts (typos, lookups, ≤20 words) bypass LLM evaluation and route directly to Fast tier.

3 Stage 2: JudgeCheap LLM

Escalated prompts are rated along 4 independent categorical axes: Blast Radius, Irreversibility, Reasoning Depth, and Ambiguity.

4 Model ExecutionFrontierBalancedFast

The prompt is dispatched to the resolved tier's model. Returns model completion, tier metadata, and computed cost/latency savings.

Architecture & System Overview

Axon is built from the ground up for production engineering workflows. Here is a summary of the core pillars documented throughout this site:

3-Tier Model System

Frontier, Balanced, and Fast tiers. Severity is a strict hard-floor; cost and latency savings are maximized without risking output quality on complex prompts.

Gate Layer (Heuristics)

Sub-millisecond keyword and length rules. Catches trivial tasks instantly while scope denylist words trigger immediate escalation to the Judge layer.

Judge Layer (Rubrics)

Rates Blast Radius, Irreversibility, Reasoning Depth, and Ambiguity. Irreversible tasks automatically trigger Frontier routing via a veto rule.

Failure Resilience & BYOK

Engineered for Bring-Your-Own-Key setups. Proactive and live health checks, automated fallback tier routing, and strict confirmation stops for irreversible requests.

Cost & Latency Baselines

Computes real savings against your chosen fallback baseline using a merged pricing map. Unknown models safely return null rather than throwing errors.

Provider Resolution & Eval

Native support for OpenAI, Anthropic, and Gemini, plus any OpenAI-compatible endpoint (Groq, Together, vLLM). Includes a 100–200 prompt CSV benchmark harness.