Circuits of LLM Classifiers

We combine LLM classifiers and decision models into adaptive query policies that best trade off classification accuracy against inference cost

Hadi Khalaf1,2* Sajani Vithana1* Kunal Talreja3 Haewon Jeong4,5 Viveck R. Cadambe3 Flavio Calmon1
1Harvard SEAS·2Kempner Institute, Harvard·3Georgia Tech·4UC Santa Barbara·5Flatiron Institute
arXiv Code
Circuit setting: Prioritize accuracy
P(harmful)=54%
Queries=0
Cost per 1M prompts=$0

Illustrative accuracies. Costs are list prices for classifying one million prompts.

Best viewed on a desktop browser.

LLMs are used for classification

Describe the task in a few sentences, add the text to classify, and the model returns a label. Researchers and companies now label data this way at scale.

Social science

Coding tweets and news articles. ChatGPT beat crowd workers on most tasks, at about a thirtieth of the cost.

Gilardi et al., 2023

Health

Grading answers against 48,562 criteria written by physicians.

HealthBench, 2025

Content moderation

Applying a written policy to user content.

OpenAI, 2023

Model evaluation

LLM judges that decide which of two answers is better.

Zheng et al., 2023

With hundreds of models to choose from, which ones should we use?

An LLM classifier is a noisy measurement instrument

Two models can give different answers to the same question, and so can one model asked twice. Every answer costs money.

We treat each model, with its prompt and settings, as an instrument: each call is a noisy measurement of the true label, at a price.

Illustrative answers. Gemma 4 E4B gives two different answers when asked twice, and the decision model Jev and Gemma 4 31B disagree. Costs are list prices for one million prompts.

Which LLMs should we ask, and in what order?When should we stop?

We call the combination of LLM classifiers a circuit

Combining classifiers is a classic problem in machine learning. Each LLM classifier runs on its own hardware, so we call their combination a circuit.

Majority voting, routing and cascades are circuits, and so is LMCircuit, the method of this paper. They differ in which models they ask, in what order, and how they turn the answers into one prediction.

Illustrative accuracies. Costs are list prices for one million prompts. All four circuits get the three example prompts right. They differ in what they spend.

A circuit decides which model to query next and when to stop

After each answer, the circuit either asks another model or stops and predicts. If we care about cost, asking one model at a time loses nothing.1

1 If we care about speed, asking several models in parallel is faster.

LMCircuit on the hard prompt, starting from its prompt classifier's guess. Illustrative accuracies; costs for one million prompts. At each step the circuit compares what each option costs with the chance of a mistake it leaves.

The best circuit is the solution of a dynamic program

Each answer changes what the next call is worth, so the choices form a tree. The dynamic program finds the best tree by working backward from the predictions.

It weighs the cost of one more answer against how much that answer lowers the chance of a mistake. Prioritize accuracy and the circuit asks more. Prioritize cost and it asks less.

Illustrative accuracies, costs for one million prompts, and no prompt classifier yet. Each stop on the slider gives a different best tree. The dashed path is the one the hard prompt takes, ending in its prediction.

Any circuit gets better with a little routing

Prompts differ in difficulty. Some are easy for every model and others split them, so one plan for every prompt wastes money.

LMCircuit adds a small prompt classifier. It reads the prompt and predicts the label and which models will get it right. It trains in minutes and runs on a CPU.

Illustrative accuracies; costs for one million prompts. On the easy prompt the circuit asks no model. On the hard one it asks only Gemma 4 31B, the model most likely to be right.

LMCircuit is a prompt classifier and a dynamic program

We fit the prompt classifier once. For any setting, from prioritize cost to prioritize accuracy, the dynamic program turns its estimates into a circuit. LMCircuit offers continuous control over how to weigh accuracy vs. cost, tracing the entire frontier.

The prompt classifier reads a text embedding of the prompt (Qwen3-Embedding-0.6B) and the answers collected so far. It returns the chance the prompt is harmful and the chance each model gets it right. It is trained on labelled prompts and recorded model answers.

The prompt classifier is fitted once. Changing the setting only reruns the dynamic program. Illustrative numbers; costs are list prices for one million prompts.

LMCircuit achieves higher accuracy at nearly every budget

It is more accurate than every single model and than eight routing and cascading methods. Here are five of the seven tasks: four judged by open-weight models and one by decision models.

Values read from Figure 4 of the paper, which reports balanced accuracy (the average of the accuracy on each class). Bars are 95% confidence intervals. The three markers show how many calls LMCircuit makes per item, and to which models, at a low, a medium and a high budget.
All seven tasks (PDF)

LMCircuit trains in 3 minutes or less on every task. Several of the baselines take hours.

Time to build each method
Time to construct each method on the seven tasks. LMCircuit fits in 10 seconds to 3 minutes; FrugalGPT and Cascade Routing take 2 to 7 hours.

LMCircuit's decisions are calibrated

When LMCircuit is 70% sure of a label, it is right about 70% of the time.

LMCircuit returns a belief: the probability that the label is 1. We group held-out prompts into ten bins by that belief, and each point compares a bin's average belief with how often its label is truly 1. Calibration error is the average distance from the diagonal, weighted by the number of prompts in each bin. LMCircuit is shown at a budget of 0.1; its calibration error is about the same at every budget from 0.01 to 1.

We are going through a revolution in classification.

Hundreds of models can now label text on demand, at prices that differ more than a thousandfold. Used together, a few of them are more accurate than any one of them at the same cost.

We show that the best circuit leans on the model that gives the most information per dollar.

Authors

Hadi Khalaf

Hadi Khalaf*

Harvard SEASKempner Institute
Sajani Vithana

Sajani Vithana*

Harvard SEAS
Kunal Talreja

Kunal Talreja

Georgia Institute of Technology
Haewon Jeong

Haewon Jeong

UC Santa BarbaraFlatiron Institute
Viveck R. Cadambe

Viveck R. Cadambe

Georgia Institute of Technology
Flavio Calmon

Flavio Calmon

Harvard SEAS

* Equal contribution.

Contact

Research inquiries
Questions about the method or the results
Collaboration
Interested in working together
Code and data
Measurements, training and every baseline
Problems with this site
sajani@seas.harvard.edu

Circuits of LLM Classifiers

Abstract. Language models make it easy to build classifiers by simply prompting a model with the sample to classify. However, their predictions are neither deterministic nor uniformly reliable across samples. How can we combine predictions from different models to improve reliability? We view LLM classifiers as noisy measurement instruments of an underlying target label. Our goal is to maximize average classification accuracy at a given average query cost. To this end, we introduce circuits that organize LLM classifiers into an adaptive sequence of queries and decisions. We then characterize the optimal circuit, obtain the fundamental limits of achievable accuracy, and provide practical insights on realizing the optimal circuit. Across seven classification tasks spanning safety, health, political science, human preference, agent evaluation, decision-making and reasoning, our method improves the cost–accuracy frontier over routing and cascading baselines.

Hadi Khalaf*, Sajani Vithana*, Kunal Talreja, Haewon Jeong, Viveck R. Cadambe, Flavio Calmon. *Equal contribution. arXiv:2610.30076, October 2026.

PDF arXiv Code

Cite

@article{khalaf2026circuits,
  title   = {Circuits of {LLM} Classifiers},
  author  = {Khalaf, Hadi and Vithana, Sajani and Talreja, Kunal and Jeong, Haewon and Cadambe, Viveck R. and Calmon, Flavio},
  journal = {arXiv preprint arXiv:2610.30076},
  year    = {2026}
}

Technical details

For a prompt z, a state s is the set of (instrument, answer) pairs observed so far. The belief is qz(s) = P(Y = 1 | z, s) and the stopping error is ez(s) = min{qz(s), 1 − qz(s)}. With a price λ on cost, the value of a state satisfies

Vλ,z(s) = min { ez(s) ,  mini ∉ M(s) [ λci(z) + Σx ∈ {0,1} Q[i](x | z) · Vλ,z(s ∪ {(i, x)}) ] }
stop and predict   vs.   pay for instrument i, then continue from the answer
01

Sequential without loss. Any circuit that queries a batch of instruments at once has a sequential counterpart with the same error and cost, so the search can be restricted to one query at a time.

02

Conditional independence. Each call uses fresh randomness, so we model answers as independent given the prompt. Errors can still be correlated across prompts, and the embedding captures this. Without seeing test labels, the fitted model predicts the correlations between instruments' answers, and between their errors, to within 4% of the measured values on six tasks (15–17% low on BIG-Bench Hard, the smallest). It predicts each instrument's accuracy to within 1.2 to 3.6 points.

03

Belief model (the prompt classifier). fθ(φ(z), s) = (ηθ(φ(z), s), νθ(1)(φ(z), s), …, νθ(n)(φ(z), s)). A learned projection of a 1024-dimensional Qwen3-Embedding-0.6B vector to 64 dimensions feeds three small heads. The model assumes each prompt belongs to one of T latent types: a type head gives the prior over types, a measurement head gives each instrument's chance of answering 1 under each type, and a label head gives the label probability under each type at state s, summing learned embeddings of the observed (instrument, answer) pairs so it is invariant to their order. The observed answers update the type by Bayes' rule and the outputs average over types. T ∈ {1, 2, 4, 8} is chosen on validation.

04

Loss. For each training prompt we draw a random state (a random subset of instruments and one recorded answer each) and minimize binary cross-entropy of η against the recorded labels, weighted by annotator counts where several annotators labeled a prompt (as on HealthBench), plus cross-entropy of each unqueried νi against every recorded call of instrument i. AdamW with early stopping on validation; one temperature for the label and one for the instruments' answers is then fit on validation states; five seeds per fold.

05

Costs. ci(z) is known before the query. On the four tasks we query ourselves it is the input tokens of z times the input price, plus the 95th percentile of the instrument's output length on training prompts times the output price. On AgentJudgeBench, BIG-Bench Hard and Decision Index it is the instrument's mean cost on training prompts, priced as in the data sheet (these benchmarks recorded no call costs). Costs are divided by the mean cost of the most expensive instrument.

06

Frontier. The dynamic program is solved for 201 prices spaced geometrically from 10−8 to 102, plus λ = 0 and λ = ∞. Each dataset is split into five folds, keeping prompts that share a source together. For each budget the price with the best validation balanced accuracy within budget is selected and its test balanced accuracy reported.

07

Pool selection. The drop in the lowest achievable error from adding an instrument is at most √(I(Y; Xnew | Xpool, Z)/2). Starting from a cheap instrument and greedily adding the one with the largest conditional information reaches the full-pool frontier with 2–3 instruments on the four tasks where we query the models ourselves, and dominates accuracy-first, cheapest-first and random orderings.

08

Scaling laws. For one prompt, the error of the best circuit falls exponentially with the amount spent, −log Eλ(z) / Cλ(z) → I(z), where I(z) is the largest information per unit cost among the models (the KL divergence between a model’s answers under the two labels, divided by its cost). The circuit then spends almost all of its budget on that model. Averaged over prompts, the error falls more slowly when many prompts are hard for every model.

Datasets and models

DatasetQuestion an instrument answersPromptsInstruments
AgentJudgeBenchDo an agent's planned tool calls meet a criterion (for example, the right tools or full coverage of the request)?20,850Prometheus-2-7B, gpt-oss-20b, gpt-oss-120b, QwQ-32B, Gemini-2.5-Pro, GPT-5.4, Claude Sonnet 4.5 (7, recorded)
BeaverTailsIs a model response unsafe?30,200Gemma 4 E2B, E4B, 26B-A4B, 31B; Qwen3 4B, 8B, 32B (7)
BIG-Bench HardWhich of two options is correct? (8 reasoning subtasks)1,86546 recorded models, 0.36B–123B; in each fold, the 8 whose cost–accuracy frontier on the training prompts comes closest to that of all 46 (9 different models across the five folds)
Decision IndexSix binary decision tasks (clinical-trial NLI, causal QA, sarcasm, multi-hop claims, humour, limericks)24,92818 leaderboard entrants, 0.4B–36B; 8 per fold, chosen the same way (10 different models across the five folds)
HealthBenchDoes the last response meet a physician-written rubric criterion? (target: share of physicians who say yes)29,131Granite 4.2 3B (reasoning off, low, default); gpt-oss-20b (low, medium, high); gpt-oss-120b (low, medium) (8)
LMArenaWhich of two responses is preferred?34,952Gemma 4, Qwen3 (7)
PolitiCAUSEDoes a sentence state a causal relation?17,767Gemma 4, Qwen3 (7)

Seven binary classification tasks. On BeaverTails, HealthBench, LMArena and PolitiCAUSE we query the instruments ourselves with each dataset's official instruction and the model's default decoding; the Gemma models answer directly and the Qwen3 models reason before answering. On AgentJudgeBench, BIG-Bench Hard and Decision Index we use the answers each benchmark recorded, priced at the model's API price (AgentJudgeBench) or at $0.01 per million tokens per billion active parameters (BIG-Bench Hard, Decision Index). Each dataset is split into five folds, keeping prompts that share a source together; each fold serves once as the test set.

$ / M tokensGemma 4 E2BGemma 4 E4BGemma 4 26BGemma 4 31BQwen3 4BQwen3 8BQwen3 32BGranite 4.2 3Bgpt-oss-20bgpt-oss-120b
Input0.040.020.0420.090.030.200.080.030.0180.03
Output0.080.100.220.340.030.200.280.120.090.17

Price of the instruments we query ourselves in USD per million tokens, from the cheapest listed provider on 24 September 2026 (OpenRouter, AWS Bedrock or Fireworks). The examples on this page use the Gemma 4 prices.

Baselines

Predictive routers pick one model per prompt. Cascades ask models one after another until a confidence rule fires. Sampling asks one model again and again until its answers agree. All are circuits. Routers and cascades over the same models are among the circuits the dynamic program compares.

  1. RouteLLM. Ong et al. Learning to route LLMs from preference data. ICLR 2025 (router)
  2. HybridLLM. Ding et al. Cost-efficient and quality-aware query routing. ICLR 2024 (router)
  3. CARROT. Somerstep et al. A cost aware rate optimal router. arXiv:2502.03261, 2025 (router)
  4. Cascade Routing. Dekoninck, Baader and Vechev. A unified approach to routing and cascading for LLMs. ICML 2025 (cascade)
  5. Consistency Cascades. Yue et al. LLM cascades with mixture of thought representations. ICLR 2024 (cascade)
  6. Cascaded Selective. Jung, Brahman and Choi. Trust or escalate: LLM judges with provable guarantees for human agreement. ICLR 2025 (cascade)
  7. FrugalGPT. Chen, Zaharia and Zou. How to use LLMs while reducing cost and improving performance. TMLR 2024 (cascade)
  8. Adaptive-Consistency. Aggarwal et al. Let's sample step by step. EMNLP 2023 (sampling)