The $0 Baseline: Before You Compare Models, Test No Model

The current debate about AI cost often begins too late.

Teams compare models, token prices, latency and cost per successful outcome after already assuming that an LLM belongs in the solution. A problem is labelled an “AI use case,” and the architecture discussion starts with model selection.

Which model is most accurate? Which model is cheapest? Should we use one large model or a coordinated group of smaller ones?

Those can be useful questions. But they belong at question three. The first questions should be:

1. What is the problem?
2. What class of mechanism can solve it?
3. If that class includes LLMs, which model provides the best outcome for the cost and operational risk?

If every arm in your experiment contains an LLM, you have not tested whether the LLM adds value. You have only tested which LLM implementation you prefer.

This is not an argument against LLMs. I use them extensively, and I spend much of my independent research time building and evaluating systems that depend on them.

It is an argument for making them earn their place.

A note on scope

This argument is primarily about operational decision systems: allocation, routing, forecasting, recovery, workflow automation and agentic systems that observe state and take actions.

Coding is an important exception.

Software generation is open-ended synthesis rather than selection from a small, legible action space. LLMs can provide substantial value by proposing implementations, explaining unfamiliar code, generating tests and accelerating debugging. The surrounding software-development ecosystem also provides unusually strong verification mechanisms: compilers, type systems, tests, static analysis, code review and controlled deployment pipelines.

Coding therefore reinforces rather than contradicts the architectural principle in this article:

The model proposes. Deterministic tools and accountable humans verify before the result becomes authoritative.

The solution class is being selected too early

Executive pressure to demonstrate AI adoption can unintentionally turn a technology into a requirement before the problem has been properly diagnosed.

“Find opportunities to use AI” becomes “solve this with AI.”

By the time engineers receive the problem, the most consequential architectural decision has already been made. The remaining discussion is about providers, prompts, context windows and token budgets.

That creates a failure of experimental discipline: teams optimise within a preselected solution class instead of testing whether that class was appropriate.

The question becomes:

Which LLM should make this decision?

Rather than:

Does this decision need an LLM at all?

The distinction matters more as systems move from occasional experiments into production workflows. At low volume, an unnecessary model call may look harmless. At enterprise scale, it becomes a permanent cost, latency and control burden embedded in the architecture.

Before optimising model cost, establish the cost and performance of the best credible alternative.

That alternative may be a rule, a hundred lines of Python, a classical machine-learning model, a search algorithm, a mathematical optimiser or a simple human workflow.

I call this the $0 baseline.

“$0” means zero model-inference or external API spend. It does not mean zero compute, engineering effort or maintenance.

What my experiments kept showing me

I run two independent research projects involving LLM agents inside simulated environments.

In Aftershock, teams coordinate disaster response in a procedurally generated town.

In Redoubtia Arena, agents operate simulated organisations. They allocate resources, respond to incidents, hire, fire, manage budgets and sometimes drive their organisations into insolvency.

The projects are different, but both repeatedly raised the same uncomfortable question:

What did the LLM actually add?

The most consistent finding so far has been that a well-designed cheap solver (a scripted heuristic, greedy policy or linear extrapolation) has been extremely difficult for an LLM to beat.

Not only for small models. For frontier models too.

Aftershock: the scripted responders tied the LLMs

Aftershock benchmarks four approaches on identical seeded disaster scenarios:

• solo: one large model coordinates the whole response
• swarm: multiple LLM agents act without a coordination mechanism
• society: LLM agents operate under a negotiation protocol and written doctrine
• scripted: deterministic heuristic agents use the same coordination structure

The scripted arm saved an average of 106.8 lives per run. The best LLM arm saved 108.4. The difference was statistically indistinguishable in the tested sample.

The uncoordinated LLM approaches performed worse:

• Solo: 95.6
• Swarm: 93.8

The scripted arm incurred no model-inference cost.

The result was not that LLM agents were useless. It was that the strongest effect came from the coordination mechanism, operating doctrine and action contract, not from probabilistic reasoning.

The structure saved lives. Once that structure existed, simple encoded doctrine rivalled the LLM agents on the measured outcome.

The full tables and run records are available in the public Aftershock field notes.

A 10× increase in model spend changed almost nothing

I then replaced all six agent roles with progressively more expensive model tiers while holding the world seeds, protocol, doctrine and action contracts fixed.

A roughly 9.7× increase in model cost produced no meaningful improvement in outcomes.

Above a minimum capability floor, the harness appeared to set the ceiling. Better doctrine, contracts and coordination mattered more than a more expensive model.

This does not mean model capability never matters. Below the floor, it matters enormously. In separate testing, a very small local model failed to sustain the protocol, while a larger model carried it successfully.

The lesson is not “always use the cheapest model.” It is:

Once the model is capable enough to operate inside the system, demonstrate that additional model capability changes the outcome before paying for it.

The cross-model exception matters

I also tested twelve models from several families as single-model coordinators.

No model produced a statistically significant outcome improvement over the cheap coordinated society in the tested sample. Most frontier models tied the outcome at substantially higher inference cost.

But there was an important exception.

One cheap frontier model tied the outcome while being roughly four times more cost-efficient than the coordinated society.

That result matters because it prevents the wrong conclusion.

These experiments did not prove that scripted systems always beat LLMs, or that multi-agent coordination always wins. They showed that the correct comparator is the strongest credible alternative, not merely another expensive model.

Sometimes the LLM will earn the seat. The point is to run the test.

Arena: the greedy policy nobody beat

Redoubtia Arena includes an incident-recovery gym.

A simulated organisation’s infrastructure is disrupted, and an agent must sequence recovery actions. The deterministic comparator is deliberately simple:

Repair the most critical recoverable component first, then repeat.

It is the kind of greedy policy that could be written in an afternoon.

Methodology

The current Arena evidence comes from:

• a procedurally generated incident-recovery environment
• identical seeded worlds across comparison arms
• a multi-model sweep that included frontier-tier systems
• multiple scaffolding configurations
• a greedy scripted recovery policy as the control
• recovery performance measured separately from cost
• no real organisations, production environments or company data

In my testing so far, across every model and configuration in the sweep, nothing beat the greedy scripted policy.

The strongest and most reliable models generally matched it. Weaker models lost to it. Erratic models sometimes selected invalid or poor recovery sequences.

I then introduced look-ahead scaffolding: before committing an action, the agent could preview likely consequences.

Look-ahead helped only the weaker and more erratic models. It did not lift any of the reliable models above the greedy control.

It acted as a safety net, not a ladder.

The important finding is not that the greedy policy is universally optimal. The environment is deliberately constrained, and the result applies to the tested mission and action space.

The finding is methodological:

Without the scripted comparator, the stronger models would have looked highly capable. With the comparator, they were shown to be matching a policy that cost almost nothing to execute.

The underlying Arena run records and analysis are retained within the research project. The methodology and aggregate findings are summarised here so the claim can be evaluated without implying that private evidence should simply be accepted on trust.

The same pattern appeared in forecasting

Arena also produces economic and operational time-series data.

I evaluated a purpose-built time-series foundation model as a possible early-warning system. A simple linear drift extrapolation produced dramatically lower forecast error on this specific data shape, while the foundation model’s uncertainty intervals were badly miscalibrated.

The near-optimal operational signal was already available through a runway calculation.

Division.

Again, this does not establish that linear methods are universally superior. It establishes that a specialised model still needs to beat the simple baseline on the actual data distribution where it will operate.

The same baseline-first pattern has also appeared across agent benchmarks, code-evolution systems, tabular learning, forecasting and formal planning research. The supporting papers are included in the endnotes rather than reviewed individually here.

The consistent lesson is not that simple methods always win.

It is that complexity does not exempt a system from comparison.

Agent count is not a capability metric

The same discipline should apply to the number of agents in a system.

Large agent swarms are increasingly presented as evidence of architectural sophistication: ten agents, fifty agents, hundreds of specialised roles. But agent count is an implementation detail, not an outcome.

Every additional agent introduces potential value, but also additional inference cost, duplicated context, latency, coordination overhead, permissions, failure modes and observability burden.

A larger swarm is justified when interaction creates something the simpler system cannot produce: useful specialisation, genuine parallelism, independent verification, adversarial challenge or better decisions under contention.

Otherwise, it may simply be distributing one task across more expensive moving parts.

The relevant experiment is an ablation:

Remove an agent, role or coordination layer. Does the measured outcome get worse?

If not, the component may be ornamental.

Each additional agent may also introduce another identity, permission set, tool surface and path for sensitive information to propagate.

A swarm is not an achievement. A result the simpler system could not produce is an achievement.

Security is part of the baseline

The cost of an LLM is not limited to tokens, latency and compute.

Every model introduced into an architecture may also create a new trust boundary, data flow, identity, dependency and failure mode. If it can use tools or modify state, it may create a new privileged actor inside the system.

That security cost should be included in the comparison with the non-LLM baseline.

A deterministic alternative can still contain vulnerabilities. The relevant distinction is not “deterministic equals secure.” It is that probabilistic components introduce additional risks and control requirements:

• untrusted content may influence model behaviour
• instructions and data can become difficult to separate
• sensitive context may be copied across prompts, agents or providers
• generated actions may be valid in structure but unsafe in meaning
• model and provider behaviour may change outside the application release cycle
• tool-enabled agents can turn a reasoning error into a real state change
• stochastic execution can make incidents harder to reproduce and investigate

Multi-agent systems compound this burden. Each additional agent may require an identity, permissions, tool access, context, logs and an explicit trust relationship with other agents.

An agent count of fifty may therefore represent fifty specialised capabilities.

Or it may represent fifty opportunities for excessive privilege, data leakage, inconsistent policy enforcement and confused authority.

Before introducing a model or agent, threat-model the authority it will receive:

1. What information can it access?
2. Which instructions or external content can influence it?
3. What tools can it invoke?
4. What state can it change?
5. What is the maximum blast radius of an incorrect action?
6. Can the decision be reconstructed from retained evidence?
7. Which deterministic control can reject, constrain or reverse the action?

The architectural principle is to separate capability from authority.

An LLM may be highly capable at interpreting language, synthesising information and proposing actions without being authorised to commit those actions directly.

The model proposes. Deterministic controls authenticate, authorise, validate and commit.

This is not merely a safety pattern. It is a security boundary.

A model should therefore earn its place twice:

First, by outperforming the cheapest credible baseline.

Second, by creating enough value to justify the additional trust and control burden it introduces.

The cheapest model per successful outcome may still be the most expensive architecture if it creates an unnecessary privileged trust boundary.

Before you add an LLM, ask six questions

1. Is the action space legible?

Can you enumerate the legal actions and their preconditions?

If the task is allocation, routing, scheduling, recovery or triage, build the heuristic or formal optimiser first.

A closed action space does not guarantee that the optimal policy is simple. But it often makes a strong baseline cheap to construct.

2. Have you built the $0 baseline?

If every comparison arm contains a model, you have not measured model value.

The baseline may be:

• a deterministic rule
• a greedy policy
• a lookup table
• regression
• gradient-boosted trees
• dynamic programming
• mixed-integer optimisation
• a human decision with lightweight automation

The baseline is not necessarily the final solution. It is the incumbent the model must beat.

3. How often will this run?

Inference cost is structural.

A model invoked three times a week is a different architectural decision from a model invoked millions of times inside a hot production loop.

In Aftershock, approximately 85% of prompt tokens were repeated static prefixes. Switching to native function calling doubled the cost in one experiment while producing statistically equivalent outcomes.

At scale, small per-call inefficiencies become permanent taxes.

4. Do you require determinism, replayability or auditability?

Hosted LLM paths are often not exactly replayable end to end. Temperature zero and sampling seeds do not guarantee identical behaviour, and some providers do not fully honour supplied seeds.

That does not mean LLMs cannot appear in auditable systems. It means the authoritative state changes should be controlled.

Ask:

Which decisions are we allowing a probabilistic component to make, and what evidence shows that the uncertainty buys us something?

5. Will the output feed a machine or a human?

Schema-valid is not the same as true.

An LLM can convert contradictory data into fluent, internally coherent and incorrect output.

In Aftershock, an after-action analyst once confidently reported that zero lives had been saved because of a faulty input field, while the correct total appeared elsewhere in the supplied data.

For human readers, prose can be useful even when it is probabilistic.

For machine-consumed outputs, use typed contracts, validators, explicit rejection reasons and safe fallbacks.

Fluent prose is not a control.

6. What authority and blast radius will the model have?

Can it be constrained, observed, investigated and safely overridden?

If a model can modify production state, initiate financial activity, change access, affect safety controls or create a material obligation, it should operate inside a constrained and observable decision loop.

Every probabilistic component increases the system’s validation, monitoring and incident-response burden.

That burden is justified when the model adds capability. It is waste when deterministic code delivers the same outcome.

Where probabilistic systems belong

The alternative to “LLMs everywhere” is not “no LLMs.”

It is deliberate placement.

I currently think about the architecture in three zones.

The probabilistic boundary

This is where LLMs are often unusually valuable:

• natural-language intake
• information synthesis grounded in source data
• summarisation
• explanation and prose
• translating human intent into candidate structured actions
• generating hypotheses and options
• handling novel or ambiguous situations
• identifying possible rules that can later be tested

These outputs may be creative, interpretive or probabilistic. They should not automatically be authoritative.

The verified bridge

The boundary communicates with the core through controls such as:

• typed contracts
• schema validation
• deterministic precondition checks
• policy enforcement
• named rejection reasons
• confidence and provenance metadata
• approval thresholds
• safe fallback paths

The LLM proposes. The bridge verifies.

The deterministic core

The authoritative core should generally retain control over:

• state transitions
• permissions
• policy decisions
• calculations
• financial ledgers
• resource constraints
• safety invariants
• audit logging

This does not mean the core can never contain probabilistic systems. It means:

Prose and information synthesis can remain probabilistic, but authentication, authorisation and state mutation should remain deterministic by default.

And more generally:

The authoritative path should not depend on probabilistic output unless that dependency measurably improves the outcome.

When an LLM is the right tool

There are several places where LLMs have earned their role in my own work.

Coding and software development

Coding is one of the strongest current use cases for LLMs.

The task combines natural-language intent, large and ambiguous solution spaces, pattern synthesis and iterative refinement. A useful output does not need to be immediately authoritative: it can be compiled, tested, reviewed and rejected before it reaches production.

That verification environment makes coding particularly well suited to probabilistic generation.

The relevant baseline is not usually “could a script write this code?” It is developer time, accepted changes, defect rates and the quality of the resulting system.

Language in, language out

Turning messy human requests into structured candidates, and structured system state into understandable prose, remains one of the clearest use cases.

Information synthesis

LLMs can synthesise large amounts of real source data into explanations, comparisons and summaries.

The data should remain the source of truth. The generated prose should remain an interpretation of that data.

The illegible middle

When the action space cannot be fully enumerated, the task is novel, or the rules are not yet known, an LLM’s learned prior may be the best available policy.

In Aftershock, unexpected events introduced during a run were among the areas where LLM flexibility became more useful.

Rule discovery, followed by distillation

An LLM can propose rules, configurations or candidate policies. A deterministic loop can then evaluate them.

Once a reliable rule has been discovered, the production system may not need to keep paying an LLM to rediscover it.

Use the model to find the cheap solver. Then consider shipping the solver.

Proposals inside validated loops

A strong production pattern is:

Model proposes → deterministic system validates → authorised component commits.

Every mechanically enforced contract removes a class of failure. The remaining semantic failures become easier to observe and investigate.

Low-frequency, high-ambiguity work

One-off analysis, drafting, summarising an unusual incident or exploring an unfamiliar domain may justify the per-call cost.

Frequency changes the economics.

Safety support for weaker models

Scaffolding, look-ahead and verification can improve the weak tail when a smaller model is required for cost, latency or deployment reasons.

But scaffolding should also be benchmarked. It is not automatically beneficial, and it may add cost without raising the ceiling.

Importance is not an argument for an LLM

A common intuition is:

This decision is important, so we should use the most capable model available.

Importance should produce the opposite instinct.

An important decision deserves:

• a strong baseline
• explicit constraints
• replayable evidence
• validation
• clear authority boundaries
• tested failure paths

A model may still be part of the answer.

But importance alone does not establish that probabilistic reasoning should sit in the authoritative path.

The meta-rule

If I compress the argument into one sentence:

The LLM should be the component you add after establishing the baseline, not the component whose necessity you assume before the experiment begins.

In both Aftershock and Arena, this principle became executable through:

• identical seeded worlds across comparison arms
• a scripted control in every benchmark
• cost reported beside the outcome
• paired comparisons that cancel environmental variance
• explicit capability floors
• retained run records
• exceptions reported alongside supporting results

In Aftershock, the world seed explained approximately 79% of the run-to-run variance. Unpaired comparisons could easily have produced confident but misleading conclusions.

This is not exotic methodology. It is ordinary experimental hygiene applied to systems that are often evaluated through persuasive demos rather than controlled comparisons.

I suspect a surprising share of today’s agent workloads would fail the first two questions in the checklist. They are allocation, routing, triage and recovery problems with relatively legible action spaces, wrapped in LLMs because the first demo was easier to build than the control.

The cheap solver is not the fallback. It is the incumbent.

Make the LLM earn the seat.

---

What is the cheapest credible baseline your LLM system has been benchmarked against?

And if the answer is “none,” what would the non-LLM version look like?

---

This article describes independent personal research. Every organisation and scenario used in the work is procedurally generated; no real company data is used. Aftershock’s run records and field notes are publicly available on GitHub.

Endnotes and supporting research

Several of the works below are recent arXiv preprints and have not been peer-reviewed; they are cited as convergent evidence for the baseline-first pattern, not as settled results. Citation metadata verified 2026-07-11 via OpenAlex / Crossref / Semantic Scholar.

1. Agent evaluation and cost: Kapoor, Stroebl, Siegel, Nadgir and Narayanan, AI Agents That Matter (Transactions on Machine Learning Research, 2024; cited_by ≈ 155, Semantic Scholar). The paper argues for jointly evaluating accuracy and cost, and demonstrates that simple agent baselines can dominate more elaborate architectures.
2. Code-evolution systems: Gideoni, Risi and Gal, Simple Baselines are Competitive with Code Evolution (arXiv preprint 2026, unrefereed). Repeated sampling and straightforward search were competitive with more elaborate code-evolution pipelines across several task classes. This challenges elaborate autonomous optimisation scaffolds; it does not imply that everyday coding assistants provide no developer-productivity benefit.
3. Agentic search: Zou et al., FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics (arXiv preprint 2026, unrefereed). A greedy hill-climbing approach approached the performance of substantially more complex tree-search systems.
4. LLMs as rule proposers: Li, Greedy Is a Strong Default: Agents as Iterative Optimizers (arXiv preprint 2026, unrefereed). Relevant to the pattern of using an LLM to propose candidate rules that are then evaluated by a classical loop.
5. Formal optimisation versus agent reasoning: Wang et al., AstroReason-Bench: Evaluating Unified Agentic Planning across Heterogeneous Space Planning Problems (arXiv preprint 2026, unrefereed). LLM agents improved over naive heuristics but remained behind specialised optimisation and reinforcement-learning approaches.
6. Tabular data: Grinsztajn, Oyallon and Varoquaux, Why Do Tree-Based Models Still Outperform Deep Learning on Typical Tabular Data? (NeurIPS 2022 Datasets and Benchmarks; cited_by ≈ 319, OpenAlex).
7. Forecasting baselines: Linear Regression as a Litmus Test for Time Series Forecasting Benchmarks (OpenReview; not indexed in OpenAlex / Crossref / Semantic Scholar at the time of writing, so refereeing status and citation count are unverified), and Toner et al., Performance of Zero-Shot Time Series Foundation Models on Cloud Data (arXiv preprint 2025, unrefereed).
8. Aftershock experimental records: Aftershock field notes, particularly the scripted-control benchmark, model-tier sweep, capability-floor study, cross-family panel and cost analysis.