The Shell Tool

how our shell tool saves you a trillion tokens vs every other agent

yet another deep dive into command code's harness engineering. all of this is live today, a month into v1, after ~401 released versions.

there are no bad models, only bad harnesses.

this series keeps arriving at that line from different directions, and nowhere does it bite harder than here. every coding agent ships a shell tool. after file reads, it is where most of your agent's tokens go, and it is the most expensive tool to get wrong, because the failure mode is not a wrong answer. the failure mode is the model sitting in a loop. it runs the build, runs sleep 5, reads the log, reads the log again, re-reads the whole thing a third time because the harness gave it no way to ask for only the new part. the build finishes at its own pace either way. you paid for every turn.

we run open models at scale. trillions of tokens a day, at ~30T tokens scale, and a wasted turn shows up in the eval score and the bill the same day. that changes what you build. a vendor whose margin lives in the model can postpone this work. we can't, so we didn't.

this is the shell edition of the series we started with tool call repairs and the read tool. same method as the read tool post: we took the top harnesses, ours included, pinned their commits, and read the source of everything shell-shaped. the command runner, the background process registry, the output reader, the kill path, the scheduler. nine open harnesses read at commits from 28 to 30 august 2026, plus claude code probed live against controlled commands since we can't read its source. 23 graded capabilities.

everything the shell tool ships, in one list:

• background exec by default: task id + pid + log path back instantly
• monitor wake-ups: the runtime calls the agent, never the reverse
• sleep that parks the agent, wakes early on user input
• durable cron, survives restarts
• background agents + agent_output to collect results
• three wait modes: now / next write / exit
• from_offset cursor reads, never the same bytes twice
• middle-out truncation with exact omitted counts
• full log on disk, path in every result
• honest exits: 137 sigkill, 143 sigterm, never a fake success
• benign-exit notes: grep 1 = "no matches", not an error
• untrusted fencing on all process output
• kill by task id, raw pid, or port
• sigterm → poll → sigkill, never a group we didn't create
• timeouts: 30s default, 600s cap, model-settable
• first-class powershell on windows
• workspace boundary enforced everywhere
• and a bunch of other small improvements that compound

the sections below cover the most important ones.

1/ pricing the waste

the usual way to present a benchmark is a feature matrix, and the full matrix is at the bottom of this page. but a checkmark hides the thing that matters, which is what each capability is worth. a monitor tool that wakes the agent when a dev server logs an error is not worth the same as a nicer truncation message. so we priced each capability instead. for every one of the 23, we estimated the tokens it saves per million tokens of shell-driven agent traffic, from our own production traces.

the top of the list:

down at the bottom, workspace boundary checks at ~2k. sum across all 23 and roughly 306k of every million shell-driven tokens is removable waste. about 30%. nearly a third of what agents spend around shells is not intelligence, it is raw context. which makes it an excellent harness engineering opportunity.

now rank the capabilities by price, and for each harness accumulate the savings it actually ships, left to right. every harness draws a curve. the upper envelope of all those curves is the efficiency frontier, the most tokens anyone saves at every step.

command code's curve is the envelope. not near it. it is the frontier, at all 23 steps, never crossed once. we save ~300k of the ~306k available, about 98%. hermes gets to 240k, openclaw 234k, grok build 219k, claude code around 211k, then a cliff: codex 155k, kilo code 120k, opencode 83k, cline 63k, pi 40k.

chip engineers have a name for this kind of target. they call it speed of light: the absolute ceiling of what the hardware can physically do, and you chase 100% of it, absolute numbers, never numbers relative to a competitor.

i think about the harness the same way. 306k is the speed of light of shell tooling, the most waste any harness could remove, and we run at 98% of it. the benchmark against nine other harnesses is the less interesting part. the interesting part is the 6k we still leave on the table.

the gap between us and second place is about 60k tokens per million. six percent of all shell traffic, which at our volume is over a trillion tokens a month that our users do not buy. two different kinds of confidence in this chart, and you should know which is which. the capability grades come from reading source and are as hard as we could make them. the prices are modeled estimates from our traces, directional, not measured. somebody will argue monitor wake-ups are worth 30k and not 40k. fine. re-weight it however you like, the envelope doesn't move, because we ship the whole list.

2/ never make the model wait

foreground-only shells are still common. pi has no background mode at all. opencode's prompt tells the model to append & itself, which orphans the process with no registry, no reader, no kill. cline detaches only when the human clicks a button. in command code, run_in_background: true returns a task id, a pid and a log path immediately, and the model moves on to the next piece of work.

grok build deserves credit here for a design we like: a foreground command that outlives a 15 second budget is automatically promoted to background instead of killed. that is the right instinct. blocking is a choice the harness should almost never make on the model's behalf.

3/ never make the model poll

this is the single most expensive behavior in the audit. open models poll. they learned it from somewhere, and given a bare shell it is the only correct strategy available to them. the fix is not a system prompt scolding the model. three harnesses literally write "do not poll" in prose while giving the model nothing else to do. the fix is harness capability, not prompt engineering.

monitor_command runs a watcher and the runtime wakes the agent on a schedule and on exit. sleep parks the agent without holding a process, and ends early if the user types. cron fires prompts later, durably, surviving restarts. background agents return control immediately and agent_output collects results when you actually need them. four tools, one principle. the agent's attention is the scarce resource, so events come to it.

the field is better here than i expected. hermes has good output-pattern notifications with rate limits and a strike-based auto-disable. openclaw wakes the agent on process exit and even returns a backoff hint on empty polls. grok pushes completion reminders and detects polling doom-loops. but nobody ships the whole stack. codex has the only other real sleep tool and nothing to wake it. kilo's process events go to a ui sidebar the model never sees.

there's a bigger reason this stack matters than the tokens it saves today. the best latency is no latency at all. the agent runs in the background, on human time scales, and you check in on it the way you check in on a colleague, once a day, not every five minutes. that's where agents are going, and background work runs on a much cheaper class of inference. throughput-optimized serving, big batches, no premium for spitting tokens out fast, because nobody is sitting there waiting. a polling agent can't use any of that. re-checking a log every few seconds demands interactive latency from infrastructure priced for patience. a woken agent doesn't care if the answer takes a minute. so the anti-polling stack is more than a token optimization. it is what makes an agent background-tolerant, and background-tolerant agents get to ride the cheapest tokens that will ever exist. our bet is that agent tokens end up overwhelmingly background within a couple of years. we built the shell tool for that future on purpose.

4/ never make the model re-read

shell_output takes from_offset and returns only what's new. it waits in three modes: return now, wait for the next write, wait for exit. that combination sounds small and it is the difference between reading a 30k character log once versus five times. no other harness has all three wait modes. tail snapshots, which several harnesses ship, re-deliver the same last 100 lines on every read. every poll costs the same as the first one, forever.

5/ never lie about how a process died

making agents more honest improves output quality. it is the tool call repairs lesson again: what looks like a dumb model is usually a harness feeding it fiction.

in pi, a process killed by a signal reports as success. node gives you code === null for a signal death, pi's check is !== 0 && !== null, and an oom-killed build comes back as a green tool result. in opencode and kilo code the exit code is captured and stored in ui metadata, and the model never sees it at all. cline collapses every signal death to exit 1. think about what the model experiences in each case. this is what makes open models loop, and looping is expensive. we learned it in v0 and fixed it for v1.

command code reports 128+n honestly, 137 for sigkill, 143 for sigterm, never a fake success. and it annotates the benign cases: grep exiting 1 says "no matches found", because to a model trained that nonzero means failure, an unannotated grep 1 is a retry loop for many models. grok does this all the time. only hermes does this too, and their exit-code interpreter, with oom notes and masked-success detection for cmd | tail pipelines, is the best implementation by any other harness in the audit.

6/ never destroy the evidence

truncation should be a view, not a loss. command code truncates middle-out, so the command echo and the final error both survive, tells the model exactly how many characters were omitted, and writes the full output to a log file whose path is in the result. the model can grep the log instead of re-running a ten minute build.

codex keeps truncated output in memory only. once it's clipped, it is gone, no file exists. openclaw is at least honest about the same choice, the result literally says "earlier output was discarded at the retention cap and cannot be recovered". i respect the honesty. i do not respect the design. either way, the model pays for that loss later, in turns and tokens.

7/ a log line should not be able to talk to your agent

shell output is untrusted input that happens to come from your own machine. an npm install of a compromised package can print anything it wants, and in nearly every harness we read, that text flows into the context indistinguishable from tool protocol. command code fences all process output in explicit untrusted-content delimiters. openclaw fences command text for its own reviewer sub-model and then hands the raw output to the main agent unfenced, which tells you the authors know the threat exists.

8/ killing a process is a feature, not an afterthought

the kill path got more of our engineering time than the run path, which sounds absurd until you watch sessions. kill_shell takes a task id, a raw pid, or a port number. the port one matters most in practice, "port 3000 already in use" is the most common dev-server failure there is, and we are the only harness where it is one call. escalation is sigterm, poll, then sigkill, and we never signal a process group we didn't create.

the other harnesses' kill paths are a museum of tradeoffs. codex's model can only type ctrl-c into the process's stdin. cline goes straight to sigkill with no grace. grok validates pgids against degenerate values so a kill can never become a broadcast, careful work, but exposes no list tool and no pid or port kill to the model. hermes guards against pid reuse by checking kernel start times before signaling, which is the kind of paranoia i want in this layer.

9/ do we win every row? no.

here is the category view, and the per-cell grades are all in the table below. we don't max every cell, and the cells we miss are the ones we're actively working on: a tree-sitter style ast walk for safe-command classification instead of our tokenizer, external-directory checks on the workspace boundary, a persistent shell session. we already have a pr ready that closes the classification and session gaps. it is not an easy ship though, the parser sits in the permission path and we don't want to eat more of your machine's memory for it, so it lands when it is lean enough. as command code improves on these, we will update this page with a dated re-audit.

10/ the point

we're building the fastest, cheapest, most capable shell tool in the world, and we were forced into it. we run open models where a wasted turn is visible in the eval score and the bill the same day, which is the entire reason any of the above got built. roughly a third of shell-driven agent traffic is removable friction. removing 98% of it is not a model achievement.

a lot of our workload is already agents running in the background, where humans check in once a day or once a week. that kind of trust means the harness manages the token budget and the agent self-administers it per task, and the harness is the only place this can be done well. tokens are only today's unit of work. the next unit is outcomes: agents taking as many shots on goal as the budget allows. the moment that flips, harness efficiency stops being a line on the bill and becomes capability. a cheaper turn is another shot on goal inside the same budget, and a harness that wastes a third of the budget takes a third fewer shots.

so, the line this post opened with, earned the long way: there are no bad models, only bad harnesses. most harnesses were built for closed models, where openai and anthropic do a lot server-side to make inference forgiving. open models are left out. skill issue applies to the harness more often than the model, and the shell tool is where the skill compounds fastest.

we're building the open-models-focused harness and inference infra, and we plan to keep sharing everything we discover that can help everyone. you can try all of this yourself in command code, and we're going open source soon, so you'll be able to read the code too. tool call repairs, the read tool, now the shell tool. let me know what y'all wanna read about next.

the token efficiency frontier (TEF) benchmark

so where does everyone else land? the top of the table is basically solved. eight of ten harnesses have a native background mode. that part is common knowledge now. the tail is where the sessions are won:

none of those rows show up in a demo. every one of them starts costing you in hour nine of a long session, when the dev server holds port 3000, the build gets oom-killed, and the model has re-read the same log tail eleven times. teams build them only after production forces it.

Related