A pilot run of four harness mods against stock Codex on 43 tasks

On 43 coding tasks with two attempts each, Codex with four experimental Oh My Diff (OMD) mods and one plugin component cost an estimated 25% less than stock Codex (95% CI 19% to 31%). The run showed no measurable change in pass rate or wall time. The results table below gives the costs, token counts, timings and confidence intervals.

Both arms used Codex 0.166.0 on a single Apple Silicon Mac on 2026-11-01. The results come from one run of one model on 43 tasks, so they show the direction of the effect and are too thin to quote as a release measurement. The mods are experimental, and no install profile includes them.

What the harness controls

A coding agent's harness is the code between the model and the work: what each request contains, how tool results are returned and when a turn may end. Codex's harness resends every earlier tool output on every request, allows a turn to end without a passing check and reports a syntax error only when the file is later executed. OMD mods can change each of these points without forking Codex. I selected techniques with published effects and implemented each as one mod.

Setup

The task set comprised 30 tasks from SWE-bench Verified (21 Django, 5 SymPy, 2 Sphinx, 1 pytest, 1 seaborn) and 13 terminal tasks I wrote for OMD, such as a race-condition fix, a git bisect and a CSV-to-SQLite import. I wrote the terminal tasks in the style of Terminal-Bench; none of them come from its task set. I excluded the astropy, matplotlib, scikit-learn and xarray repositories and Django versions before 3.0, and dropped six Sphinx tasks whose reference patches failed on local arm64 images.

Both arms used the same sandbox: the workspace, including .git, was writable, and network access was disabled. Each SWE-bench attempt was scored by the SWE-bench 4.1.0 harness and each terminal task by its own check script. Cost is estimated from token counts at OpenAI's standard prices for gpt-6.1-sol: $2.00 per million input tokens, $0.10 per million cached input tokens and $10.00 per million output tokens. The runs themselves were billed to a ChatGPT subscription. The confidence interval is a bootstrap over tasks.

Tuning rounds

Before the final run, I tuned the mods on 23 of the tasks, with one attempt each. The first round reduced input tokens by 18% but cost by only 5%, because input that is not served from the provider's prompt cache costs 20 times as much as input that is.

The first request of 18 of the 23 runs had no cached tokens at all, whereas stock Codex had no such request in 111 runs. Every Codex user sends the same tool list and system prompt, about 12K tokens, so the provider almost always has that prefix cached. Two OMD mods changed the tool list: omd_plan_anchor enabled Codex's update_plan tool, and omd_plan_removal, which OMD+ enabled at the time, removed request_user_input. Either change gave every thread a prefix that no other client had sent. I disabled both in the mods arm and confirmed that the first two items of its first request matched stock's.

In the final run, the first request of both arms carried the same 24 tools and the same 39K characters of tool schemas. Instructions varied only with the number of skills Codex listed in a run, and the first request was about 15.8K input tokens in both arms, including the task prompt. Tool and instruction lengths are taken from the Codex session log, which does not record what a mod adds while sending a request. The token counts are reported by the provider and include everything.

omd_env_bootstrap added a snapshot despite the similar token counts. Codex listed fewer skills in the mods arm, which made its first-request session log about 1,500 characters shorter than stock's. On the stock runs, the log's length predicts the provider's count to within 34 tokens (one standard deviation). The same prediction falls short by about 290 tokens on the mods arm's first requests, and that excess is the text the mod inserted. A later 23-run round, in which the host retained each thread's snapshot, shows a similar excess (242 tokens) with snapshots averaging 656 bytes.

omd_verify_stop also needed a change. SWE-bench checkouts have no installed dependencies and no network access, so a test often could not run at all. The mod nevertheless asked the agent to verify, and three tasks took about twice as many tool calls with no improvement. I changed the mod so that a failed check reporting a missing module, a missing command or a network error marks the turn as blocked, and the mod does not continue it. The second round reduced cost by 21% and passed all 23 tasks.

Results

Token counts, cost and wall time are totals over each arm's 86 runs. Their intervals come from a paired bootstrap over tasks of the ratio of the two arms' totals.

The mods were tuned on 23 of the 43 tasks, so the final run could overstate what they do on unseen work. On the 20 tasks never used for tuning, estimated cost fell by 26% (95% CI 16% to 35%), against 24% (16% to 31%) on the 23 tuning tasks.

Both suites moved in the same direction on every measure. The agent made fewer tool calls in both: 4.3 instead of 5.9 per terminal run and 11.2 instead of 14.6 per SWE-bench run. Wall time fell less than cost in both suites, and least on SWE-bench.

Of the 43 tasks, 36 became cheaper. The median task saved 24% on SWE-bench and 22% on the terminal tasks. Seven tasks cost more, the largest increase being django__django-16145 at +27%.

On SWE-bench, the mods arm solved at least once three tasks that stock Codex failed in both attempts (django-10973, django-14376 and django-14608), and failed both attempts of one task that stock solved twice (django-11400). With 30 tasks, this difference is within the noise.

The $2.08 saved came in roughly equal parts from uncached input, cached input and output. The cache hit rate barely changed, from 0.896 to 0.892 on terminal tasks and from 0.926 to 0.914 on SWE-bench, so the mods arm sent less without breaking the cache. Reasoning tokens fell the most, from 40.6K to 15.7K, a reduction of 61%.

Attribution to individual mods

I tested all five components together, so I can't tell how much each contributed. From the traces I can see which mods acted. omd_verify_stop and the output-budget component acted often. omd_observation_mask and omd_edit_guard almost never did, because the runs were short, few outputs were large and edits rarely broke syntax. omd_plan_anchor was off in this arm. The reduction in tool calls is consistent with omd_env_bootstrap answering the first questions an agent asks about a workspace. The snapshot was present in every run's first request, but no run omitted it, so its share of the saving is unknown.

I also found costs in the traces that no mod addressed. About 5% of requests in both arms lost the provider's cache down to the shared 12K-token prefix partway through a run. Those requests accounted for about 21% of cost, and the provider's cache is outside a mod's control.

Changes after the run

After the final run, I made three code changes based on what I found in the traces. None of them was measured here:

• 196 of the 290 edits in the benchmark were shell commands that wrote files, which omd_edit_guard did not observe. It now also checks files changed by a shell command.
• Another experimental mod, omd_loop_guard, notifies the agent when the same call repeatedly returns the same error or output.
• omd_plan_removal no longer changes the tool list, and OMD+ installs it disabled, so plan mode works in Codex and in applications such as T3 Code. omd_plan_anchor no longer adds update_plan either, so it now does nothing on gpt-6.1-sol.

OMD+ with the mods on the nightly build

After those changes, I ran the 23 tuning tasks once more, with one attempt each, on the OMD nightly of 2026-11-01 (v1.3.1-nightly.20261101150218, Codex 0.166.0). In this round the arm was a full OMD+ install with its default mods enabled, plus the four mods above, omd_loop_guard and the output-budget component. I disabled two OMD+ defaults that change the tool list, since the nightly predates the fix for them:

• omd_plan_removal, which OMD+ now installs disabled.
• omd_plan_panel, which enables update_plan to draw the plan above the prompt. This adds update_plan to the tool list, so every new OMD+ thread missed the shared 12K-token prefix on its first request, the same problem described above. Earlier runs did not show it because they disabled the interface mods. OMD+ now installs it disabled.

With both disabled, the first request's tools and system prompt were byte-identical to stock's, and none of the 23 first requests was uncached. The comparison is against the stock runs of the same tasks.

Cost and tokens fell by about as much as in the final run, but wall time did not. The agent sent 9% fewer requests, but each took 7.1 seconds of model time on average, against 5.6 seconds for stock and 6.0 seconds for the same mods in the morning. This round ran in the evening while the others ran overnight, and I ran small local checks alongside it, so I cannot determine how much of the model-time slowdown is attributable to OMD.

Tool calls took 49 seconds longer in total over the 23 runs (168 against 119). That is consistent with OMD's own overhead, since the edit and shell-change checks run inside tool calls, but the local checks running alongside may account for part of it. omd_edit_guard now asks Git for the list of changed files only when a shell command writes to a path it can't read from the command. omd_verify_stop acted in 4 runs and the output-budget component in 3. The other mods did not act, and the snapshot was present in all 23 first requests.

Limitations

• The run does not separate individual mods. The next run is an ablation, with one mod disabled at a time, on the same 43 tasks.
• The tasks were short. Observation masking and the plan anchor are designed for runs with dozens of large outputs, and these runs had few.
• Every run used gpt-6.1-sol at medium effort, so other models and effort levels are untested.
• The cost is estimated from list prices, and the runs were billed to a subscription, so actual billing was not measured.

The benchmark harness is not published. The full experiment, including tuning and the nightly round, used 290 runs of a 300-run budget. I plan a larger benchmark covering all of OMD+, with ablation, once further budget is available.