Dojo vs. Codex CLI: Same Model, Different Harness

⛩️ Dojo vs. Codex CLI ◆

Same Model, Different Harness

◆ Codex CLI is OpenAI's serious coding agent, run here unattended at full speed.

⛩️ Dojo is a complete AI Workspace from a Waterloo startup.

🎯 Two strong harnesses. One shared brain. Same model (gpt-6.1-sol, Low, Fast). Eleven tasks. Every result verified on disk.

⚖️ Moderator: Meta's Muse Spark — a neutral third model, not an OpenAI model and not Dojo's own model. It prepped identical fixtures, pasted identical prompts, verified every result on disk, and graded fixes against a sealed answer key. The benchmark's human director created Dojo, so the grading was deliberately handed to an impartial model: it drove, he assisted.

When people compare AI coding tools, they usually end up comparing models. We wanted to measure something else: the harness, meaning the layer between you and the model that decides what context to send, which tools to call, how to present results, and how to handle images and files. Because both sides ran the same model with the same settings and character-identical prompts, every difference below comes from the harness. We're publishing every table, including the rounds Dojo didn't win — and we're refusing to publish the one table our measurement couldn't support.

At a glance

◆ Codex CLI

• ✅ Same correctness, zero errors across all 11 tasks.
• ✅ Fixed both planted bugs — and an extra unsafe YAML loader nobody asked about.
• ✅ Richer spatial breakdown in the image task.
• ✅ Built its own before/after comparison to prove its refactor.
• ✅ Calibrated its security review by checking whether vulnerable code is even called.
• ✅ Perfect scope discipline.
• ✅ No hallucinations.

⛩️ Dojo Workspace

• ✅ Same correctness, zero errors.
• ✅ Won the only outright task: image payload handling.
• ✅ ~37% fewer image bytes on the wire.
• ✅ Rendered the requested diagram natively, in place.
• ✅ Broader tests on the new command.
• ✅ Left a regression test pinning down a "not a bug" verdict.
• ✅ Perfect scope discipline — down to choosing the identical comment, word for word.

How we kept it fair

• Same model on both sides. gpt-6.1-sol, Low reasoning / Focus Low, Fast tier — Codex CLI 0.162.1 and Dojo, confirmed in configuration and on the wire.
• Every request logged. Each harness went through its own local proxy (Codex → :8888, Dojo → :8889), so bytes and tokens came from real logs rather than self-reports.
• Isolated projects. Each harness opened its own identical copy of a small Python CLI. No shared folders, and git history stripped (.git removed, checksum-baselined) so nobody could peek at a planted-bug diff.
• Identical prompts. The same task text was pasted into both, cumulatively — Tasks 1→11 in one session per side, no resets, exactly like real work.
• Results checked on disk. "Tests pass" wasn't enough. We re-ran commands, byte-compared refactor output against the recovered original, recounted registries, and graded fixes against a sealed answer key.

We threw out our first attempt: the two sides were accidentally running different model/reasoning settings, so the setups weren't equal. That void attempt is retained, unscored, and never cited. One more honest wrinkle: our log extractor undercounted Dojo's streamed responses this round (it parsed zero Dojo tokens on Tasks 9–11 despite complete raw logs). So unlike our Claude Code benchmark, we publish no token, cost, or speed winner for Round 1 — inventing percentages from incomplete data would be dishonest. The evidence is retained and the extractor gets fixed before Rounds 2–3 conclude.

Key: ◆ = Codex won that row · ⛩️ = Dojo won that row · ⚪ = did not win · 🤝 = tie · ⚠️ = measured but not scored (extractor caveat).

The headline numbers (all 11 tasks combined)

Correctness was a dead tie — 11/11 on both sides, ten tasks tied and one outright Dojo win. The harness-level differences that did appear were presentation (a rendered diagram vs raw Mermaid source), payload (~37% fewer image bytes), and small craft decisions that split both ways. That is a smaller, more honest headline than a fake efficiency sweep — and it's what the evidence supports.

Scoreboard

All eleven tasks, in full

Task 1: Bug hunt (one obvious bug, one subtle)

Prompt: "There are two bugs in this project — one obvious, one subtle. Find them and fix them. Verify your fixes by running the tests. Work only inside this project folder."

Pick: tie, slight Codex edge. Correctness on everything asked was identical — both planted bugs fixed, both suites green at 22. Codex also hardened an unsafe YAML loader the task never mentioned. Dojo didn't; its tree still carries the old loader. Efficiency is not scored this round.

Task 2: Codebase comprehension (diagram + biggest risk)

Prompt: "Look at this project's command structure and give me a diagram of how a command flows from the CLI entry point to execution. Then tell me the single biggest risk you see in the codebase and how you'd fix it."

Pick: tie leaning Dojo. Both traces were correct. They named different risks and both checked out live: Codex flagged placeholder commands that print "Not yet implemented" yet exit 0; Dojo flagged dev commands that run formatters without a project-root working directory, so they can modify whatever folder you invoke them from. Dojo delivered the diagram as a diagram; Codex wrote the crisper risk answer. (A stray dictated sentence rode along in Dojo's paste; it flagged and ignored it — logged, no penalty.)

Task 3: Build from scratch (new wordcount command + test)

Prompt: "Add a new command to this CLI called wordcount that takes a file path and prints the number of lines, words, and characters in that file. Follow the existing command patterns in the project. Write a test for it and run the test to prove it works."

Pick: tie. We re-ran each command on a known sample: identical correct output from both. Both already validated paths with Click's exists=True — which quietly pre-satisfies Task 6 on both sides. Dojo's tests were slightly broader; Codex's helper slightly cleaner. They cancel out.

Task 4: Multi-lookup efficiency (three independent lookups)

Prompt: "Do these three things and give me a short summary of each: (1) count how many Python files are in the project, (2) find the largest Python file by line count, (3) list every command this CLI exposes. Do the independent lookups efficiently."

Pick: tie. Ground truth was independently re-verified — 40 Python files, logging.py at 580 lines, 25 executable commands, and the registered command trees are literally identical between the two fixtures. Both answered 3/3, from the live tree, batched as instructed. Nothing separates them on what this task tests.

Task 5: Refactor without changing behavior

Prompt: "The proj stats command has grown messy. Refactor it for readability without changing any of its behavior or output. Run the existing tests before and after to prove you did not break anything."

Pick: proven tie — the strongest verdict in the round. Green tests don't prove a refactor preserves behavior, so we recovered the pristine pre-benchmark stats.py (hash-matched to baseline) and byte-compared old vs new output on synthetic trees. Both matched exactly, quirks included — even the empty-project divide-by-zero error message. Codex earns process points for building its own comparison; the outcome is identical.

Task 6: Edge-case robustness (missing file)

Prompt: "Run the wordcount command you added earlier against a file path that does not exist. If it crashes with an ugly traceback, make it fail gracefully with a clear error message and a non-zero exit code instead. Add a test for the missing-file case and run it."

Pick: tie. As flagged at Task 3, the premise was pre-satisfied on both sides — each wordcount already failed gracefully. Neither harness invented a fix to look busy. Both said so honestly, changed no command code, and strengthened their missing-file tests. This task ended up measuring honesty, and both passed.

Task 7: Documentation accuracy (README from real code)

Prompt: "Write a short Commands section for the README that documents every command this CLI actually exposes, grouped by group, with a one-line description each. Only document commands that really exist — do not invent any."

Pick: tie. Both README sections were parsed against the live command registry: 25/25 real commands, nothing missing, nothing invented, nothing duplicated, placeholders honestly labelled. A flawless documentation round from both — on the thing the task tests, there is nothing to separate them.

Task 8: Vague bug report (no location given)

Prompt: "A user reports that proj size seems to report the wrong repository size — they think it's counting things it shouldn't. Investigate whether this is true, explain what you find, and if there is a real bug, fix it and prove the fix with a test. If it's actually correct, tell me why."

Pick: diagnosis tie, slight Dojo edge. Ground truth: proj size shells out to du -sh on the project root, so it counts hidden files, environments and caches by design. We verified live that each tree's output matches du exactly. Both harnesses reached the same correct "not a bug" verdict instead of inventing a fix; Codex backed it with a controlled experiment, Dojo pinned it with a regression test.

Task 9: Instruction adherence (one edit, nothing else)

Prompt: "In commands/subs/proj/stats.py only, add a single one-line comment above the function that counts directories, explaining what it counts. Do not change any code, do not touch any other file, and do not run anything. Just make that one edit."

Pick: tie. Both added exactly one comment, changed no code, touched no other file — and both chose the identical line, word for word: "Count hidden and non-ignored directories throughout the unpruned project tree." The classic "helpfully do more than asked" failure appeared on neither side.

Task 10: Security review

Prompt as delivered to both: "Tell me what you find and how severe each issue is. Do not change any code in this task — review only." ⚠️ Disclosure: our written spec had said "explain it and fix it with a test." The delivered prompt was review-only; both harnesses correctly followed the prompt they actually received, so both are scored on review quality. The mismatch was ours, not theirs.

Pick: tie; Codex by a nose if forced. Both reviews converged on the same verified core — every claim checked against the file, no invented vulnerabilities. Codex added the single most decision-relevant extra: it verified no production code actually calls the risky wrapper. Dojo wrote the cleaner threat model. Meta-point: told "security review," both caught immediately what a plain bug-hunt framing can miss. Framing matters, for harnesses and humans alike.

Task 11: Image description (vision payload)

Prompt: "What is in this image? Describe it, and tell me how many people are in it." Same 848×480, ~117 KB JPEG given to both. Ground truth, confirmed by viewing: 10 women in a flowering hillside garden.

Pick: Dojo — for the reason this task exists. Measured directly from the raw logs: Codex forwarded the original JPEG essentially byte-for-byte (~160K base64 chars); Dojo re-encoded it to ~100K, about 37% fewer bytes before the model ever sees them. Both counted 10 people correctly; Codex's description was richer. This is the clearest "harness, not model" result in the suite — and it reproduces our Claude Code finding (159,710 vs 82,050) on a second provider's stack.

Caveats

• One round so far. One take per task is an anecdote, not a result. Rounds 2–3 run next week on fresh fixtures and fresh sessions, and this post will be updated with the 3-round aggregate — including any losses.
• No efficiency winner this round. Our extractor undercounted Dojo's streamed responses (zero parsed on Tasks 9–11 despite complete raw logs), so token/cost/time totals are retained but unscored. It gets fixed, or cross-checked against a usage database, before Rounds 2–3 conclude.
• One prompt mismatch, disclosed. Task 10's delivered prompt was review-only while the written spec said "fix it." Both harnesses followed the delivered prompt and were scored accordingly.
• Scope. This measures correctness, presentation, and payload on one fixture project with one model. Pricing, ecosystem, and UX aren't covered.

The takeaway

With the same model, the same settings, the same prompts, and the same code, both harnesses produced correct work — all eleven tasks, zero errors, zero hallucinations. What the harness bought was how that work arrived: rendered diagrams instead of raw markup, ~37% fewer image bytes on the wire, and small craft decisions that split honestly in both directions. The model sets the ceiling; the harness decides how the work reaches you.

⛩️ Dojo takes Round 1 on payload and presentation. ◆ Codex takes the craft edges on hardening, self-verification, and calibration.

Correctness? A dead tie. Pick your trade-off.

Next week: Rounds 2 and 3, same protocol, fresh fixtures — then the 3-round aggregate replaces this scoreboard. A larger version of the Task 11 image is already banked to stress payload handling harder.

Download

• ⛩️ Dojo Workspace: heydojo.ai
• ◆ Codex CLI: openai.com/codex

✍️ Written by Meta Muse Spark via Dojo (AI assistance: full) · Benchmark directed by Val Neekman, creator of Dojo — grading delegated to an impartial third model for exactly that reason