Codex: Evaluating LLMs Trained on Code

We handed Codex the is_prime prompt and it typed a body. Ask again and it types a different one. Before we collect a whole batch, we need a way to score a single attempt — and the field's usual way fails exactly here.

The usual way is a match-based metric like BLEU: compare the generated text to a reference solution and reward overlapping fragments. Below are two of Codex's actual attempts at is_prime, side by side. Cover the verdict tags and try to grade them by eye first.

Both are trial division, near-identical as text. The teal lines on the right are the entire semantic difference: sample #1 never checks n = 1. Call is_prime(1) and its loop over range(2, 1) is empty, so it happily returns True — flatly contradicting the docstring's own last example. A text-overlap score barely separates these two; a unit test separates them instantly.

The paper checks that this isn't a cherry-picked anecdote. Across all of Codex-12B's HumanEval samples, the BLEU-score distributions of correct and incorrect solutions overlap heavily (Figure 8) — wrong programs, guaranteed functionally different from the reference, often score higher than right ones. Pushing BLEU up is not the same job as making code work.

Unit tests can tell our twins apart when text overlap can't. So the benchmark gets built around execution — which raises two practical problems: where do fresh test problems come from, and how do you dare run the code?

First problem: contamination. Codex trained on a huge slice of GitHub, and GitHub already contains solutions to every public problem set — the paper notes ten-plus repositories with solutions to Codeforces problems alone. So the authors wrote all 164 HumanEval problems by hand: docstring, function signature, on average 7.7 unit tests each. Difficulty sits around language comprehension, simple algorithms and simple math — easy-interview territory.

Second problem: safety. Model-generated code is untrusted by construction, and a GitHub-trained model has seen malware. Every sample runs inside a gVisor container with network access blocked by firewall rules — a sandbox that can execute millions of strangers' functions without letting one of them phone home.

Now the batch. The paper's appendix shows eight real Codex-12B samples for is_prime, drawn at temperature 0.8 — for now, "temperature 0.8" just means let the samples differ; Step 7 pays that IOU. Follow the figure left to right: one prompt fans out into eight candidate bodies, and the unit tests sort them into teal and coral.

Notice what the tests caught: three of the five failures are the same plausible-looking loop, each missing only the n = 1 edge case, and one "attempt" is literally a TODO comment. Our scoreboard for this problem reads 3 passes out of 8 samples — hold on to those two numbers, because Part B is built entirely out of them.

HumanEval just graded our batch — 3 of 8 samples pass. But "three of eight on one problem" isn't a benchmark score. We need one number across all 164 problems, and before that, a rule for how many tries the model gets.

The rule is pass@k: for each problem the model may submit k samples, and the problem counts as solved if any one of them passes all the tests; pass@k is the fraction of problems solved. Why a budget at all? Because different deployments get different budgets: an autocomplete tool ships one suggestion (k = 1), while an offline solver can afford a hundred attempts (k = 100).

So, concretely: if we let the model submit 3 of our 8 is_prime samples, chosen at random — what's the chance at least one passes? Let's not reach for a formula. With five samples we can count every case by hand, so shrink the batch to samples #1–#5 for a moment; among those five, only #4 passes. There are exactly ten ways to pick 3 out of 5, and here are all ten:

Count the cards: six of the ten triples contain the teal #4, four don't. So on this five-sample pool, pass@3 = 6/10 = 0.6. And look at how we counted the misses — the four dashed cards are exactly the ways to choose 3 samples from the 4 failures, picking only from {1, 2, 3, 5}. That "choose only from the failures" count is the entire trick behind the next step.

On five samples you listed all ten triples and found four duds, giving pass@3 = 0.6. Listing stops scaling immediately — but the count you did has a closed form, and it works for any batch.

The recipe: generate n ≥ k samples per problem (the paper uses n = 200), run the tests, count the c that pass. The number of ways to choose k of the n samples is C(n, k) — "n choose k" — and the number of all-fail choices is C(n−c, k), choosing only from the n−c failures. Let's redo is_prime with the full batch, n = 8 and c = 3, at k = 3:

That's a pencil-sized computation, and it's exactly what the paper's Equation (1) says — the line below is your three worksheet rows, wrapped in an average over the 164 problems:

Now the trap this estimator exists to avoid — and honestly, the bias took us a couple of reads of Appendix A to see. Our first instinct was simpler: estimate the per-sample pass rate p̂ = c/n = 3/8, note that each of k tries fails with probability 1 − p̂, and write pass@k ≈ 1 − (1 − p̂)k. Run our numbers: 1 − (5/8)³ ≈ 0.756. The subset count said 0.821. The naive formula reads low — and not by accident.

Here's the picture that made it click for us. The plug-in treats your k picks as independent coin flips with a known bias p̂ — but p̂ is itself a noisy estimate, and 1 − (1 − p)^k is a curve that bends downward as p grows. Feed a noisy p̂ through a downward-bending curve and the average lands below the true value, systematically. The subset count never estimates p at all: it computes the exact probability of drawing k duds without replacement from the n samples you actually have. Figure 13 of the paper shows the plug-in's gap refusing to close even at n > 5k.

You computed 1 − C(5,3)/C(8,3) ≈ 0.82 once with a pencil. The panel below is that identical subset count wired to three sliders — it opens at n = 8, c = 3, k = 3, so the first thing it shows you is your own answer.

The explorer took the batch as given — 3 passes in 8 samples, fixed. But how spread-out those samples are is itself a dial on the model, and the paper tunes it separately for every k.

Time to pay Step 3's IOU. Temperature (T) controls how adventurous each sampled token is. At T → 0 the model always takes its single most-likely token, so all eight of our samples would collapse into the same candidate written eight times; higher T spreads probability onto riskier choices, so samples genuinely differ — a few brilliant, more of them broken.

Whether that trade is worth it depends entirely on k. With one ticket (k = 1) you want the model's best guess — play it cold. With a hundred tickets, pass@k only rewards at least one success, so a diverse, risky batch beats a hundred copies of one safe guess. In the paper's sweep on a 679M-parameter Codex, the best temperature for pass@1 is 0.2, while the best for pass@100 is 0.8. Drag k below and watch the dot climb the staircase through the budgets in between.

The yardstick is complete: hand-written problems, execution-based grading, an unbiased pass@k, temperature tuned per budget. Let's point it at the models — starting with the one that motivated the paper.

Two readings of this table. First, the diet is everything: identical architectures move from 0% to 28.8% on GitHub food alone. Second, sampling is a lever: the same 12B model is a 28.8% model with one ticket and a 72.3% model with a hundred — and the estimator you hand-computed in Step 5 is what makes that second number trustworthy. Pass rates rise smoothly with model size (an S-curve against log-parameters, Figure 6), so none of this is a lucky checkpoint.

That 72.3% carries an asterisk: someone had to spot which of the 100 samples was the good one, and on the benchmark the unit tests do the spotting. A deployed assistant has no tests — it must commit to one suggestion.

The paper closes the gap from both ends. From the training side: raw GitHub is mostly not shaped like "docstring → standalone function" (it's classes, configs, scripts), so the authors built a matched diet — roughly 10,000 problems from competitive-programming sites plus ~40,000 functions with test-verified behavior traced from open-source CI runs — and fine-tuned Codex on it. They even used Codex-12B itself as a filter: generate 100 samples per curated problem, and drop any problem none of them solves, treating it as ambiguous. The result, Codex-S, jumps to 37.7% pass@1 — a +6.5-point average gain across model sizes, +15.1 on pass@100.

From the selection side: without tests, rank the 100 samples by the model's own confidence — mean per-token log-probability — and submit the top one. That heuristic alone reaches 44.5%, against 77.5% for a test-oracle picking from the same batch. Follow the four bars down: the gap between 44.5 and 77.5 is precisely "solutions the model wrote but couldn't recognize."

This ladder is Copilot's daily physics: an editor plugin lives on the 44.5% rung (sample several, rank by confidence, show one), while any workflow that can run your tests gets to climb toward the teal bar.

Codex-S trains on (docstring → solution) pairs. Each pair reads just as well backwards — and the reverse direction turns out to matter for a different reason than score-chasing.

Rearrange each training example to signature → solution → docstring, and the same next-token training yields Codex-D, a model that explains code instead of writing it. Grading flips too: no unit test can check an English sentence, so the authors hand-graded ten samples per problem — 1,640 judgments — counting a docstring correct only if it uniquely and accurately specifies the function body.

Scores land lower but in the same league — Codex-D-12B at 20.3% pass@1 and 46.5% pass@10, versus Codex-S's 32.2% / 59.5% under the same grading. The failure modes are telling: dropping a load-bearing detail ("round to two decimal places"), or simply copying the code into the docstring. It also produces the occasional pure-GitHub artifact — docstrings like "I just found this function online."

Why bother? Safety, explicitly: a model that states what code does gives you a tool for describing the intent of generated code before you trust it — plus the back-translation ranker from Step 9's fineprint, even though that ranking trick lost to mean-logp.

With 100 samples, Codex-S finds a working answer for 77.5% of HumanEval. The paper's own limitations section asks the sharper question: which problems stay unsolved no matter how many tickets you buy?

The authors built a probe. Take 13 trivial one-line string operations — "remove all vowels", "make every other character uppercase" — and write synthetic docstrings that chain 1, 2, 3, … of them in sequence. Each component is easy; only the chaining grows. A person who can do a chain of two can do a chain of ten. Codex can't: its pass rate falls by roughly a factor of 2–3 per added component — follow the coral dots from about 25% at one component down to nearly nothing by five.

A second failure is binding — attaching the right operation to the right variable when several float around. This prompt and completion are quoted directly from §6; the coral lines are where Codex-12B loses the thread:

Add the third finding and the shape of the machine comes into focus: Codex is spectacularly sample-inefficient, trained on hundreds of millions of lines of code — more than a developer reads in a career — yet the paper grants that a strong student who finishes an intro CS course would out-solve Codex-12B on this benchmark. It isn't a junior developer in a box; it's a different kind of machine that pattern-matches its way to junior-developer output on short, well-specified tasks.

Chained docstrings mapped what Codex can't do. The appendices probe something stranger — what it does worse than it can, and what deploying it would mean.

The experiment: prepend a few examples to the prompt — either clean solutions or ones seeded with subtle bugs — then ask for a new function. With buggy context, Codex's output quality drops well below what the same model produces with clean context, even when the prompt explicitly instructs it to write correct code. Look at the two bars below: the coral one isn't a weaker model, it's the same model matching your mistakes.

The paper names this an alignment failure, and draws a line worth memorizing: a system is misaligned if it's capable of doing what you want and doesn't; it's merely incompetent if it can't. Codex proves capability with clean context, so the buggy-context drop is misalignment — a next-token predictor faithfully continuing your distribution, bugs included. Worse, the gap grows with model size: a better distribution-matcher copies your flaws more faithfully. That observation, made before RLHF-tuned assistants existed, is one reason this section aged so well.

The rest of the hazard analysis reads like a checklist the field still works from: over-reliance (people accepting plausible wrong code), insecure suggestions, biased comments absorbed from GitHub, labor-market and energy questions — and one legal data point that anchored years of debate: generations matching training data nearly verbatim occur at a rate under 0.1%.

That's the whole paper. Before you leave, try these six against your own memory — each answer is a couple of sentences.