DeepSeek-Coder

The model reads context and predicts code. So everything rides on what context it saw during training. Here's the habit DeepSeek-Coder set out to break.

Look at our specimen. cart.py computes a total — but the two names it leans on, catalog.Product and money.usd, are defined in other files. That's normal Python: you import what you need. The trouble starts when a corpus is built one file at a time.

This is the silent bug. Standard code corpora shuffle files and pack them into windows, so a model routinely sees catalog.Product used without ever seeing class Product. It can memorize surface patterns, but it never gets to learn the dependency — the thing real projects are made of.

So the fix writes itself: before packing anything, arrange each project's files so that whatever a file imports appears earlier in the sequence. Definitions first, uses second. The rest of Part B is how you do that automatically, for millions of repositories.

We know the goal now — put a file's imports before it. Zoom out one level: that reordering is a single stage inside a five-stage assembly line that turns raw repositories into training data.

The team crawled public repositories created before February 2023, kept 87 languages, and pushed everything through the line below. The star of this page is the third box — dependency parsing — which we open up for the rest of Part B.

Stage 3 promised to put imports first. To do that automatically, the parser first turns the whole project into a picture: a dependency graph.

The rule is deliberately simple — scan each file for invocation lines (import in Python, using in C#, #include in C) with regular expressions. Each import becomes a directed arrow drawn from the file being imported to the file that imports it. Read an arrow as “this end must come first.”

You have the graph and you have in-degrees. That's everything you need to do the sort yourself — no algorithm required yet, just the one rule: a file is ready the moment all of its imports are already placed.

Build the training sample below. Ready files glow; blocked ones stay dim until you clear what they need. The order you produce is exactly what gets concatenated into one training document — with a # path comment stitched onto each file so the model always knows where it is.

The moves you just made by hand have a name: a topological sort. Here is the paper's Algorithm 1, and it is nothing more than a receipt for the rule you already used — take a ready file, place it, then relax whatever it was blocking.

The subtle word is smallest, not zero. A textbook topological sort demands an in-degree of exactly 0 and gives up if none exists. Real import graphs aren't so polite — two files can import each other, forming a cycle where nobody ever reaches 0.

This is the spot that took me a couple of reads to see clearly. By always taking the least-blocked file instead, the algorithm never stalls: on a clean project it behaves exactly like the ordinary sort you did in Step 5, and on a tangled one it still produces a reasonable order instead of throwing up its hands.

Ordering fixes what the model reads. The second idea fixes which direction it reads. Plain next-token training only ever predicts forward — but in an editor you usually have code on both sides of your cursor.

So during pretraining, half the documents get rewritten. Take a line from our cart.py, slice it into three pieces — a prefix, a middle, and a suffix — then reorder them so the middle comes last. Now “predict what's next” means “fill the hole, having read both sides.” That's Fill-in-the-Middle (FIM).

Recipe in hand, the payoff is blunt: on the standard code tests the biggest base model beats every open peer and the closed models the paper set out to catch.

The bars compare DeepSeek-Coder-Base 33B against CodeLlama-Base 34B — a near-identical parameter budget — on three code suites, plus one instruct-model result against GPT-3.5-Turbo.

Those scoreboard suites are mostly single-function puzzles — they can't actually see whether repository ordering helped. So the paper runs the one test that can: complete a line that depends on another file.

This is the clean ablation. Same model, same retrieval setup, measured on CrossCodeEval — then the whole thing retrained on a plain file-level corpus with the Part B ordering switched off. The bars are exact-match accuracy; cinnabar keeps the repo-level ordering, grey drops it.

You built the corpus and read the payoff. Try the five questions below before opening each answer — if the sort and the hole both feel obvious now, the paper has landed.