AI Research

Uses activation patching and attention head ablation to identify circuits responsible for factual recall, indirect object identification, and basic arithmetic in Claude 3-class models. Finds 12 circuit classes that generalize across model scales.

The full technical report for Llama 3's flagship 405B parameter model. Covers training data curation at 15T tokens, hardware infrastructure (24K H100s), instruction tuning methodology, safety measures, and comprehensive benchmark comparisons.

AlphaFold 3 extends the original protein structure prediction breakthrough to cover all biological molecules. Uses a diffusion architecture rather than the original MSA transformer, achieving state-of-the-art accuracy across protein-DNA, protein-RNA, and protein-ligand interactions simultaneously.

Proposes DPO as a simpler alternative to RLHF — instead of training a separate reward model and running PPO, DPO directly optimizes language model outputs using preference data. Achieves comparable alignment results at significantly lower computational cost.

The technical document behind OpenAI's o3 model — a system that performs extended chain-of-thought reasoning before answering. Covers the training methodology combining supervised learning with reinforcement learning on verifiable problems, and the compute scaling behavior of extended reasoning.

Introduces a draft-model-free approach to speculative decoding that uses tree-structured attention to generate multiple candidate token continuations simultaneously. Achieves 3–5× inference speedup without the overhead of maintaining and running a separate draft model.