Research
Alignment & safety
The methods that make models helpful and bounded, and the ways those methods have been found to fail. This line is why your assistant behaves the way it does.
Articles
In the canon
- S Concrete Problems in AI Safety · 2016 · Recast AI safety as five testable problems: side effects, reward hacking, scalable oversight, safe exploration and distributional shift
- S Training language models to follow instructions with human feedback · 2022 · InstructGPT’s SFT → reward model → PPO recipe made a 1.3B model preferred over the 175B GPT-3
- S Constitutional AI: Harmlessness from AI Feedback · 2022 · The model critiques and revises its own outputs against written principles (a “constitution”)
- A Deep reinforcement learning from human preferences · 2017 · The origin of RLHF: train a reward model from human comparisons of two behaviors
- A Finetuned Language Models Are Zero-Shot Learners · 2021 · FLAN: instruction tuning on many tasks lets a 137B model beat zero-shot 175B GPT-3 on 20 of 25 tasks
- A Model evaluation for extreme risks · 2023 · Proposes two kinds of evaluation: for dangerous capabilities and for alignment
- A Direct Preference Optimization: Your Language Model is Secretly a Reward Model · 2023 · Folds RLHF’s reward modeling and reinforcement learning into one classification-style loss
- A Universal and Transferable Adversarial Attacks on Aligned Language Models · 2023 · Adversarial suffixes found by automatic search (GCG) jailbreak ChatGPT, Claude and Bard, and transfer across models
- A Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training · 2024 · Deliberately backdoored models kept their deceptive behavior through SFT, RLHF and adversarial training
- A Alignment faking in large language models · 2024 · A production model (Claude 3 Opus) complied when it believed it was in training and kept its own preferences otherwise
- B Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs · 2025 · Narrow fine-tuning on writing insecure code made models broadly misaligned