Research
Foundations
The skeleton every large model shares, the laws that made scale predictable, and the engineering that makes them faster and cheaper. This line explains most of what you see in context limits, prices and speed.
Articles
注意力就是全部:Transformer 如何改写了序列建模SChinese
它把「按顺序读」换成「同时看全句、按相关性取信息」,让训练可以完全并行;此后所有大模型都长在这副骨架上。
Deep read规模律:为什么「把模型做大」从信仰变成了工程SChinese
语言模型的损失随参数量、数据量与算力按平滑的幂律下降,三者之间有可预测的最优配比;它让扩大规模成为可以提前算账的决策。
PickIn the canon
- S Attention Is All You Need · 2017 · Introduced the Transformer, replacing recurrence with self-attention
- S Scaling Laws for Neural Language Models · 2020 · Test loss falls as a smooth power law in parameters, data and compute
- S Language Models are Few-Shot Learners · 2020 · At 175B parameters, GPT-3 learns new tasks from a few examples in the prompt, with no weight updates
- S Learning Transferable Visual Models From Natural Language Supervision · 2021 · CLIP’s contrastive training on 400M image–text pairs puts images and text in one shared space
- S Training Compute-Optimal Large Language Models · 2022 · Chinchilla: at fixed compute, scale parameters and data equally (about 20 tokens per parameter)
- S FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness · 2022 · IO-aware exact attention: several times faster, with memory linear in sequence length
- A BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding · 2018 · Established pretrain-then-fine-tune and masked language modeling
- A Denoising Diffusion Probabilistic Models · 2020 · Turned diffusion models into practical, high-quality image generators with a clear training objective
- A An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale · 2020 · Feeds 16×16 image patches to a standard Transformer as if they were words
- A Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity · 2021 · Simplified mixture-of-experts to one expert per token and scaled past a trillion parameters
- A LoRA: Low-Rank Adaptation of Large Language Models · 2021 · Freezes the weights and trains small low-rank updates, cutting trainable parameters by up to 10,000×
- A High-Resolution Image Synthesis with Latent Diffusion Models · 2021 · Diffusion in a compressed latent space brings high-resolution image generation to consumer GPUs
- A Emergent Abilities of Large Language Models · 2022 · Catalogued abilities that appear abruptly as models scale up
- A Fast Inference from Transformers via Speculative Decoding · 2022 · A small model drafts and the large model verifies in parallel: 2–3× faster, same output distribution
- A LLaMA: Open and Efficient Foundation Language Models · 2023 · Trained 7B–65B models on public data alone; the 13B beat GPT-3 on most benchmarks
- A Visual Instruction Tuning · 2023 · LLaVA links a CLIP encoder to a language model with one projection layer and GPT-4-generated instruction data
- A Efficient Memory Management for Large Language Model Serving with PagedAttention · 2023 · Pages the KV cache like an operating system, nearly eliminating memory fragmentation
- A DeepSeek-V3 Technical Report · 2024 · A 671B MoE model that neared the closed frontier on about 2.8M H800 GPU hours
- B Are Emergent Abilities of Large Language Models a Mirage? · 2023 · Argues many “emergent” abilities are artifacts of nonlinear or discontinuous metrics