The Information Bottleneck

What comes after scaling?

We talk with Sara Hooker, co-founder and CEO of Adaptation Lab, about why the next generation of AI may look very different from today's static models. Sara argues that models should continuously adapt to new tasks, data, users, and environments—and that doing this efficiently will require rethinking much more than fine-tuning.
We discuss continual learning, AutoScientist and automated research, why non-verifiable tasks may become the next major bottleneck, and why interfaces could be as important as the models themselves. We also get into open vs. closed models, distillation and Chinese AI labs, AI regulation and safety, cybersecurity and biorisk, AI companionship, and what may eventually come after Transformers and tokenization.
Topics
Continuous learning and adaptive AI
Fine-tuning, memory, and AutoScientist
AI agents and automated research
Non-verifiable tasks and human feedback
Adaptive interfaces
Open vs. closed models and distillation
AI safety, regulation, cyber risk, and biorisk
AI companionship and persuasion
The limits of Transformers
Multilingual models and tokenization
Chapters
00:00 — Introduction
02:15 — Why start another AI lab? The return of research
05:46 — What continuous learning actually means
12:04 — Should every company have its own adapting model?
13:59 — Fine-tuning and platforms like Tinker
18:04 — AutoScientist and automated optimization
22:52 — Can AI really improve its own research?
28:38 — The problem of non-verifiable tasks
31:30 — Human feedback and the limits of exponential progress
34:43 — Why the AI interface matters
40:36 — Distillation, China, and open models
49:05 — Open-model licensing
52:19 — Will open models catch closed models?
58:43 — AI regulation and compute thresholds
1:03:07 — AI safety and agent failures
1:10:19 — Biorisk vs. cybersecurity
1:14:03 — Persuasion, AI companionship, and overlooked risks
1:20:41 — Where will AI have the biggest real-world impact?
1:25:41 — What is missing from current AI architectures?
1:29:03 — Neurosymbolic AI
1:31:30 — Multilingual models and tokenization
1:34:02 — Byte-level models and alternatives to tokenization
1:35:03 — Closing

Music
"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Alexia Jolicoeur-Martineau is a Principal Researcher at Microsoft and the author of "Less is More: Recursive Reasoning with Tiny Networks," the paper behind the Tiny Recursive Model that hit about 45% on ARC-AGI-1 with a fraction of the parameters of frontier systems. It won the 2025 ARC Prize paper award.
She read the hierarchical reasoning paper, thought the potential was real and the explanation was not, and rebuilt it without the mouse brains: a small network that carries a hidden state and a current answer, thinks for a few steps, updates, and repeats, with the gradient truncated at each loop. We get into why puzzles suit this and autoregression doesn't, why she thinks LLMs are bad at molecules and more data won't fix it, and what she'd do with a trillion dollars.

Timeline
00:01 Intro
01:06 Leaving biostatistics, and why the field stagnated
06:47 GANs, diffusion, and research on four GPUs
12:58 What was wrong with the hierarchical reasoning paper
16:31 Tiny recursive models explained without the biology
22:35 Why puzzles favor recursion over left to right generation
24:15 Is the bitter lesson really bitter?
27:28 With infinite compute, would you still want small models?
32:00 Self improvement, memory, and a trillion dollars
37:01 Test time compute beyond chain of thought
40:41 Why chain of thought fails on molecules
45:17 Is there a universal representation?
48:06 What people are already building with TRM
55:22 Fixed point models and DEQ
Music

"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0
Topics
Tiny Recursive Models and the ARC-AGI results
What the hierarchical reasoning model was really doing
Deep supervision and truncated backprop
Looping transformers and parameter efficiency
Why puzzles favor whole-context iteration over left to right generation
Test time compute beyond chain of thought
Latent reasoning and the Coconut line of work
Why LLMs fail on chemistry and physics
Representation learning and whether a universal representation exists

Rohan Anil spent eleven and a half years at Google, where he went from writing memory allocators to large-scale linear solvers, then optimization at Google Brain, where he co-developed distributed Shampoo and led optimization for PaLM and Gemini pre-training, including the work that produced Gemini Flash. He then joined Anthropic's pre-training team, and left before the IPO to co-found Core Automation with Jerry Tworek (ex-VP of Research at OpenAI). We talk with him about how Brain worked at its peak, why he left two of the world's best labs, and what he thinks is missing from today's models.
Rohan's view is that pre-training and RL were split by organizational convenience rather than by science. Pre-training builds a prior, and RL sharpens it to the tasks we care about, and neither gives a model a way to absorb new data or learn from its own experience once it is deployed. Post-training more every day plateaus, on-policy distillation plateaus, and in-context learning only goes as far as the context does. He argues the next architecture needs better ways to fold in new knowledge at inference time, and that this is a fundamental optimization question rather than a harness-engineering one.
We also get into why coding agents still fail on low-level systems work, his take on Muon, why second-order methods matter once you leave the noise-dominated regime, and why nobody can yet use a few million GPUs for a single training run.

Timeline
00:00 Intro
01:09 From computer vision to Google systems engineering
02:37 Large-scale linear solvers and sparse features
06:01 Getting into optimization: SDCA and Yonghui Wu's team
07:29 Joining the Shampoo crew
09:35 The Google Brain ethos, and why 2017 to 2019 was special
14:23 Is open research going to keep winning?
15:53 Frontier models are only as good as the prior you give them
17:30 Missing the language model wave, then Common Crawl and online distillation
18:31 Paternity leave, DALL-E Mini, and the 14 days that became two years
20:32 PaLM, Gemini pre-training, and Gemini Flash
23:59 The Shampoo origin story: Tomer Koren's two-week proof
26:54 Why leave Google for Anthropic
30:20 Why leave Anthropic for a startup
31:33 Meeting Jerry Tworek at Dolores Park
34:00 What Core Automation is building
36:17 Continual learning and the pre-training vs RL split
39:11 Why coding agents fail at kernels and low-level pipelines
42:00 The QR factorization kernel competition and reward hacking
44:57 Numerics, verification, and hardware that keeps changing
46:07 Are LLMs creative, or just good at search?
49:50 Getting models to extrapolate instead of interpolate
52:03 Why did we ever call it pre-training?
55:02 What RL is really learning
56:27 Competing with the big labs with fewer people
58:31 Will kernel generation keep old GPUs alive? Amdahl's law
1:01:58 Open source plans
1:02:55 Audience question: agentic optimizers
1:04:53 Audience question: Muon, Shampoo, and the future of second-order methods
1:08:58 Hiring at Core Automation

key topics
Journey from Google Brain to startup
Evolution of AI research and optimization
Pre-training and reinforcement learning
Kernel optimization and system efficiency
Open source AI and collaborative research
Challenges in AI creativity and exploration
Future directions in continual learning and model scaling

Music
"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

John Langford, one of the heads of Microsoft's AI Labs, the creator of Vowpal Wabbit, and a co-inventor of CAPTCHA, joins us to talk about world models. Transformers need orders of magnitude more data than humans to learn the same thing, and John argues a compact, implicit world model is how you close that gap. He explains why he's skeptical of JEPA-style objectives, why a transformer's KV cache is the Ptolemaic epicycle model of belief states, and what his Next Latent work does differently.
We also get into whether research still matters in the age of scale; open versus closed models; agent-driven research after running 2,000 pre-training experiments in 90 days; the origin story of CAPTCHA; and why Muon and orthonormal optimizers actually work.

Topics:
Implicit vs. explicit world models, and the case against JEPA-style objectives
Compact belief states: why compression beats a growing KV cache
Does research still matter in the age of scale? The Kimi K3 argument
Agent-driven research: 2,000 pre-training experiments in 90 days
The invention of CAPTCHA
Optimizers from SGD and Vowpal Wabbit to Muon

Chapters
00:00 Why world models: the sample-complexity gap
09:48 The case against JEPA; a transformer-style implicit world model
15:52 Compact belief states: epicycles vs. heliocentrism
23:41 Does research still matter? The Kimi K3 argument
27:35 Open vs. closed models
35:57 Recursive self-improvement and agent-driven research
42:30 2,000 pre-training experiments in 90 days; weak baselines and reproducibility
54:54 The invention of CAPTCHA
1:00:53 Optimizers: from Vowpal Wabbit to Muon
Music

"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Description
Tabular data is still where most of machine learning actually happens in industry, and the field has changed a lot in the last few years. In this episode we talk with David Holzmüller, a researcher at INRIA and one of the people behind TabArena, TabICL and RealMLP, about what the state of the art looks like right now and how to pick a model for your own data.
We cover the shift to TabPFN-style foundation models that learn to learn from whole tables, why TabArena was built and what earlier benchmarks got wrong, what Google's new TabFM means for the leaderboard, and when gradient boosted trees are still the right tool. David explains why LLMs struggle with tables, shares an early result comparing Claude Opus against TabICL on tiny datasets, and walks through how to embed text columns for tabular models. We also get into time series vs tabular data, the open research problems he thinks matter most, and why classical ML libraries are so bad out of the box.

Links:
TabArena: https://tabarena.ai

Topics
Tabular foundation models and in-context learning on tables
TabArena and Beyond Arena: building a benchmark that stays honest
TabFM, TabPFN, TabICL and the tradeoffs between them
When boosted trees and MLPs still win (large data, CPU, fast inference)
Why LLMs are inefficient on tabular data and where they might help
Embedding text columns with language models
Explainability, calibration and class imbalance
Time series vs tabular data
Open problems: invariances, synthetic data, uncertainty, scaling down
Where the field is heading in the next five years

Chapters
0:00 Intro
0:31 What changed in tabular ML: TabPFN-style foundation models
2:22 Which model to try first? TabArena and how it was built
5:14 What older benchmarks got wrong, and Beyond Arena
8:45 GPU AutoML vs foundation models
10:40 Reading the leaderboard: TabFM, TabPFN, TabICL and the tradeoffs
12:47 Calibration, class imbalance and small vs large data
19:45 Explainability for black-box tabular models
21:34 Why LLMs are bad at tabular data
25:39 Claude Opus 4.6 vs TabICL on tiny datasets
27:51 New classifiers, five-year outlook, real vs synthetic pretraining
33:13 Embedding text columns for tabular foundation models
36:13 Time series vs tabular data
39:59 When gradient boosted trees still win, and feature engineering
45:31 Open research problems and where the field is heading
52:54 Better MLPs and why classical defaults are bad out of the box
Music"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0