The Optimal Inductive Bias

AI, Babies, and the Speed of Light

I hear a lot of questions about AI these days. Is it dangerous? Will it take my job? How smart can it become? But the question I wish we were talking about is: what is the optimal inductive bias?

Inductive bias is a fancy term used in machine learning and AI for constraints put on a model. For example, let's say we were trying to have a machine learning model read and understand English. An intuitive person might assume that constraining the model to reading and writing in appropriate grammar would improve the model's performance. Right? English should follow English grammar rules.

But any inhabitant of the internet can tell you, it seldom does. Even common colloquialisms like “long time no see” make no sense grammatically. So if you constrained your model to think and reason only in those terms, a great deal of English would be nonsense.

So we found out the fewer constraints you put on your model's thought, the better. But that comes with other tradeoffs. A model with fewer inductive biases requires more data and takes longer to learn.

My favorite way of thinking about inductive biases is thinking about babies. When a human baby is born, they are helpless. They can't talk, they can't walk, and they certainly can't provide for themselves. However, when a baby deer is born, it's only minutes before they start walking. Why in God's name would evolution hamper us so much as to giving birth to such helpless, albeit cute, little creatures? I suspect it's for the same reasons we put fewer constraints on our models. We may take longer to learn, but we come out far smarter.

So let’s take this to the extreme. If we thoroughly unconstrain the way that our machine learning models think, how can we even begin to fathom how smart they could become? If a model learns to optimize itself, will its cognition be God-like compared to human beings?

To a physicist, I think the answer might be apparent. Oftentimes, philosophers and computer scientists forget that there's no way to remove all inductive biases. There's no way to think with zero constraints.

There's a great little book that demonstrates this perfectly, The Three Body Problem. In this book, they talk quite a lot about a mysterious universal constraint, the speed of light. And they imagine what changes by increasing or decreasing it. Most things in a human’s world won’t change. The speed at which we throw a ball, fly a supersonic jet, or fall in love, all happen glacially compared to the speed of light. There is one thing that would dramatically change though: computers.

Computers seem miraculous to a lay person. They seem to do so many things so smartly. But in reality, computers are quite dumb. The key to computers is that they do countless dumb things very very quickly. And this speed is ultimately what powers them to think and learn.

Computers don't learn like humans do, instead machine learning is like finding a needle in a haystack. The needle is the perfect function that takes a group of ones and zeros (like a photo) and spits out the perfect output (hotdog or not). The hay is all the other billions of functions that don’t work. The computer’s job is to quickly search through that massive pile of hay to find any needles. The human’s job is to give the computer the right pile of hay which constrains what the model can think and is (in large part) the inductive bias.

In old school machine learning models, the pile of hay was tiny. Computers weren't as fast, so they needed a long time to find the needle. But as computers have gotten faster, they can find needles in farmhouses full of hay.

So how fast can these computers get? Well, electricity moves through transistors near the speed of light, therefore the biggest constraint on how fast and how smart computers can become happens to be a universal one.

So when I hear people asking about how smart AI is going to be in the future, I often wonder to myself, whether they've thought about the fundamental inductive bias of the world. Humans, computers, atoms, and stars all exist in reality. A reality that’s fundamentally governed by universal constraints, like the force of gravity, or the speed of light. So given these constraints, we can ask: what is the theoretical limit of intelligence?

Machines cannot become unboundedly smart, because no machine can surpass the speed of light. We can only pack transistors in so densely, and electricity can only pass through them so fast. So, with a touch of physics and math, one can find the true limit to the speed of thought. What is the biggest haystack we can search through? What is the optimal inductive bias?