From Simple Parts

Emergence Everywhere

On Tuesday, you built a network from simple neurons — each one just multiplies inputs by weights, adds them up, and applies an activation function. Nothing fancy. Yet when you connected thousands of them and trained the network, it could recognize handwritten digits. Nobody programmed digit recognition — it emerged from training.

The first reading shows this same pattern everywhere in nature: water molecules that don't "know" about ice, ants that don't "know" about bridges, birds that don't "know" about flocking. Simple parts following simple rules → complex collective behavior that seems to come from nowhere.

Discuss with your partner: What makes emergence surprising? Is it genuinely surprising that the digit network worked, or does it feel obvious in hindsight? What's the difference between "emergence" and "just a complicated system"?

The Neuron Gap

Geoff just shared a finding that may change how you think about Tuesday's exploration: it takes about 1,000 artificial neurons to approximate what a single biological neuron does. The artificial neurons you explored on Tuesday are radically simpler than their biological counterparts.

Your digit network had ~13,000 artificial neurons. A single cubic millimeter of brain tissue has ~50,000 biological neurons, each one as complex as a 1,000-node deep network. The human brain has roughly 86 billion of them.

Discuss with your partner: Does this change how you think about what you built on Tuesday? If AI neurons are so much simpler than biological ones, why does AI work at all? What does this gap mean for the analogy between brains and neural networks?

What Emerges at Scale

The digit network has about 13,000 parameters. GPT-4 has hundreds of billions — tens of millions of times more. Same basic building blocks (weighted sums + activation functions), incomprehensibly different scale.

From the second reading: researchers tested 204 tasks and found that some abilities appear suddenly at specific model sizes. Models below a threshold score essentially zero — random guessing. Then at some scale, performance jumps dramatically. Models trained only to predict the next word can suddenly do arithmetic, identify movies from emojis, translate proverbs.

But there's a debate: are these "real" emergent abilities, or are they measurement artifacts — abilities building gradually that our tests miss until they cross a visible threshold?

Discuss with your partner: Are emergent abilities in LLMs "real" emergence or measurement artifacts? Does it matter? What does it mean for predicting what future, larger models will be able to do?

So What?

Here's the thread: simple artificial neurons → digit recognition emerging from training → vastly more complex biological neurons → language models with billions of parameters displaying abilities nobody predicted.

Complex intelligence — whether in brains or in neural networks — emerges from simple mathematical operations repeated at enormous scale. Nobody designed it. Nobody fully understands it. And nobody can reliably predict what will emerge next.

Discuss with your partner: If complex intelligence can emerge from simple mathematical operations, what does that imply? For understanding our own minds? For what AI might become? For how we should think about the systems we use every day?