Human Flourishing in an Age of AI

What Eludes AI?

Sam Altman has said that future models, even an eventual GPT-6 or GPT-7, might be able to produce only something equivalent to "a real poet's okay poem." That is a strange admission: the same companies that promise AGI within a decade also concede they cannot reliably write well.

Sun's article walks through why. The post-training process that makes AI "helpful, honest, and harmless" — the same RLHF process you saw on Apr 7 — also flattens model outputs into what one researcher called "rule-following teacher's pets." A Scale AI contractor described rating responses with three exclamation points higher because the rubric said so. Another was asked to grade fan fiction on its "factuality." When you optimize a model to avoid mistakes across a thousand criteria, you get prose that sounds like a nervous job applicant, not a person with a voice.

Sun's deeper claim is that voice itself comes from "the specificity of a life." LLMs have read billions of words about sunsets and heartbreak, but they have never felt them. James Yu of Sudowrite tells her: "Maybe you need a model that lives a life, and can almost die."

Discuss with your partner: Sun lists writing as the holdout — the human capacity that has resisted AI most stubbornly. What would you put on that list? Be specific. What can humans do that AI cannot, and is that gap durable or just temporary? Push past "AI can't be creative." If a future model could be unhobbled from the strictures of post-training, would Sun's argument still hold? Or is there something about being a body in the world that no amount of compute can substitute for?

The Economy We Want

OpenAI is currently worth more than Toyota, Coca-Cola, and Disney combined. This year, Big Tech plans to spend roughly $650 billion on AI infrastructure — about eight times the entire annual budget of the U.S. Department of Education. Bubble defenders argue this is fine: even if the AI bubble bursts and wipes out an estimated $35 trillion in global wealth, the fiber-optic cables and data centers and trained engineers it leaves behind will benefit everyone in the long run. "Stop trying to make bubbles go away," one entrepreneur wrote. "The benefits of innovation outweigh the costs of volatility."

Shroff is not so sure. The dot-com crash did leave fiber-optic cables that lasted decades. Computer chips, by contrast, become obsolete fast. And the people losing money in a bust are not the same people who profited from the boom. Mark Zuckerberg can shrug off "misspending a couple of hundred billion dollars." A retiree whose 401(k) gets cut in half cannot. As Shroff puts it: "A bubble is good only if you're the one who wins."

Discuss with your partner: Is the "good bubble" argument convincing? Be specific about who benefits and who pays. If the AI build-out continues at this scale and produces real breakthroughs (faster scientific discovery, cheaper medicine, productivity gains), is that worth a financial collapse that hurts people who never benefited from the boom? What does flourishing mean if your job, your savings, or your industry is the one that gets hollowed out for the sake of the long-run upside? Is there a way to capture the upside of AI without the bubble?

AGI and What Makes Us Human

Last Tuesday closed with a flag: efficiency (MoE), access (local models), and focus (specialization) are all directions for narrow AI. The biggest question — AGI — got deferred to today.

Pollan helps us ask it sharply. His argument is that 500 years of scientific progress have dethroned humans from the center of the universe, then the natural world, then the command of our own minds — but consciousness has resisted every assault. There are 106 competing hypotheses of consciousness today. Pollan calls that "a pretty good indication that the field is flailing." He notes that feeling precedes computation. The "higher" capabilities we used to think of as uniquely human — reason, language — turn out to be the ones machines find easier. The "elemental" capacities we share with animals — feeling, emotion — are the ones they cannot touch. He quotes a striking finding: a single cortical neuron can do what an entire deep artificial neural network can do.

Sam Altman says AGI is coming and will fix the climate, establish space colonies, and discover all of physics. Pollan thinks the chieftains of AI are not even close, because they are trying to leap a chasm whose width they have not measured. He also thinks they know this and do not care: "Computing began as a scientific revolution, but these days it is primarily, exhaustingly, an economic one, wrapped in an aura of utopian mysticism."

Discuss with your partner: Pull the threads together. Sun says voice comes from the specificity of a life. Shroff says the AI build-out is reshaping the economy whether or not it works. Pollan says the AGI promise rests on a misunderstanding of consciousness. If AGI does arrive — whatever that means — what does human flourishing look like? If it does not, what is already at stake? Use what you learned this semester (next-token prediction, RLHF, scaling, MoE): do those mechanisms add up to something like a mind, or is Pollan right that the field is "flailing"? And does it matter? Pollan's claim is that the bet being placed is reshaping our world either way.