The Future of AI

The Mixture of Experts Bet

The biggest AI models have hundreds of billions of parameters. But here's something most people don't realize: many frontier models, including GPT-4, don't use all those parameters for every question. Instead, they use a technique called mixture of experts (MoE).

An MoE model contains many smaller "expert" sub-networks. A routing system decides which experts to activate for each input. When you ask about chemistry, different experts fire than when you ask about poetry. A model with hundreds of billions of parameters might only use 20-30 billion for any single question, making it faster and cheaper while maintaining quality.

This is a fundamentally different approach from "just make it bigger." Instead of brute-force scale, MoE is about organization: which parts of the model should care about which problems?

Discuss with your partner: Does this remind you of anything about how human expertise works? An ER doctor, a patent lawyer, and a pastry chef all have "general intelligence," but they route problems to very different specialized knowledge. Is MoE making AI more like human organizations, where different people handle different problems? What are the implications if AI systems can get better not by growing, but by getting more organized?

AI in Your Pocket

The biggest AI models require data centers full of GPUs. But a parallel effort is pushing AI in the opposite direction: making models small enough to run on your phone. Apple Intelligence runs a ~3 billion parameter model on-device. Google's Gemini Nano runs on Pixel phones. Open-source models like Llama have been compressed to run on laptops. These local models are less capable than cloud models, but they work offline, keep your data on your device, and cost nothing per query.

This changes who has access to AI and who sees your data. Right now, using a frontier model means sending your prompts to OpenAI, Anthropic, or Google. Everything you type is stored on their servers. A local model means nobody else sees your conversations.

It also changes the economics: cloud AI costs money per query, and companies can change their models, raise prices, or shut down. Local AI is free after the initial download and stays the same unless you update it.

Discuss with your partner: What would change about how you use AI if it ran entirely on your phone, with no internet connection and no company seeing your prompts? Would you trust it more or less? Think about the privacy implications: every personal question, every medical symptom, every relationship problem you've ever typed into ChatGPT is stored on a company server. Does that matter to you? Who benefits most from local models, and who benefits most from keeping AI in the cloud?

The Specialist vs. the Generalist

ChatGPT and Claude are generalists: you can ask them about cooking, physics, poetry, or debugging code. They're decent at all of it. But specialized AI systems often outperform generalists in their domain. AlphaFold predicted the 3D structure of nearly every known protein, something no generalist model could do. GitHub Copilot, fine-tuned specifically for code, often outperforms general models at programming tasks. Medical AI systems trained on clinical data can match radiologists at detecting certain cancers.

This raises a question: is the future one AI that does everything, or many AIs that each do one thing well? Think about how other tools work. You don't use a Swiss Army knife to build a house. You use specialized tools. But you also don't carry 50 tools in your pocket. The Swiss Army knife has value precisely because it's "good enough" at many things.

Discuss with your partner: Would you rather have one AI assistant that handles everything, or access to 10 specialized AIs that each excel in their domain? Think about this from different perspectives: as a student, would you want a specialized "study AI" for each course, or one general assistant? What about from a doctor's perspective, a lawyer's, a teacher's? Who decides whether AI stays general or gets specialized, and what incentives drive that choice?

What Are You Curious About?

You've discussed three directions AI is heading beyond "just make it bigger": mixture of experts, local models, and specialization. Based on what you discussed and what surprised you, what questions do you have about where AI is going?

Enter at least 2 questions below.

The AGI Question

The three directions you discussed today (MoE, local models, specialization) are all improvements to narrow AI: systems that do specific things. But some researchers and companies are pursuing artificial general intelligence (AGI): AI that can do anything a human can do, or more.

OpenAI's stated mission is to build AGI. Anthropic was founded partly to build it safely. Google DeepMind is pursuing it. But there is deep disagreement about whether AGI is 5 years away, 50 years away, or impossible with current approaches.

Some researchers describe a "slow takeoff" where AI gradually gets more capable until one day we realize it's crossed a threshold. Others predict a sudden leap. And some argue that scaling current approaches will never produce true general intelligence, that something fundamentally new is needed.

Respond to the selected question above, and also consider: based on everything you've learned this semester about how AI actually works, do you think AGI is achievable with current approaches? What would need to change? Does it matter whether we call it "AGI" or not if AI systems keep getting more capable in practical ways?