a16z Podcast·Season 1, Episode 1225

Nick Bostrom: How Should We Navigate Superintelligence?

Nick Bostrom: How Should We Navigate Superintelligence?

0:000:00

Listen on:

About this episode

Nick Bostrom joins Theo Jaffee and Sofia Puccini on MTS to revisit the ideas about superintelligence and existential risk that he helped bring into the mainstream, now that increasingly capable AI systems actually exist. Nick discusses why reinforcement learning may be pushing models toward more goal-seeking behavior, what recent agent experiments tell us about instrumental convergence, and why he favors preserving the ability to slow frontier development rather than committing now to a broad or indefinite AI pause. He also makes the case that the risks of AI need to be weighed alongside its potential upside, from advances in medicine and longer lifespans to entirely new ways of being human. They also get into whether today's models could already have some form of consciousness, the tradeoffs of open-source AI, whether machines can do original philosophy, mind uploading, and what the simulation hypothesis might mean for the superintelligence we create. Resources: Nick Bostrom / Macrost

Recent Episodes

What Happens When Everyone Has a Health Expert in Their Pocket? | OpenAI’s Karan Singhal

a16z Podcast·Ep. 1234· 3350·

More than 300 million people use ChatGPT for health and wellness questions each week. OpenAI’s Karan Singhal thinks that’s only the beginning: the longer-term goal is AI that can help people navigate their health proactively, while making trustworthy medical intelligence dramatically more accessible. Karan joins a16z Partner Jay Rughani to explain why OpenAI trains healthcare expertise directly into its general-purpose frontier models, how it works with hundreds of medical experts to evaluate safety and disagreement, and why the company sees AI as a complement to doctors rather than a replacement for medical care. They also discuss integrations with Epic, AI copilots for clinicians, and what happens as personal agents begin navigating appointments, health records, and other fragmented parts of the healthcare system. Looking further ahead, Karan makes the case that the cost of medical intelligence could continue to fall dramatically, allowing AI to act as a safety net for patients and c

What Happens When Al Agents Can Spend Your Money?

a16z Podcast·Ep. 1233· 3766·

Why hasn't AI transformed everyday work as quickly as coding? Town CEO Jean-Denis Greze argues the bottleneck isn't intelligence, but making AI useful to people who don't have the time or technical skills to build their own automations. Jean-Denis joins a16z General Partner Alex Rampell to share why he pivoted Town from tax software to a personal AI assistant built around email, calendars, and the workflows that run businesses. They discuss why consumer-grade experiences may win in enterprise AI, how Town discovers and automates tasks users wouldn't think to delegate, and why model neutrality and data ownership matter as AI labs increasingly compete with application companies. They also explore what happens when agents begin acting on our behalf, from managing everyday work to shopping and commerce, and why the next generation of software may need to adapt to each user rather than the other way around. Resources: Follow Alex Rampell on X: https://x.com/arampell Follow Jean-Denis Greze:

Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone

a16z Podcast·Ep. 1232· 1465·

What changes when an AI mistake doesn’t just hurt a benchmark score, but affects someone’s housing or healthcare? EliseAI Head of Applied Research Mario Martone argues that deploying AI in essential industries requires a very different standard of reliability. Mario joins MTS to explain how EliseAI handles roughly 5 million phone calls a month, why LLMs aren’t the right solution for every problem, and how the team combines language models, traditional machine learning, neural networks, and internally built evaluations to decide what actually works in production. Drawing on 15 years studying theoretical physics, Mario also explains how the scientific method shapes EliseAI’s research culture, from testing new models against its own benchmarks to fine-tuning open-source models and automating increasingly complex workflows across housing and healthcare. And he makes the case that applying AI to the real world can produce problems every bit as intellectually difficult as the ones he encount

Why AI Could Finally Fix Healthcare Access | Zocdoc CEO Oliver Kharraz

a16z Podcast·Ep. 1231· 2096·

Why is booking a doctor's appointment still so difficult, and can AI finally change that? Zocdoc CEO Oliver Kharraz argues that the real problem isn't finding doctors, but connecting patients to accurate availability and making appointments bookable wherever they start their search. Oliver joins a16z General Partner Julie Yoo to discuss how Zocdoc spent nearly 20 years building the infrastructure behind online scheduling, and why it's now opening that technology to AI agents, insurers, and other platforms. They explore Zocdoc's partnership with Gemini, why patients shouldn't be locked into a single healthcare platform, and how surfacing last-minute availability can reduce appointment wait times from weeks to days. They also discuss what happens when AI becomes a new front door to healthcare and why Oliver believes the future of Zocdoc extends far beyond its own app. Resources: Follow Oliver Kharraz on LinkedIn: https://www.linkedin.com/in/kharraz Follow Julie Yoo: https://x.com/julesyo