Episode
21

Why Great Data Still Leads to Bad Decisions (And How to Fix It) with Amy Edmondson and Michael Luca

Amy Edmondson (Harvard Business School) and Mike Luca (Johns Hopkins) join High Signal to unpack what actually drives good decisions in data‑rich organizations. Using contrasts like the Bay of Pigs vs. the Cuban Missile Crisis and product cases such as Airbnb’s work on measuring discrimination, they show how decision quality tracks conversation quality—framing options, surfacing uncertainty, and challenging assumptions. We cover common failure modes (correlation vs. causation, anchoring, hierarchy, false precision), practical meeting designs that raise the signal, and where algorithms and LLMs help or hinder human judgment.
August 5, 2025
Listen on
spotify logoApple podcast logo
Guest
Amy Edmondson

Harvard Business School

,

Amy C. Edmondson is the Novartis Professor of Leadership and Management at the Harvard Business School, a chair established to support the study of human interactions that lead to the creation of successful enterprises that contribute to the betterment of society.

Edmondson has been recognized by the biannual Thinkers50 global ranking of management thinkers since 2011, and most recently was ranked #1 in 2021 and 2023; she also received that organization’s Breakthrough Idea Award in 2019, and Talent Award in 2017.  She studies teaming, psychological safety, and organizational learning, and her articles have been published in numerous academic and management outlets, including Administrative Science Quarterly, Academy of Management Journal, Harvard Business Review and California Management Review. Her 2019 book, The Fearless Organization: Creating Psychological Safety in the Workplace for Learning, Innovation and Growth (Wiley), has been translated into 15 languages. Edmondson’s latest book, Right Kind of Wrong (Atria), builds on her prior work on psychological safety and teaming to provide a framework for thinking about, discussing, and practicing the science of failing well. First published in the US and in the UK (Penguin) in September, 2023, the book is due to be translated into 24 additional languages, and was selected for the Financial Times and Schroders Best Business Book of the Year award.

‍

Guest
Michael Luca

Johns Hopkins University

,

Michael Luca is a professor and the director of the Technology and Society Initiative at the Johns Hopkins University, Carey Business School, and a research associate at the NBER. Professor Luca's research, teaching, and advisory work focuses on the design of online platforms, and on the ways in which data can inform managerial and policy decisions.His research has been published in academic journals including the Journal of Economic Perspectives, Management Science, the Proceedings of the National Academy of Sciences, American Economic Review: Papers and Proceeding, the American Economic Journal: Applied Economics, and the American Economic Journal: Microeconomics.He has also written about behavioral economics and online platforms for media outlets including The Wall Street Journal, Harvard Business Review, The Atlantic, Wired, and Slate. His research has been written about in a variety of media outlets including The Wall Street Journal, New York Times, New Yorker, Atlantic, Economist, Washington Post, Financial Times, Guardian, Huffington Post, Harvard Business Review, Time, USA Today, Boston Globe, LA Times, San Francisco Chronicle, Fortune, Mashable, GQ, Wired, and Vox.

HOST
Hugo Bowne-Anderson

Delphina

Hugo Bowne-Anderson is an independent data and AI consultant with extensive experience in the tech industry. He is the host of the industry podcast Vanishing Gradients, a podcast exploring developments in data science and AI. Previously, Hugo served as Head of Developer Relations at Outerbounds and held roles at Coiled and DataCamp, where his work in data science education reached over 3 million learners. He has taught at Yale University, Cold Spring Harbor Laboratory, and conferences like SciPy and PyCon, and is a passionate advocate for democratizing data skills and open-source tools.

Key Quotes

Key Takeaways

Decision quality is conversation quality.
Bay of Pigs vs. Cuban Missile Crisis shows the delta wasn’t data: iit was the rigor of debate, exploration of options, and pressure-testing assumptions.

‍

Design the decision, not just the analysis.
Good outcomes come from structured meetings: name the uncertainty, surface dissent, switch sides, and expand the option set before choosing.

‍

Measurement choices shape strategy.
What you track becomes what you optimize. Airbnb’s shift (measuring discrimination, adding guardrails, and changing product flows) shows how metrics redirect behavior.

‍

Causality is earned, not assumed.
Correlation doesn’t settle managerial bets. Leaders must ask how the claim was identified (experiment? quasi-experiment?), what confounders remain, and what would change the decision.

‍

Treat uncertainty as a first-class input.
Ranges, confidence intervals, and priors beat single point estimates. Decide for robustness across plausible outcomes, not for a fragile best guess.

‍

Culture can overpower evidence.
Anchoring, hierarchy (the HiPPO), and confirmation bias routinely swamp data. Create norms and language to call these out in the room before they harden into decisions.

‍

Weak signals deserve investigation.
“Not significant” isn’t “not real.” Probe subgroup effects, moderators, and time horizons: early warnings often hide in heterogeneity and short windows.

‍

Generalize with care.
Ask what worked, why it worked, and where the mechanism holds. Internal validity is not external validity; portability requires understanding the “why,” not just the “what.”

‍

Right-size rigor to stakes.
Save heavyweight process for high-stakes, high-uncertainty choices. Trivial decisions can run on heuristics; consequential ones need designed debate.

‍

Algorithms assist; humans are accountable.
LLMs and models expand options, but outsourcing judgment is a risk. Use machines for scale and patterning; use humans for context, values, and novel possibility.

‍

You can read the full transcript here.

‍

Timestamps

‍

00:00 Introduction to High-Stakes Decision Making

01:06 Meet the Experts: Amy Edmondson (Harvard Business School) and Mike Luca (Johns Hopkins)

02:25 The Importance of High-Quality Conversations

11:06 Case Study: Airbnb and Discrimination

13:57 Strategies for Effective Decision-Making

26:40 Pay Attention to Weak Signals

27:15 Statistical Significance and Variability

28:21 Confirmation Bias in Analytics

29:26 Generalizing Findings Across Markets

29:54 Bayesian Approach to Data Integration

32:59 Leadership and Decision-Making

36:09 The Role of Algorithms and Human Judgment

40:50 Data-Driven Leadership

46:34 Behavioral Economics and Decision Pitfalls

48:53 Concluding Thoughts and Takeaways

‍

Links From The Show

LINKS

‍

Transcript

featured

In the spotlight: Our most popular episodes

Episode
15

Eoin O'Mahony on Why Good Metrics Still Lead to Bad Decisions — and How to Fix It

Eoin O'Mahony—data science partner at Lightspeed, former Uber science lead, and one of the early architects of the system that kept NYC’s Citi Bikes available across the city—argues that positive metrics are meaningless if you don’t understand the mechanism behind them. At Uber, he was careful to make sure his launches both looked good on paper and made sense in practice. Now in venture, he’s applying that same rigor to unstructured data—using GenAI to scale a kind of work that’s long resisted systematization.

With
Eoin O'Mahony
Partner, Data Science
,
Lightspeed Ventures
With
,
Play episode
Episode
10

Ari Kaplan on Why AI Won't Save You But Data Intelligence Will

Ari Kaplan—Global Head of Evangelism at Databricks and a pioneer in sports analytics—explains why businesses fixated on AI often overlook the real advantage: making better decisions with their own data. He shares lessons from his work building analytics teams for Major League Baseball, advising McLaren’s F1 strategy, and helping companies apply AI where it actually works—without falling into hype-driven traps.
With
Ari Kaplan
Global Head of Evangelism
,
Databricks
With
,
Episode
5

Gabriel Weintraub on The Hard Truth About Building AI Systems and What Most Leaders Miss About AI

In this episode of High Signal, Gabriel Weintraub (the Amman Professor of Operations, Information, and Technology at Stanford Graduate School of Business), brings his expertise in market design, data science, and operations, enriched by his experience with global platforms like Uber and Mercado Libre, to a conversation that spans practical strategies, cultural insights, and global perspectives on data and AI.
With
Gabriel Weintraub
Amman Mineral Professor of Operations, Information & Technology
,
Stanford Graduate School of Business
With
,
most recent

Listen up: Our latest discussions

Hear the hottest takes on data science and AI.

See all episodes
Episode
21

Why Great Data Still Leads to Bad Decisions (And How to Fix It) with Amy Edmondson and Michael Luca

Amy Edmondson (Harvard Business School) and Mike Luca (Johns Hopkins) join High Signal to unpack what actually drives good decisions in data‑rich organizations. Using contrasts like the Bay of Pigs vs. the Cuban Missile Crisis and product cases such as Airbnb’s work on measuring discrimination, they show how decision quality tracks conversation quality—framing options, surfacing uncertainty, and challenging assumptions. We cover common failure modes (correlation vs. causation, anchoring, hierarchy, false precision), practical meeting designs that raise the signal, and where algorithms and LLMs help or hinder human judgment.
With
Amy Edmondson
Novartis Professor of Leadership and Management
,
Harvard Business School
and
Michael Luca
Professor and Director, Technology and Society Initiative
,
Johns Hopkins University
Episode
20

Daragh Sibley on Incentives, Accountability, and the Data Leader’s Dilemma

Daragh Sibley, Chief Algorithms Officer at Literati and former data-science leader at Stitch Fix, joins High Signal to unpack how machine-learning moves from slide-deck promise to bottom-line impact. He walks through his shift from academic research on how kids learn to read to owning inventory and personalization algorithms that decide which five books land in every child’s box. We dig into the moment a data leader stops advising and starts owning P&L-critical calls, why some problems deserve simple analytics while others need high-dimensional models, and how to design workflows where human judgment and algorithmic predictions share accountability. Along the way we talk incentive design, balancing exploration and exploitation in inventory, and measuring success in dollars—not dashboards.
With
Daragh Sibley
Chief Algorithms Officer
,
Literati
and
,
Episode
19

Defaults, Decisions, and Dynamic Systems: Behavioral Science Meets AI

Lis Costa, Chief of Innovation and Partnerships at the Behavioural Insights Team, joins High Signal to explore how behavioral science is reshaping public policy, digital platforms, and machine learning. She explains how defaults influence behavior at scale, why personalization and chatbots are unlocking new kinds of interventions, and what happens when AI systems meet real-world complexity. We also discuss the limits of nudging, the promise of boosting, and why building for human decision-making requires more than just good models.
With
Lis Costa
Chief of Innovation and Partnerships
,
Behavioural Insights Team
and
,

Get the latest episodes in your inbox

Never miss an episode of High Signal by signing up for the Delphina newsletter.

By clicking Sign Up you're confirming that you agree with our Terms and Conditions.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.