Security runs on metrics, but do those metrics reflect real risk?
Security runs on metrics, but do those metrics reflect real risk?
In this episode of The Watchtower, Ash Hunt sits down with Wade Baker - co-founder of Cyentia Institute and longtime architect of the Verizon DBIR - to dismantle the cybersecurity metrics that feel right but consistently lead programs astray. They take down "average cost per breach," expose why MTTR makes security teams look great while 99% of their vulnerabilities sit untouched, and introduce the half-life metric that actually tracks risk. Plus: why metrics are weaponized more often than they're used, and how AI agents are (finally) democratizing rigorous risk quantification.
Key Takeaways:
- Cost-per-data-record is a survey artifact — there's no linear correlation between breach cost and records lost
- MTTR only measures the vulnerabilities you remediate — so you can post a great MTTR while ignoring 99% of your environment
- Survival analysis / half-life is the better metric — it tracks burn-down against a defined finish line, not raw speed
- Think like a general, not a sniper: zero vulnerabilities is the wrong objective; the right 80% is
- Metrics are weaponized to justify budget more often than they're used to manage program effectiveness
- You don't need a stats PhD — AI agents are democratizing rigorous risk modeling
Wade Baker on LinkedIn: https://www.linkedin.com/in/drwadebaker/
Cyentia Research: cyentia.com/research
Chapters
00:00 Are we measuring the right things?
01:18 Which cybersecurity metrics are most misunderstood
02:48 The psychology of measuring what's easy
04:20 "We've got to measure something" — and the trap that creates
05:30 The real problem: security doesn't agree what "good" looks like
07:40 Sniper vs general: the thinking style CISOs need
09:28 Doing security things vs achieving security goals
10:25 The $215-per-record myth — and why it won't die
12:13 Metrics as weapons: the real reason the number survives
14:31 The needle-in-the-haystack reality of real breaches
15:45 Risk quantification was solved decades ago — in other industries
17:24 The MTTR indictment: measuring only what you fix
18:48 Survival analysis and the half-life metric
21:07 Fixed-speed decay: metrics as decision engineering
23:57 Event landscape vs threat landscape
27:19 AI agents as scenario-analysis partners
30:05 Democratizing risk modeling without a stats PhD
31:13 What security leaders should actually measure
34:15 Your metrics are not your boss's metrics
36:07 Data storytelling: testing a metric's "so what?"
37:03 What's next from Cyentia Institute