Sign in Subscribe
ci

The 15-Minute Rule: PR Feedback SLOs

The 15-Minute Rule: PR Feedback SLOs

Engineering teams love to talk about speed. “We deploy 20 times a day.” “We can spin up a new service in an hour.” “We cut release time from weeks to hours.” But hidden beneath all those flashy stats is a quieter, less glamorous number that might matter even more: how long it takes to get feedback on a pull request.

That’s the soul of the 15-Minute Rule.

Why PR Feedback Time Matters

When a developer opens a pull request (PR), two clocks start ticking:

  1. The human clock - how long until another engineer reviews it?
  2. The machine clock - how long until CI smoke tests give a signal?

The second one is where most teams stumble. If your smoke suite takes 30, 45, 60+ minutes, developers switch contexts. They Slack someone, they hop into another ticket, or they check Twitter. By the time the results land, they’ve mentally checked out. That context switch is productivity death.

Fast PR feedback isn’t just about speed; it’s about developer happiness. Nothing feels better than pushing a branch, grabbing coffee, and seeing your smoke suite pass before you’re back at your desk. Conversely, nothing kills momentum like waiting an hour just to learn you forgot a semicolon or broke a test selector.

The 15-Minute Rule

Here’s the principle:

Every pull request should have smoke test results in under 15 minutes.

Not “most PRs.” Not “when CI is healthy.” Every PR. That’s your service level objective (SLO).

Fifteen minutes isn’t arbitrary. It’s just long enough to cover real smoke checks (login, signup, critical APIs, homepage render) but short enough to stay inside a developer’s attention span. It’s the sweet spot where you can maintain flow.

Defining PR Feedback SLOs

SLOs (Service Level Objectives) are borrowed from Site Reliability Engineering. You define what “good” looks like, measure it, and hold yourself accountable.

For PR feedback, you can define two key SLOs:

  • p50 < 15 minutes
    Half of all PR smoke runs should finish under 15 minutes.
  • p90 < 25 minutes
    90% should finish under 25 minutes.

That way, you’re not chasing perfection. A one-off PR with an unusually heavy setup step might take longer, but the system as a whole keeps developers in flow.

Why p50 and p90?

  • p50 tells you what the typical developer sees. If that’s under 15 minutes, you’ve kept momentum for most of the team.
  • p90 tells you about the tail. If 1 in 10 PRs drags out to 40 minutes, you’ll feel it in morale. Keeping the tail tight protects against frustration.

Together, these two stats give you both median experience and worst-case experience.

What Counts as Smoke?

This is where teams trip themselves up. “Smoke tests” are not your full regression suite. They are the critical checks that prove the system isn’t fundamentally broken.

Examples:

  • Auth: Can you log in and log out?
  • Payments: Can you make a basic transaction?
  • Navigation: Does the homepage load without console errors?
  • API: Do key endpoints return 200s?

Think P0 tests only. No edge cases. No deep journeys. Just “can the lights turn on.”

If you overload your smoke suite with P2/P3 tests, you’ll blow past 15 minutes and ruin the point.

How to Enforce the 15-Minute Rule

1. Ruthless Test Selection

Every test in the smoke suite should justify itself:

  • Does it cover a P0 business risk?
  • Does it fail fast when broken?
  • Does it run reliably?

If the answer isn’t a clear yes, cut it. Save it for regression.

2. Sharding and Parallelization

Split tests across workers. Modern frameworks like Playwright, Cypress, or pytest-xdist make this easy. If you have 100 smoke tests, don’t run them serially; shard across 10 workers and finish in minutes.

3. Stable Environments

Nothing tanks PR feedback faster than environment flakiness. If your smoke tests hit a dev cluster that’s constantly restarting, you’ll waste minutes chasing ghosts. Invest in stable staging sandboxes or ephemeral test environments.

4. Cached Dependencies

Avoid cold installs on every run. Cache Playwright browsers, npm packages, pip dependencies. The first run might take 20 minutes, but every run after should leverage caching to stay under 15.

5. Fail Fast

Configure your test runner to bail on the first catastrophic failure (like the app won’t boot). No need to burn 15 minutes proving every test is broken.

6. Monitoring and Alerts

Track your p50 and p90 continuously. Post them to Slack. Add them to dashboards. If they drift, make it visible. Like uptime, PR feedback is an SLO that deserves attention.

Why Developer Happiness Depends on It

Think about the alternative. If your smoke suite takes 40 minutes:

  • Developers start batch-pushing PRs and ignoring results.
  • Reverts increase because regressions slip through.
  • Morale dips because CI feels like a punishment, not a tool.
  • Releases slow down because nobody trusts the signals.

CI should feel like an enabler, not a gatekeeper.

When PR feedback is fast, engineers:

  • Stay in flow.
  • Fix issues immediately while context is fresh.
  • Trust the system because it feels responsive.
  • Ship with confidence.

In other words, the 15-Minute Rule isn’t about CI at all; it’s about culture. It’s about building a dev experience that says: your time matters.

Antipatterns to Avoid

  • Overloading Smoke: Treating smoke as regression. Resist the temptation.
  • Ignoring Flake: A flaky 10-minute suite is worse than a stable 20-minute one.
  • Unsharded Suites: Running everything serially because “it’s simpler.”
  • No SLOs: If you don’t measure, you won’t improve.

The difference between a high-performing team and a frustrated one often boils down to whether they’ve taken these seriously.

A Real Example

At my company, our initial smoke suite ran in 45–50 minutes. Developers hated it. They’d go to lunch after opening a PR because there was no point waiting around.

We restructured:

  • Cut smoke down to true P0 tests only.
  • Sharded across 3 CI runners.
  • Cached Playwright browsers.
  • Added monitoring of p50/p90.

Within a week, we hit p50 = 12 minutes, p90 = 20 minutes.

The change in morale was instant. PRs merged faster. Bug escapes dropped. Developers stopped complaining about CI and started trusting it.

Making It Visible

One of the best tricks is to publish your PR feedback metrics. Add a Slack bot that posts weekly:

  • PR smoke suite p50 = 13:42
  • p90 = 21:09
  • ✅ SLO met

When developers see those numbers, they feel proud. They start treating fast feedback as part of team identity. And if the numbers slip, everyone knows it’s time to dig in.

What's your P50 and P90?

The 15-Minute Rule sounds simple: “make PR smoke tests finish in 15 minutes.” But it’s one of those deceptively simple ideas that changes everything.

It forces you to prioritize ruthlessly.
It forces you to care about infrastructure.
It forces you to respect developer time.

And in return, you get a happier, faster, more confident team.

So here’s the challenge: measure your PR smoke suite today. What’s your p50? Your p90? If you’re over 15 and 25 minutes, start cutting. Start sharding. Start treating developer happiness as an SLO worth defending.

Because the truth is: the best developer experience isn’t about 20 deploys a day. It’s about never waiting more than 15 minutes to know if your code is safe. That’s the 15-Minute Rule.

👉 Want more posts like this? Subscribe and get the next one straight to your inbox.  Subscribe to the Blog or Follow me on LinkedIn

Read next

Comments ()