Matthias Wisniowski
Writing
EntrepreneurshipBuildingCraft

What safety-critical engineering taught me about shipping

At Cruise, a bug could put two tons of steel through a red light. Here's what that pressure taught me about moving fast without being reckless.

3 min read

DRAFT — replace with Matthias’s own words.

For four years, the code I wrote decided whether a car stopped at a red light. Not “showed a warning.” Stopped. When your failure mode is a two-ton vehicle in a crosswalk, you develop a very particular relationship with the word ship.

People assume safety-critical work is slow. It can be. But the best teams I worked with at Cruise were fast and careful at the same time — and I’ve come to believe those two things aren’t in tension. They’re the same discipline, viewed from two angles.

Speed comes from knowing what you can’t get wrong

The instinct under pressure is to slow everything down uniformly. Test everything twice. Review every line. That doesn’t make you safe; it makes you slow and still unsafe, because attention spread evenly is attention wasted.

What actually worked was ruthless triage of consequence. We drew a hard line between the code that could hurt someone and the code that couldn’t, and we spent our fear budget accordingly.

Move fast on the reversible. Move deliberately on the irreversible. Almost everything is reversible.

The traffic-light detector — camera selection, calibration, the GPU image pipeline, the detector and tracker that got us through 99.9% of San Francisco’s intersections1 — that got the paranoid treatment. The internal dashboard that visualized its output? We shipped that on a Tuesday afternoon and fixed it in prod. Both decisions were correct.

A checklist for the things that scared us

Over time the “irreversible” work converged on a small set of habits. Roughly:

  • Make the failure observable before you make the feature fast. If you can’t see it fail in a log, a metric, or a replay, you can’t trust it.
  • Write the test that would have caught the last incident. Not a hypothetical one. The real one.
  • Ship behind a flag you can pull in seconds, not a deploy you can roll back in an hour.
  • Assume the input is adversarial. Sensors lie. Users mistype. The world sends you glare, rain, and a cyclist doing something no dataset predicted.

That last one is where a mental model helps more than a process:

def decide(observation):
    # The dangerous bug is never the case you thought about.
    # It's the confident wrong answer on the case you didn't.
    if not observation.is_trustworthy():
        return fail_safe()          # slow, boring, correct
    return act(observation)

The fail_safe() branch is unglamorous. It’s also the whole job. A system that degrades gracefully under uncertainty beats a clever one that’s occasionally, confidently wrong.

What carried over to startups

I’m a founder now, not a perception engineer, and the stakes look completely different. Nobody gets hurt if our voice agent mispronounces a word. But the discipline transferred almost intact.

A startup’s version of the red light is trust. Spend it recklessly and you don’t crash — you just quietly stop being believed, which for a young company is the same thing. So I still ask the old question about every release: what, here, is irreversible? Usually the answer is “very little,” and we move fast. Occasionally the answer is “a user’s trust in whether we tell the truth,” and we slow all the way down.

Safety-critical work didn’t teach me to be cautious. It taught me to be precise about where caution is owed — and to spend everywhere else like I mean it.

Footnotes

  1. Figure to confirm before publishing. The point stands regardless of the exact decimal. ↩