TupleTrace Detection you can audit.Diagnosis you can read.
← Blog

Write the test down before you run it

Bruno Ambrozio ·

Four steps: write the test, commit it, run, report every run.

Here’s a habit that sounds dull and has turned out to be one of the most useful things we do.

Before we run an experiment, we write a short note: what we’re testing, what result would count as a win, and what we’ll do if it doesn’t win. We commit that note to version control. Only then do we run anything and look at the numbers.

Clinical trials have worked this way for years; it’s called pre-registration. In machine learning it’s still rare, and it’s easy to see why. It feels like paperwork standing between you and the fun part.

Why bother

Anomaly detection makes it very easy to fool yourself. Anomalies are rare, so a test set might hold only a handful of real events, and a handful of events can swing a score a long way. Try enough settings and something will look great by luck. Decide what “great” means after seeing the results, and you’ll find it every time.

Writing the test down first has already caught us out more than once:

None of these were fun to find. All of them were much better found by us than by someone relying on the result.

Losing counts too

The other half of the habit: we report every run, including the ones that lose. A negative result tells you where the real problem is, and it’s often not where you expected.

Try it

Our note fits on half a page:

  1. The question, in one sentence.
  2. The data, and how it’s split into training and test.
  3. The metric, and the number that counts as a win.
  4. What we’ll do if it wins, and what we’ll do if it doesn’t.
  5. The date and the commit, before any results exist.

That’s all there is to it, and it has saved us from a few very convincing mistakes.


TupleTrace builds anomaly detection for spacecraft telemetry that explains every alert. See how it works, or download the brochure.